Merge c92d3d6 into 99752e5

containerbuildsystem · Apr 20, 2020 · 09edbd9 · 09edbd9
2 parents 99752e5 + c92d3d6
commit 09edbd9
Show file tree

Hide file tree

Showing 8 changed files with 169 additions and 124 deletions.
diff --git a/.mdlint_style b/.mdlint_style
@@ -0,0 +1,3 @@
+all
+rule 'MD013', :code_blocks => false 
+
diff --git a/.mdlrc b/.mdlrc
@@ -0,0 +1,2 @@
+style "./.mdlint_style"
+
diff --git a/.travis.yml b/.travis.yml
@@ -26,6 +26,8 @@ env:
       OS_VERSION=31
       PYTHON_VERSION=3
       ENGINE=docker
+    - ACTION=markdownlint
+      ENGINE=docker
 install:
  - pip install coveralls
 script:

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -1,6 +1,9 @@
 # Contributing to koji-containerbuild
 
-Refer to the [OSBS documentation](https://osbs.readthedocs.io/) for information
-about how the containerbuild Koji build plugin fits into the OSBS architecture.
+Refer to the [OSBS documentation][] for information about how the containerbuild
+Koji build plugin fits into the OSBS architecture.
 
-Please read the [review checklist](https://osbs.readthedocs.io/en/latest/contributors.html#submitting-changes) when submitting pull requests.
+Please read the [review checklist][] when submitting pull requests.
+
+[OSBS documentation]: https://osbs.readthedocs.io
+[review checklist]: https://osbs.readthedocs.io/en/latest/contributors.html#submitting-changes
diff --git a/docs/build-architecture.md b/docs/build-architecture.md
@@ -1,43 +1,42 @@
-This document describes some general concepts behind container build architecture.
+# Container Build Architecture
 
-Let's say we have CLI named `rpkg` which is client of `pyrpkg` library. How koji configuration comes into container-build workflow:
+Let's say we have CLI named `rpkg` which is client of `pyrpkg` library. This is
+how koji configuration comes into container-build workflow
 
 1. Developer clones dist-git repository (`rpkg clone` command)
+1. Developer switches branch (`rpkg switch-branch`). Let's say branch is
+   `${BRANCH}`
+1. In this branch there is a Dockerfile which contains several `LABEL`
+   directives. One of them is `BZComponent`. Let's call this value
+   `${COMPONENT}`
+1. Developer wants to build the container so they issue `rpkg container-build`
+   command
 
-2. Developer switches branch (`rpkg switch-branch`). Let's say branch is `${BRANCH}`.
+Now some automagic comes into the game
 
-3. In this branch there is a Dockerfile which contains several `LABEL` directives. One of them is `BZComponent`. Let's call this value `${COMPONENT}`.
-
-4. Developer wants to build the container so they issue `rpkg container-build` command.
-
-Now some automagic comes into the game:
-
-1. `rpkg` constructs Koji target by appending `-docker-candidate` to the branch name.
-   In our case it is `${BRANCH}-docker-candidate`.
-
-2. During the build Koji buildroot tag is used as yum repository. *The
+1. `rpkg` constructs Koji target by appending `-docker-candidate` to the branch
+   name. In our case it is `${BRANCH}-docker-candidate`
+1. During the build Koji buildroot tag is used as yum repository. *The
    semantic of buildroot tag differs from rpm builds* as it should contain
-   either:
-
-   - candidate (rpm) packages (to be released) - a tag ending with `-candidate`.
-
-   - released (rpm) packages - a tag without any suffix.
-
-3. Usually neither candidate nor released tags have architecture set in
+   either
+   - Candidate (rpm) packages (to be released) ― a tag ending with `-candidate`
+   - Released (rpm) packages ― a tag without any suffix
+1. Usually neither candidate nor released tags have architecture set in
    Koji. This means a yum repository isn't generated for these tags. To
    workaround this we don't add architecture to these tags but create special
    "build" tag for containers. These builds tags have suffix `-container-build`
-   and inherit from either candidate or released tags.
-
-4. After the container build succeeds it is tagged to the destination tag.
+   and inherit from either candidate or released tags
+1. After the container build succeeds it is tagged to the destination tag.
    Due to Koji policy, the build needs to be added to (whitelisted in) this
    destination tag. Use `${COMPONENT}` to add the package to this destination
-   tag.
+   tag
 
-This automagic is hardcoded in the `pyrpkg` library but its clients (like `fedpkg`) can override this behaviour if needed.
+This automagic is hardcoded in the `pyrpkg` library but its clients (like
+`fedpkg`) can override this behaviour if needed.
 
-Example:
+Example
 
+```shell
     $ rpkg co rsyslog-docker
     $ cd ./rsyslog-docker
     $ rpkg switch-branch extras-rhel-7.1
@@ -62,3 +61,4 @@ Example:
     Package                 Tag                     Extra Arches     Owner
     ----------------------- ----------------------- ---------------- ---------------
     rsyslog-docker          extras-rhel-7.1                          foo-owner
+```
diff --git a/docs/build-process.md b/docs/build-process.md
@@ -1,7 +1,13 @@
-1. User calls `buildContainer` XMLRPC call - e.g. by `container-build` command of `rpkg`.
-2. Hub XMLRPC handler `buildContainer` creates `buildContainer` method.
-3. Builder method `buildContainer`:
-    1. Checks that target and SCM are correct
-    2. Checks that build with given NVR doesn't exist (unless it's a scratch or autorelease task)
-    3. For each architecture [creates build in OSBS](https://github.com/containerbuildsystem/koji-containerbuild/blob/master/koji_containerbuild/plugins/builder_containerbuild.py#L413)
-    4. Watches logs and sends them to hub to save
+# Build Process
+
+1. User calls `buildContainer` XMLRPC call ― e.g. by `container-build` command
+   of `rpkg`
+1. Hub XMLRPC handler `buildContainer` creates `buildContainer` method
+1. Builder method `buildContainer`
+   1. Checks that target and SCM are correct
+   1. Checks that build with given NVR doesn't exist (unless it's a scratch or
+      autorelease task)
+   1. For each architecture, [creates build in OSBS][]
+   1. Watches logs and sends them to hub to save
+
+[creates build in OSBS]: https://github.com/containerbuildsystem/koji-containerbuild/blob/master/koji_containerbuild/plugins/builder_containerbuild.py#L413
diff --git a/docs/pull_request_template.md b/docs/pull_request_template.md
@@ -1,8 +1,7 @@
+# Maintainers will complete the following section
 
-
-Maintainers will complete the following section:
 - [ ] Commit messages are descriptive enough
 - [ ] "Signed-off-by:" line is present in each commit
 - [ ] Code coverage from testing does not decrease and new code is covered
 - [ ] JSON/YAML configuration changes are updated in the relevant schema
-- [ ] Pull request includes link to an osbs-docs PR for user documentation updates
+- [ ] Pull request has a link to an osbs-docs PR for user documentation updates
diff --git a/test.sh b/test.sh
@@ -8,108 +8,138 @@ OS_VERSION=${OS_VERSION:="7"}
 PYTHON_VERSION=${PYTHON_VERSION:="2"}
 ACTION=${ACTION:="test"}
 IMAGE="$OS:$OS_VERSION"
-# Pull fedora images from registry.fedoraproject.org
-if [[ $OS == "fedora" ]]; then
-  IMAGE="registry.fedoraproject.org/$IMAGE"
-fi
-
 CONTAINER_NAME="koji-containerbuild-$OS-$OS_VERSION-py$PYTHON_VERSION"
-RUN="$ENGINE exec -ti $CONTAINER_NAME"
-if [[ $OS == "fedora" ]]; then
-  PIP_PKG="python$PYTHON_VERSION-pip"
-  PIP="pip$PYTHON_VERSION"
-  PKG="dnf"
-  PKG_EXTRA="dnf-plugins-core git-core
-             python$PYTHON_VERSION-koji python$PYTHON_VERSION-koji-hub"
-  BUILDDEP="dnf builddep"
-  PYTHON="python$PYTHON_VERSION"
-else
-  PIP_PKG="python-pip"
-  PIP="pip"
-  PKG="yum"
-  PKG_EXTRA="yum-utils git-core koji koji-hub"
-  BUILDDEP="yum-builddep"
-  PYTHON="python"
-fi
-# Create container if needed
-if [[ $($ENGINE ps -q -f name="$CONTAINER_NAME" | wc -l) -eq 0 ]]; then
-  $ENGINE run --name "$CONTAINER_NAME" -d -v "$PWD":"$PWD":z -w "$PWD" -ti "$IMAGE" sleep infinity
-fi
 
-# Install dependencies
-if [[ $OS != "fedora" ]]; then $RUN $PKG install -y epel-release; fi
-$RUN $PKG install -y $PKG_EXTRA
-$RUN $BUILDDEP -y koji-containerbuild.spec
-# Install pip package
-$RUN $PKG install -y $PIP_PKG
-if [[ $PYTHON_VERSION == 3 && $OS_VERSION == rawhide ]]; then
-  # https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
-  $RUN mkdir -p /usr/local/lib/python3.6/site-packages/
+if [[ $ACTION == "markdownlint" ]]; then
+  IMAGE="ruby"
+  CONTAINER_NAME="koji-containerbuild-$ACTION-$IMAGE"
 fi
 
-# Install other dependencies for tests
-if [[ $PYTHON_VERSION == 3 ]]; then
-  OSBS_CLIENT_DEPS="python3-PyYAML"
-else
-  OSBS_CLIENT_DEPS="PyYAML"
-fi
-$RUN $PKG install -y $OSBS_CLIENT_DEPS
-
-# Install latest osbs-client by installing dependencies from the master branch
-# and running pip install with '--no-deps' to avoid compilation
-# This would also ensure all the deps are specified in the spec
-$RUN rm -rf /tmp/osbs-client
-$RUN git clone https://github.com/projectatomic/osbs-client /tmp/osbs-client
-[[ ${PYTHON_VERSION} == '3' ]] && WITH_PY3=1 || WITH_PY3=0
-$RUN $BUILDDEP --define "with_python3 ${WITH_PY3}" -y /tmp/osbs-client/osbs-client.spec
-
-if [[ ${OS} == "centos" && ${PYTHON_VERSION} == 2 ]]; then
-    # there is no package that could provide more-itertools module on centos7
-    # latest version with py2 support in PyPI is 5.0.0, never version causes
-    # failures with py2
-    $RUN $PIP install 'more-itertools==5.*'
-fi
-
-$RUN $PIP install --upgrade --no-deps --force-reinstall git+https://github.com/projectatomic/osbs-client
-
-# Install the latest dockerfile-parse from git
-$RUN $PIP install --upgrade --force-reinstall \
-    git+https://github.com/containerbuildsystem/dockerfile-parse
-
-# CentOS needs to have setuptools updates to make pytest-cov work
-# setuptools will no longer support python2 starting on version 45
-if [[ $OS != "fedora" ]]; then
-  $RUN $PIP install -U 'setuptools<45'
-
-  # Watch out for https://github.com/pypa/setuptools/issues/937
-  $RUN curl -O https://bootstrap.pypa.io/2.6/get-pip.py
-  $RUN $PYTHON get-pip.py
-fi
-
-# https://github.com/jaraco/zipp/issues/28
-if [[ $PYTHON_VERSION == 2 ]]; then
-  $RUN $PIP install zipp==1.0.0
-fi
+RUN="$ENGINE exec -ti $CONTAINER_NAME"
 
-# configparser no longer supports python 2
-if [[ $PYTHON_VERSION == 2 ]]; then
-  $RUN $PIP install configparser==4.0.2
+# Use arrays to prevent globbing and word splitting
+engine_mounts=(-v "$PWD":"$PWD":z)
+for dir in ${EXTRA_MOUNT:-}; do
+  engine_mounts=("${engine_mounts[@]}" -v "$dir":"$dir":z)
+done
+
+# Create or resurrect container if needed
+if [[ $($ENGINE ps -qa -f name="$CONTAINER_NAME" | wc -l) -eq 0 ]]; then
+  $ENGINE run --name "$CONTAINER_NAME" -d "${engine_mounts[@]}" -w "$PWD" -ti "$IMAGE" sleep infinity
+elif [[ $($ENGINE ps -q -f name="$CONTAINER_NAME" | wc -l) -eq 0 ]]; then
+  echo found stopped existing container, restarting. volume mounts cannot be updated.
+  $ENGINE container start "$CONTAINER_NAME"
 fi
 
-# Install koji-containerbuild
-$RUN $PYTHON setup.py install
+function setup_kojic() {
+  # Pull fedora images from registry.fedoraproject.org
+  if [[ $OS == "fedora" ]]; then
+    IMAGE="registry.fedoraproject.org/$IMAGE"
+  fi
+
+  if [[ $OS == "fedora" ]]; then
+    PIP_PKG="python$PYTHON_VERSION-pip"
+    PIP="pip$PYTHON_VERSION"
+    PKG="dnf"
+    PKG_EXTRA="dnf-plugins-core git-core
+              python$PYTHON_VERSION-koji python$PYTHON_VERSION-koji-hub"
+    BUILDDEP="dnf builddep"
+    PYTHON="python$PYTHON_VERSION"
+  else
+    PIP_PKG="python-pip"
+    PIP="pip"
+    PKG="yum"
+    PKG_EXTRA="yum-utils git-core koji koji-hub"
+    BUILDDEP="yum-builddep"
+    PYTHON="python"
+  fi
+  # Create container if needed
+  if [[ $($ENGINE ps -q -f name="$CONTAINER_NAME" | wc -l) -eq 0 ]]; then
+    $ENGINE run --name "$CONTAINER_NAME" -d -v "$PWD":"$PWD":z -w "$PWD" -ti "$IMAGE" sleep infinity
+  fi
+
+  # Install dependencies
+  if [[ $OS != "fedora" ]]; then $RUN $PKG install -y epel-release; fi
+  $RUN $PKG install -y $PKG_EXTRA
+  $RUN $BUILDDEP -y koji-containerbuild.spec
+  # Install pip package
+  $RUN $PKG install -y $PIP_PKG
+  if [[ $PYTHON_VERSION == 3 && $OS_VERSION == rawhide ]]; then
+    # https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
+    $RUN mkdir -p /usr/local/lib/python3.6/site-packages/
+  fi
+RUN="$ENGINE exec -ti $CONTAINER_NAME"
 
-# Install packages for tests
-$RUN $PIP install -r tests/requirements.txt
+  # Install other dependencies for tests
+  if [[ $PYTHON_VERSION == 3 ]]; then
+    OSBS_CLIENT_DEPS="python3-PyYAML"
+  else
+    OSBS_CLIENT_DEPS="PyYAML"
+  fi
+  $RUN $PKG install -y $OSBS_CLIENT_DEPS
+
+  # Install latest osbs-client by installing dependencies from the master branch
+  # and running pip install with '--no-deps' to avoid compilation
+  # This would also ensure all the deps are specified in the spec
+  $RUN rm -rf /tmp/osbs-client
+  $RUN git clone https://github.com/projectatomic/osbs-client /tmp/osbs-client
+  [[ ${PYTHON_VERSION} == '3' ]] && WITH_PY3=1 || WITH_PY3=0
+  $RUN $BUILDDEP --define "with_python3 ${WITH_PY3}" -y /tmp/osbs-client/osbs-client.spec
+
+  if [[ ${OS} == "centos" && ${PYTHON_VERSION} == 2 ]]; then
+      # there is no package that could provide more-itertools module on centos7
+      # latest version with py2 support in PyPI is 5.0.0, never version causes
+      # failures with py2
+      $RUN $PIP install 'more-itertools==5.*'
+  fi
+
+  $RUN $PIP install --upgrade --no-deps --force-reinstall git+https://github.com/projectatomic/osbs-client
+
+  # Install the latest dockerfile-parse from git
+  $RUN $PIP install --upgrade --force-reinstall \
+      git+https://github.com/containerbuildsystem/dockerfile-parse
+
+  # CentOS needs to have setuptools updates to make pytest-cov work
+  # setuptools will no longer support python2 starting on version 45
+  if [[ $OS != "fedora" ]]; then
+    $RUN $PIP install -U 'setuptools<45'
+
+    # Watch out for https://github.com/pypa/setuptools/issues/937
+    $RUN curl -O https://bootstrap.pypa.io/2.6/get-pip.py
+    $RUN $PYTHON get-pip.py
+  fi
+
+  # https://github.com/jaraco/zipp/issues/28
+  if [[ $PYTHON_VERSION == 2 ]]; then
+    $RUN $PIP install zipp==1.0.0
+  fi
+
+  # configparser no longer supports python 2
+  if [[ $PYTHON_VERSION == 2 ]]; then
+    $RUN $PIP install configparser==4.0.2
+  fi
+
+  # Install koji-containerbuild
+  $RUN $PYTHON setup.py install
+
+  # Install packages for tests
+  $RUN $PIP install -r tests/requirements.txt
+}
 
 case ${ACTION} in
 "test")
+  setup_kojic
   TEST_CMD="pytest -vv tests --cov koji_containerbuild"
   ;;
 "bandit")
+  setup_kojic
   $RUN $PIP install bandit
   TEST_CMD="bandit-baseline -r koji_containerbuild -ll -ii"
   ;;
+"markdownlint")
+  $RUN gem install mdl
+  TEST_CMD="mdl -g ."
+  ;;
 *)
   echo "Unknown action: ${ACTION}"
   exit 2