Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump and update for fresh chartpressing #1004

Merged
merged 2 commits into from Nov 19, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
22 changes: 20 additions & 2 deletions ci/test-helm
Expand Up @@ -34,21 +34,39 @@ if [[ ! -z "$GITHUB_ACCESS_TOKEN" ]]; then
EOF
fi

# FIXME: Move to a dedicated build stage in travis like in z2jh
#
# ref: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/.travis.yml
if [[ "$TRAVIS_PULL_REQUEST" == "false" && "$TRAVIS_BRANCH" == "master" ]]; then
openssl aes-256-cbc -K $encrypted_d8355cc3d845_key -iv $encrypted_d8355cc3d845_iv -in travis.enc -out travis -d
chmod 0400 travis
export GIT_SSH_COMMAND="ssh -i ${PWD}/travis"
docker login -u ${DOCKER_USERNAME} -p "${DOCKER_PASSWORD}"
PUSH="--push --publish-chart"

if [ "${TRAVIS_TAG:-}" == "" ]; then
# Using --long, we are ensured to get a build suffix, which ensures we don't
# build the same tag twice. The --extra-message influences the additional
# text to the commit message on the gh-pages branch of
# jupyterhub/helm-chart.
#
# ref: https://github.com/jupyterhub/helm-chart
PUSH="--push --publish-chart --long --extra-message \"${TRAVIS_REPO_SLUG}$(git log -1 --pretty=%B | head -n1 | sed 's/^.*\(#[0-9]*\).*/\1/')\""
else
# Setting a tag explicitly enforces a rebuild if this tag had already been
# built and we wanted to override it.
PUSH="--push --publish-chart --tag ${TRAVIS_TAG}"
fi
else
PUSH=""
fi



echo "building helm chart"
cd helm-chart
helm repo add jupyterhub https://jupyterhub.github.io/helm-chart
helm dependency update binderhub
chartpress --commit-range ${TRAVIS_COMMIT_RANGE} ${PUSH} --extra-message "${TRAVIS_REPO_SLUG}$(git log -1 --pretty=%B | head -n1 | sed 's/^.*\(#[0-9]*\).*/\1/')"
chartpress ${PUSH}
cd ..
# git diff will show us the result of the chartpress render.
# This should only include the tags for chartpress images.
Expand Down
2 changes: 1 addition & 1 deletion dev-requirements.txt
Expand Up @@ -6,5 +6,5 @@ pytest-asyncio
pytest-cov
requests
ruamel.yaml>=0.15
chartpress==0.3.1
chartpress>=0.4.3,<0.5
jupyterhub
25 changes: 15 additions & 10 deletions helm-chart/chartpress.yaml
@@ -1,23 +1,28 @@
# For a reference on this configuration, see the chartpress README file.
# ref: https://github.com/jupyterhub/chartpress
charts:
- name: binderhub
imagePrefix: jupyterhub/k8s-
repo:
git: jupyterhub/helm-chart
published: https://jupyterhub.github.io/helm-chart
paths:
- ..
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#927 is the PR that added this. Check out the issue linked in it for some details on the bug that was fixed by adding this here. What has changed that we don't need this any more?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before, chartpress did not consider changes to the images it depended on so this was a hack to resolve it. But, now chartpress does this in this code section, so we don't need this workaround.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that code section is not in chartpress version 0.4.3, that's must be the reason of #919 (comment)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was meant to use the info about the file-requirements of the images alongside its own, so it should not had mattered. The issue is that this may still be required but wasnt meant to still be required.

There are tests etc setup in chartpress, but ive apparently missed this test case.

Copy link
Collaborator

@bitnik bitnik Nov 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I think I was not clear or I couldn't understand you.

What I mean is PR jupyterhub/chartpress#68 is not included in v0.4.3. And that PR fixes the problem we have in binderhub right now. I think we need a version upgrade in chartpress or we add the same hack back in chartpress.yaml in binderhub.

resetTag: local
resetVersion: 0.2.0
# NOTE: All paths will be set relative to this file's location, which is in the
# helm-chart folder.
images:
image-cleaner:
valuesPath: imageCleaner.image
binderhub:
# Context to send to docker build for use by the Dockerfile
# Context to send to docker build for use by the Dockerfile. We pass the
# root folder in order to allow the image to access and build the python
# package. Since we do that, chartpress will always react to changes in
# documentation and other things, and always consider the chart version
# to change along with the image version.
contextPath: ..
# Dockerfile path relative to chartpress.yaml
# Since we changed the contextPath, we must also change the
# dockerfilePath. This is because chartpress assume the Dockerfile will
# reside in the contextPath folder, and since we overrode the default of
# images/binderhub it will be the wrong folder.
dockerfilePath: images/binderhub/Dockerfile
valuesPath: image
# additional paths relevant to the image
# in addition to image directory itself
# paths are relative to chartpress.yaml
paths:
- ../setup.py
- ../binderhub
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume these are being proposed for removal because the contextPath already covers them? Is there a way to check that? I have a vague memory that contextPath is the build context sent to dockerd but that the paths are used to determine if an image needs to be built at all.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, but the build context is used as one foundational path alongside those, so these don't add anything as they are part of the build context.

39 changes: 31 additions & 8 deletions helm-chart/images/binderhub/Dockerfile
@@ -1,22 +1,45 @@
FROM buildpack-deps:stretch
# Using multi-stage builds
# ref: https://docs.docker.com/develop/develop-images/multistage-build/
FROM buildpack-deps:stretch as build-stage

RUN echo 'deb http://deb.nodesource.com/node_8.x artful main' > /etc/apt/sources.list.d/nodesource.list
RUN echo 'deb http://deb.nodesource.com/node_8.x artful main' > /etc/apt/sources.list.d/nodesource.list \
&& curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -

RUN curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -
RUN apt-get update && \
apt-get install --yes nodejs python3 python3-pip python3-wheel python3-setuptools
apt-get install --yes \
nodejs \
python3 \
python3-pip \
python3-wheel \
python3-setuptools

# Copy the whole git repository to /tmp/binderhub
COPY . /tmp/binderhub
WORKDIR /tmp/binderhub

# Build binderhub the python library
RUN python3 setup.py bdist_wheel

# The final stage
# ---------------
FROM python:3.6-stretch
WORKDIR /

COPY --from=0 /tmp/binderhub/dist/*.whl .
ADD helm-chart/images/binderhub/binderhub_config.py .
ADD helm-chart/images/binderhub/requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir *.whl -r /tmp/requirements.txt
# Copy the built binderhub python wheel from the build-stage
# to the current directory within the container.
COPY --from=build-stage /tmp/binderhub/dist/*.whl .

# Copy the additional Python requirements for our Docker container from the
# build context. These can be certain pinned versions or peer dependencies.
COPY helm-chart/images/binderhub/requirements.txt .
RUN pip install --no-cache-dir \
*.whl \
-r requirements.txt

# Copy the binderhub configuration to use.
# FUTURE: Follow the approach taken in Z2JH:
# https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/1478
COPY helm-chart/images/binderhub/binderhub_config.py .

CMD ["python3", "-m", "binderhub"]
ENV PYTHONUNBUFFERED=1
Expand Down