Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker #340

Merged
merged 4 commits into from Mar 27, 2019
Merged

Docker #340

Changes from 2 commits
Commits
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -11,13 +11,13 @@ git:
before_install:
- source scripts/website/setup-build-tools.sh
sudo: required
services:
- docker
stages:
- name: test
- name: release
if: (branch = master AND type = push) OR (tag IS present)
- name: upload-launcher

This comment has been minimized.

Copy link
@alexarchambault

alexarchambault Mar 25, 2019

Member

Why remove the upload-launcher stage? Couldn't it be kept along the new upload-artifacts?

This comment has been minimized.

Copy link
@sbrunk

sbrunk Mar 25, 2019

Author Member

Because, if my understanding of travis is correct, jobs run only in parallel if they're in the same stage.

This comment has been minimized.

Copy link
@sbrunk

sbrunk Mar 26, 2019

Author Member

I've put the upload-launcher stage back in as it was before.

if: tag IS present
- name: update-docker-images # after upload-launcher, that waited for the sync to Maven Central
- name: upload-artifacts
if: tag IS present
- name: update-versioned-docs
if: tag IS present
@@ -48,10 +48,10 @@ jobs:
script: amm scripts/site.sc --publishLocal true
- stage: release
script: sbt ci-release
- stage: upload-launcher
- stage: upload-artifacts
name: "Upload launcher"
script: scripts/upload-launcher.sh
- stage: update-docker-images
- stage: upload-artifacts
name: "Update docker images"
script: scripts/update-docker-images.sh
- stage: update-versioned-docs
@@ -0,0 +1,39 @@
# Dockerfile with support for creating images with kernels for multiple Scala versions.
# Expects ALMOND_VERSION and SCALA_VERSIONS to be set as build arg, like this:
# docker build --build-arg ALMOND_VERSION=0.3.1 --build-arg SCALA_VERSIONS="2.11.12 2.12.8" .

# Set LOCAL_IVY=yes to have the contents of ivy-local copied into the image.
# Can be used to create an image with a locally built almond that isn't on maven central yet.
ARG LOCAL_IVY=no

FROM almondsh/almond:coursier as local_ivy_yes
USER $NB_UID
ONBUILD RUN mkdir -p .ivy2/local/
ONBUILD COPY --chown=1000:100 ivy-local/ .ivy2/local/

FROM almondsh/almond:coursier as local_ivy_no

FROM local_ivy_${LOCAL_IVY}
ARG ALMOND_VERSION
# Set to a single Scala version string or list of Scala versions separated by a space.
# i.e SCALA_VERSIONS="2.11.12 2.12.8"
ARG SCALA_VERSIONS
USER $NB_UID
RUN [ -z "$SCALA_VERSIONS" ] && echo "SCALA_VERSIONS is empty" && exit 1; \

This comment has been minimized.

Copy link
@alexarchambault

alexarchambault Mar 25, 2019

Member

NIT Maybe we could put that command in a script, to make it easier to tweak it in the future.

This comment has been minimized.

Copy link
@sbrunk

sbrunk Mar 25, 2019

Author Member

Agreed. That would also make it easier to reuse it outside of a docker build. I'll try to move it out.

This comment has been minimized.

Copy link
@sbrunk

sbrunk Mar 26, 2019

Author Member

done

[ -z "$ALMOND_VERSION" ] && echo "ALMOND_VERSION is empty" && exit 1; \
for SCALA_FULL_VERSION in ${SCALA_VERSIONS}; do \
# remove patch version
SCALA_MAJOR_VERSION=${SCALA_FULL_VERSION%.*}; \
# remove all dots for the kernel id
SCALA_MAJOR_VERSION_TRIMMED=$(echo ${SCALA_MAJOR_VERSION} | tr -d .); \
echo Installing almond ${ALMOND_VERSION} for Scala ${SCALA_FULL_VERSION}; \
coursier bootstrap \
-r jitpack \
-i user -I user:sh.almond:scala-kernel-api_${SCALA_FULL_VERSION}:${ALMOND_VERSION} \
sh.almond:scala-kernel_${SCALA_FULL_VERSION}:${ALMOND_VERSION} \
--default=true --sources \
-o almond && \
./almond --install --log info --metabrowse --id scala${SCALA_MAJOR_VERSION_TRIMMED} --display-name "Scala ${SCALA_MAJOR_VERSION}" && \
rm -f almond; \
done; \
rm -rf .ivy2
@@ -1,49 +1,34 @@
#!/usr/bin/env bash
set -euv

if [[ ${TRAVIS_TAG} != v* ]]; then
echo "Not on a git tag"
exit 1
fi

SCALA212_VERSION="$(grep -oP '(?<=def scala212 = ")[^"]*(?<!")' project/Settings.scala)"
SCALA211_VERSION="$(grep -oP '(?<=def scala211 = ")[^"]*(?<!")' project/Settings.scala)"

VERSION="$(git describe --tags --abbrev=0 --match 'v*' | sed 's/^v//')"

mkdir -p target
cd target

git clone "https://$GH_TOKEN@github.com/almond-sh/docker-images.git" -b template
cd docker-images

BRANCHES=()

for sv in "$SCALA211_VERSION" "$SCALA212_VERSION"; do

BRANCH="almond-$VERSION-scala-$sv"
git checkout -b "$BRANCH" template
BRANCHES+=("$BRANCH")
ALMOND_VERSION="$(git describe --tags --abbrev=0 --match 'v*' | sed 's/^v//')"

mv Dockerfile Dockerfile.tmp
cat Dockerfile.tmp |
sed 's@{SCALA_VERSION}@'"$sv"'@g' |
sed 's@{VERSION}@'"$VERSION"'@g' > Dockerfile
rm -f Dockerfile.tmp
DOCKER_REPO=almondsh/almond
IMAGE_NAME=${DOCKER_REPO}:${ALMOND_VERSION}

git add Dockerfile
git commit -m "almond $VERSION, scala $sv"
done
sbt '+ publishLocal'

This comment has been minimized.

Copy link
@alexarchambault

alexarchambault Mar 25, 2019

Member

Do we want to do that for releases? They should already be on Maven Central when this stage runs.

Maybe we could run that stage for snapshots (and push them with a snapshot tag say), and rely on publishLocal only for them.

This comment has been minimized.

Copy link
@sbrunk

sbrunk Mar 25, 2019

Author Member

Right now it runs in parallel to the upload-launcher (see my comment about the stage) so the artifacts aren't available yet. I can change it back to run afterwards and use the published artifacts. Then only do a publishLocal for snapshots builds as you've suggested.

This comment has been minimized.

Copy link
@alexarchambault

alexarchambault Mar 25, 2019

Member

I guess we can run them sequentially, yes. Doing so allows to avoid the +publishLocal, so maybe it's not even slower…

This comment has been minimized.

Copy link
@sbrunk

sbrunk Mar 26, 2019

Author Member

The script in combination with .travis.yml differentiates now between releases and a push to master. +publishLocal runs only on master now and creates a snapshot docker image from the current HEAD. On a tag, it should grab the published artifacts instead.

cp -r $HOME/.ivy2/local/ ivy-local/

DEFAULT_BRANCH="almond-$VERSION-scala-$SCALA212_VERSION"
docker build --build-arg ALMOND_VERSION=${ALMOND_VERSION} --build-arg=LOCAL_IVY=yes \
--build-arg SCALA_VERSIONS="$SCALA211_VERSION $SCALA212_VERSION" -t ${IMAGE_NAME} .
docker build --build-arg ALMOND_VERSION=${ALMOND_VERSION} --build-arg=LOCAL_IVY=yes \
--build-arg SCALA_VERSIONS="$SCALA211_VERSION" -t ${IMAGE_NAME}-scala-${SCALA211_VERSION} .
docker build --build-arg ALMOND_VERSION=${ALMOND_VERSION} --build-arg=LOCAL_IVY=yes \
--build-arg SCALA_VERSIONS="$SCALA212_VERSION" -t ${IMAGE_NAME}-scala-${SCALA212_VERSION} .

BRANCH="almond-$VERSION"
git checkout -b "$BRANCH" "$DEFAULT_BRANCH"
BRANCHES+=("$BRANCH")
echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin
This conversation was marked as resolved by alexarchambault

This comment has been minimized.

Copy link
@alexarchambault

alexarchambault Mar 25, 2019

Member

Going to create a dedicated docker user with the right to push to the almondsh org.

This comment has been minimized.

Copy link
@alexarchambault

alexarchambault Mar 25, 2019

Member

(Done, will add its credentials on Travis.)

This comment has been minimized.

Copy link
@sbrunk

sbrunk Mar 25, 2019

Author Member

Ah yes, forgot to mention that.


BRANCH="latest"
git checkout -b "$BRANCH" "$DEFAULT_BRANCH"
BRANCHES+=("$BRANCH")
docker push ${IMAGE_NAME}-scala-${SCALA211_VERSION}
docker push ${IMAGE_NAME}-scala-${SCALA212_VERSION}
docker push ${IMAGE_NAME}

for b in "${BRANCHES[@]}"; do
if [ "$b" = "latest" ]; then
git push -f origin "$b"
else
git push origin "$b"
fi
done
docker tag ${IMAGE_NAME} ${DOCKER_REPO}:latest
docker push ${DOCKER_REPO}:latest
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.