Add Dockerfile and Dockertag generation from templates #12

iesahin · 2021-05-13T17:16:14Z

This PR is a major update for the Docker image production.

Divided build-all.bash into two (build-all.bash and push-all.bash) and added generate-all.bash that generates Dockerfiles from Dockerfile.template files.
Instead of static Dockerfile and Dockertags, the script first generates a set of Dockerfile and Dockertags by calling generate.* scripts in directories.
generate.* scripts can be written in bash, sh, zsh, or python. generate-all.bash checks the extension of generate.EXT script and calls the appropriate interpreter.
RELEASE_HASH and TAG_PREFIX can be set before calling the script now. Hence running

$ TAG_PREFIX=my_docker_username ./generate-all.bash

produces a set of Dockertag files using my_docker_username/ Docker handle.

DVC Get Started Images

These images are produced by start/generate.bash

For each Git tag in dvc-get-started, a Dockerfile-Dockertag pair is generated in start/build/. Dockerfiles are all depend to python:3.8 public Docker image. Dockertag files use ${TAG_PREFIX}/doc-start:${GIT_TAG} format. For prefix dvcorg, and Git tag signal-file, the produced Docker image is tagged as dvcorg/doc-start:signal-file

Katacoda Image changes

Formerly Katacoda images were produced from static files. In this version, these are also converted to Dockerfile.template files and generated using TAG_PREFIX variable. In the previous version, Dockerfiles had FROM dvcorg/doc-katacoda:base lines. In this version, these are replaced by FROM ${TAG_PREFIX}/doc-katacoda:base, so when the user changes ${TAG_PREFIX}, they also change dependencies among Dockerfiles. This allows for third parties to use these images in their own Dockerhub (or other repository) accounts.

Closes #7
Closes #11
Related iterative/example-repos-dev#34

casperdcl · 2021-05-13T19:26:29Z

build-all.bash


-RELEASE_HASH=$(curl -s https://api.github.com/repos/iterative/dvc/releases/latest | sha256sum | cut -c -8)
+./generate-all.bash


Why create a separate generate-all.bash yet not use it anywhere else? Either keep things in one file or re-use them in multiple places, surely?

Removed this call to generate-all.bash. For debugging and security, generate, build, push should be called separately IMO.

build-all.bash

generate-all.bash

katacoda/generate.bash

casperdcl · 2021-05-13T19:34:11Z

katacoda/get-started/03-accessing/Dockerfile.template


-ARG RELEASE_HASH 


why remove? (also same with all other similar cases...)

Because these are set by environment variables now, with envsubst in generate.bash. Previously RELEASE_HASH was supplied as --build-arg in docker build. ARG in Dockerfile is misleading.

push-all.bash

start/generate.bash

build-all.bash

shcheklein · 2021-05-14T02:19:04Z

LGTM in general, I didn't check the code behind it though. I hope @casperdcl did this :)

shcheklein · 2021-05-14T02:19:20Z

Why does CI fail?

casperdcl · 2021-05-14T05:06:59Z

LGTM in general, I didn't check the code behind it though. I hope @casperdcl did this :)

will need a second pass - I suggested obvious fixes in case that helps with CI

iesahin · 2021-05-14T15:25:09Z

CI fails because of missing Docker username/password in PR. I removed run in PR from GA. I'll test it afterward.

iesahin · 2021-05-14T16:43:51Z

It looks the scripts are running as expected. I run

rm katacoda/build start/build -rf && TAG_PREFIX=emresult ./generate-all.bash && TAG_PREFIX=emresult ./build-all.bash && TAG_PREFIX=emresult ./push-all.bash

and all images are generated and pushed.

e.g. https://hub.docker.com/repository/registry-1.docker.io/emresult/doc-start/tags?page=1&ordering=last_updated

I'll run these containers, test a few commands if they work and push containers to dvcorg.

iesahin · 2021-05-14T16:45:28Z

I'll add example-get-started and dvc-checkpoints-mnist images as well. But let's push these first.

casperdcl

Should probably use the GH interface (https://github.com/iterative/dvc-doc-containers/pull/12/files) to respond to reviews (including e.g. "add suggestion to batch" functionality to commit changes)

casperdcl · 2021-05-17T05:40:33Z

.github/workflows/build-all.yml

+  # pull_request:
+  #   branches: [ master ]


TODO: revert

casperdcl · 2021-05-17T05:41:32Z

.github/workflows/build-all.yml

+      - 
+        name: Generate containers from templates
+        run: ./generate-all.bash


appropriately named one-line run command makes name meaningless (same for the cases below)

Suggested change

-

name: Generate containers from templates

run: ./generate-all.bash

- run: ./generate-all.bash

casperdcl · 2021-05-17T05:44:19Z

build-all.bash

+    dockerdir=$(dirname "${filepath}")
+    tagfile="${dockerdir}/Dockertag"
+    if [ -f "${tagfile}" ] ; then
+        TAG=$(head -n 1 "${tagfile}")
+    else
+        TAG=$(echo "${dockerdir:3:100}" | tr '/ ' '--')


Suggested change

dockerdir=$(dirname "${filepath}")

tagfile="${dockerdir}/Dockertag"

if [ -f "${tagfile}" ] ; then

TAG=$(head -n 1 "${tagfile}")

else

TAG=$(echo "${dockerdir:3:100}" | tr '/ ' '--')

dockerdir="$(dirname "${filepath}")"

tagfile="${dockerdir}/Dockertag"

if [ -f "${tagfile}" ] ; then

TAG="$(head -n 1 "${tagfile}")"

else

TAG="$(echo "${dockerdir:3:100}" | tr '/ ' '--')"

casperdcl · 2021-05-17T05:44:48Z

generate-all.bash

+HERE="$( cd "$(dirname "$0")" ; pwd -P )"
+export TAG_PREFIX="${TAG_PREFIX:-dvcorg}"
+
+find . -name generate.* | sort | while read -r filepath ; do 


Suggested change

find . -name generate.* | sort | while read -r filepath ; do

find . -name 'generate.*' | sort | while read -r filepath ; do

casperdcl · 2021-05-17T05:45:13Z

generate-all.bash

+    filename=$(basename -- "$filepath")
+    dirname=$(dirname -- "$filepath")
+    ext="${filename##*.}"
+    pushd $dirname


Suggested change

filename=$(basename -- "$filepath")

dirname=$(dirname -- "$filepath")

ext="${filename##*.}"

pushd $dirname

filename="$(basename -- "$filepath")"

dirname="$(dirname -- "$filepath")"

ext="${filename##*.}"

pushd "$dirname"

casperdcl · 2021-05-17T05:48:11Z

katacoda/generate.bash

+find ${HERE} -name Dockerfile.template | sort | while read -r filepath  ; do
+    ABSOLUTE_SOURCE=$(dirname "$filepath")
+    SOURCE_DIR=$(realpath --relative-to="${HERE}" "${ABSOLUTE_SOURCE}")
+    TARGET_DIR="${HERE}/build/${SOURCE_DIR}"
+    if [ ! -d "${TARGET_DIR}" ] ; then
+        mkdir -p "${TARGET_DIR}"
+    fi
+    cp -fr ${ABSOLUTE_SOURCE}/* "${TARGET_DIR}"


Suggested change

find ${HERE} -name Dockerfile.template | sort | while read -r filepath ; do

ABSOLUTE_SOURCE=$(dirname "$filepath")

SOURCE_DIR=$(realpath --relative-to="${HERE}" "${ABSOLUTE_SOURCE}")

TARGET_DIR="${HERE}/build/${SOURCE_DIR}"

if [ ! -d "${TARGET_DIR}" ] ; then

mkdir -p "${TARGET_DIR}"

fi

cp -fr ${ABSOLUTE_SOURCE}/* "${TARGET_DIR}"

find "${HERE}" -name Dockerfile.template | sort | while read -r filepath ; do

ABSOLUTE_SOURCE="$(dirname "$filepath")"

SOURCE_DIR="$(realpath --relative-to="${HERE}" "${ABSOLUTE_SOURCE}")"

TARGET_DIR="${HERE}/build/${SOURCE_DIR}"

if [ ! -d "${TARGET_DIR}" ] ; then

mkdir -p "${TARGET_DIR}"

fi

cp -fr "${ABSOLUTE_SOURCE}"/* "${TARGET_DIR}"

casperdcl · 2021-05-17T05:50:04Z

push-all.bash

+    dockerdir=$(dirname "${filepath}")
+    tagfile="${dockerdir}/Dockertag"
+    if [ -f "${tagfile}" ] ; then
+        TAG=$(head -n 1 "${tagfile}")
+    else
+        TAG=$(echo "${dockerdir:3:100}" | tr '/ ' '--')


Suggested change

dockerdir=$(dirname "${filepath}")

tagfile="${dockerdir}/Dockertag"

if [ -f "${tagfile}" ] ; then

TAG=$(head -n 1 "${tagfile}")

else

TAG=$(echo "${dockerdir:3:100}" | tr '/ ' '--')

dockerdir="$(dirname "${filepath}")"

tagfile="${dockerdir}/Dockertag"

if [ -f "${tagfile}" ] ; then

TAG="$(head -n 1 "${tagfile}")"

else

TAG="$(echo "${dockerdir:3:100}" | tr '/ ' '--')"

casperdcl · 2021-05-17T05:53:04Z

start/Dockerfile.template

@@ -1,28 +1,25 @@
-FROM ubuntu:20.04
+FROM python:3.8


does the katacode base have to be changed as well?

should this be slim?

Suggested change

FROM python:3.8

FROM python:3.8-slim

casperdcl · 2021-05-17T05:54:27Z

start/generate.bash

+
+if [[ -d build ]] ; then
+    echo "Please delete ${PWD}/build directory first."
+    exit


Suggested change

exit

exit 1

casperdcl · 2021-05-17T05:56:42Z

start/generate.bash

+
+    export GIT_TAG="${TAG}"
+
+    git clone build/dvc-get-started -b "${GIT_TAG}" "${TAG_DIR}/dvc-get-started"


Suggested change

git clone build/dvc-get-started -b "${GIT_TAG}" "${TAG_DIR}/dvc-get-started"

git clone "file://$PWD/build/dvc-get-started/.git" -b "${GIT_TAG}" "${TAG_DIR}/dvc-get-started"

iesahin added 6 commits May 13, 2021 16:57

Split push from build-all and added Dockerfile generator

952e4a1

added Dockerfile generation for dvc-get-started

5cbd699

Renamed TAGPREFIX to TAG_PREFIX

cf0db6b

moved start-script to start

cc248de

updated katacoda docker generation to be script based

bf9d59c

updated Github Action with the script and fixed paths

340ede7

iesahin requested review from casperdcl and shcheklein May 13, 2021 17:18

casperdcl suggested changes May 13, 2021

View reviewed changes

shcheklein reviewed May 14, 2021

View reviewed changes

build-all.bash Outdated Show resolved Hide resolved

iesahin added 11 commits May 14, 2021 12:05

removed generate-all call from build

54d42d7

fixed zsh indexing in auto tag generation

179a15d

checks whether the TAG has TAG_PREFIX as prefix and doesn't add if so

2399964

fix TAG_PREFIX check

05d8c1a

the workflow runs the generate, build, push separately now

0c5165f

added -u options to all shell scripts

45f9c11

disabled action on pull request

37c2b0e

removed RELEASE_HASH from top level scripts

32dc461

minor fixes on quotations and DIR->HERE

7f8f944

repository is first cloned to build/ now

db05f23

cd -> pushd/popd

2eb442c

iesahin added 5 commits May 14, 2021 16:20

quotation marks around paths

b9a12ab

find $HERE -> find .

7dff837

add TAG_PREFIX to Dockertag

2676eef

run dvc pull

b2ee0d5

for -> while

dc44686

iesahin requested a review from casperdcl May 15, 2021 15:12

casperdcl suggested changes May 17, 2021

View reviewed changes

iesahin closed this Nov 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Dockerfile and Dockertag generation from templates #12

Add Dockerfile and Dockertag generation from templates #12

iesahin commented May 13, 2021 •

edited

Loading

casperdcl May 13, 2021

iesahin May 14, 2021

casperdcl May 13, 2021

iesahin May 14, 2021

shcheklein commented May 14, 2021

shcheklein commented May 14, 2021

casperdcl commented May 14, 2021

iesahin commented May 14, 2021

iesahin commented May 14, 2021

iesahin commented May 14, 2021

casperdcl left a comment

casperdcl May 17, 2021

casperdcl May 17, 2021

casperdcl May 17, 2021

casperdcl May 17, 2021

casperdcl May 17, 2021

casperdcl May 17, 2021

casperdcl May 17, 2021

casperdcl May 17, 2021

casperdcl May 17, 2021

casperdcl May 17, 2021


		RELEASE_HASH=$(curl -s https://api.github.com/repos/iterative/dvc/releases/latest \| sha256sum \| cut -c -8)
		./generate-all.bash

	find . -name generate.* \| sort \| while read -r filepath ; do
	find . -name 'generate.*' \| sort \| while read -r filepath ; do


		export GIT_TAG="${TAG}"

		git clone build/dvc-get-started -b "${GIT_TAG}" "${TAG_DIR}/dvc-get-started"

	git clone build/dvc-get-started -b "${GIT_TAG}" "${TAG_DIR}/dvc-get-started"
	git clone "file://$PWD/build/dvc-get-started/.git" -b "${GIT_TAG}" "${TAG_DIR}/dvc-get-started"


		ARG RELEASE_HASH

Add Dockerfile and Dockertag generation from templates #12

Add Dockerfile and Dockertag generation from templates #12

Conversation

iesahin commented May 13, 2021 • edited Loading

DVC Get Started Images

Katacoda Image changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shcheklein commented May 14, 2021

shcheklein commented May 14, 2021

casperdcl commented May 14, 2021

iesahin commented May 14, 2021

iesahin commented May 14, 2021

iesahin commented May 14, 2021

casperdcl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iesahin commented May 13, 2021 •

edited

Loading