Docker Hub build failures

[andrew-gardener]

Summary:

Seems that docker hub image builds have been failing for the past couple months

https://hub.docker.com/r/instructure/canvas-lms/builds/

Last stable image update 4 months ago (master and beta 5 months ago)

https://hub.docker.com/r/instructure/canvas-lms/tags/

Steps to reproduce:

N/A

Expected behavior:

Docker Hub images should build successfully

Additional notes:

Not sure if you are already aware or not

[tierra]

Hi @andrew-gardener , we've been aware of these for a while, but haven't been able to come up with a game plan on how to fix them just yet. There's actually nothing wrong with the docker image builds as far as the software side is concerned, I'm able to build these images locally just fine. It seems to be an issue with Docker Hub's lack of system resources for building the images. There's just not enough memory or CPU to handle building Canvas LMS assets for the image within the 2 hour time limit that Docker Hub imposes.

We're looking at other options, notably, building the image on our own infrastructure at Instructure. I'll try to keep you updated if we're able to resolve this.

In the mean time, if you need updated images, you can always build them yourself, and push them up somewhere for your own builds.

[andrew-gardener]

I've just tried building the image locally off of stable and got

Step 23/28 : RUN bundle install --jobs 8   && yarn install --pure-lockfile
 ---> Running in 6164dc653f23
/home/docker/.gem/ruby/2.4.0/bin/bundle:23:in `load': cannot load such file -- /usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.1/exe/bundle (LoadError)
	from /home/docker/.gem/ruby/2.4.0/bin/bundle:23:in `<main>'
The command '/bin/sh -c bundle install --jobs 8   && yarn install --pure-lockfile' returned a non-zero code: 1

Which is the same error that docker hub is getting https://hub.docker.com/r/instructure/canvas-lms/builds/b9l8nahsj3b38pec5myktcq/

It might be something simple like a missing file from a clean checkout?

[tierra]

That was fixed with #1230 in 150dfe1 already, and we don't plan to backport that to stable branch.

[andrew-gardener]

@tierra thanks for the info

[andrew-gardener]

@tierra

I changed the docker file with the fix in #1230 and now I get:

Step 28/28 : RUN COMPILE_ASSETS_NPM_INSTALL=0 bundle exec rake canvas:compile_assets
 ---> Running in c1a35d0bd2c3
...
...
--> Building DEVELOPMENT webpack bundles
--> Starting: 'js:webpack_production'
--> Building PRODUCTION webpack bundles
yarn run v1.3.2
yarn run v1.3.2
$ NODE_ENV=production gulp rev & node --max_old_space_size=4096 $(yarn bin)/webpack --color --config webpack.production.config.js
$ gulp rev & webpack --progress --color
[22:24:19] Using gulpfile /usr/src/app/gulpfile.js
[22:24:19] Starting 'rev'...
[22:24:19] Using gulpfile /usr/src/app/gulpfile.js
[22:24:19] Starting 'rev'...
Happy[babel]: Version: 3.1.0. Using cache? Yes. Threads: 4 (shared pool)
Happy[babel]: No cache was found, starting fresh.
Happy[babel]: All set; signaling webpack to proceed.
Happy[coffee]: Version: 3.1.0. Using cache? Yes. Threads: 4 (shared pool)
Happy[coffee]: No cache was found, starting fresh.
Happy[coffee]: All set; signaling webpack to proceed.
Happy[handlebars-i18n]: Version: 3.1.0. Using cache? Yes. Threads: 4 (shared pool)
Happy[handlebars-i18n]: No cache was found, starting fresh.
Happy[handlebars-i18n]: All set; signaling webpack to proceed.
Happy[handlebars-ember]: Version: 3.1.0. Using cache? Yes. Threads: 4 (shared pool)
Happy[handlebars-ember]: No cache was found, starting fresh.
Happy[handlebars-ember]: All set; signaling webpack to proceed.
 10% building modules 1/1 modules 0 activeHappy[babel]: Version: 3.1.0. Using cache? No. Threads: 4 (shared pool)
Happy[babel]: All set; signaling webpack to proceed.
Happy[coffee]: Version: 3.1.0. Using cache? No. Threads: 4 (shared pool)
 10% building modules 1/2 modules 1 active ...pts/vendor/mediaelement-and-player.jsHappy[coffee]: All set; signaling webpack to proceed.
Happy[handlebars-i18n]: Version: 3.1.0. Using cache? No. Threads: 4 (shared pool)
Happy[handlebars-i18n]: All set; signaling webpack to proceed.
Happy[handlebars-ember]: Version: 3.1.0. Using cache? No. Threads: 4 (shared pool)
Happy[handlebars-ember]: All set; signaling webpack to proceed.
 10% building modules 2/2 modules 0 active[22:24:24] [22:24:24] Finished 'rev' after 4.74 s
Finished 'rev' after 4.68 s
 69% building modules 3294/3348 modules 54 active ...app/app/jsx/bundles/question_banks.js[BABEL] Note: The code generator has deoptimised the styling of "/usr/src/app/public/javascripts/translations/gradebook.js" as it exceeds the max of "500KB".
 68% building modules 3352/3416 modules 64 active ...busAppointmentGroupsCollection.coffee[BABEL] Note: The code generator has deoptimised the styling of "/usr/src/app/public/javascripts/translations/gradebook.js" as it exceeds the max of "500KB".
 95% emittingKilled                                                                       
error Command failed with exit code 137.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

which seems to hang indefinitely. Is there another fix on master for this one?

Also is it possible for you to manually push the stable image to docker hub since you seem to be able to build it locally? Thanks

[omarkhan]

Looks like it could be webpack/webpack#2908. Are you using windows?

[andrew-gardener]

using macOS High Sierra 10.13.2

[omarkhan]

Does your docker VM have enough memory? 8gb is recommended

[andrew-gardener]

@omarkhan Thanks, using current stable branch and increasing the docker VM memory limit to 8gb has allowed me to successfully build the image locally.

[omarkhan]

Great! Closing

[andrew-gardener]

Shouldn't this be closed when images are successfully published to docker hub again (the original issue)?

Building images locally is not something I want to do long term if you plan on eventually pushing to docker hub again.

[tierra]

You're right, and sorry about that. We are still looking into options for re-enabling the Docker Hub image builds.

[bootstraponline]

I setup bitrise to build the docker images.