Skip to content

chore: Build docker compose images nightly and use them in CI#2579

Closed
MOZGIII wants to merge 20 commits intomasterfrom
nightly-images
Closed

chore: Build docker compose images nightly and use them in CI#2579
MOZGIII wants to merge 20 commits intomasterfrom
nightly-images

Conversation

@MOZGIII
Copy link
Copy Markdown
Contributor

@MOZGIII MOZGIII commented May 12, 2020

This is an attempt to finally streamline the CI and ensure we have quick builds in CI and a unified environment configs across local and CI tasks.

To make things work, I had to introduce the following changes:

  • configure images for all the services at docker-compose.yml
  • refresh the CI images build script
  • set up automated nightly builds for CI images (the auto builds at the docker hub end are to be disabled)
  • made the per-push CI flow use pre-built docker images - this allows us to properly control build environment at CI, but eliminates the image build step, dramatically speeding up the flow
  • solve the issue with tmp files being on overlayfs (cause of issues with tests failing in docker)

While working on this, I uncovered a few issues that I'm not planning on solving in this PR:

  • build image Dockerfiles are not cache-friendly, we should correct this
  • docker-compose.yml, Makefile and .github/workflow/code.yml each contains configuration for the same "unit"s, but it's all coded separately; there's way too much attention required to implement keep things in order and consistent, and it's hard to maintain; we should revisit the approach, and root all those places in a single source of truth

@MOZGIII MOZGIII changed the title chore: Build docker compose images nightly and used them in CI chore: Build docker compose images nightly and use them in CI May 12, 2020
@MOZGIII MOZGIII requested a review from binarylogic May 12, 2020 09:47
@MOZGIII MOZGIII marked this pull request as ready for review May 12, 2020 09:47
@MOZGIII MOZGIII requested a review from a user May 12, 2020 09:47
@MOZGIII
Copy link
Copy Markdown
Contributor Author

MOZGIII commented May 12, 2020

The rest of the CI failures are actual errors that went undetected with the current CI setup. Let's merge this and fix the errors in other PRs.

@fanatid fanatid mentioned this pull request May 12, 2020
@MOZGIII MOZGIII removed the request for review from a user May 12, 2020 18:24
@MOZGIII MOZGIII force-pushed the nightly-images branch 4 times, most recently from 0f655c6 to eccffa6 Compare May 12, 2020 22:24
@MOZGIII
Copy link
Copy Markdown
Contributor Author

MOZGIII commented May 13, 2020

This PR still has one failing job - the test-unit. The interesting part is that it only consistently fails if we’re running the job within docker.
@LucioFranco you mentioned you were working on tests troubleshooting - is this among the things you’re already looking into?

@MOZGIII MOZGIII force-pushed the nightly-images branch 5 times, most recently from 49c16ee to 6a36ff4 Compare May 14, 2020 12:48
MOZGIII added 10 commits May 14, 2020 16:36
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
MOZGIII added 10 commits May 14, 2020 16:36
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
…script

Signed-off-by: MOZGIII <mike-n@narod.ru>
…thout docker

Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
19.04 is deprecated and is not longer updated at Docker Hub.
20.04, in turn, in the fresh latest LTS release.

Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
@MOZGIII
Copy link
Copy Markdown
Contributor Author

MOZGIII commented May 18, 2020

This is blocked by #2618

@Hoverbear
Copy link
Copy Markdown
Contributor

Is this still relevant @MOZGIII ?

@MOZGIII
Copy link
Copy Markdown
Contributor Author

MOZGIII commented Jun 12, 2020

The PR - definitely not. We might still want to build an environment image every night though.

@MOZGIII MOZGIII closed this Jun 12, 2020
@MOZGIII MOZGIII deleted the nightly-images branch June 12, 2020 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants