Fix race condition when looking up reaper (ryuk) container #2508
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for testcontainers-go ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
This was referenced Apr 22, 2024
|
Hey team, just bumping this PR. Would love to hear feedback and if there's anything I can do to help move it forward 🙏 Thanks for all your efforts on Testcontainers! Great project. |
Hi @emetsger I cannot reproduce this yet: I've run run it even with count=10, with no errors. Is there anything you think I can do to reproduce it? |
|
Thank you for trying to reproduce!
I’m using Rancher, so I wonder if that may be contributing to the
discrepancy.
What container engine/version are you on? I’ll be back at my computer
later this weekend and I can do some more testing on my side. Thank you!
…On Fri, Apr 26, 2024 at 11:35 AM Manuel de la Peña ***@***.***> wrote:
Clone this test case repo
<https://github.com/emetsger/testcontainers-ryuk-testcase> and run go
test -count=1 -v ./... and observe the errors.
Hi @emetsger <https://github.com/emetsger> I cannot reproduce this yet:
go test -count=1 -v ./...
=== RUN TestA
2024/04/26 17:27:38 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 78+testcontainerscloud (via Testcontainers Desktop 1.10.1)
API Version: 1.43
Operating System: Ubuntu 22.04.4 LTS
Total Memory: 15779 MB
Resolved Docker Host: tcp://127.0.0.1:54509
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: a077cc8ef4acf0841cd4b227aeac15cf8611190b47530d735cb90d3993d5d084
Test ProcessID: b81dbac4-46a1-4e91-a577-a7e47d57518e
2024/04/26 17:27:38 🔥 Reaper obtained from Docker for this test session 494993ccefba558a0d64b9ddce7d8db30444e35a49bf12d9cbd8ef205c23e05f
2024/04/26 17:27:38 🐳 Creating container for image docker.io/postgres:16-alpine
2024/04/26 <http://docker.io/postgres:16-alpine2024/04/26> 17:27:38 ✅ Container created: 0b7ae9cbdeb1
2024/04/26 17:27:38 🐳 Starting container: 0b7ae9cbdeb1
2024/04/26 17:27:39 ✅ Container started: 0b7ae9cbdeb1
2024/04/26 17:27:39 🚧 Waiting for container id 0b7ae9cbdeb1 image: docker.io/postgres:16-alpine. Waiting for: &{timeout:<nil> deadline:0x140003c6e88 Strategies:[0x140003e2c30]}
2024/04/26 17:27:41 🔔 Container is ready: 0b7ae9cbdeb1
2024/04/26 17:27:41 🐳 Terminating container: 0b7ae9cbdeb1
2024/04/26 17:27:41 🚫 Container terminated: 0b7ae9cbdeb1
--- PASS: TestA (3.65s)
PASS
ok tc-debug/a 4.012s
=== RUN TestB
2024/04/26 17:27:37 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 78+testcontainerscloud (via Testcontainers Desktop 1.10.1)
API Version: 1.43
Operating System: Ubuntu 22.04.4 LTS
Total Memory: 15779 MB
Resolved Docker Host: tcp://127.0.0.1:54509
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: a077cc8ef4acf0841cd4b227aeac15cf8611190b47530d735cb90d3993d5d084
Test ProcessID: 2a7fd6eb-d055-4e31-aaa6-00cf7cd47542
2024/04/26 17:27:37 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/04/26 17:27:38 ✅ Container created: 494993ccefba
2024/04/26 17:27:38 🐳 Starting container: 494993ccefba
2024/04/26 17:27:38 ✅ Container started: 494993ccefba
2024/04/26 17:27:38 🚧 Waiting for container id 494993ccefba image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/04/26 17:27:38 🔔 Container is ready: 494993ccefba
2024/04/26 17:27:38 🐳 Creating container for image docker.io/postgres:16-alpine
2024/04/26 <http://docker.io/postgres:16-alpine2024/04/26> 17:27:38 ✅ Container created: 8386f359812f
2024/04/26 17:27:38 🐳 Starting container: 8386f359812f
2024/04/26 17:27:39 ✅ Container started: 8386f359812f
2024/04/26 17:27:39 🚧 Waiting for container id 8386f359812f image: docker.io/postgres:16-alpine. Waiting for: &{timeout:<nil> deadline:0x1400037ef28 Strategies:[0x1400039ebd0]}
2024/04/26 17:27:41 🔔 Container is ready: 8386f359812f
2024/04/26 17:27:41 🐳 Terminating container: 8386f359812f
2024/04/26 17:27:41 🚫 Container terminated: 8386f359812f
--- PASS: TestB (3.92s)
PASS
ok tc-debug/b 4.104s
=== RUN TestC
2024/04/26 17:27:38 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 78+testcontainerscloud (via Testcontainers Desktop 1.10.1)
API Version: 1.43
Operating System: Ubuntu 22.04.4 LTS
Total Memory: 15779 MB
Resolved Docker Host: tcp://127.0.0.1:54509
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: a077cc8ef4acf0841cd4b227aeac15cf8611190b47530d735cb90d3993d5d084
Test ProcessID: 0b1569fa-f57c-46af-afe3-b26dc818fbc8
2024/04/26 17:27:38 🔥 Reaper obtained from Docker for this test session 494993ccefba558a0d64b9ddce7d8db30444e35a49bf12d9cbd8ef205c23e05f
2024/04/26 17:27:38 🐳 Creating container for image docker.io/postgres:16-alpine
2024/04/26 <http://docker.io/postgres:16-alpine2024/04/26> 17:27:38 ✅ Container created: b2b8530b534d
2024/04/26 17:27:38 🐳 Starting container: b2b8530b534d
2024/04/26 17:27:38 ✅ Container started: b2b8530b534d
2024/04/26 17:27:38 🚧 Waiting for container id b2b8530b534d image: docker.io/postgres:16-alpine. Waiting for: &{timeout:<nil> deadline:0x1400037ef28 Strategies:[0x1400039ebd0]}
2024/04/26 17:27:40 🔔 Container is ready: b2b8530b534d
2024/04/26 17:27:40 🐳 Terminating container: b2b8530b534d
2024/04/26 17:27:41 🚫 Container terminated: b2b8530b534d
--- PASS: TestC (3.28s)
PASS
ok tc-debug/c 3.756s
=== RUN TestD
2024/04/26 17:27:38 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 78+testcontainerscloud (via Testcontainers Desktop 1.10.1)
API Version: 1.43
Operating System: Ubuntu 22.04.4 LTS
Total Memory: 15779 MB
Resolved Docker Host: tcp://127.0.0.1:54509
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: a077cc8ef4acf0841cd4b227aeac15cf8611190b47530d735cb90d3993d5d084
Test ProcessID: bb20c031-95bb-4791-af3b-e4b1ecf5d818
2024/04/26 17:27:38 🔥 Reaper obtained from Docker for this test session 494993ccefba558a0d64b9ddce7d8db30444e35a49bf12d9cbd8ef205c23e05f
2024/04/26 17:27:38 🐳 Creating container for image docker.io/postgres:16-alpine
2024/04/26 <http://docker.io/postgres:16-alpine2024/04/26> 17:27:38 ✅ Container created: 2f51f8a61406
2024/04/26 17:27:38 🐳 Starting container: 2f51f8a61406
2024/04/26 17:27:39 ✅ Container started: 2f51f8a61406
2024/04/26 17:27:39 🚧 Waiting for container id 2f51f8a61406 image: docker.io/postgres:16-alpine. Waiting for: &{timeout:<nil> deadline:0x140003f8e58 Strategies:[0x14000458bd0]}
2024/04/26 17:27:41 🔔 Container is ready: 2f51f8a61406
2024/04/26 17:27:41 🐳 Terminating container: 2f51f8a61406
2024/04/26 17:27:41 🚫 Container terminated: 2f51f8a61406
--- PASS: TestD (3.32s)
PASS
ok tc-debug/d 3.933s
I've run run it even with count=10, with no errors. Is there anything you
think I can do to reproduce it?
—
Reply to this email directly, view it on GitHub
<#2508 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABD4GRM6OFYD2Q3WX5QWJDY7JX5DAVCNFSM6AAAAABGR5VJXSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZZGYZDQOJRHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I'm using both Docker for Mac and Testcontainers Cloud. |
|
I am experiencing a similar issue in (Jenkins) CI, although it's flaky - only on some runs there is a connection refused error as follows. Note that I am unable to reproduce this on my machine (with different container engine & version). |
|
Do you know what docker engine/version is running in your CI?
…On Thu, May 2, 2024 at 6:00 AM Simon Bos ***@***.***> wrote:
I am experiencing a similar issue in (Jenkins) CI, although it's flaky -
only on some runs there is a connection refused error as follows.
[2024-05-02T09:39:21.954Z] 2024/05/02 09:39:12 github.com/testcontainers/testcontainers-go - Connected to docker:
[2024-05-02T09:39:21.954Z] Server Version: 20.10.23
[2024-05-02T09:39:21.954Z] API Version: 1.41
[2024-05-02T09:39:21.954Z] Operating System: Rocky Linux 9.2 (Blue Onyx)
[2024-05-02T09:39:21.954Z] Total Memory: 15982 MB
[2024-05-02T09:39:21.954Z] Resolved Docker Host: unix:///var/run/docker.sock
[2024-05-02T09:39:21.954Z] Resolved Docker Socket Path: /var/run/docker.sock
[2024-05-02T09:39:21.954Z] Test SessionID: 9644673c1b6ea45d4962460bb5e7a2a9ba32c1f293bf4166d0e43de70c021a86
[2024-05-02T09:39:21.954Z] Test ProcessID: 05a16b84-4e96-4b61-ac3d-29f63de4d258
[2024-05-02T09:39:21.954Z] 2024/05/02 09:39:12 🐳 Creating container for image testcontainers/ryuk:0.7.0
[2024-05-02T09:39:21.954Z] 2024/05/02 09:39:13 🔥 Reaper obtained from Docker for this test session 1179229b008b48672081fd3372999389793505d2b013d054c454a6838a6b5544
[2024-05-02T09:39:21.954Z] <redacted>: dial tcp <redacted>:35412: connect: connection refused: Connecting to Ryuk on <redacted>:35412 failed: connecting to reaper failed: failed to create container
Note that I am unable to reproduce this on my machine (with different
container engine & version).
—
Reply to this email directly, view it on GitHub
<#2508 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABD4GQZUZCQO7VRD4VO4U3ZAIFEHAVCNFSM6AAAAABGR5VJXSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJQGA3TCMRYGE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
It's running "Docker Engine - Community" version 20.10.23. There is some extra info in the logs above. |
|
@simonbos what docker engine/version are you running on your local machine? |
|
Edit: updated with Colima results (thanks @lachauj for bringing this to our attention!)
(link to gist) |
|
@mdelapenya I ran the test case against a variety of container managers/docker engines (see table above) I was able to successfully run my test case against all the container managers except for Rancher.
@mdelapenya can you install Rancher 1.13.1 (latest at the time of this writing) and attempt to reproduce on your side? Edit: test matrix table updated to include Colima, which also fails 🚫 |
|
Wow! @emetsger thank you so much for dedicating that amount of time to reproduce the potential bug in so many container environments. Very proud of you 🙇 I'll install rancher today and try to reproduce it. Will ping you back here |
Bablzz
reviewed
May 6, 2024
It's the least I can do: trying to pay it forward! |
|
Thanks for the effort @emetsger !
Docker Desktop for Mac v4.29.0. |
|
Same issue with Colima on MacOS Sonoma, works randomly (95% of the time it triggers the issue) but works at 100% with Docker Desktop. |
|
@lachauj I could repeat your failure with my test case ✅ I updated the test matrix table above to include Colima. Edit: This PR and the associated moby/ryuk patch does not fix Colima. I rebased testcontainers/moby-ryuk#128 against the latest Edit #2: The only way I can get the test case to pass with colima was to introduce an artificial delay, here, before returning the ryuk container (e.g. with So there seems to be something else going on with Colima that isn't addressed by the healthcheck patches.
|
Summary Findings
DiscussionAfter testing a number of different container managers, it's clear that their behavior differs.
I think adding healthcheck support is a good idea, but it isn't going to solve the root issue, whatever that may be. In the meantime, there are two different workarounds:
The fact that this bug is only raised when multiple test binaries are executing testcontainers-go code concurrently makes me think that there is some kind of state where the ryuk container can get returned "early", before the container manager is ready to accept connections to it. |
FixOk, I believe cd0a106 has the fix. It works with Rancher and Colima, 100% of the time ✅ 🎉 There is a race between DescriptionThe container request composed by It is possible for cd0a106 updates @mdelapenya @Bablzz: what do you think? With this fix do you think we can move this PR forward? @simonbos @lachauj: can you try this PR out and see if it fixes your errors? |
emetsger
changed the title
|
hello @emetsger ! I hope you are doing well:) |
|
Hi @emetsger ! Thanks a lot, that's works perfectly! |
lachauj
reviewed
May 20, 2024
…er before returning.
… being started by newReaper in a separate process. A race between newReaper and lookUpReaperContainer occurs: - newReaper creates the container with a ContainerRequest.WaitingFor = wait.ForListeningPort(listeningPort) - newReaper starts the container - lookUpReaperContainer obtains the container and returns. - newReaper invokes the readiness hook wait.ForListeningPort(listeningPort).
|
Hey @emetsger, applying this fix seems to work for me - I haven't seen any |
mdelapenya
reviewed
May 21, 2024
There was a problem hiding this comment.
@emetsger thanks a lot again for your detailed issue report and the work done for reproducing the bug. As a maintainer, it helped me a lot to identify the root cause, specially this: #2508 (comment)
I added a few comments, let's address and I think we are done here, not needing the healthcheck in the Ryuk container (please see @kiview's comment)
|
@emetsger I added a commit on top for simplification. If you agree, I'm fine merging this one. Thanks! |
|
Folks, I'm prioritising this fix, so please take a look at the few commits I added on top of @emetsger's fantastic work. I'd appreciate your feedback so I can make this into the project asap. Thanks! |
|
BTW I had time to install Rancher Desktop and configure the tests for it, and this branch works perfectly. I verified the repro code with and without this branch being applied, and the fix is effective: Applying this branch: For that reason I'm merging it into main. Thanks everybody here providing such valuable feedback 🙇 |
mdelapenya
added a commit
that referenced
this pull request
Jun 10, 2024
* If the ryuk container has a health status, wait for a healthy container before returning. * Use docker/types for health status, but also check for the zero value. * Wait for the ryuk port to be available in case the container is still being started by newReaper in a separate process. A race between newReaper and lookUpReaperContainer occurs: - newReaper creates the container with a ContainerRequest.WaitingFor = wait.ForListeningPort(listeningPort) - newReaper starts the container - lookUpReaperContainer obtains the container and returns. - newReaper invokes the readiness hook wait.ForListeningPort(listeningPort). * Fix whitespace. * chore: simplify * chore: make lint * chore: change emoji in log output when waiting * chore: move inside reuseReaper --------- Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
mdelapenya
added a commit
to bearrito/testcontainers-go
that referenced
this pull request
Jun 11, 2024
* main: (48 commits) Fix race condition when looking up reaper (ryuk) container (testcontainers#2508) chore: bring golangci-lint back (testcontainers#2571) docs(compose): Fix typo docker compose docs (testcontainers#2565) Handle error properly during port forwarding initialization. (testcontainers#2550) chore: pin vearch version (testcontainers#2568) feat: add vearch module (testcontainers#2560) chore: run tests against latest Docker engine, nightly (testcontainers#2566) chore(deps): bump mkdocs-include-markdown-plugin from 6.0.4 to 6.0.7 (testcontainers#2562) Fix network accessor for port-forwarding feature (testcontainers#2551) --- (testcontainers#2549) fix: update search bar eval in mkdocs (testcontainers#2547) docs: improve contributing docs for code snippets (testcontainers#2546) chore: use a virtualenv for working with the docs site (testcontainers#2545) docs: document test session semantics (testcontainers#2544) feat(ryuk): allow to configure ryuk timeouts using env variables (testcontainers#2541) docs: fix CircleCI docs (testcontainers#2539) fix: add import to module generation (testcontainers#2537) chore: prepare for next minor development cycle (0.32.0) chore: use new version (v0.31.0) in modules and examples feat(mongodb): add replica set support via opts (testcontainers#2469) ...
|
Great! Thank you for carrying this forward and merging - I’ve been out of
pocket, appreciate the effort and community! 💪🙏
…On Mon, Jun 10, 2024 at 11:56 AM Manuel de la Peña ***@***.***> wrote:
Merged #2508
<#2508> into main.
—
Reply to this email directly, view it on GitHub
<#2508 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABD4GT437CEQL4MYIY3TR3ZGW5DRAVCNFSM6AAAAABGR5VJXSVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJTGEYDEOJXGMYDCNA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
.com>
|
|
I agree that the ryuk health check is not needed!
…On Tue, May 21, 2024 at 5:32 AM Manuel de la Peña ***@***.***> wrote:
***@***.**** commented on this pull request.
@emetsger <https://github.com/emetsger> thanks a lot again for your
detailed issue report and the work done for reproducing the bug. As a
maintainer, it helped me a lot to identify the root cause, specially this: #2508
(comment)
<#2508 (comment)>
I added a few comments, let's address and I think we are done here, not
needing the healthcheck in the Ryuk container (please see @kiview
<https://github.com/kiview>'s comment
<testcontainers/moby-ryuk#128 (comment)>
)
------------------------------
In docker.go
<#2508 (comment)>
:
> @@ -1590,6 +1592,11 @@ func containerFromDockerResponse(ctx context.Context, response types.Container)
return nil, err
}
+ // the health status of the container, if any
+ if health := container.raw.State.Health; health != nil {
Is there any case where State can be nil? If so, we can return the
container with a default healthStatus 🤔
------------------------------
In reaper.go
<#2508 (comment)>
:
> + // if a health status is present on the container, and the container is not healthy, error
+ if r.healthStatus != "" {
+ if r.healthStatus != types.Healthy && r.healthStatus != types.NoHealthcheck {
+ return fmt.Errorf("container %s is not healthy, wanted status=%s, got status=%s", resp[0].ID[:8], types.Healthy, r.healthStatus)
+ }
+ }
+
Do you think we could add this check as a readiness hook? Then I think
this fix would be more intentional. Thoughts?
------------------------------
In reaper.go
<#2508 (comment)>
:
> + Logger.Printf("⏳ Waiting for Reaper port to be ready")
+
+ var containerJson *types.ContainerJSON
+
+ if containerJson, err = reaperContainer.Inspect(ctx); err != nil {
+ return nil, fmt.Errorf("failed to inspect reaper container %s: %w", reaperContainer.ID[:8], err)
+ }
+
+ if containerJson != nil && containerJson.NetworkSettings != nil {
+ for port := range containerJson.NetworkSettings.Ports {
+ err := wait.ForListeningPort(port).
+ WithPollInterval(100*time.Millisecond).
+ WaitUntilReady(ctx, reaperContainer)
+ if err != nil {
+ return nil, fmt.Errorf("failed waiting for reaper container %s port %s/%s to be ready: %w",
+ reaperContainer.ID[:8], port.Proto(), port.Port(), err)
+ }
+ }
+ }
+
Do you think we can add this logic to the reuseReaperContainer function so
that all invocations of that function benefit for this code?
—
Reply to this email directly, view it on GitHub
<#2508 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABD4GWBY27JFPCZY4W6PZDZDMBBHAVCNFSM6AAAAABGR5VJXSVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDANRXHEYTEOBTGQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
mdelapenya
added a commit
that referenced
this pull request
Jun 18, 2024
* If the ryuk container has a health status, wait for a healthy container before returning. * Use docker/types for health status, but also check for the zero value. * Wait for the ryuk port to be available in case the container is still being started by newReaper in a separate process. A race between newReaper and lookUpReaperContainer occurs: - newReaper creates the container with a ContainerRequest.WaitingFor = wait.ForListeningPort(listeningPort) - newReaper starts the container - lookUpReaperContainer obtains the container and returns. - newReaper invokes the readiness hook wait.ForListeningPort(listeningPort). * Fix whitespace. * chore: simplify * chore: make lint * chore: change emoji in log output when waiting * chore: move inside reuseReaper --------- Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
mdelapenya
added a commit
to mdelapenya/testcontainers-go
that referenced
this pull request
Jun 24, 2024
* main: docs: document disabling ryuk with properties (testcontainers#2603) fix: allow compose files and readers to be used together (testcontainers#2598) chore(deps): bump urllib3 from 2.2.1 to 2.2.2 (testcontainers#2590) docs: example for NATS cluster (testcontainers#2591) docs: fix deprecation for material extensions emoji (testcontainers#2592) gha: update golangci-lint to v1.59.1 (testcontainers#2588) network: WithCheckDuplicate: don't set CheckDuplicate value (testcontainers#2589) fix: proper synchronisation for start/stop log production (testcontainers#2576) chore(deps): bump github.com/docker/docker from v25.0.5 to v26.1.4 (testcontainers#2584) docs: update contributing guide (testcontainers#2586) feat: add testcontainers labels to the built images (testcontainers#2579) feat(mongodb): Make the replica set name configurable (testcontainers#2538) chore: skip reaper tests if ryuk is disabled (testcontainers#2582) chore: increase check time to 60 secs instead of 20 (testcontainers#2581) fix: prepend substitutors for built images (testcontainers#2577) docs: update colima docs (testcontainers#2578) fix(cockroachdb): Fixes cockroachdb wait strategy handling (testcontainers#2456) Fix race condition when looking up reaper (ryuk) container (testcontainers#2508) chore: bring golangci-lint back (testcontainers#2571) docs(compose): Fix typo docker compose docs (testcontainers#2565)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Updates
lookUpReaperContainerto consider the health of the ryuk container.lookUpReaperContainerwill require a healthy container before returning.lookUpReaperContainerwill return the container.Why is it important?
The ryuk container can be exposed by the Docker API before it is ready to accept connections, resulting in messages like:
Tests will fail when they would otherwise pass.
This is especially the case in Go, where test binaries are run in parallel, one for each package. If multiple packages use
testcontainers, they can obtain a reaper instance from the Docker API before the container is ready to serve connections, leading to dialing errors likeconnect: connection refused.The current workaround is to run test binaries serially, e.g.,
go test -p 1 ./..., which is not optimal, even for modest codebases.Related issues
See complementary PR, which adds a
HEALTHCHECKto the ryuk container: testcontainers/moby-ryuk#128How to test this PR
To recreate the initial problem and run the workaround
Clone this test case repo and run
go test -count=1 -v ./...and observe the errors.Run
go test -count=1 -p 1 -v ./...and tests should pass.To test the fix
Clone moby-ryuk, apply pull/128, then build a new ryuk container, and tag it:
docker build -f linux/Dockerfile -t local/ryuk .docker tag local/ryuk testcontainers/ryuk:0.7.0Check out this branch/PR locally.
Clone the test case repo and update
go.modto point to this branch via a replace, e.g.:replace github.com/testcontainers/testcontainers-go => /Users/esm/workspaces/testcontainers-goRunning
go test -count=1 -v ./...should pass.