moby-ryuk containers leak with SBT

[alex-vv]

I use version 1.6.0 with testcontainers-scala 0.14 to run scala integration tests with sbt. When running tests in the interactive mode I see that moby-ryuk container is not stopped and it is created again for each run, so after few runs docker ps shows the following:

CONTAINER ID        IMAGE                     COMMAND             CREATED              STATUS              PORTS                     NAMES
832de2504d77        bsideup/moby-ryuk:0.2.2   "/app"              31 seconds ago       Up 52 seconds       0.0.0.0:32962->8080/tcp   testcontainers-ryuk-fb8fc138-b54f-46da-ae76-f72f77ff6bc5
66330a95d32c        bsideup/moby-ryuk:0.2.2   "/app"              51 seconds ago       Up About a minute   0.0.0.0:32959->8080/tcp   testcontainers-ryuk-1abe1265-f87b-4ca8-a426-fdc005da58d0
9ce16c780c80        bsideup/moby-ryuk:0.2.2   "/app"              About a minute ago   Up About a minute   0.0.0.0:32956->8080/tcp   testcontainers-ryuk-6f6c27a5-cd03-4c1c-b6b0-e8f3633c2bfe
e6af83fa93f6        bsideup/moby-ryuk:0.2.2   "/app"              About a minute ago   Up 2 minutes        0.0.0.0:32953->8080/tcp   testcontainers-ryuk-a6b83d36-4b9b-4c93-bfd3-e37d67442092

Once I exit sbt these containers are stopped. I think this problem is not scala or sbt-specific but may appear in any case when you run a test suite repeatedly on the same JVM.

Tried version 1.5.1 and it doesn't have this problem. I guess the issue is that ResourceReaper is started in DockerClientFactory but I could not find that it is stopped anywhere.

[bsideup]

@alex-vv this container will not exit until you have an open TCP connection to it, but rest assured it will once you close it. It seems that SBT keeps some JVMs running, maybe daemon thing or something, this is why it happens.

ryuk containers are very lightweight, implemented in Golang and will not consume too many resources on your machine, you can safely ignore them since there is no leak (as you said they disappear once you exit sbt)

FYI 1.5.1 doesn't have it because we introduced Ryuk in 1.6.0 :)

[alex-vv]

Yes, in the interactive mode sbt keeps JVM running and that is expected behavior. Do I understand correctly that TCP connection to ryuk container is managed internally by DockerClientFactory and there is no way for the library user to manually close it without stopping JVM?

It is good that ryuk is lightweight but if I re-run tests continuously it still doesn't feel right to have a lot of them hanging around. Maybe if you don't want to automatically stop it, that's ok, but is it possible to provide some API to do that manually if needed?

[bsideup]

@alex-vv I'm quite sure you will have as many Ryuks as JVMs in SBTs, so it's not "a lot" but 5-8 or how many JVMs SBT keeps running?

The problem is that if you kill them manually and then some of the tests will run on that daemonized JVM your containers will not get cleaned.

[alex-vv]

@alex-vv I'm quite sure you will have as many Ryuks as JVMs in SBTs, so it's not "a lot" but 5-8 or how many JVMs SBT keeps running?

no, in my case a new ryuk gets created for each new test run, and they are all in one JVM. I can try to do some minimal reproducible case if needed.

[alex-vv]

@bsideup you can check https://github.com/alex-vv/testcontainers-sbt

[bsideup]

@alex-vv thanks, I'll check it soon 👍

[corbs9-zz]

@alex-vv we use test containers with sbt and Scala and saw similar problems. What we did was essentially ‘fork in Test := true’ which ensured that the test itself ran in a separate JVM and when exited, killed off the containers. At this point, sbt in interactive mode is still running.

[iNikem]

I believe I have the same issue with Gradle and Testcontainers 1.9.rc-1. Ryuk containers just stay lingering after test completes. Or maybe only after test is canceled. Anyway I have now 48 Ryuk containers :) In status "Exited (0)"

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

[kubukoz]

+1 for fixing this, I don't want to fork tests for the sole purpose of killing the containers after the tests. Perhaps there could be a method for killing them manually?