Add port monitoring test script

[jandubois]

It verifies that when a container is destroyed, its ports can be reused immediately and are not subject to being freed by a polling loop.

I wrote this to validate #4066, but I think we should keep things like this in the repo, just in case we need them again. They also serve as samples to write other similar tests in the future.

It should not be run in CI; it is not totally reliable: there is always a chance that the server will take longer to actually respond to requests after opening the port. The test works around it by retrying once on curl exit code 52, but I have it seen fail once by refusing to connect.

Anyways, running the test on master fails:

❯ time TEMPLATE=docker ./lib/bats-core/bin/bats -T extras/port-monitor.bats
port-monitor.bats
 ✗ Verify that the container is working [4038]
   (from function `assert_success' in file lib/bats-assert/src/assert_success.bash, line 45,
    from function `verify_port' in file extras/port-monitor.bats, line 47,
    in test file extras/port-monitor.bats, line 55)
     `verify_port' failed
   docker.io/library/nginx:latest
   7f19e4ccde1a38c72b96b0be4824dc0eb06dcd11445fc030faa0a5ff4191171f

   -- command failed --
   status : 7
   output :
   --

 ✗ Stop and restart the container multiple times [558]
   (from function `assert_success' in file lib/bats-assert/src/assert_success.bash, line 45,
    from function `verify_port' in file extras/port-monitor.bats, line 47,
    in test file extras/port-monitor.bats, line 62)
     `verify_port' failed
   nginx
   nginx
   4866814feab1b4dfdec7833af5206c945dc4fff58248883f2f5e4cbb16cf882f

   -- command failed --
   status : 7
   output :
   --


2 tests, 2 failures in 53 seconds

TEMPLATE=docker ./lib/bats-core/bin/bats -T extras/port-monitor.bats  3.20s user 2.77s system 11% cpu 53.326 total

I've been running on the #4066 PR branch multiple times, with both containerd and docker, and it seems to be passing regularly:

❯ time TEMPLATE=docker ./lib/bats-core/bin/bats -T extras/port-monitor.bats
port-monitor.bats
 ✓ Verify that the container is working [4930]
 ✓ Stop and restart the container multiple times [54854]

2 tests, 0 failures in 101 seconds

TEMPLATE=docker ./lib/bats-core/bin/bats -T extras/port-monitor.bats  16.64s user 14.79s system 31% cpu 1:41.11 total

❯ time TEMPLATE=default ./lib/bats-core/bin/bats -T extras/port-monitor.bats
port-monitor.bats
 ✓ Verify that the container is working [5298]
 ✓ Stop and restart the container multiple times [67924]

2 tests, 0 failures in 120 seconds

TEMPLATE=default ./lib/bats-core/bin/bats -T extras/port-monitor.bats  17.57s user 15.77s system 27% cpu 2:00.04 total

[AkihiroSuda]

Should this be part of test-templates ?

Anyway we should establish the criteria to clarify what should be in test-templates and what else should be in bats.
Maybe we should also consider rewriting test-templates in bats.
https://lima-vm.io/docs/dev/testing/

[jandubois]

Maybe we should also consider rewriting test-templates in bats.

I think we should, but it will take some time. Until then we can run them in parallel. New tests should be in BATS, I think, if possible with reasonable effort.

I don't have the time to rework the whole test-suite at once, but will add a little bit whenever I have some extra time.

[jandubois]

Should this be part of test-templates ?

The reason this particular test has a TEMPLATES variable is so that you can use it to test both containerd and docker, as the 2 container engines may have different port binding mechanisms, and they should be tested both. There is no point in running this test with all the templates.

Potentially it could be extended to run with rootful containerd/docker, but I didn't think that was necessary.

When we have more tests we can figure out a set of tags to run different groups under different scenarios, but I think it is still too early to bother with that: https://bats-core.readthedocs.io/en/stable/writing-tests.html#tagging-tests

[AkihiroSuda]

Thanks