Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to execute tests when they are running in Jenkins with Kubernetes plugin when Pods are running in EKS #243

Closed
ikolomiyets opened this issue Sep 17, 2020 · 0 comments

Comments

@ikolomiyets
Copy link
Contributor

ikolomiyets commented Sep 17, 2020

Describe the bug
The pipeline is running in Jenkins instance with Kubernetes plugin. Both Jenkins and Kube plugin pods are running in the EKS.
Tests are failing when trying to create Reaper container, i.e. it times out when waiting for the Reaper to come alive.

To Reproduce
Set up Jenkins server with a Kubernetes cloud defined which connectes to a EKS instance.
A test scenario uses Testcontainers to create a container.
A test step just run go test.
The following error appears in the output:

$ go test
go: downloading github.com/testcontainers/testcontainers-go v0.8.0
2020/09/17 15:50:09 Starting container id: d3eb2438b8d6 image: quay.io/testcontainers/ryuk:0.2.3
2020/09/17 15:50:10 Waiting for container id d3eb2438b8d6 image: quay.io/testcontainers/ryuk:0.2.3
test: 2020/09/17 15:50:10 failed to create container: creating reaper failed: could not start container: port not found
exit status 1

Expected behavior
Test succeeds

** docker info **

$ docker info
Client:
Debug Mode: false

Server:
Containers: 52
  Running: 41
  Paused: 0
  Stopped: 11
Images: 59
Server Version: 19.03.6-ce
Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: ff48f57fc83a8c44cf4ad5d672424a98ba37ded6
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
  seccomp
   Profile: default
Kernel Version: 4.14.193-149.317.amzn2.x86_64
Operating System: Amazon Linux 2
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.69GiB
Name: ip-10-0-2-81.eu-west-1.compute.internal
ID: P2FL:MSHL:6TGF:WSMZ:XMA2:3HXS:KUTP:NKKN:LM6T:JCTP:7WOE:WXGN
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
  127.0.0.0/8
Live Restore Enabled: true

Additional context

There are two issues related to the EKS and similar setup:

  1. Docker instance on the EKS node by default has bridge network disabled. This pretty much means that any port exposure has no effect but does not fail either as in this case none network becomes default which means no networking. Reaper has currently no option to use a non default network which leads to the failure.
  2. While trying to resolve the first issue by creating a new network with bridge driver and using it instead of default one, it appeared that in this setup the IP returned by the GetGatewayIP is non-routable, again it is probably because bridge network does not exist, althouh I'm surprised that it did not fail in this case.

I have a working code that rectifies the above error but need to do a bit more cleaning prior to submitting PR.

ikolomiyets added a commit to ikolomiyets/testcontainers-go that referenced this issue Sep 17, 2020
… to run the tests in the Docker environment with disabled "bridge" network.

            The CreateContainer or CreateNetwork function will check if bridge network exists and if does not it creates a reaper_default
            network and ensure that container is attached to the reaper_default network.
@ikolomiyets ikolomiyets changed the title Unable execute tests when they are running in Jenkins with Kubernetes plugin when Pods are running in EKS Unable to execute tests when they are running in Jenkins with Kubernetes plugin when Pods are running in EKS Sep 17, 2020
gianarb added a commit that referenced this issue Sep 18, 2020
Issue #243: Introducing the default network in case "bridge" is disabled
@gianarb gianarb closed this as completed Sep 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants