Ability to specify container to host port mappings

[adbourne]

I notice the ExposedPorts functionality, that allows for a container port to be specified such that it is mapped to a random host port e.g 0.0.0.0:32911->9092/tcp. Is there a way to specify the port mappings such that the container port is mapped to a specific host port e.g 0.0.0.0:9092->9092/tcp?

I'm attempting to use dockertest to spin up an Apache Kafka container spotify/kafka, for use in an integration test. The app is making use of the Shopify/sarama Kafka client. As expected, dockertest spins up the Kafka container successfully. In fact, all works fine up until Sarama attempts to consume from the Kafka topic. The reason for this is that the Kafka broker advertises its port number to Sarama, which Sarama then uses, but it's the container port and not the host port. For example, dockertest spins up a Kafka container with port mappings 0.0.0.0:32911->9092/tcp, Sarama connects initially on port 32911 and queries for the broker information, "localhost:9092" is returned to Sarama, Sarama attempts to connect but fails as port 9092 is not available.

I'm sure there's probably a more elegant solution rather than ensuring that the container port is mapped to the exact same host port, but specific port mappings would certainly be a very useful feature in dockertest.

For reference, the code I'm working on can be found here

[aeneasr]

You need to set the port in Sarama to 9092, you can get the current port mapping by doing, in the case of mysql, resource.GetPort("3306/tcp"))):

db, err = sql.Open("mysql", fmt.Sprintf("root:secret@(localhost:%s)/mysql?parseTime=true", resource.GetPort("3306/tcp")))

For kafka this is probably something like resource.GetPort("32911/tcp"). You can dump the ports to stdout to see which values are available

[adbourne]

Thanks for the reply @arekkas.

In regards to mysql this would certainly be the solution, however things have gotten a bit trickier since trying dockertest with Kafka.

I use resource.GetPort("9092/tcp") initially to get the port number of Kafka that Docker has assigned (e.g 32911). This is used for the successful initial connection. The challenge is that this is only used to get metadata from Kafka, after that Sarama uses the port number Kafka tells it about, which is the container port number and not the host port number:

Sarama                           Kafka
  |                                |
  | get metadata (localhost:32911) |
  |------------------------------->| 
  |                                |
  | broker on localhost:9092       |
  |<-------------------------------| 
  |                                |
  | consume topic (localhost:9092) |
  |-------------------------------X|
  |                                | 

For the moment, I have "solved" the issue by using fsouza/go-dockerclient directly. This allows for specifying the host ports the container ports should be mapped to: https://github.com/adbourne/go-archetype-kafka-processor/blob/master/app_integration_test.go#L124

If I understand correctly, as dockertest uses fsouza/go-dockerclient, it should be possible to use a similar approach to allow for dockertest to offer this feature. Perhaps by allowing a ":" to be present in an ExposedPorts declaration such that a host port and container port can be specified e.g 9092:9092 . This would be a similar approach to the Docker cli. The logic could live here: https://github.com/ory/dockertest/blob/v3/dockertest.go#L119

[aeneasr]

If I understand correctly, as dockertest uses fsouza/go-dockerclient, it should be possible to use a similar approach to allow for dockertest to offer this feature. Perhaps by allowing a ":" to be present in an ExposedPorts declaration such that a host port and container port can be specified e.g 9092:9092

Yes, this was actually the case until the v3 release, but was replaced in favor of dynamic port allocation which avoids port conflicts, which happened a lot. I'm open to having an option to force specific ports though.

[ruz]

Same problem here. I understand issue with port conflicts during testing.

Is there way to know port beforehand so I can configure Kafka properly via environment variables?

Is there way to use some sort of substitutions in Env variables?

Is there way to run a command to form environment variable?

[ruz]

For people looking for answer.

It's possible to get fixed port mappings using PortBindings. See kafka example.

Also, got single container variant working:

	resource, err := pool.RunWithOptions(
		&dockertest.RunOptions{
			Repository: "docker.io/bitnami/kafka",
			Tag:        "3.2",
			Env: []string{
				"KAFKA_ENABLE_KRAFT=yes",
				"KAFKA_BROKER_ID=1",
				"KAFKA_CFG_PROCESS_ROLES=broker,controller",
				"KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER",
				"KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093",
				"KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT",
				"KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092",
				"KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@127.0.0.1:9093",
				"ALLOW_PLAINTEXT_LISTENER=yes",
			},
			ExposedPorts: []string{"9092/tcp"},
			PortBindings: map[docker.Port][]docker.PortBinding{
				"9092/tcp": {{HostIP: "localhost", HostPort: "9092/tcp"}},
			},
		})