Habitat should hunt for an available port if 9631 and 9638 are unavailable

[irvingpop]

As an operator of Habitized applications, I may want to use a networking mode like ECS Host-mode networking.

Since all ports are mapped to the host port, it means that my 2nd supervisor won't be able to successfully start - because ports 9631 and 9638 are already taken.

In these cases, it would be idea for the supervisor to automatically attempt to find an available port - hunting through an available range of ports. Chef-Zero used to have this same problem and was solved by chef/chef-zero#67

[reset]

This may be a bit difficult to get done seeing we have an "initial peer" problem already, but it's something we should keep in mind as we attempt to solve for initial peer

[smith]

The default behavior on Linux is to pick a random port if 0 is given to a program as a port number, so right now you can do hab sup run --listen-http 0.0.0.0:0 and it will run on a random port, but it won't tell you which port it's running on, you'll have to get that from lsof or netstat or something.

So, it's possible now but very user unfriendly; would be awesome if it was nice instead.

[irvingpop]

that's a neat idea @smith ! unfortunately not usable for the gossip port, because it advertises itself to the census as having a gossip port and http port of 0:

          "sys": {
            "ip": "172.31.31.122",
            "hostname": "ip-172-31-31-122",
            "gossip_ip": "172.31.31.122",
            "gossip_port": 0,
            "http_gateway_ip": "0.0.0.0",
            "http_gateway_port": 0
          },

[irvingpop]

agreed @reset - I've solved for the initial peer problem in my configuration, but you gotta gotta know which one is Habitat Prime and that one must be on the default port.

[irvingpop]

Interesting, I may have actually pissed off all the supervisors in my ring by introducing that node listening on port 0:

ERROR:habitat_butterfly::server::outbound: Failed Ping to 172.31.31.122:0: Invalid argument (os error 22)
ERROR:habitat_butterfly::server::outbound: Failed Ping to 172.31.31.122:0: Invalid argument (os error 22)
Could not send rumor to "9fc4302dc0b14ea09dc1e4ba55e28d24" @ "172.31.31.122:0"; ZMQ said: Resource temporarily unavailable
Could not send rumor to "9fc4302dc0b14ea09dc1e4ba55e28d24" @ "172.31.31.122:0"; ZMQ said: Resource temporarily unavailable

[christophermaier]

@irvingpop Makes sense... Rust's network code will pick a port for you if you specify 0, but Habitat's code isn't expecting you to do that, so it doesn't subsequently check to see what port was ultimately picked.

[rsertelon]

I don't know if it should really be habitat's job to figure out which port to use. It seems more like an infrastructure constraint, and thus should be configured from outside habitat explicitly I think.

I figure that if you'd deploy two different containers with webservers in them, they might both use port 80, and it would be on the ops part to map/use something to configure another port for example, shouldn't it?

[bixu]

I feel a little bit uncomfortable about automating port selection by default. I would be much more comfortable having this turned on with a flag.

I'm reminded of this RFC I opened: #1939 - is the core problem we are trying to solve automatic bootstrapping of the initial ring?

[adamhjk]

I would be okay with it being a flag, but I'm not okay with it being on by default.

[eeyun]

I think we're all in agreement that it would be a totally reasonable feature to have so long as it's implemented appropriately (e.g. behind some kind of toggle or flag). For now we're going to sort this into help wanted. If anyone with an interest in implementing this is ready to get started hacking feel free to reach out for some implementation guidance from the core team!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[stale]

This issue has been automatically closed after being stale for 400 days. We still value your input and contribution. Please re-open the issue if desired and leave a comment with details.