Command line option for setting HTTP port

[jtobin]

Both #1264 and #1266 have been looking for a way to set the HTTP port from the command line -- I've just extracted this request more directly and stuffed it here.

From the latter:

My urbit ships come up listening on port 8080 if it is available, and if not, it increments to 8081, 8082, etc, until it finds an open port to use. This is not ideal, because i need to start my ships in the correct ordering so my personal proxies are configured correctly. If i could specify a port when booting a ship, this ordering wouldn't matter.

and the former:

[..] it seems like it'd be good to have the ability to specify the http(s) ports, and have that show up on the command line help.

[joemfb]

I think I have written this up a few times before, but a brief search hasn't turned up this information in any public facing documents. Here are the details and some rational for Urbit's port-binding behavior.

%ames uses a UDP socket. The port is a fixed offset of the ship-number for galaxies, but a random available UDP port (or the value of the -p argument) for everyone else. Failure to open the port terminates the process.

Next, we have TCP ports for the HTTP server(s). These can vary dramatically based on the ship state and host environment.

The first of these is the "lens" server, the "control plane" server bound to the local loopback. It attempts to bind port 12321, and increments until a port is successfully bound.

Next we have the plain HTTP server. It attempts to bind port 80, falling back to 8080 and then incrementing on failure until a port is successfully bound. Unless the ship is configured to proxy for it's sponcees (this currently applies only to stars), in which case it's the proxy server that tries to bind port 80, and it falls back to 9080 on failure. When proxied, the http server starts at 8080. Both ports individually incremented until successfully bound.

Finally, we have the HTTPS server. This is activated if %eyre has been configured with a TLS certificate. It starts at port 443, falling back to 8443 and then incrementing until bound. Unless proxied (securely, via the Server Name Indicator of the TLS ClientHello), in which case it's the secure proxy server that starts at 443, and it falls back to 9443. When proxied, the HTTPS server starts at 8443. Both ports are individually incremented until successfully bound.

There are two common situations in which it is important for a ship to be listening on or able to bind the standard 80/443 ports.

The first is in retrieving a LetsEncrypt certificate via the :acme app. The LetsEncrypt service performs domain validation by making HTTP requests on port 80 to the domain in question, so no ship will be able to get a certificate on less it can service requests to ship.arvo.network:80. For galaxies and stars, this means they must be able bind port 80 themselves, or be run in an environment where TCP traffic over port 80 is forwarded to them. For planets, this means their star must have its proxy server listening on port 80.

The second situation is when acting as a proxy server (again, currently only stars, although this could be easily configured for other ships). This is necessary for the star's proxied, sponcee ships to able use ship.arvo.network without needing to learn of which non-standard port their sponsor is using (and, again, for them to be able to complete domain validation).

The question of whether a process can bind ports 80 and 443 depends on the platform, host environment, permissions of the user account running the process, the capabilities of the binary itself, and the availability of the port in question. Running the process with sudo, setting CAP_NET_BIND on the binary, or simply using a recent version of OS X are the obvious ways to accomplish this. Alternately, urbit can be left to bind whatever ports it can get, and traffic can be routed by configuring some type of port-forwarding / TCP proxy in the host environment.

Back to the issue at hand, it's not clear to me how this behavior (and the underlying requirements) should be integrated with command-line option(s). The naive approach would require quite a few new command-line options, and we are already drowning in them. I'm opposed to terminating the process if we're unable to bind a default port (in other words, I'm attaching to the port-incrementing behavior), as that would make it inconvenient to run multiple ships in development. But I agree that this socket initialization behavior should be controllable from outside urbit.

[mrdomino]

Great write-up, that actually demystifies what's going on in my case (planet on OSX, galaxy on OpenBSD).

[dpc]

As for command-line. There should an option to force a given port (so it can reliably be used in scripts and such). Failure to bind to a request port should result in termination.

Setting up port forwarding is a massive PITA, and starting as root would be a great simplification, but only if urbit binary dropped the privileges right after obtaining the necessary resources.

[wbobeirne]

I'd really like to see a way to control this as well, it's very standard for most web server applications. I do a lot of development on my machine, and a lot of my projects also default to 8080, which means I don't really have a deterministic expectations for which port urbit or my projects will be running on. The only thing I've been able to think of to work around this is to run urbit in a docker container, and then expose the container's port 8080 over my desired port.

If a command line option is undesirable, maybe it's time to have an .urbitrc or a $HOME/.config/urbit file where you can put these configurations?

[philipcmonk]

The upcoming "King Haskell" release has this feature. Some people use it already, but its official release has slipped several times, unfortunately.

[belisarius222]

It would be great to remove all unnecessary privileges from the Urbit daemon and worker processes. Pull requests welcome :)

[mikegogulski]

+1 to this. I run multiple ships on a single machine, the startup order may not be deterministic, so I'd like to explicitly set the HTTP port for each of them. This will let me do things like bind a DNS name that points to the machine to an Nginx reverse-proxy that puts the HTTP port behind SSL and HTTP basic authentication.

FWIW, I agree that adding 4, 6, or more commandline options would be a bit of a mess. Instead, a single commandline option that points to a config file (differing from @wbobeirne 's suggestion of a single config file per *nix user) sounds good.

[oneingan]

The upcoming "King Haskell" release has this feature. Some people use it already, but its official release has slipped several times, unfortunately.

We can not use it after network breach #4113

[odyslam]

@mikegogulski, coming back to this. I am trying myself to do what you describe, but I think there is an issue with how Landscape requests the source files.

Thus far, I haven't managed to make reverse-proxy work.