[supervisor] Add port management

[csweichel]

• /werft ws-feature-flags=registry_facade
• /werft https

This PR adds holistic port management to supervisor.

Until now Theia had multiple tasks when it comes to ports. For one, it was listening to ports being served in the workspace (moved to supervisor in #1827), it's listening for workspace instance status updates to determine which ports are exposed, and it combines those two sources to auto-expose ports when neccesary. It also offers a UI where users can expose ports publicly.

We're looking to move this functionality out of Theia and into supervisor. To this end, this PR

• extends the supervisor port status API to support port exposure and served status
• implements auto-exposure of served ports
• starts the infrastructure in supervisor to combine the served/exposed ports state
• implements a first version of an ExposedPorts source (by polling openPorts on the server)
• introduces a comprehensive set of unit tests for that state merge.

This PR also changes the termination log behaviour of workspaces to include the last few lines of log output. This makes debugging supervisor crashes tenable.

Open Points

• implement a ExposedPortsInterface that listens to workspace instance updates rather than polling. To this end one would first need to extend the Go Gitpod API support to provide access to incoming notifications.
• make the Gitpod API websocket connection automatically reconnect if it looses connection (kind of like what gRPC connections do)
• remove the old port handling code from Theia and consume the supervisor API instead
• hook up into Code

How to test

• start some workspace
• create server.js file with the following content:

const http = require('http');

const requestListener = function (req, res) {
  res.writeHead(200);
  res.end('Hello, World!');
}

const server = http.createServer(requestListener);
server.listen(4040, 'localhost');

• run node server.js to test that the localhost service gets served and exposed via the proxy as private
• there should be a notification offering to open it in browser/preview or make public
• try to toggle visibility and check that changes are reflected in the ports view and status bar
• stop the script, check that view and status bar updated
• now change the script to serve on '0.0.0.0', start the server and check everything again
• update .gitpod.yml with a new record for the unexposed port yet, try different action and visibilities, each time change server.js to use such port and start the server
• don't forget to try port ranges similarly to the previous point
• check that auto resolving localhost URIs work:
  • in terminal echo localhost:5000 and then click on a link, given that 5000 is not exposed yet
• additional points:
  • grant yourself devops role, switch to Code as the default IDE and do testing again: update d_b_user set rolesOrPermissions = '["admin","devops"]' where name = 'akosyakov';
  • try different example workspaces and compare to production
  • check that webpack HMR working when deployed on localhost: http://cw-supervisor-ports.staging.gitpod-dev.com/#https://github.com/gitpod-io/contribute.dev

[csweichel]

It took me a minute to understand this function. If I understand correctly, when there's no config, service.parser.parseInstanceConfigs(ports) will always return empty lists. In that case we could exit early returning false.

[akosyakov]

i don't think it would be correct, if the file was removed we should remove the previous state as well

[csweichel]

I've tested the PR with

# bind globally
curl lama.sh | sh
curl lama.sh | LAMA_PORT=9264 sh

# bind to localhost
curl lama.sh | sh -s -- -l
curl lama.sh | LAMA_PORT=9264 sh -s -- -l

all worked as intended.

[akosyakov]

@csweichel I've addressed all your comments and retested everything. Could you please have a look again?

[csweichel]

Thank you for the changes.
I've re-tested the PR and everything works well.

As I'm the original author of the PR, I cannot approve it.
But from my end, this is ✅

[akosyakov]

Reviewed and tested by me and Chris.