HTTP status page for Polykey Agent

[CMCDragonkai]

Specification

It would be nice to have an HTTP/s status page for Polykey Agent. The basic idea that it would be possible to use curl, http or a browser to hit the PK agent, and acquire basic status information.

For this status page, it is a public page, so no authentication is necessary to access this status page. It would be similar to calling polykey agent status. But it will only show "public" information, and nothing private.

This status page can be designed similar to other public status pages like for example https://www.githubstatus.com/ or https://status.gitlab.com/

As for the port it uses, we may want to reuse the PK_CLIENT_PORT. But by default HTTP uses 80 and HTTPS uses 443. Therefore PK_CLIENT_PORT and --client-port must be capable of taking multiple ports. Like PK_CLIENT_PORT=80,443,1315, and then PK will bind to all of those ports.

It will be important that the system can differentiate HTTP from HTTPs traffic however and server accordingly. If the client initiates as HTTP, we respond with HTTP, if the client initiates with HTTPs we respond with HTTPs.

If we move to graphql or jsonrpc, we also need to differentiate such an HTTP(s) protocol to regular API requests.

So a protocol demuxer will need to be put in place.

Alternatively the status page is just on an entirely different port which makes it simpler, but increases configuration overhead. That means an additional configuration for status page for both HTTP and HTTPs.

With HTTPs, it will offer its own root cert chain as the certificate data. The cert doesn't need to be signed by a third party certificate authority #154.

As for https://testnet.polykey.io, we would want to show the status of all the nodes in the testnet. One way to do this is to add server somewhere that answers the request but gathers information about the entire cluster #403. Alternatively each node in the testnet cluster can also show status information about the other nodes in the cluster, DHT style. Which would mean that we could route this to the NLBs and avoid having to run another piece of infrastructure.

Additional context

• Third Party Integration - HTTP API, OAuth2 Provider API, Plugin API, Library vs Framework, Smart Tokens #166
• CSR feature for Keys domain to allow KN root certificates to be signed by an external CA #154

Tasks

1. ...
2. ...
3. ...

[CMCDragonkai]

With the change to JSON RPC, this should break our dependency on the underlying transport protocol. So it would be possible to support an HTTP router for specific RPC handlers. However because it would be expected be HTML, we would also need to be capable of using an alternative format than JSON. But this should be similar to having to deal with protobuf... etc. A single blessed endpoint can be used to provide an HTTP status page.

[tegefaulkes]

Serving the content should be pretty simple. We can generate the HTML using template strings and a template. I'm not sure it's good to serve it over the same socket as the RPC system. Ideally we'd just have a separate TCP socket serving the HTTP content.

[CMCDragonkai]

The RPC system doesn't have knowledge about sockets at all. Starting an RPC server would depend on some event handler (external to the RPC system) to provide ReadableWritablePair objects. So the RPC system wouldn't care where these objects come from.

TCP sockets/websockets/QUIC sockets are all things that could be handing over a ReadableWritablePair to the RPC system.

[CMCDragonkai]

It would nice to show the status of the testnet and mainnet, specifically if they are online.

In the future, we could actually have something that gathers status from all the nodes in testnet/mainnet and show that aswell. I'm not sure if this will be shown on the status page of a single agent, or if it's custom service that we deploy to cloudflare.

Like a testnet/mainnet status.

As for the agent's own status page, that's something that be done with the uWebsocket library because it serves HTTP as well. This appears to be on the same port that uws is binding to. This makes it quite easy, since we reuse the same port.

[tegefaulkes]

As part of #540 , the WebSocketServer is running a node https server. We should use that the serve an HTTP page.

[CMCDragonkai]

During some development on js-quic, we discovered as of now the quiche system which uses boring ssl doesn't yet implement support for multiple X509 certificates. Usually one can present multiple X509 certificates signed by different key algorithms and the server and client can present one or the other depending on negotiation. The work was done here: MatrixAI/js-quic#17.

The main usage for this is to preserve the root key as ed25519, and then deterministically generate other subkeys from it. That way, then we could generate another X.509 cert with a different key algorithm such as one supported by browsers. https://security.stackexchange.com/questions/269725/what-is-the-current-april-2023-browser-support-for-ed25519-certificate-signatu

Node's own TLS does in fact support this.

And now that our client service websocket system js-ws actually uses a regular HTTPS server (not http2 because no ws server implementation is available on http2), then this is possible now. We could serve up an HTTP status page for each agent too!

And this enables #166 to have an HTTP-based RESTful API as an alternative to the WS system if necessary for clients that don't understand or require a simpler integration (possibly allowing us to create vault-compatibility mode or other API compatibility modes as suggested by some users).

[CMCDragonkai]

The https://testnet.polykey.com and https://mainnet.polykey.com provider global status page for the entire network. While a status page here provides individual status over a single Polykey node.

I was also debating whether adding a TUI would be cool to just give a quick terminal "dashboard-like" overview of the current node - but an HTTP status page is likely to be more useful anyway. Limited development resources of course.