RFC: Overhaul HTTPS support

[lydell]

Yesterday I got some great feedback from @dsimunic on using elm-watch behind a proxy and with HTTPS on Discord. That discussion ended up in the following plan:

1. Add a way to configure the WebSocket URL. This could be "webSocketUrl": "wss://example.com:12345/whatever" in elm-watch.json. It means that the elm-watch client will effectively do new WebSocket("wss://example.com:12345/whatever"). Fixes Ability to set domain for websocket server #60 and Support different client and server ports #46.

2. If "webSocketUrl" is unset:

   • On http:// pages, everything will work as before.
   • On https:// pages, elm-watch won’t even try to connect. Instead it will show a helpful message, with a link to docs about how to use elm-watch with HTTPS. Which means setting up an HTTPS server yourself, and proxying the WebSocket.

3. Remove HTTPS support in elm-watch. This makes Add ability to specify certificate #47 not needed any more – you instead control all things HTTPS yourself, at the expense of having to proxy the WebSocket for elm-watch.

@chazsconi and @bdukes would this work for you?

---

There is just one potential downside of this I can think of. By default, elm-watch sets the WebSocket URL using window.location.hostname. On a phone you might visit http://192.168.1.123 (given that you dev server is exposed on the network and your phone is on the same wifi) and then elm-watch tries to connect to ws://192.168.1.123:45678/elm-watch. So testing on a phone just works. Now, if you for some reason use HTTPS – like https://localhost – then you would have to set "webSocketUrl": "wss://localhost/elm-watch" (since elm-watch no longer does anything on https:// pages without config). But then you wouldn’t be able to test on your phone anymore, without temporarily editing elm-watch.json to say "webSocketUrl": "wss://192.168.1.123:45678/whatever". My plan is to use my classic way of documenting this and saying “if you need this, please open an issue”. An idea I have already could be to also have an environment variable ELM_WATCH_WEBSOCKET_URL that overrides elm-watch.json. Then you can easily set it temporarily, or even script it to ask the computer for the IP address and put that in.

[bdukes]

I'd be interested to see the documentation for setting up a WebSocket proxy, I'm not sure that I know of a quick or straightforward way to get that working in our dev environments. As I understand it, it's definitely more setup for us than configuring certificate files. However, if we can automate a simple proxy setup, I can make that work for our use case.

[lydell]

@bdukes Good point, I should make sure proxying WebSockets is easy enough before doing this. What dev server do you use? Or, what is the thing handling HTTPS in you dev setup? Trying to decide what proxy solutions I should look at first.

[bdukes]

We typically use IIS (the web server built into Windows) or Kestrel, the ASP.NET Core web server.

[dsimunic]

@bdukes You only need to set up normal http proxy for a single location /elm-watch: web socket is HTTP, so the first request will be an ordinary http request, and then it will switch. The proxy itself doesn't need to know much about it beyond allowing the upgrade header to go through.

For example, nginx is set up like this:

location /elm-watch {
  proxy_pass http://localhost:61879; #port as specified in elm-watch.json
  # This also works: proxy_pass ws://localhost:61879
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "Upgrade";
  proxy_set_header Host $host;
}

ChatGPT says this is the translation of that location definition to Kestrel (using ws:// upstream, you may ask it for http:// upstream it's almost the same). No idea if it's correct, but seems complex enough to be the real thing:

Setting up a reverse proxy location in a Kestrel web server for a .NET application to correspond to a specific NGINX location definition involves several steps. The NGINX configuration you provided is for proxying requests to a WebSocket server. To achieve similar functionality in Kestrel, you would typically use ASP.NET Core middleware to configure the reverse proxy.

Here is a general outline of the steps you need to follow:

Install the Required NuGet Package:
First, ensure that you have the Microsoft.AspNetCore.App package, which should include the necessary dependencies for reverse proxy setup.

Configure the Startup Class:
In your Startup.cs file, you need to configure the ConfigureServices and Configure methods.

ConfigureServices Method:
In the ConfigureServices method, you typically add services needed for the application. For reverse proxy, there might not be specific services to add unless you're using a specific middleware package.

Configure Method:
This is where you set up the request pipeline. You'll use middleware here to intercept and route requests.

Set Up the Reverse Proxy Middleware:
In the Configure method, you would set up the middleware to handle WebSocket connections and proxy them to the desired endpoint. You can use the UseWebSockets method for enabling WebSocket support and then write custom middleware or use existing libraries to proxy the requests.

Here is a basic example of what the code might look like:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    app.UseWebSockets();

    app.Use(async (context, next) =>
    {
        if (context.Request.Path.StartsWithSegments("/elm-watch"))
        {
            if (context.WebSockets.IsWebSocketRequest)
            {
                // Handle WebSocket request
                using var webSocket = await context.WebSockets.AcceptWebSocketAsync();
                await ProxyWebSocketRequestAsync(webSocket, "ws://localhost:61879");
            }
            else
            {
                // Handle non-WebSocket request
                context.Response.StatusCode = 400;
            }
        }
        else
        {
            await next();
        }
    });

    // Other middleware (e.g., static files, MVC)
}

private async Task ProxyWebSocketRequestAsync(WebSocket clientWebSocket, string wsDestinationUri)
{
    // Implement the logic to proxy WebSocket requests to wsDestinationUri
    // This involves receiving messages from the clientWebSocket, sending them to the server at wsDestinationUri,
    // and then sending the server's response back to the client.
}

[dsimunic]

Again ChatGPT:

Translating the NGINX location configuration to a configuration suitable for IIS (Internet Information Services) involves using URL Rewrite and Application Request Routing (ARR) modules in IIS. These modules allow you to configure reverse proxy behavior similar to what you've defined in NGINX.

To achieve a similar setup in IIS, follow these steps:

Install URL Rewrite and ARR:
Make sure you have the URL Rewrite module and Application Request Routing installed in your IIS. These can be installed via the Web Platform Installer.

Open IIS Manager:
Launch the IIS Manager and navigate to the site for which you want to set up the reverse proxy.

URL Rewrite Rule:

Go to the site in IIS Manager and open the "URL Rewrite" feature.
Click "Add Rule(s)..." in the right-hand pane.
Select "Reverse Proxy" from the list of rule templates.
If you haven't installed ARR, you will be prompted to do so. Follow the instructions to install it.
Configure the Reverse Proxy Rule:

In the "Add Reverse Proxy Rules" dialog, set the "Inbound Rules" section.
For the "From" field, enter the pattern that matches your NGINX location, like ^elm-watch(/.*)?.
In the "To" field, enter http://localhost:61879.
Make sure to enable "Rewrite the domain names of the links in HTTP responses".
Additional Settings:

To replicate the proxy_set_header directives, you may need to add server variables or custom rewrite rules.
For WebSocket support, ensure "Enable SSL offloading" is unchecked and "Reverse rewrite host in response headers" is checked.
Apply the Configuration:
After setting up the rule, click "Apply" in the actions pane to save your configuration.

Test the Configuration:
It's important to test your configuration to ensure that requests to /elm-watch are correctly proxied to http://localhost:61879.

Remember that IIS's approach to reverse proxying might not be a 1-to-1 match with NGINX's capabilities and syntax. Certain complex behaviors might require additional rules or custom modules. The steps above provide a general guide and might need adjustments based on your specific requirements and the IIS version.

[dsimunic]

Finally, YARP. Again ChatGPT.

Configuring YARP (Yet Another Reverse Proxy) in a .NET application to match the functionality of the given NGINX configuration involves creating and configuring a YARP proxy in your ASP.NET Core application. YARP allows you to set up sophisticated reverse proxy scenarios with .NET, and it's particularly useful for routing, load balancing, and other proxy-related tasks.

Install YARP:
Add the YARP NuGet package to your ASP.NET Core project. You can do this via the NuGet Package Manager or using the Package Manager Console:

Install-Package Microsoft.ReverseProxy -Version <specific_version>

Configure Services:
In your Startup.cs or Program.cs (for .NET 5 or later), configure the services for YARP in the ConfigureServices method:

public void ConfigureServices(IServiceCollection services)
{
    services.AddReverseProxy()
        .LoadFromConfig(Configuration.GetSection("ReverseProxy"));
}

Add YARP Configuration:
In your appsettings.json or a similar configuration file, define the YARP routes and clusters to match your NGINX setup:

{
  "ReverseProxy": {
    "Routes": [
      {
        "RouteId": "elm-watch-route",
        "ClusterId": "elm-watch-cluster",
        "Match": {
          "Path": "/elm-watch/{**catch-all}"
        }
      }
    ],
    "Clusters": {
      "elm-watch-cluster": {
        "Destinations": {
          "elm-watch-destination": {
            "Address": "http://localhost:61879"
          }
        }
      }
    }
  }
}

Configure the Proxy Middleware:
In the Configure method of your Startup.cs or Program.cs, add the YARP middleware:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    app.UseRouting();
    
    app.UseEndpoints(endpoints =>
    {
        endpoints.MapReverseProxy();
    });
}

Run Your Application:
When you run your application, YARP will use the configuration to route requests that match /elm-watch to http://localhost:61879.

[dsimunic]

To set up a Node.js-based web server to proxy WebSocket connections in a manner similar to the NGINX configuration you provided, you can use the http-proxy module, which is a popular choice for creating HTTP and WebSocket reverse proxies in Node.js.

const http = require('http');
const httpProxy = require('http-proxy');

// Create a proxy server with custom application logic
const proxy = httpProxy.createProxyServer({});

// Handle WebSocket requests
proxy.on('upgrade', function (req, socket, head) {
  proxy.ws(req, socket, head, {
    target: 'http://localhost:61879',
    ws: true
  });
});

// Create your server and then proxies the request
const server = http.createServer(function (req, res) {
  proxy.web(req, res, { target: 'http://localhost:61879' });
});

server.on('upgrade', function (req, socket, head) {
  proxy.ws(req, socket, head);
});

server.listen(3000, () => {
  console.log('Proxy server listening on port 3000');
});

[bdukes]

Setting up a simple node.js proxy server would probably be the approach we'd take. Thanks!

[chazsconi]

Yesterday I got some great feedback from @dsimunic on using elm-watch behind a proxy and with HTTPS on Discord. That discussion ended up in the following plan:

1. Add a way to configure the WebSocket URL. This could be `"webSocketUrl": "wss://example.com/whatever"` in elm-watch.json. It means that the elm-watch client will effectively do `new WebSocket("wss://example.com:12345/whatever")`. Fixes [Ability to set domain for websocket server #60](https://github.com/lydell/elm-watch/issues/60) and [Support different client and server ports #46](https://github.com/lydell/elm-watch/issues/46).

2. If `"webSocketUrl"` is unset:
   
   * On `http://` pages, everything will work as before.
   * On `https://` pages, elm-watch won’t even try to connect. Instead it will show a helpful message, with a link to docs about how to use elm-watch with HTTPS. Which means setting up an HTTPS server yourself, and proxying the WebSocket.

3. Remove HTTPS support in elm-watch. This makes [Add ability to specify certificate #47](https://github.com/lydell/elm-watch/pull/47) not needed any more – you instead control all things HTTPS yourself, at the expense of having to proxy the WebSocket for elm-watch.

@chazsconi and @bdukes would this work for you?

There is just one potential downside of this I can think of. By default, elm-watch sets the WebSocket URL using window.location.hostname. On a phone you might visit http://192.168.1.123 (given that you dev server is exposed on the network and your phone is on the same wifi) and then elm-watch tries to connect to ws://192.168.1.123:45678/elm-watch. So testing on a phone just works. Now, if you for some reason use HTTPS – like https://localhost – then you would have to set "webSocketUrl": "wss://localhost/elm-watch" (since elm-watch no longer does anything on https:// pages without config). But then you wouldn’t be able to test on your phone anymore, without temporarily editing elm-watch.json to say "webSocketUrl": "wss://192.168.1.123:45678/whatever". My plan is to use my classic way of documenting this and saying “if you need this, please open an issue”. An idea I have already could be to also have an environment variable ELM_WATCH_WEBSOCKET_URL that overrides elm-watch.json. Then you can easily set it temporarily, or even script it to ask the computer for the IP address and put that in.

This approach sounds good and will work for me in a use case I have when I run the development environment on my laptop, and here I can remove the code I have to monkey patch the WebSocket class (this overrides the port to 443 if the pathname is /elm-watch.)

However, I have another use case which involves running the development environment remotely (like Github Codespaces, Gitpod etc). Here hostname on which the site runs changes dynamically - a developer creates a new environment and gets a new hostname with a functioning SSL cert e.g. https://env1234.foo.com or https://env5678.foo.com etc.

So it would be nice to be able to override the port,path and scheme of the webSocketUrl, but retain the hostname as this changes dynamically. I realise this is an edge case, and I can continue to use the WebSocket monkey patch to handle this if you think this is too complex.

[lydell]

That’s a valid use case. From what I heard on podcasts, remote code spaces are supposed to take over the dev scene :)

Here are some ways I can think of solving it:

• Support "webSocketUrl": "wss://$current_hostname/whatever" where $current_hostname is a special cased string that is replaced with the current host name. Downsides: A bit weird, might result in an invalid URL, opens up for eventually having a full little templating language in there.
• Support the ELM_WATCH_WEBSOCKET_URL env var like I suggested. Then you could start elm-watch like this: ELM_WATCH_WEBSOCKET_URL="wss://$(somehow-get-gitpod-hostname)/whatever" elm-watch hot. Downside: How to implement somehow-get-gitpod-hostname?
• Support something like window.__elmWatchWebSocketUrl = `wss://${window.location.hostname}/whatever`;. Downside: The config becomes more spread out, and you might want to only include that line in dev builds (not production builds) so then you need to set that up as well. (If you use the postprocess feature anyway, it might be possible to inject it there (except in optimize mode).)
• Don’t support it and require people to use the WebSocket monkey patch. But since your reason is “because of remote dev containers” and not “because of the super specific ancient tech debt setup at work” it feels like it should be supported.

[chazsconi]

I think the ELM_WATCH_WEBSOCKET_URL option is the simplest.

In my experience these remote development environments always provide the hostname in an environment variable in the container or virtual machine in which you run so getting somehow-get-gitpod-hostname shouldn't be an issue.

[lydell]

The changes described in this issue are now released as 2.0.0-beta.1. With one difference: elm-watch still automatically uses wss:// on HTTPS pages – the only difference now is that I show a link to the HTTPS docs instead of to the page where you could accept the self-signed certificate (which is no longer a thing).

On top of that, if you don’t want to create a proxy you can also achieve HTTPS like so:

import * as fs from "node:fs";
import * as https from "node:https";
import * as path from "node:path";
import * as url from "node:url";
import elmWatch from "elm-watch";

const DIRNAME = path.dirname(url.fileURLToPath(import.meta.url));

// Deal with certificates and HTTPS options in whatever way you’d like:
const CERTIFICATE = {
  key: fs.readFileSync(path.join(DIRNAME, "certificate", "dev.key")),
  cert: fs.readFileSync(path.join(DIRNAME, "certificate", "dev.crt")),
};

elmWatch(process.argv.slice(2), {
  createServer: ({ onRequest, onUpgrade }) =>
    https.createServer(CERTIFICATE, onRequest).on("upgrade", onUpgrade),
})
  .then((exitCode) => {
    process.exit(exitCode);
  })
  .catch((error) => {
    console.error("Unexpected elm-watch error:", error);
  });

See the updated HTTPS and Server docs for more info.

It’ll take a while before 2.0.0 will leave beta (because there are some other minor, potentially-breaking changes I’d like to make and the Node.js world has complicated things for me), but if some of you would like to start trying this out already, you now can.

[bdukes]

We don't have any active projects using elm-watch at this moment, so I don't have an easy opportunity to test, but this looks like it would handle all of our issues. Thanks!