Use regexp for harbor endpoints registration

[toots]

This PR completely revamps the harbor HTTP endpoint implementation:

Here's the new doc:

---

Simple API

The harbor.http.register.simple function provides a simple, easy to use registration API for quick
HTTP response implementation. This function receives a record describing the request and returns
the HTTP response.

For convenience, a HTTP response builder is provided via harbor.http.response. Here's an example:

def handler(request) =
  log("Got a request on path #{request.path}, protocol version: #{request.http_version}, \
       method: #{request.method}, headers: #{request.headers}, query: #{request.query}, \
       data: #{request.data}")

  harbor.http.response(
    content_type="text/html",
    data="<p>ok, this works!</p>"
  )
end

harbor.http.register.simple(port=8080, method="GET", path, handler)

where:

• port is the port where to receive incoming connections
• method is for the http method (or verb), one of: "GET", "PUT", "POST", "DELETE", "OPTIONS" and "HEAD"
• path is the matched path. It can include named fragments, e.g. "/users/:id/collabs/:cid". Named named framents are passed via request.query, for instance: req.query["cid"].

Node/express API

The harbor.http.register function offers a higher-level API for advanced HTTP response implementation.
Its API is very similar to the node/express API. Here's an example:

def handler(request, response) =
  log("Got a request on path #{request.path}, protocol version: #{request.http_version}, \
       method: #{request.method}, headers: #{request.headers}, query: #{request.query}, \
       data: #{request.data}")

  # Set response code. Defaults to 200
  response.status_code(201)
  
  # Set response status message. Uses `status_code` if not specified
  response.status_message("Created")
  
  # Replaces response headers
  response.headers(["X-Foo", "bar"])
  
  # Set a single header
  response.header("X-Foo", "bar")
 
  # Set http protocol version
  response.http_version("1.1")
  
  # Same as setting the "Content-Type" header
  response.content_type("application/liquidsoap")

  # Set response data. Can be a `string` or a function of type `()->string` returning an empty string
  # when done such as `file.read`
  response.data("foo")
  
  # Advanced wrappers:

  # Sets content-type to json and data to `json.stringify({foo = "bla"})`
  response.json({foo = "bla"})
  
  # Sets `status_code` and `Location:` header for a HTTP redirect response. Takes an optional `status_code` argument.
  response.redirect("http://...")
  
  # Sets content-type to html and data to `"<p>It works!</p>"`
  response.html("<p>It works!</p>")
end

harbor.http.register(port=8080, method="GET", path, handler)

where:

• port is the port where to receive incoming connections
• method is for the http method (or verb), one of: "GET", "PUT", "POST", "DELETE", "OPTIONS" and "HEAD"
• path is the matched path. It can include named fragments, e.g. "/users/:id/collabs/:cid". Matched named framents are passed via request.query, for instance: req.query["cid"].

The handler function receives a record containing all the information about the request and fills
up the details about the response, which is then used to write a proper HTTP response to the client.

Named fragments from the request path are passed to the response query list.

Advanced usage

All registration functions have a .regexp counter part, e.g. harbor.http.register.simple.regexp. These function accept
a full regular expression for their path argument. Named matches on the regular expression are also passed via the request's query parameter.

It is also possible to directly interact with the underlying socket using the simple API:

  # Custom response
  def handler(req) =
    req.socket.write("HTTP/1.0 201 YYR\r\nFoo: bar\r\n\r\n")
    req.socket.close()

    # Null indicates that we're using the socket directly.
    null()
  end

  harbor.http.register.simple("/custom", port=3456, handler)

Examples

These functions can be used to create your own HTTP interface. Some examples
are:

Redirect Icecast's pages

Some source clients using the harbor may also request pages that
are served by an icecast server, for instance listeners statistics.
In this case, you can register the following handler:

# Redirect all files other
# than /admin.* to icecast,
# located at localhost:8000
def redirect_icecast(request, response) =
  response.redirect("http://localhost:8000#{request.path}")
end

# Register this handler at port 8005
# (provided harbor sources are also served
#  from this port).
harbor.http.register.regexp(
  port=8005,
  method="GET",
  r/^\/(?!admin)/,
  redirect_icecast
)

Get metadata

You can use harbor to register HTTP services to
fecth/set the metadata of a source. For instance,
using the JSON export function json.stringify:

meta = ref([])

# s = some source
s.on_metadata(fun (m) -> meta := m)

# Return the json content of meta
def get_meta(_, response) =
  response.json(!meta)
end

# Register get_meta at port 700
harbor.http.register(port=7000,method="GET","/getmeta",get_meta)

Once the script is running,
a GET request for /getmeta at port 7000
returns the following:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "genre": "Soul",
  "album": "The Complete Stax-Volt Singles: 1959-1968 (Disc 8)",
  "artist": "Astors",
  "title": "Daddy Didn't Tell Me"
}

Set metadata

Using insert_metadata, you can register a GET handler that
updates the metadata of a given source. For instance:

# s = some source

# Create a source equiped with a `insert_metadata` method:
s = insert_metadata(s)

# The handler
def set_meta(request, response) =
  # Filter out unusual metadata
  meta = metadata.export(request.query)

  # Grab the returned message
  ret =
    if meta != [] then
      s.insert_metadata(meta)
      "OK!"
    else
      "No metadata to add!"
  end

  response.html("<html><body><b>#{ret}</b></body></html>")
end

# Register handler on port 700
harbor.http.register(port=7000,method="GET","/setmeta",set_meta)

Now, a request of the form http://server:7000/setmeta?title=foo
will update the metadata of source s with [("title","foo")]. You
can use this handler, for instance, in a custom HTML form.

---

Also worth noting, I don't think that you can provide backward compatible APIs here or, perhaps more accurately, I think we should avoid it. Adapting existing use of the harbor HTTP endpoints shouldn't be too difficult and I'd rather encourage that.

This PR also unifies the terminology used for HTTP queries between harbor and libcurl (http.{get,post,put,...}), in particular it switches to:

• status_code/status_message
• http_version

Lastly, the PR introduces support for named match for regexp in the language by adding a groups method to the list of matches. This is pretty much on-par with the javascript implementation.

TODO

• Test endpoint resolution order
• Test manually
• Write tests
• Custom response
• Update doc

[smimram]

This is very nice, but I am not sur that we want to use regexps by default. The thing is that a mountpoint of stream.mp3 would match xxxstreamXmp3yyy which is not what most users expect I guess. Maybe can we have an explicit option regexp=true if we want the string to be handled as a regexp?

[toots]

I have a bigger change pending that will make the whole thing much closer to how express works in node. Default operator will use absolute strings e.g. "/users". There will be an operator that takes regexp for advanced usage. Also the handler will be passed request and response records and won't have to return a low level html response string. The socket will be available as an escape hatch to do low level stuff if needed.

[vitoyucepi]

I'd like to see

1. literal path matching for routes;
2. capture groups;
3. maybe function parameter mapping, but req.query.foo and req.body.bar will be fine to.

[toots]

@smimram @vitoyucepi I believe that this one is ready. Description and documentation are fully updated.

[smimram]

I have not read the PR in details (please leave me some time to do so) but I have two comments from the description.

Taking one record parameter (request) instead of many ones is nice. It's much simpler to write handlers and it is future-proof (we should be able add more parameters later on without changing the functions). Nice!

However, I am not fan of taking the pre-filled response as an answer. What was wrong with the previous http.response (which had reasonable default values)?

[toots]

I have not read the PR in details (please leave me some time to do so) but I have two comments from the description.

Taking one record parameter (request) instead of many ones is nice. It's much simpler to write handlers and it is future-proof (we should be able add more parameters later on without changing the functions). Nice!

However, I am not fan of taking the pre-filled response as an answer. What was wrong with the previous http.response (which had reasonable default values)?

The main reasons for me are:

• It's closer to the node/express API, which will make adoption easier
• It is also future proof!
• It allows for higher level processing such as:
  • Responding directly using the socket
  • More importantly: wrapping the data parameter into chunked output while sending initial headers as single string.

I think that, from a user perspective, this is much more user-friendly than using a one-size-fits-all wrapper over the low-level HTTP response string..

[toots]

Code for the http response wrapper is here: https://github.com/savonet/liquidsoap/pull/2599/files#diff-4106ab6d9bc3a4ec0b7b38bb791ee42d1f950031f2e208c0aae454484da075fdR272-R322

[toots]

Ok, I've reviewed everything and here's what I'm thinking here:

• We need to make the end-user life's easier
• We need to pre-fill the response, otherwise, each call needs to do annoying things like passing the http_version from request to response.
• We need to wrap some low-level HTTP response logic so that each harbor user does not have to re-invent the wheel and/or know about HTTP response internals
• We can add the pre-filled response to the request but this'll end up bloating the type, making it barely readable.
• Once we have a pre-filled response, there isn't much value in expecting the user to return it. We are already wrapping up the handler call with it so we might as well return it without having to expect the user to do it.
• We need low-level escape hatch
• We want to do as much as possible in the standard scripting library

Thus, I have:

• Added harbor.{http,https}.register.core which takes only one request and can return any {string}?, null being the custom response.
• Moved the http response logic to the scripting library.

I think that this is a good compromise and that we should merge this now!