Move interface from HTTP to JavaScript

[EvanHahn]

In short: I propose we move much of mapeo-map-server’s API from HTTP to JS. I made a draft PR to move one of the routes. It might be a bad idea!

Currently, mapeo-map-server exposes all its functionality over HTTP, except for starting the server: that's a regular JavaScript function call to createMapServer().

In other words, there are two interfaces: an HTTP API (where most stuff happens) and a JavaScript API (where you start the server).

This document proposes moving most of the HTTP API to the JavaScript API.

Instead of making an HTTP POST to /tilesets/import, you'd call something like importTileset(). Some routes, such as GET /tilesets/:tilesetId/:zoom/:x/:y, would be unaffected, as they are needed for the map renderer.

Here’s a draft PR that moves one of the routes: #110

This might be a bad idea!

What?

Conceptually, I see the current system like this:

CoMapeo function call → HTTP request → Route handler → Internal API

I see the new system like this:

CoMapeo function call → External API

Some routes, such as GET /tilesets/:tilesetId/:zoom/:x/:y, would be unaffected, as they are needed for the map renderer. In other words, there would still be an HTTP server.

Roughly, I imagine an interface like this:

import MapServer from '@mapeo/map-server'

// Create the server instance. Doesn't do much.
const server = new MapServer({
  database: new Database('./example.db'),
})

// Start the server.
await server.listen(3000)

// Start an import.
// You could fetch this import later with `server.getImport(importId)`.
const mbtileImport = server.importMbtiles('/path/to/mbtiles_file')
mbtileImport.on('progress', ({ soFar, total }) => {
  const ratio = soFar / total
  const percent = Math.round(ratio * 100)
  console.log(`Imported ${percent}%`)
})
await mbtileImport.completion()

// Get a list of all the styles.
const styles = server.getStyles()

// Stop the server.
await server.close()

Again, this is a rough sketch; please ignore the specifics.

Why?

I think this idea provides several advantages:

• Less code. We could remove most of our route handlers. CoMapeo wouldn't need HTTP wrapper functions. Our server-side events code could be removed, as could many of our parameter validations.

• Better performance. Though HTTP requests incur minimal overhead, in-process function calls should incur even less.

• Easier type checking. Adding an HTTP boundary means that we either need to validate arguments and return values at runtime; not so with regular function calls.

• Improved security. You could imagine a malicious web app making HTTP requests to the server and performing various undesirable operations. If the server were read-only, that would be less of a threat.

• Simpler ergonomics for the consumer (and our tests). It's usually easier to call a function than to make an HTTP request. For example, you currently create an import and then request its status. What if this was a single function call?

• Easier documentation. Many editors let you hover over a function and see its JSDoc information, but few do the same for HTTP endpoints.

I'm not aware of any significant downsides to this, but (1) I may be overconfident in my idea (2) I don't know the map server very well!

How?

Currently, we have a component called ApiPlugin, which handles most of the business logic. I suspect we could remove most routes and expose this plugin's functionality.

I think this can be done piecemeal. For example, we could migrate a single route in the first PR, like #110.

[achou11]

Heads up for @rudokemper and @luandro in case you have any thoughts on what's being proposed here. Gregor and I had conversations related to doing this prior to Evan making a more structured proposal for it, as it would make the maintenance more straightforward and the usage less opinionated.

[achou11]

Some routes, such as GET /tilesets/:tilesetId/:zoom/:x/:y, would be unaffected, as they are needed for the map renderer. In other words, there would still be an HTTP server.

Maybe I'm forgetting something, I don't think it would be necessary to keep this. I would lean towards exposing a method like getTile(...) that takes the relevant params as an argument, and then this can be used by an end-user that decides to set up that route.

Basically, I don't really see any reason for the library to include it's own server instance of any kind as part of the main implementation. This also means no inclusion of a listen() method or anything similar as a result. Thoughts?

[luandro]

I think this makes a lot of sense. If there's not much overhead added, I like the idea of exposing at least the GET for tilesets, which could be consumed by other applications in the future.

[achou11]

If there's not much overhead added, I like the idea of exposing at least the GET for tilesets, which could be consumed by other applications in the future.

The way I'm imagining this is that there would be a method that we expose e.g. getTilesets() and we'd leave it up to an application developer to wrap that with their own HTTP server implementation (whether it's Express, Fastify, raw Node HTTP, etc). think we'd have a Fastify plugin for sure out of those options, but the other options are viable and will be left for the application developer to implement

[rudokemper]

To be honest, I'm not super up to date with the anatomy of mapeo-map-server and whether the current API routes could be leveraged in an environment that isn't CoMapeo, but one advantage of an HTTP API is that it is language-agnostic and so Python (or other) tooling could seamlessly work with the API without the need for any bridging. If the idea behind mapeo-map-server is to be modular, interoperable, and easily plugged into other toolchains (regardless of language), that might be a consideration. But even so there are workarounds and based on reading the thread, it definitely seems like a parsimonious solution for CoMapeo.

[EvanHahn]

To be honest, I'm not super up to date with the anatomy of mapeo-map-server and whether the current API routes could be leveraged in an environment that isn't CoMapeo, but one advantage of an HTTP API is that it is language-agnostic and so Python (or other) tooling could seamlessly work with the API without the need for any bridging. If the idea behind mapeo-map-server is to be modular, interoperable, and easily plugged into other toolchains (regardless of language), that might be a consideration. But even so there are workarounds and based on reading the thread, it definitely seems like a parsimonious solution for CoMapeo.

Good point.

Currently, you need to use Node to start the server and set up the database, so it wouldn't currently work with Python (or other).

We could address this by making this repo a standalone executable with its own HTTP server and database, but that would make things a bit harder for CoMapeo.

Instead, I think the ideal design (in the short term) is to use regular JavaScript function calls in this module. In the future, one could easily wrap these functions with an HTTP server, or a Python bridge, or FIFOs, or whatever is needed by the consumer.

[EvanHahn]

Given the discussion here, I'm going to get started on this. Merged the first patch in f374a64.

I'm not going to completely remove the HTTP server as part of this work, but I'll plan to move the rest.