Loader/Connector API

[helfer]

Sorry in advance for the long text. This is essentially a braindump because I'm still figuring things out.

Having thought about it a little more, I think what I've been calling loaders until now is actually three layers that can be separated (I'm not sure if the names I found are really the best):

1. Connectors
2. Loaders (or Fetchers)
3. Transports

Connectors
Each type has its own connector, which means they're schema specific and not that reusable between projects. They're also somewhat optional because that code could be written in the resolve function, but I would recommend using the connector abstraction anyway for modularity.

Connectors are exposed to the resolve functions via the context. Resolve functions call the connectors, which will have an API that has methods like the following:

• get by id (needed anyway for node interface. Should be able to specify the fields wanted)
• get by attribute (i.e. name on Person)
• custom ones for arguments, for example matching a RegExp or specifying a range. Most likely the resolver would just pass its arguments to the connector with little processing.

Connectors will also have to support writing, not just fetching. So you should be able to update an object through the connector. Create, Update and Delete are the basic cases. Arbitrary mutations are still confusing me a bit. If they implement complex logic maybe that should be in the connector as well and not in the resolve function? I’ll have to think more about it.

In general, the methods on the connectors will most likely closely reflect the backend operations available, but they may also abstract over the backend operations by combining multiple of them into a single operation. For the discourse API we need to get a CSRF token sometimes, but that’s not an operation that a connector would expose, but only use internally.

I think it’s important that we aim for simplicity over efficiency at first. What I mean by that is that if you wanted to delete a whole list of objects, you would have to call delete for each of the objects. If folks need to optimize further, it should be easy to do if they can swap out the db loader at any time and replace it with their own.

Hm… is that too much boilerplate? Too much abstraction? It should be easy to get started with, and it shouldn’t scare people off. Ideally it would be the right thing for a small project, as well as for a huge project.

Loaders
The loaders are the place where batching and caching/memoizing happens. They are specific to the backend, which means they are reusable between projects.

If the backend is standardized - like SQL for example - then we can write just one generic loader for it. If the backend is not standardized - eg. most RESTful APIs - then we will most likely have to write a loader for each API flavor, of which there may be many. We may even have to write a loader for each endpoint the API exposes if the endpoints don't share a common style.

SQL, Mongo & most other DBs : the loader is essentially a query builder (like knex). It's not an ORM.
RESTful APIs: the loader is custom to that API, maybe even to the particular endpoint. This is necessary because there's usually no standard way of batching requests, and because the . If the API doesn't provide any way to batch requests or cache individual nodes then an inefficient standard loader could be used which essentially caches the whole response, but only for that exact request.

DataLoader should be helpful for this. Initially we could just build our loaders with DataLoader, but I think we'll want to have more control and more efficiency later so we'll probably move beyond it.

Let’s say we have requests to load a bunch of objects by attribute or ID in SQL. Without you needing to write code, the loader should be able to batch these requests. SELECTs from the same table and selecting on the same set of fields should happen in one query (concatenated OR, then match back to original request when data is returned - for efficiency, you might attach that condition to the columns of the query so you know which one it matched). It should be possible to turn batching on and off and to bypass it for individual elements. SQL loader can probably extend DataLoader. SQL loader v1 should be pretty generic, if possible it won't make use of any Postgres, MySQL, MSSQL or Oracle specific stuff. It should produce queries, which are then passed to a DB adapter (transport) that deals with sending requests and getting responses.

For REST, the story is a bit different. There we cannot really do any batching unless the backend supports it. I think the right way to go there is to put a bit of pressure on the REST backends to support this kind of batched fetching. It’s only required as an optimization though, so people can get started without it. If we wait long enough, these classical REST endpoints will all but disappear, I believe.

Transport
The transport (not a great name, a bit of a collision) is what sends the actual data over the wire as far as we're concerned. It just takes the request given by the loader and passes back the response when it arrives. Examples for transports: node http, DDP, node MySQL adapter, node Postgres adapter etc.

Okay, that was quite a lot of text. I feel like I need to work on some specific examples to refine the ideas, because right now I don't know if this really fits in practice, but I wanted to write this here so we can start a discussion.

Let me know what you think @jbaxleyiii @stubailo .

PS: ORMs fit into the picture above, but not perfectly. They fulfill the role of Connector and Loader, but they don't do any batching for us afaik. Maybe they don't even do caching. To let people use these, we'll have to figure out how to make them support batching and caching in the way we want. It may be easy, but I haven't looked into that yet.

[helfer]

This should move to Apollo Server or to separate repos for loaders.