Get something basic working

[coffeemug]

In order for us to start building various APIs in Fusion, parallelizing the work, defining the server protocol, etc. we should get a very basic version of the server and the client library up and running.

I'm going to describe a minimal client library API below that will allow connecting to the Fusion server, inserting data, and setting up a stream. For the moment, let's ignore the question of what the server protocol will look like -- once we get a basic server up and running that handles some basic commands, we can go back and define a proper server-side protocol. The following is a basic client-library API the user would use in the browser.

Connecting and getting a reference to a table/collection:

// Connect to the Fusion server
var ref = new Fusion('http://localhost:8181');

// Access a specific collection (which translates to a RethinkDB table in the `test` database).
// If the table doesn't exist, Fusion server will automatically create it.
var items = ref('todo-items');

Listening on events:

// Listen on events on a table. Returns an EventEmitter.
items.on('added', item => {
  // `item` was added to the db
}).on('removed', item => {
  // `item` was removed from the db
});

Inserting data:

// Insert a json object. Returns a promise.
items.insert({ item: 'Get milk.' }).then((id) => {
  // Item inserted successfully, with generated `id`
}).catch(err => {
  // Something bad happened
});

Once we get the basic client library/server infrastructure going for this, it'll be much easier to add more commands. I'll start speccing out the API in the meantime, but in general I think we should support the following operations:

• Reads: get, getAll, between, orderBy (indexed only), limit
• Writes: insert, delete, update, replace

(We'll probably rename/consolidate these so people don't get confused when they start using ReQL)

[mike-marcacci]

This looks great - I think skip goes along with limit here.

I'd like to suggest taking a look at json-patch as a candidate for the REST API. I've considered building a json-patch RQL builder for several different projects, and I think doing so would be almost trivial.

One of the biggest challenges I've faced when building web APIs that support arbitrary data is performing partial updates to embedded arrays. While RQL gives a developer excellent control over this by accepting a mutation function, Fusion will probably have to take on much of this previously-delegated responsibility. This problem is greatly complicated by the disparate states that live in each browser tab or window, because the index is not a reliable reference for an array's value. I don't have a very good answer to this problem, but it's definitely one that any project in this space needs to get right. In all my implementations, I've either designed away the need for arrays, or simulated them with objects (sometimes in pure RQL). The latter might be the best approach (and is similar to Firebase I believe), but it requires every array value to have a unique ID.

[coffeemug]

Yes, partial array changes are a tricky one. I'm not sure how/if we'll support it, or if it's necessary yet (it probably is). Another option would be to support an API that abstracts efficient table joins, which I think would be very nice. We'll see how this'll work out once we get a basic API going.

[marshall007]

From the README, it sounds like one of the primary goals of Fusion is to abstract away a good portion of ReQL:

... empirically there is still too much friction for most developers. To get started they have to learn ReQL, understand changefeeds, and figure out how to thread changefeeds through their backend code.

While I think it totally makes sense for the Fusion Client to have high-level abstractions for the common cases (especially changefeeds), I don't see a lot of value in abandoning ReQL altogether or using different terminology. By doing so, we'd simply invert the problem such that people already familiar with ReQL now have to learn another set of APIs (even though the operations are mostly identical and ReQL terms are already pretty concise and intuitive).

Furthermore, from a tooling/implementation perspective, I think it would be really convenient if the ASTs Fusion generates were simply a subset of standard ReQL.

So, IMHO the Fusion Client would consist of some subset of ReQL that gets fleshed out over time + higher-level APIs.

[coffeemug]

I don't see a lot of value in abandoning ReQL altogether or using different terminology.

We talked about this a bit, and the problem with using a subset of ReQL is that it results in code that isn't idiomatic for the front-end. For example, r.table('foo').get(bar).run(baz) is a good API for a database, but would look pretty jarring in a front-end app.

Now suppose you change it to fusion('foo').get(bar).on(...) -- that looks close enough to ReQL that someone might try and paste this into the data explorer/their ReQL code, where it'll break. So I think it would be really bad if the API mostly looks like ReQL, but really isn't, and it would be bad if it's a proper subset of ReQL.

An obvious solution is to change the names of the commands so nobody is confused, but I do see how that would force existing developers to learn a whole new API (though I don't think it's a big problem, because the API will be really simple).

[danielmewes]

@mike-marcacci

This looks great - I think skip goes along with limit here.

We've considered that, but skip has some technical problems that might make it a bad choice for the first version. The first problem is that you can't have changefeeds on skip queries at the moment. The second one is that skip can become very slow with how it's currently implemented.

Pagination should be done through between by passing in the highest key seen so far and using an open left bound. This isn't completely intuitive, so we should probably try to find a very nice and easy API to do pagination, that we can implement with between under the hood. That should be a separate step after we have the basic system working I think.

[mike-marcacci]

@danielmewes, thanks for the clarification - that makes total sense.

[marshall007]

Now suppose you change it to fusion('foo').get(bar).on(...) -- that looks close enough to ReQL that someone might try and paste this into the data explorer/their ReQL code, where it'll break.

That was kinda the idea behind Fusion/ReQL generating the same ASTs, actually. That fusion expressions could in fact be runnable in the Data Explorer and interpreted by other tools with relative ease.

I was basically thinking that the higher-level Fusion APIs would build standard ReQL expressions under the hood.

[deontologician]

Ok, on master now we have something basic working