Example Falcor DataSource/Model to wrap existing RESTful resources

[baer]

So I've now read all of the docs and watched the keynote + the 8 getting started videos and I'm still not that clear on how Falcor orchestrates data fetching. Furthermore since very few of us are lucky enough to work in greenfield projects I'd imagine a very common use case would be to start by wrapping an existing RESTful API. There are two specific scenariaos that I'm not sure about how to deal with:

---

• What would be the preferred way to expose the GET lists vs. GET details pattern. These two models effectively nested.

GET /users

[{
  "id": "123",
  "givenName": "Brandon",
  "surame": "Pricher"
}, {
  ...
}]

GET /users/123

{
    "id": "123",
    "givenName": "Brandon",
    "surame": "Pricher",
    "customerLifetimeValue": "909869.00",
    "preferredLocale": "en-US",
}

---

• To extend the above, lets say my view needs the Customer + the customer's address. Lets assume we have a user model but the address is an external lookup (USPS address validation for example). On the client sideI might deal with this as follows in the controller but the model seems to have some of this information already encoded in it. Furthermore, if a Falcor client doesn't request address, there is no reason to make that call. Does Falcor orchestrate dependent queries or is there a proper way to handle this?

getCustomerViewModel(id) {
  let customer = getCustomer(id)
  let address = customer.then(getCustomerAddress)

  return Promise.all(customer, address)
    .then((customer, address) => {
      return customer.address = address;
    }).catch(errorHandler)
}

This is what the view actually needs to render the page

{
    "id": "123",
    "givenName": "Brandon",
    "surame": "Pricher",
    "customerLifetimeValue": "909869.00",
    "preferredLocale": "en-US",
    "address": {
      "street": 123,
      "city": "Memphis",
      "zip": "38120-0098" 
    }
}

[dallonf]

The latter example you gave (with address as an external resource) is actually fairly well documented and represented by the Router examples, so I'll stick with talking about what I've found regarding REST APIs, which is much less clear.

Quick warning, I will be (ab)using ES2015 and beyond syntax in this example, particularly async/await, arrow functions, and enhanced object literals.

Also, while this particular code is untested, it's based on some of my early experiments wrapping a REST API with a Falcor Router.

Alright, enough rambling, let's code!

Code

A naive approach is to build out a Router something like this:

[{
  route: 'customers.length',
  async get(pathSet) {
    var customers = await request.get('/customers');
    return {
      path: ['customers', 'length'],
      value: customers.length
    };
  }
}, {
  route: 'customers[{integers:indices}]',
  async get(pathSet) {
    var customers = await request.get('/customers');
    return pathSet.indices.map(index => ({
      path: ['customers', index],
      // NOTE: This will actually throw errors if you request out-of-bounds indexes,
      // but for the purposes of demonstration I'll keep it concise
      value: { $type: 'ref', value: ['customersById', customers[index].id] }
    }));
  }
}, {
  route: 'customersById[{keys:ids}]["id", "givenName", "surname", "customerLifetimeValue", "preferredLocale"]',
  async get(pathSet) {
    var attributes = pathSet[2];
    // Using lodash for conciseness
    return _.flatten(await Promise.all(
      pathSet.keys.map(pathSet.ids, async id => {
        var customer = await request.get('/customers/' + id);
        return attributes.map(attribute => ({
          path: ['customersById', id, attribute],
          value: customer[attribute]
        }));
      });
    ));
  }
}]

This will produce pretty much exactly the results you'd want. Let's say you want to fetch some data for the first three customers, as well as the total number of customers so you know whether you should show pagination UI:

// On the client
model.get([
  'customers[0..2]["id", "givenName", "surname", "customerLifetimeValue"]',
  'customers.length'
]);
// Async callback elided for brevity

But now let's look at what kind of requests the server is making:

• GET /customers
• GET /customers
• GET /customers/123
• GET /customers/456
• GET /customers/789

This is very wasteful! All the data we needed was available in one GET /customers call. What we need to do is cache the results of that call temporarily so that the customersById route and the length route can skip their queries if the data is already available.

The official examples suggest newing up a Router instance for every request to the Falcor endpoint, so that's a decent place for our short-lived cache, especially since the Router instance is exposed as this in the router methods. Let's see what we can do...

[{
  route: 'customers.length',
  async get(pathSet) {
    // Begin new code:
    var customers = this.customersListCache;
    if (!customers) {
      customers = await request.get('/customers');
      this.customersListCache = customers;
    }
    // End new code
    return {
      path: ['customers', 'length'],
      value: customers.length
    };
  }
}, {
  route: 'customers[{integers:indices}]',
  async get(pathSet) {
    // Begin new code:
    var customers = this.customersListCache;
    if (!customers) {
      customers = await request.get('/customers');
      this.customersListCache = customers;
      if (!this.customerCache) {
        this.customerCache = {};
      }
      customers.forEach(customer => {
        this.customerCache[customer.id] = customer;
      });
    }
    // End new code
    return pathSet.indices.map(index => ({
      path: ['customers', index],
      value: { $type: 'ref', value: ['customersById', customers[index].id] }
    }));
  }
}, {
  route: 'customersById[{keys:ids}]["id", "givenName", "surname", "customerLifetimeValue", "preferredLocale"]',
  async get(pathSet) {
    var attributes = pathSet[2];
    return _.flatten(await Promise.all(
      pathSet.keys.map(pathSet.ids, async id => {
        // Begin new code:
        var customer = this.customerCache && this.customerCache[id];
        if (!customer) {
          customer = await request.get('/customers/' + id);
          if (!this.customerCache) {
            this.customerCache = {};
          }
          this.customerCache[id] = customer;
        }
        // End new code
        return attributes.map(attribute => ({
          path: ['customersById', id, attribute],
          value: customer[attribute]
        }));
      });
    ));
  }
}]

Perfect! Now let's see what's happening on the server...

• GET /customers
• GET /customers

Well, it's certainly saved us those detail requests, but it's still requesting the list twice. This is because Falcor optimizes for maximum parallelism, and it's actually trying to resolve the customers.length route and the customers[{integers:indices}] route at the same time. Of course, we only put the value in the cache after it's loaded, so that doesn't stop us from requesting it twice.

Fortunately, since we're using promises and async/await, this isn't as terrible to fix as it could be; we just need to move some code around and cache the promises instead of the result:

[{
  route: 'customers.length',
  async get(pathSet) {
    var customersRequest = this.customersListCache;
    if (!customers) {
      customersRequest = request.get('/customers'); // Remove "await"
      this.customersListCache = customersRequest;
    }
    var customers = await customersRequest; // And put it here
    return {
      path: ['customers', 'length'],
      value: customers.length
    };
  }
}, {
  route: 'customers[{integers:indices}]',
  async get(pathSet) {
    var customersRequest = this.customersListCache;
    if (!customers) {
      customersRequest = request.get('/customers'); // Remove "await"
      this.customersListCache = customersRequest;
      if (!this.customerCache) {
        this.customerCache = {};
      }
      var customers = await customersRequest; // And put it here
      customers.forEach(customer => {
        this.customerCache[customer.id] = Promise.resolve(customer); // Wrap this in a Promise
      });
    }
    return pathSet.indices.map(index => ({
      path: ['customers', index],
      value: { $type: 'ref', value: ['customersById', customers[index].id] }
    }));
  }
}, {
  route: 'customersById[{keys:ids}]["id", "givenName", "surname", "customerLifetimeValue", "preferredLocale"]',
  async get(pathSet) {
    var attributes = pathSet[2];
    return _.flatten(await Promise.all(
      pathSet.keys.map(pathSet.ids, async id => {
        var customerRequest = this.customerCache && this.customerCache[id];
        if (!customerRequest) {
          customerRequest = request.get('/customers/' + id); // Remove await
          if (!this.customerCache) {
            this.customerCache = {};
          }
          this.customerCache[id] = customerRequest;
        }
        var customer = await customerRequest; // And put it here
        return attributes.map(attribute => ({
          path: ['customersById', id, attribute],
          value: customer[attribute]
        }));
      });
    ));
  }
}]

(Seriously, if you're not using async/await, you should be. I would not look forward to that kind of refactoring with callbacks!)

And now on the server, it will only request GET /customers and return all the data you asked for.

Conclusion

Well, we managed to wrap a very simple REST API and it makes the minimum number of requests necessary to fulfill our query. But there's certainly room for improvement. That's a lot of boilerplate code to do something very common, and I can't shake the feeling that I'm reinventing the wheel - the hard part here is arguably caching and batching, and that's what Falcor is already good at - at least on the client.

I think an ideal workflow here would involve a Falcor Model constructed on the server for each request. Each route would not only return its own data, but add all the data it found to its proper place on the Model so it wouldn't be requested again. In a Router like that, I would only need to write something like this:

[{
  // Support multiple routes for a single getter
  routes: [
    'customers.length',
    'customers[{integers:indices}]'
  ],
  async get(pathSet) {
    var customers = await request.get('/customers');
    // Instead of returning PathValues, return the entire subset of the Graph we found
    var customersById = {};
    customers.forEach(customer => {
      customersById[id] = customer;
    });
    return {
      customers: customers.map(customer => ({
        $type: 'ref',
        path: ['customersById', customer.id]
      })),
      customersById
    };
  }
}, {
  route: 'customersById[{keys:ids}]["id", "givenName", "surname", "customerLifetimeValue", "preferredLocale"]',
  async get(pathSet) {
    var customersById = {};
    await Promise.all(pathSet.ids.map(async id => {
      var customer = await request.get('/customer/' + id);
      customersById[id] = customer;
    }));
    return {
      customersById
    }
  }
}]

There's a lot of implicitness here, though, and that does seem to go against some of Falcor's design principles of explicitly declaring the data you want to provide.

[MarkHerhold]

@dallonf many thanks for this! It has cleared so much up for me! 👍 👍 👍

---

I still have some fundamental concerns (as others do I am sure) that maybe you could help me clear up.

Falcor seems to be defining a new JSON API specification, (specifically a flavor of HATEOAS) for REST between the client and server. I am unnerved by the implications this has on existing "standards" (or ones competing to be standards) like the JSON:API, JOSN-HAL, JSON-LD, or simply "traditional" object mapping to/from JSON. Your example covers the client side case showing that you don't need to rewrite your REST server (yay), but there is so much boilerplate that servers should arguably just dedicate an entire set of endpoints or a dedicated router to serving up Falcor-spec routes. I am concerned about the additional workload and lock-in that this will cause and how other clients will be supported (e.g. Java, C#/.Net, etc, etc).

Caching and graphs are really nice, but I am having trouble seeing the cost-benefit given the risk (when is the next Falcor going to appear?). Maybe these are just silly concerns or I am over estimating the work that supporting Falcor will incur.

[baer]

@dallonf thank you so so so much for a very thorough and well written response. I really appreciate that.

[trxcllnt]

@MarkHerhold I am unnerved by the implications this has on existing "standards" (or ones competing to be standards) like the JSON:API, JOSN-HAL, JSON-LD, or simply "traditional" object mapping to/from JSON.

Falcor was designed to be as close to your last example as possible: the minimal-possible set of type and operation definitions to make interacting with a JSON object on a different machine as easy as interacting with a JSON object in memory. We identified the types and operations that were necessary to solve this problem at Netflix, but have no opinion on JSON standards that apply to anything beyond that.

[MarkHerhold]

@trxcllnt Ok, makes sense. Thank you.

[barsukov]

But it looks really monsterable, do we have any other solutions to not implement so many workaround which should be done by others ? I mean even I have ability to change the backend response (but backend has written in another language(ruby) this is example of task #593

[tivac]

I've got a partial implementation of router-on-the-client talking to an existing HTTP API working, I'm actually pretty happy w/ it.

https://github.com/tivac/falcor-experiment

[baer]

I'd love to see a Falcor wrapper of https://swapi.co/. It would be a great way to prove out these ideas.

[wilkerlucio]

@dallonf hey, a bit off topic, but can you please tell me what you are using for the async/await on the server? When I looked for it I found this one https://github.com/yortus/asyncawait but this uses a function syntax while your example seems to use native implementation, I'm using Node 4.2, do you have a tutorial to point out? Thanks.

[dallonf]

@wilkerlucio Babel for the win 😄! Alternatively, Node 4.x natively supports generators, so something like Bluebird.coroutine or co can get you almost all the way there even without transpilation.

[wilkerlucio]

@dallonf thanks, I was using babel on the client already and just didn't realized I was one require("babel-core/register") away from having on the server too, so much to refactor now.

[greim]

My existing REST service returns nested resources. In order to map these into json graph, my idea was to pull them into the top level and return them, even though the client didn't ask for them. For example:

model.get(['usersById', 4, 'username']);

that causes the router to retrieve this:

// GET /users/4
{
  id: 4
  username: 'greim'
  avatar: {
    id: 5
    src: 'http://.../foo.jpg'
  }
}

which gets converted to:

{
  usersById: {
    4: {
      username: 'greim'
      avatar: { $type: 'ref', path: ['mediaById', 5] }
    }
  }
  mediaById: {
    5: {
      src: 'http://.../foo.jpg'
    }
  }
}

Since the service did the heavy lifting to pull that info out of the DB, we might as well give it to the client to use later. I verified it comes across the wire to the client, but mediaById doesn't get cached in the model, so if I subsequently do:

model.get(['usersById', 4, 'avatar', 'src']);

that makes another call to the service. Which would defeat the scalability wins I was hoping for. If I was designing the REST API I'd do it differently, but for now that's what I have to work with. Is there any way to tell Falcor to cache stuff that gets returned but which the client didn't ask for?