Best practices for search endpoints with multiple criteria

[c0]

This is an extension of #713.

I have a search endpoint that accepts multiple criteria. Not all of them are required. It may look like:

{
  city: 'New York',
  state: 'New York',
  lowPrice: 1000,
  highPrice: 3000
}

I didn't see where path sets include key-value pairs, so the best I could come up with for the router is:

search[{keys:criteriaKeys}][{keys:criteriaValues}]['name']

An example path:

['search', ['city', 'lowPrice' ], ['New York', 1000], ['name']]

Is there a best practice for how to search?

[greim]

As far as I can tell, once you get to where you need to support arbitrary combinations of parameters, you have to start using querystrings (or something similar) as keys:

// router
"searches[{keys:query}][{ranges:index}]"

// pathset
['searches', 'foo=bar&baz=qux', { from: 0, to: 19 }]

I'd love to hear better ideas though. That's just the best one I've heard so far.

[joshdmiller]

Take this for what it's worth, but I'm not too fond of the idea of using query strings. To me, the idea of modelling our data on a javascript object is based on predictability (which is how caching and cache-first are able to work). The query string method limits that predictability.

For example, in most reasonable implementations, we would treat the order of the query parameters as largely irrelevant. As far as object modelling is concerned, however, these become two different strings. Our application is now responsible for maintaining strict ordering, lest we end up with duplicate data in the cache and unnecessary fetches. Ditto for optional properties: ?bar=hello&foo=false and ?bar=hello are potentially equivalent.

But it's problematic for me too conceptually. We're forcing query-type fetches into an object structure. It would be better, at least conceptually, to remove those kinds of queries from the falcor model entirely. Perhaps with a standard REST endpoint that can return an array of refs that the client would then set to the model. The parts pushed to the model are predictable and well-modelled, but the querying was left to the part of the tech stack to which it was best suited.

I'd certainly love to hear the Falcor team's thoughts here. I've seen a lot of semi-related comments on other issues.

[greim]

I have the same concerns actually, but I think everything you listed is a symptom of the more general anti-pattern of exposing a querying service to the world that allows a combinatorial explosion of possibilities. Expressing that through Falcor merely surfaces the problems with that in Falcor-specific ways.

The right thing to do in my view would be to restrict what the world can query down to the smallest possible whitelist of combinations. Falcor can put those known combinations into a graph without resorting to query strings.

"searches[{keys:query}]['byDate','byTitle']['asc','desc'][{ranges:index}]"
"searches[{keys:query}]['byDate','byTitle']['asc','desc'].length"

.
`--searches
   `--foo
      |--byDate
      |  |--asc
      |  |  |--length
      |  |  |--0
      |  |  |--1
      |  |  `--...
      |  `--desc
      |     |--length
      |     |--0
      |     |--1
      |     `--...
      `--byTitle
         |--asc
         |  |--length
         |  |--0
         |  |--1
         |  `--...
         `--desc
            |--length
            |--0
            |--1
            `--...

[joshdmiller]

@greim Totally agreed. For the vast majority of searching use cases, the solution you outlined above is precisely what I'd recommend. However, there are edge cases (particularly in applications that also have intelligence/decision support capabilities) where more sophisticated querying is important. In such cases, assuming Falcor even still makes sense, an external querying service may be the best solution.

[omerts]

@greim, @joshdmiller What would you do for multiple filters of the same key?
For example:

{
    todosById: {
        "44": {
            name: "Init",            
            status: 0
            prerequisites: []
        },
        "54": {
            name: "Started",
            status: 1
            prerequisites: [{ $type: "ref", value: ["todosById", 54] }]
        },   
        "58": {
            name: "Paused",
            status: 3
            prerequisites: [{ $type: "ref", value: ["todosById", 54] }]
        },           
        "64": {
            name: "Completed",
            status: 2
            prerequisites: []
        }
    },
    todos: [
        { $type: "ref", value: ["todosById", 44] },
        { $type: "ref", value: ["todosById", 54] },
        { $type: "ref", value: ["todosById", 58] },
        { $type: "ref", value: ["todosById", 64] }
    ]
}

I would like to get only todos that are either status 0|1.
I could theoretically send two separate requests, but the order of the todos in the responses could be important. For example if i need 10 todos, sending two requests for 5 & 5, would most likely give me a different order, and number of each type, of todos, than sending directly to the REST API one request with an 0 | 1 as filter parameteres.

Of course, I wouldn't want to have to create a key for each combination of statuses, especially if I might have 10 different statuses.

Maybe using call operations?

[greim]

The approach I'd prefer is to just expose a bunch of different "keys" on the JSON graph. In the route handler it would then just construct endpoint URLs based on which "key" was matched by the route:

• todos => /api/todos (you already have this one)
• completed_todos => /api/todos?status=complete
• incomplete_todos => /api/todos?status=incomplete
• foobar_todos => /api/todos?foo=bar

Obviously the caveat is that maybe you couldn't possibly anticipate them all (the afore-mentioned combinatorial explosion) in which case you have to resort to some kind of hack or workaround, as discussed above.

[nickretallack]

It's a shame that Falcor distinguishes itself as "not a query language". Search queries are likely to return $refs, so you'd want them to take advantage of Falcor's cache. You could build a separate API for searching and setCache the results back into Falcor I guess, but can you teach Falcor about refs this way? You'd also be giving up the ability to ask for specific fields from the referenced search results.

Falcor already has a syntax for passing parameters. "foo[0..10].name" compiles to ["foo", {"from": 0, "to": 10}]. So why can't we say ["foo", {"city": "New York", "state": "New York"}, "name"]? Unfortunately, the Falcor client strips out anything it doesn't understand, but we could fork it. This feature could manifest in paths like 'foo[{"city": "New York", "state": "New York"}]'.

Failing that, querystrings seem like a decent option. It's a standard, and it's something your browser already knows how to do. JSON might work as well if you stringified it.

An option that falls more in line with how Falcor wants you to do things is to just not allow for optional fields. Write your full faceted search as one big route and require the client-side code to pass in a value for every facet every time, even if that value is just "ignore me". So lets say you can query on all the fields in the example above, but you decide to omit the city. Your query might look like this: "apartments.state["New York"].city["any"].lowPrice[1000].highPrice[3000].name". Each time you add a new facet on the server side you'll need to add another route and leave the old one for backward compatibility.

Another option is to parse the query yourself on the server side instead of using the default router. Just establish a rule that keys and values alternate, and you'll be able to handle queries of the form ["apartments", "state", "New York", "city": "New York", "foo", "bar"] as if it were {"state": "New York", "city": "New York", "foo": "bar"}

You might even be able to express this inside the standard router by using recursive $refs, but that's probably a silly idea.

[greim]

It would be interesting to hear from someone familiar with GraphQL, to see if it has a general-purpose approach for tackling multi-faceted search. Also might be nice if the Falcor core team would officially weigh in on this issue, but yeah I'd expect the answer to echo past statements about why allowing the public to make open-ended queries might be a bad idea.

[eddieajau]

I would consider something like:

interface Range {
  from?: number;
  to?: number;
  length?: number;
  where?: RangeWhere;
  order?: RangeOrder;      
}

where RangeWhere is something along the lines of a Mongo style syntax and RangeOrder is something like this.

[abetkin]

Possibly, the issue described here can be solved by addressing a larger issue of being able to pass arguments along with a set of queried paths. That one is tracked in #826

[trxcllnt]

@eddieajau see #826 (comment)

[steveorsomethin]

I'm currently performing issue triage as we get ready to perform a proper release, and closing/tagging as I go.

I've also commented in #826 reinforcing Paul's guidance. Closing.