[super] Data Query Support

[rufuspollock]

Query support would involve supporting things like:

• size / limit (already supported in a limited way)
• offset
• order_by / sort
• filter

This will clearly depend on backend capabilities.

Implementation

• New Query model object attached to Dataset as query attribute #49 - New Query model object attached to Dataset as query attribute (we make into a Model so we can have event handlers)
  • Dataset can then listen for changes to this object and trigger queries to backend as needed.
  • This also means other things such as a Query editor view within or outside of Recline can take take of updating this object
• Support in Backends for various of the operations
  • Limit / offset
  • Sorting - Memory backend done in 1491ae5
  • Filtering
• View support - query editor - Query editor view #53
  • Limit and offset support (plus pagination?)
  • Simple search box
  • Full Query editor - Query editor view #53
     * sort support in DataTable - done in 5fc4fa9

Query Object Proposal

NB: It would be up to specific backends how to implement and support this query object. Different backends might choose to implement things differently or not support certain features.

Propose to base directly ElasticSearch query language. Query object would therefore have following key attributes:

• size (=limit): number of results to return
• from (=offset): offset into result set - http://www.elasticsearch.org/guide/reference/api/search/from-size.html
• sort: sort order - http://www.elasticsearch.org/guide/reference/api/search/sort.html
• query: Query in ES Query DSL http://www.elasticsearch.org/guide/reference/api/search/query.html
• fields: set of fields to return - http://www.elasticsearch.org/guide/reference/api/search/fields.html
• facets: TODO - see http://www.elasticsearch.org/guide/reference/api/search/facets/

Additions:

• q: either straight text or a hash will map directly onto a query_string query in backend
  • Of course this can be re-interpreted by different backends. E.g. some may just pass this straight through e.g. for an SQL backend this could be the full SQL query
• filters: dict of fields with for each one specified a filter like term, terms, prefix, range
  • Value for a field can just be text in which case this becomes a term query on that field
  • E.g. `my-field: 'abc' - would only match results with abc in that field
  • This is a quick way to do filtering.

Examples

{
   q: 'quick brown fox',
   filters: {
     'owner': 'jones'
   }
}

References

• http://www.elasticsearch.org/guide/reference/api/search/
• http://www.elasticsearch.org/guide/reference/query-dsl/

Simple 'DIY' Proposal

A query object will have the following attributes:

• q: a free text query string. It will be up to specific backend implementations how to interpret this.
  • Examples: could be a solr style free text query string
  • Example: could be SQL for an SQL style setup
• limit: limit the number of items returned by the query
• offset: offset for start of query results
• filter: dict/hash of filters keyed by field (header) id. Each filter is a single text string which is used to match against that field name.

[pudo]

big +1 obviously.

[rufuspollock]

@pudo TODO: specify what this looks like in JSON (I need that to serialize it into the url for persistence at the very least ...)

[rufuspollock]

@maxogden @pudo @kindly I'd be interested in your thoughts and feedback on this proposal.

[pudo]

First steps on this are done, now its missing hash url serialization I think?

[rufuspollock]

As this includes sorting this now subsumes #23.

[rufuspollock]

I'm closing this now. Some backends don't support all of this but we have solid query support not in place in several and it is not clear what exactly remains. As new items arise we can open new tickets.