Support queries retrieving disjoint sets of keys

[inexorabletash]

To allow getAll() (and friends) to retrieve a disjoint set of keys, introduce IDBKeySet which can be used for any of the get-type methods (get, getKey, getAll, getAllKeys) instead of a key-or-key-range.

[Constructor(sequence<any>), Exposed=(Window,Worker)]
partial interface IDBKeySet {
  readonly setlike<any>;
};

... hrm, okay, now that seems silly. Why not just an ES6 Set where non-keys result in failure? Need to discuss with others...

[inexorabletash]

Noodling:

// valid?
typedef (unrestricted double or Date or DOMString or BufferSource or sequence<IDBKey>) IDBKey;

// old:
typedef (IDBKey or IDBKeyRange) IDBQuery;

// new ???

[inexorabletash]

Web IDL typedefs can't be self-referential, so that won't quite work - sigh

...

Anyway, options include:

• IDBKeyRange.inList(...) (or any(...)) - disjoint "range".
  • Pro: easy entry point. minimal impact to other API IDLs
  • Con: not a range; .lower/.upper/etc cease to have meaning. See public-webapps discussion
• Introduce IDBKeySet (or IDBKeyList) w/ constructor that takes an iterable (of keys or throw)
  • Pro: does what it says - new IDBKeySet(1, 2, 3) is pretty straightforward
  • Con: yet another IDL-defined type
• Just use Set in the API
  • Pro: no type for type's sake
  • Con: need to define the binding for Set. Also, does it need to be a set or any iterable? But then it's easily confused with an Array (a valid key type)

Also: can you compose these with ranges? e.g. a set of keys and ranges?

[inexorabletash]

My assumption is that we'd support this anywhere the spec uses query as a parameter and accepts a key-or-range today, namely:

• IDBObjectStore
  • delete — deletes all
  • count — counts all
  • get — gets first
  • getKey — gets first key
  • getAll — gets all
  • getAllKeys — gets all keys
  • openCursor — iterates over all
  • openKeyCursor — iterates over all
• IDBIndex
  • count — counts all
  • get — gets first
  • getKey — gets first key
  • getAll — gets all
  • getAllKeys — gets all keys
  • openCursor — iterates over all
  • openKeyCursor — iterates over all

Order of results needs to be defined for everything except delete and count. E.g., assume a store that contains records with keys 1, 2, 3, 4:

store.openCursor(new IDBKeySet(4, 3, 2, 1)) — Iteration: what order?
store.getAll(new IDBKeySet(4, 3, 2, 1)) — Compound result: what order?
store.get(new IDBKeySet(4, 3, 2, 1)) — Single result: but which?

This impacts how we'd write algorithms like steps for retrieving a value from an object store when we use a generic query instead of a range as input, which e.g. currently says Let record be the first record in store's list of records whose key is in range, if any. If we treat "set" as a new type of query then logically we'd say "first record in store's list of records whose key matches query" and then define "matches query" to handle sets or ranges.

[inexorabletash]

If we introduce a new type like IDBKeySet then:

• it should be iterable and have be ordered in key order, duplicates elided
• it should have an includes() or has() member

[jfroelich]

This may be naive, but what about supporting a generic predicate function akin to Array.prototype.find or the NodeFilter function argument to createNodeIterator? How hard is it to translate an arbitrary, user-defined predicate into something that can still provide great performance? Is it simply not feasible? I understand that a predicate could provide too much flexibility. Maybe there could some some simple extra rules about what a predicate can contain? Maybe some validation of the predicate occurs once at the start, throws some type of DOM error is not allowed, maybe there is some upfront compilation of the function's intent into C++ land, maybe the function is invalid if it tries to access variables in outer scope or globals, and so forth.

var request = store.openCursor(function myCustomPredicateFunction(object) {
return object.id in [1, 4, 3, 2];
});

[inexorabletash]

@jfroelich That fits more with #45. It's not a naive or crazy thought, but the devil is in the details:

Arbitrary JS is not a win over the existing cursor mechanism. There's always room for improvement in implementations, but that's a "quality of implementation" thing. There's the notion of lightweight stateless isolated computation environments - Worklets are the new hotness here. But even then you're paying for some overhear. In Chrome, for example, we don't run JS in the process that touches the database, so we would still have the IPC overhead. Also, database entries require deserialization before JS can see them.

So all of this turns into "is there a domain-specific language we could use to express queries that is executed directly in the database engine?" - whether it looks like a subset of JS or is a declarative structure (which is what e.g. ranges or sets a simple cases of) is what needs consideration and designing. Again, this is what #45 is pointing towards.

[inexorabletash]

Rebooting, and referencing https://lists.w3.org/Archives/Public/public-webapps/2013AprJun/0848.html which emphasizes the benefits of lists over sets:

[Constructor(sequence<any>), Exposed=(Window,Worker)]
interface IDBKeyList {
  iterable<any>;
};

Anywhere in the spec that takes any query as an argument would be augmented to allow an IDBKeyList in addition to keys and IDBKeyRange. This includes: get(), getKey(), getAll(), getAllKeys(), count(), openCursor(), openKeyCursor() and delete(). This would be done by changing convert a value to a key range to "convert a value to a query". A query gets a definition like a key range where keys can be in a query.

The database operations (retrieval, delete, count) change to taking a query.

Cursor iteration is more intrusive. Maybe we disallow?

[domenic]

I tried to gather context from the rest of the thread, but couldn't quite understand why a new interface is needed, instead of using sequences.

[inexorabletash]

Keys can already be arrays, i.e.:

store.get([1,2,3]);

... already means something - [1,2,3] is a compound key, referencing a single record in the database.

[inexorabletash]

(An Array subclass might be fine, not sure what the hotness is these days though.)

[domenic]

Got it. Maybe instead just create a simple wrapper type around the array, by instead of making it iterable, just expose a getter (maybe setter too!) to operate on the array. Then people can use all their favorite array methods with no fuss.

You could even do this as a dictionary, maybe. dictionary IDBKeyList { sequence<any> keys; }.

[inexorabletash]

TPAC 2019 Web Apps breakout:

• A list was considered easier to reason about than a set
• Allow using with get, getAll, getKey, getAllKeys, count and delete
• Disallow with openCursor and openKeyCursor since it's confusing if out of order or has duplicates

Exact API still needs sketching.