IndexedDB is a transactional database for client-side storage. Each record in the database contains a key-value pair. getAll() enumerates database record values sorted by key in ascending order. getAllKeys() enumerates database record primary keys sorted by key in ascending order.
This explainer proposes a new operation, getAllRecords(), which combines getAllKeys() with getAll() to enumerate both primary keys and values at the same time. For an IDBIndex, getAllRecords() also provides the record's index key in addition to the primary key and value. Lastly, getAllRecords() offers a new direction option to enumerate records sorted by key in descending order.
Decrease the latency of database read operations. By retrieving the primary key, value and index key for database records through a single operation, getAllRecords() reduces the number of JavaScript events required to read records. Each JavaScript event runs as a task on the main JavaScript thread. These tasks can introduce overhead when reading records requires a sequence of tasks that go back and forth between the main JavaScript thread and the IndexedDB I/O thread.
For batched record iteration, for example, retrieving N records at a time, the primary and index keys provided by getAllRecords() can eliminate the need for an IDBCursor, which further reduces the number of JavaScript events required. To read the next N records, instead of advancing a cursor to determine the range of the next batch, getAllRecords() can use the primary key or the index key retrieved by the results from the previous batch.
This explainer proposes adding getAllRecords() to both IDBObjectStore and IDBIndex. getAllRecords() creates a new IDBRequest that queries its IDBObjectStore or IDBIndex owner. The IDBRequest completes with an array of IDBRecord results. Each IDBRecord contains the key, primaryKey and value attributes. For IDBIndex, key is the record's index key. For IDBObjectStore, both key and primaryKey return the same value. The pre-existing IDBCursorWithValue interface contains the same attributes and values for both IDBObjectStore and IDBIndex. However, unlike getAllRecords(), a cursor may only read one record at a time.
// Define a helper that creates a basic read transaction using `getAllRecords()`.
// Wraps the transaction in a promise that resolves with the query results or
// rejects after an error. Queries `object_store_name` unless `optional_index_name`
// is defined.
async function get_all_records_with_promise(
database,
object_store_name,
query_options,
optional_index_name
) {
return await new Promise((fulfill, reject) => {
// Create a read-only transaction.
const read_transaction = database.transaction(
object_store_name,
"readonly"
);
// Get the object store or index to query.
const object_store = read_transaction.objectStore(object_store_name);
let query_target = object_store;
if (optional_index_name) {
query_target = object_store.index(optional_index_name);
}
// Start the getAllRecords() request.
const request = query_target.getAllRecords(query_options);
// Resolve promise with results after success.
request.onsuccess = (event) => {
fulfill(request.result);
};
// Reject promise with error after failure.
request.onerror = () => {
reject(request.error);
};
read_transaction.onerror = () => {
reject(read_transaction.error);
};
});
}
// Read the first 5 records from an object store in the database.
const records = await get_all_records_with_promise(
database,
kObjectStoreName,
/*query_options=*/ { count: 5 }
);
console.log(
"The second record in the database contains: " +
`primaryKey: ${records[1].primaryKey}, key: ${records[1].key}, value: ${records[1].value}`
);Developers may use the results from getAllRecords() to construct a new Map that contains a key-value pair for each database record returned by the query.
// This example uses the `get_all_records_with_promise()` helper defined above.
//
// Read the last 9 records from an index.
const records = await get_all_records_with_promise(
database,
kObjectStoreName,
/*query_options=*/ { count: 9, direction: 'prev' },
kIndexName
);
// Map the record's index key to the record's value
const map = new Map(records.map(({ key, value }) => [key, value]));
// Returns the database record value for the index `key` when the record exists
// in `map`.
const value = map.get(key);
// Use the following to create an iterator for each database record in `map`:
const index_key_iterator = map.keys();
const value_iterator = map.values();
const entry_iterator = map.entries(); // Enumerate both index keys and values.Many scenarios read N database records at a time, waiting to read the next batch of records until needed. For example, a UI may display N records, starting with the last record in descending order. As the user scrolls, the UI will display new content by reading the next N records.
To support this access pattern, the UI calls getAllRecords() with the options direction: 'prev' and count: N to retrieve N records at a time in descending order. After the initial batch, the UI must specify the upper bound of the next batch using the primary key or index key from the getAllRecords() results of the previous batch.
// This example uses the `get_all_records_with_promise()` helper defined above.
//
// Create a batch iterator where each call to `next()` retrieves `batch_size` database
// records in `direction` order from `object_store_name` or `optional_index_name`.
async function* idb_batch_record_iterator(
database,
object_store_name,
direction,
batch_size,
optional_index_name
) {
let is_done = false;
// Begin the iteration unbounded to retrieve the first or last `batch_size` records.
let query;
while (!is_done) {
const records = await get_all_records_with_promise(
database,
object_store_name,
/*query_options=*/ { query, count: batch_size, direction },
optional_index_name
);
if (records.length < batch_size) {
// We've iterated through all the database records!
is_done = true;
return records;
}
// Store the lower or upper bound for the next iteration.
const last_record = records[records.length - 1];
if (direction === "next" || direction === "nextunique") {
query = IDBKeyRange.lowerBound(last_record.key, /*exclusive=*/ true);
} else { // direction === 'prev' || direction === 'prevunique'
query = IDBKeyRange.upperBound(last_record.key, /*exclusive=*/ true);
}
yield records;
}
}
// Create a reverse iterator that reads 5 records from an index at a time.
const reverse_iterator = idb_batch_record_iterator(
database,
"my_object_store",
/*direction=*/ "prev",
/*batch_size=*/ 5,
"my_index"
);
// Get the last 5 records.
let results = await reverse_iterator.next();
let records = results.value;
console.log(
"The first record contains: " +
`primaryKey: ${records[0].primaryKey}, key: ${records[0].key}, value: ${records[0].value}`
);
// Get the next batch of 5 records.
if (!results.done) {
results = await reverse_iterator.next();
}Similar to getAllRecords() but provides results as an array of entries. Each entry is a two or three element array containing the record's key, value and optional index key. For example:
IDBObjectStore entries provide array values with two elements: [ [primaryKey1, value1], [primaryKey2, value2], ... ]
IDBIndex entries provide array values with three elements: [ [primaryKey1, value1, indexKey1], [primaryKey2, value2, indexKey2], ... ]
Developers may directly use the entry results to construct a Map or Object since the entry results are inspired by ECMAScript's Map.prototype.entries(). However, getAllEntries() has unusual ergonomics, requiring indices like 0 and 1 to access the record properties like key and value. Also, IndexedDB database records do not map cleanly to ECMAScript entries. For IDBIndex, the results contain a third element for index key. For an alternate form, [[ indexKey1, [ primaryKey1, value1]], [ indexKey2, [ primaryKey2, value2]], ... ], the index key cannot always serve as the entry's key since the index key may not be unique across all records.
This will be pursued separately. Join the discussion at w3c/IndexedDB#130. Providing the direction option on getAllKeys() might be useful for reverse iteration scenarios that don't need to load every value enumerated.
dictionary IDBGetAllRecordsOptions {
// A key or an `IDBKeyRange` identifying the records to retrieve.
any query = null;
// The maximum number of results to retrieve.
[EnforceRange] unsigned long count;
// Determines how to enumerate and sort results.
// Use 'prev' to enumerate and sort results by key in descending order.
IDBCursorDirection direction = 'next';
};
interface IDBRecord {
// For `IDBIndex` records, `key` is the index key. For `IDBObjectStore`
// records, `key` is the same as `primaryKey`.
readonly attribute any key;
readonly attribute any primaryKey;
readonly attribute any value;
};
[Exposed=(Window,Worker)]
partial interface IDBObjectStore {
// After the `getAllRecords()` request completes, the `IDBRequest::result` property
// contains an array of records:
// `[[primaryKey1, value1], [primaryKey2, value2], ... ]`
[NewObject, RaisesException]
IDBRequest getAllRecords(optional IDBGetAllRecordsOptions options = {});
}
[Exposed=(Window,Worker)]
partial interface IDBIndex {
// Produces the same type of results as `IDBObjectStore::getAllRecords()` above,
// but each entry also includes the record's index key at array index 2:
// `[[primaryKey1, value1, indexKey1], [primaryKey2, value2, indexKey2], ... ]`
[NewObject, RaisesException]
IDBRequest getAllRecords(optional IDBGetAllRecordsOptions options = {});
}- Web Developers: Positive
- Developers have reported the limitations addressed by
getAllRecords(). A few examples:
- Developers have reported the limitations addressed by
- Chromium: Positive
- Webkit: No signals
- Gecko: No signals
Special thanks to Joshua Bell who proposed getAllRecords() in the W3C IndexedDB issue.
Many thanks for valuable feedback and advice from: