Add support for db.import

[Natim]

For some of the Mozilla platform use-cases, it's desirable to have the ability to import initial data. Since we only need it here, for the time being, we can prototype the idea in the Firefox storage adapter and apply what we learn to the other adapters.

[n1k0]

As discussed previously and as a first easy step before we implement full support for transactions, the base adapter interface should require implementation of an import(<array>) method, to import a large list of records using a single transaction, mostly for performance reasons.

This method should return a Promise, with a simple result like the number of imported records. It should reject when the transaction fails, probably simply by forwarding the backend error.

This method should probably be exposed at the Collection level as well, so users can bulk import large datasets right from their collection instance; sample api:

const coll = kinto.collection("articles");
const recordsToImport = [{title: "art00001", ..., }, ..., {title: "art99999", ...}];
coll.import(recordsToImport)
  .then(res => {
    console.log(`${res} records were imported.`);
  });

[Natim]

We've added adapter import method for LocalStorage and IndexedDB with @leplatrem.

[leplatrem]

Remaining tasks:

• Documentation
• Move of updated last modified to adapter tests
• Update collection last modified only if superior

[Natim]

In order to update the FirefoxStorage adapter, we need to add tests in Firefox.

Here is the flow to run the FirefoxStorage tests:

1. kinto.js> npm run dist-fx
2. cp ~/kinto.js/dist/moz-kinto-client.js ~/moz-central/services/common/
3. moz-central> ./mach build
4. Update services/common/tests/unit/test_storage_adapter.js and services/common/tests/unit/test_kinto.js
5. Run the tests with ./mach xpcshell-test services/common/tests/unit/test_storage_adapter.js

[Natim]

Ok so we are almost there I guess.

The remaining problem to handle is the collection_timestamp handling.

We do not want import to overwrite records in a more recent version.
So we need to make sure that the collection_timestamp we are importing is either null or lower than the minimal last_modified value for the records we are importing.

[n1k0]

How about rejecting at first invalid record encountered instead? If we provide the id, that's useful for debugging.

[leplatrem]

with some() ?

[Natim]

An even better solution would be to import only records of the dump with a last_modified value newer than the current collection_timestamp.

[n1k0]

Ditto

[mozmark]

The difficulty with only importing items with lastModified later than the newest value is that you will end up keeping things that should be removed. I'd suggest either 1) Only allow import of a new collection or 2) Allow a collection to be completely replaced if (and only if) the imported data has a last modified value newer than the old collection.

[Natim]

Yes that's the idea.

Also you raise an important point, we should probably handle tombstones in our import in order to remove deleted records.

[Natim]

• Rename import to loadDump
• Skip records that were already updated (local record status: synced and last_modified value more recent than the one in the dump or status: updated).
• Result of promise should return the list of modified records.

[mozmark]

This looks good to me. I guess we r? @n1k0 when ready.

[n1k0]

Nit: Importing a data dump locally

[n1k0]

LGTM, r+

[rnewman]

This…