Indexed data synchronization
Author: Christopher J. Brody firstname.lastname@example.org
I can hereby testify that this project is completely my own work and not subject to agreements with any other parties. In case of code written under direct guidance from sample code the link is given for reference. In case I accept contributions from any others I will require CLA with similar statements. The license may be changed at some point in the future.
Status: EXPERIMENTAL: WIP under development with API subject to change
Data is stored in key-value format along with optional index items. The key must be a globally unique string such as GUID, each indx item is a string, and the value is a JSON string.
Data with a unique key can be stored and optionally deleted at some point in time. Data for a key can never be changed and a key may not be reused once it is deleted.
Applications can store a large number of arbitrary data records with fields that are used to reference other records. The goal is to support multiple arbitrary nested collections with a higher-level client-side API. Mutable items would be simulated by adding attribute records that can be "overwritten" (delete the old and add the new).
Data is committed to a change log table on the server side. There are two types of changes:
- ADD a new key-value record
- DELETE a record for a key
A client keeps a local copy of the data and keeps track of which data has been successfully stored on the server.
The implementation is maintained in CoffeeScript which is compiled by the following command:
One or more indexed key-value stores may be stored in a sqlite database using node-sqlite3.
A log table is CREATEd for each store.
There is a sample Express REST interface in the
express-sample subdirectory. Note that Express is favored over just using Connect to support use with PassportJS.
FUTURE TBD: Swappable storage layers possible for SQLite, Postgres, redis, etc.
NOTICE: This project assumes a recent version of Node.js with a Promise implementation.
- SQLite data storage layer: see
- Sample Express application
- data synchronization client factory with sample client-side storage stub, see
- TODO: sample client-side AJAX proxy
- Client-side storage using SQLite plugin/Web SQL/IndexedDB with failure-safe transactions
- Fix server-side to use failure-safe transactions (BEGIN/COMMIT/ROLLBACK)
- Document the API/usage
- Improve factory API consistency
- Improve/further testing of two-way client/server data synchronization
- Check that each key is really unique and that a record for a key is only deleted once
- More automatic testing, with emphasis on: error handling, verify key is really unique, delete for key that is not present or already deleted
- Distinguish temporary errors vs permanent errors during add/delete/sync
- Subscriptions to store changes (support some kind of a "notification" system)
- Periodic cleanup of old change history
- VACUUM or AUTOVACUUM
May be in a higher-layer API library:
- Multi-user management
- Limited multi-user shared stores
- REST multi-user security using something like PassportJS
- Server-side data clustering
Other FUTURE TBD
- Swappable storage layers possible for SQLite, Postgres, redis, etc.
- Replace CoffeeScript with ES5/ES6
- Drop store from a database
Sources of inspiration: