Synchronisation design

Overview

• In order to maintain up-to-date data, mSupply Mobile regularly syncs with an mSupply server whenever there is internet available.
• Mobile synchronisation uses a similar mechanism to that used by satellite sync sites in a desktop/server setup. For more information on mSupply synchronisation, see the mSupply user manual.

Tracking record updates

• Record tracking is done using a queue data structure, referred to as the sync queue.
• When a local record is modified, a sync record is enqueued, including:
  • The type of change (create, update, delete).
  • The type of the updated database object.
  • The ID of the object that was changed.
  • The time of the change.
• If the change is a delete, any previous sync records for the object are removed from the sync queue.
• If a duplicate sync record exists the queue, the new record is not enqueued.

Sync protocol

"Safe" sync

• To ensure records sent by the remote site are received by the central server, all records remain in the sync queue until an acknowledgement response is received from the server.
• To ensure records sent by the central server are received by the remote site, all records remain in the sync queue until a POST request is made to the acknowledged_records endpoint. mSupply Mobile will not make this request until all records from the central server have been successfully integrated into local storage.
• Incoming record integration is performed inside a transaction, such that records can be rolled back in the event of a crash. No acknowledgement will be sent in this event, maintaining consistency between local and remote databases.
• In the rare event that records are integrated before an acknowledgement can be sent, duplicate records will be appropriately handled during the next sync.

"Push-pull"

• mSupply Mobile synchronises with the server using a "push-pull" protocol, meaning local changes are pushed to the server before remote changes are pulled down.
• If any local changes have been heard since the last sync, these will be pushed from local storage to the server database.
• If any remote changes have been made since the last sync, these will be pulled down from the server database and into local storage.
• By pushing before pulling, responsibility for handling inconsistencies or conflicts is handed off to the sync server. If a conflict occurs between local and remote data, the server is responsible for detecting this and appropriately reconciling the data sets.
• The following describes the algorithm for pushing local records to the central server:

While the sync queue is not empty
  Get the 20 most recent records in the sync queue
  For each record in the current batch
    Convert the record to JSON
    POST the JSON to /sync/v2/queued_records
    Await an acknowledgment record from the server
      Delete the record from the sync queue

• The following describes the algorithm for pulling local records from the central server:

Send a GET request to /sync/v2/queued_records/count
While there are queued records
  Send a GET request to /sync/v2/queued_records for 20 records
  For each record in the current batch
    If the record is valid
      If an existing placeholder record exists in the local database
        Overwrite the placeholder record
      Else
        Add the record to the local database
      Add or update any related records or placeholders in the local database
  POST the batch record IDs to /sync/v2/acknowledged_records

Sync events

Initial sync

• Initial sync is performed during initial setup, immediately after the sync site is authenticated. See authentication for more details on the authentication process.
• The initial sync process consists of the following steps:
  • A request is made to the /sync/v2/initial_dump endpoint, which triggers the server to generate sync records for all data belonging to the site to be synchronised.
  • A follow up request is made to the server for all generated records.
• If an error occurs, the sync can be manually resumed. The initial dump should not be regenerated.
• After initial sync is successfully completed:
  • The initialisation status is updated in local storage.
  • The sync queue is enabled.
  • The database listener is enabled for changes to local data.

Automatic sync

• mSupply Mobile will automatically attempt to synchronise with the server every 10 minutes.
• If any error occurs during the sync process, the process will terminate and be resumed at the next time interval.

Manual sync

• Synchronisation can be manually triggered using the sync icon in the upper-right of interface.

Sync "gotchas"

• mSupply synchronisation is complex. Any changes to sync logic should be done carefully and thoroughly reviewed!
• A few points to be aware of:
  • Sanity checking may not detect all malformed records (currently there is no type checking of field values).
  • mSupply mobile currently only supports record statuses: 'nw, 'sg', 'cn', 'fn', 'wp', and 'wf'. Any additional statuses may cause unexpected behaviour.
  • Centrally created stocktakes may not correctly sync. See issue #371.