Sending Records Back

Sending Records Back

People in the field capture records (visits, inspections, readings, receipts), often offline, and the owner files them into the shared file. Only the owner key can change a shared file, so each person sends their records back in a small file of their own, locked with their submission key.

sequenceDiagram
  participant P as Field app (access key only)
  participant S as Filing service (owner key)
  participant F as shared.jzm
  P->>F: open with access key
  F-->>P: own rows + submission key
  P->>P: write records.jzm locked with the submission key
  P->>S: upload records.jzm + access key id
  S->>F: accessKeyOf(id): does the grant still exist?
  S->>S: open the upload with owner.submissionKey(id)
  S->>F: append rows into the sender's own partition

Loading

How the submission key works

• Each access key has one. It's sealed in that key's slot of the shared file, so a person gets it only by opening the shared file with their access key (plus an unlock token for an online grant): reader.submissionKey.
• The owner derives the same key from the owner key and the access key's id: owner.submissionKey(keyId).
• What a submitted file proves: it was made by someone who opened the shared file with that access key. An access key that leaked, without the shared file, isn't enough to send anything.
• Safety: the submission key opens nothing in the shared file, and neither the owner key nor the access key can be worked out from it.

The whole round trip

import { JazminKey, accessKeyOf, append, open, write } from '@smithsoft-studios/jazmin';

// The owner: a shared file of site visits, one partition per field worker
const owner = JazminKey.generate();
const bob = owner.createAccessKey();
const columns = [
  { name: 'id', type: 'string', nullable: false },
  { name: 'worker', type: 'string', nullable: false },
  { name: 'site', type: 'string' },
  { name: 'note', type: 'string' },
];
write('shared.jzm', [
  { id: 'v-1', worker: 'bob', site: 'Depot A', note: 'scheduled' },
  { id: 'v-2', worker: 'ann', site: 'Depot B', note: 'scheduled' },
], { columns, key: owner, access: { partitionBy: 'worker', grants: [{ key: bob, rows: ['bob'], label: 'Bob' }] } });

// Bob's phone: holds only Bob's access key. Opening the shared file gives the submission key.
const shared = open('shared.jzm', { key: bob.export() });
const submissionKey = shared.submissionKey;
shared.close();
const upload = write(null, [
  { id: 'v-1', worker: 'bob', site: 'Depot A', note: 'done, gate repaired' }, // a correction to a filed record
  { id: 'v-3', worker: 'ann', site: 'Depot C', note: 'new visit' },          // claims to be Ann's: filed as Bob's
], { columns, key: submissionKey });

// The filing service: holds the owner key; told the sender's key id with the upload
function fileUpload(bytes, keyId) {
  accessKeyOf('shared.jzm', owner, keyId); // throws if the key has no grant (revoked, expired, unknown)
  const sent = open(bytes, { key: owner.submissionKey(keyId) }); // throws if not made with this sender's key
  const partition = 'bob'; // in a real service: the partition(s) of the sender's grant
  const rows = [...sent.rows()].map((r) => ({ ...r, worker: partition })); // own rows only
  sent.close();
  return append('shared.jzm', { key: owner, upsert: rows, keyColumns: ['id'] });
}
const { inserted, updated } = fileUpload(upload, bob.id);
console.log(`filed: ${inserted} new, ${updated} updated`);

// A file not made with Bob's submission key is refused
try {
  fileUpload(write(null, [{ id: 'v-9', worker: 'bob', site: 'X', note: 'forged' }], { columns, key: JazminKey.generate() }), bob.id);
} catch (e) {
  console.log('forged upload refused:', e.message);
}

// Bob opens the shared file again and sees his filed records
for (const row of open('shared.jzm', { key: bob.export() }).rows()) console.log(row);

filed: 1 new, 1 updated
forged upload refused: Decryption failed - wrong key/password or the data was tampered with
{
  id: 'v-1',
  worker: 'bob',
  site: 'Depot A',
  note: 'done, gate repaired'
}
{ id: 'v-3', worker: 'bob', site: 'Depot C', note: 'new visit' }

In .NET the same keys are reader.SubmissionKey, owner.SubmissionKey(keyId) and JazminFile.AccessKeyOf(path, owner, keyId):

var key = JazminReader.Open("shared.jzm", new JazminReadOptions { AccessKey = bob }).SubmissionKey;   // field app
JazminFile.AccessKeyOf("shared.jzm", owner, keyId);                                                  // filing service
using var sent = JazminReader.Open(upload, new JazminReadOptions { Key = owner.SubmissionKey(keyId) });

Rules for filing

• Own rows only: write each upload's rows into the sender's own partition, whatever the file says, so nobody can file rows as someone else.
• Give every record an id. A record whose id is already filed replaces the filed one (upsert), so corrections made on a phone reach the shared file. A file sent twice changes nothing.
  • Last arrival wins when two people change the same record.
  • Who may change a record: anyone whose grant covers its partition. A change can't move a record to another partition.
  • Send only what changed: in the ready-made service, an empty value (null, missing or '') leaves the field as it is, so a phone can't clear a field.
  • Only what the sender can see: columns the sender's grant doesn't cover are ignored, in new records too. So are columns the shared file doesn't have (reported as ignoredColumns).
• Revoked keys: once a key is revoked, accessKeyOf no longer finds it, and its uploads are refused.
• Keys that expire: file an upload only if it was both written and received before the expiry. The phone's clock (reader.writtenAt) can be set to anything; your server's clock can't. File each upload as it arrives: the first write to the shared file after a key expires removes its grant.
• Older shared files written before submission keys existed have none (submissionKey is null) until the owner's next rewrite or compact().

Files with records

A record can carry photos or PDFs. The phone embeds them in the file it sends and lists each record's files in a json column, attachments:

await JazminBrowser.write([
  { id: 'v-17', note: 'site visit', attachments: ['v-17/receipt.pdf', 'v-17/photo.jpg'] },
], { columns, key: submissionKey, files: [
  { path: 'v-17/receipt.pdf', content: receiptFile },
  { path: 'v-17/photo.jpg', content: photoFile },
] });

The ready-made filing service then checks that every listed file is in the batch and every file in the batch is listed; checks each file's real type from its first bytes (PDF, JPEG, PNG and WebP by default, whatever name the phone gives); limits sizes (10 MB per file, 50 MB per batch by default); stores each file once at attachments/<key id>/<sha256>.<ext>, visible only to keys that see the record's partition; and rewrites the record's attachments to [{ path, name, type, size }], so a reader opens a file with readFile(path).

Filing with the library (1.6.0)

fileSubmission() (.NET JazminFile.FileSubmission) applies every rule above in one call: one read of the shared file and one append. It ships in 1.6.0 of both packages.

// not run: inside your filing service, which gives upload, owner and keyId
import { JazminSubmissionRejectedError, fileSubmission } from '@smithsoft-studios/jazmin';

try {
  const result = fileSubmission('shared.jzm', upload, { key: owner, keyId, receivedAt: Date.now() });
  // { sender, filed, updated, duplicates, files, ignoredColumns }
} catch (error) {
  if (error instanceof JazminSubmissionRejectedError) { /* tell the sender error.message; keep the upload aside */ }
  else throw error; // the shared file busy, a full disk: try again later
}
fileSubmission('shared.jzm', upload, { key: owner, keyId, dryRun: true }); // what would be filed, writing nothing

// fragment
try
{
    var result = JazminFile.FileSubmission("shared.jzm", upload, new JazminFileSubmissionOptions { Key = owner, KeyId = keyId, ReceivedAt = DateTimeOffset.UtcNow });
}
catch (JazminSubmissionRejectedException rejected) { /* tell the sender rejected.Message */ }

Options: idColumn (id), onDuplicate (replace, or skip to keep the filed version), filesColumn (attachments), fileTypes (PDF, JPEG, PNG, WebP), maxFileBytes (10 MB), maxBatchFileBytes (50 MB), dryRun.

A ready-made filing service

js/examples/filing-service in the repository applies all of these rules (from 1.6.0, by calling fileSubmission()):

• fileBatch(sharedPath, ownerKey, { keyId, batch }) files one upload and returns { filed, updated, duplicates, files, ignoredColumns }, or throws RejectedBatch with a reason to show the user.
• inbox.mjs files every batch waiting in a folder (oldest first), moves them to filed/ or rejected/, and compacts the shared file after 50 appends. Run it on a schedule where the owner key is kept.

Speed: filing a batch takes about 20–25 ms whether it holds 1 record or 50, whether the shared file holds 10,000 records or 250,000, and with 1,000 people about as with 100.

Writing these files in a browser, including offline, is covered in Viewer and Browser.