storing files / blobs

[donl]

The server documentation references storing images and processing them.

Is it advisable to store files within AceBase?

If so, any cautionary advice? Are there more efficient methods than just storing files as base64 encoded strings?

[appy-one]

It's perfectly fine to store files in AceBase, you can store them as binary data (ArrayBuffer).

If you are only storing files in the database, you might want to increase the default storage settings: increase the recordSize to 1024 bytes and pageSize to 8192 records to allow AceBase to allocate more data in less records/pages. An 8KB file requires only 8 records with those settings, and 1000 8KB files can be stored in 1 page. The default storage settings (recordSize: 128 bytes, pageSize: 1024 records) are better for storage of JSON objects: an 8KB file would require 64 records, and only 16 would fit in 1 page.

const db = new AceBase('mydb', { storage: { recordSize: 1024, pageSize: 8192 } });
const binary = fs.readFileSync('./image.jpg').buffer;
await db.ref('files').push(binary);

Note that these storage settings can only be defined upon database creation and cannot be changed afterwards.
If your average files are in the MB range, you might want to try a recordSize of up to 32768¹ (2MB requiring 64 records). Always use powers of 2 (2ⁿ) for performance benefits.

Footnotes

1. 32768 is 2¹⁵ and is currently the highest working recordSize and pageSize. The max should be 65536 (2¹⁶) but there currently is an issue with that, will be fixed shortly. ↩

[donl]

Thanks, that is great information.

Here is some of your existing test code for anyone else looking for a starting point.

acebase/spec/data-types.spec.js

Lines 171 to 184 in 2762836

	data = new Uint8Array(5 * 1000 * 1000);
	for(let i = 0 ; i < data.length; i++) {
	data[i] = i % 255;
	}
	await db.ref('datatypes').update({ binary: data });
	snap = await db.ref('datatypes/binary').get();
	expect(snap.val()).toBeInstanceOf(ArrayBuffer);
	// expect(new Uint8Array(snap.val())).toEqual(data); // <-- Disabled because Jasmine is very slow to perform this comparison!
	const stored = new Uint8Array(snap.val());
	let isEqual = true;
	for(let i = 0; i < data.length && isEqual; i++) {
	isEqual = data[i] === stored[i];
	}
	expect(isEqual).toBeTrue();

[donl]

When storing files this way, would you recommend storing metadata fields (name, MIME-type, etc) in a separate database?

If so, I see previously (and maybe again for version 2??) the auth database was separate from the main database. Would it be possible now or in the future to share the auth database between a multiple database setup such as this (as a single node process)?

This also makes me ponder about abusing the separate auth database for the file metadata and ACLs.

[appy-one]

I would recommend storing the metadata in the database with auth enabled, and the binary file data in another as an (authless) storage bucket. You'd have to consider if that's feasible security-wise.

It's currently not possible to share AceBase's auth between server instances, but implementing it is on my wishlist. Like you said, that would allow one to authenticate at an AceBase auth server instance, and use the access token with other AceBaseServer instances configured to use that external auth server.