[RFC] Automatic chunked messaging

[martonlederer]

How do you envision this feature/change to look/work like?

The new system would automatically split the message data into pieces that wouldn't reach the size limitation of the browser messaging API. Each piece would then be sent as an ArrayBuffer to the recipient (ArrayBuffers can only be sent as number[]) where it would be stored till it receives all chunks of the data. After that the recipient would reassemble it using the split data and extra information submitted with the chunk. After the reconstruction of the message, the chunks need to be deleted.

There would be 3 different types of chunks:

Init chunks

The first chunk to send, it tells the recipient to start collecting chunks for this chunkCollectionId.

interface InitChunk {
	// chunk type
	type: "init";
	// the index of the chunk (helps to reconstruct the data in order)
	// for init chunks this should always be 0
	index: number;
	// an unique ID for the chunks for this specific message
	// this helps to handle multiple messages at once and not
	// confuse the chunks with other messages' chunks
	chunkCollectionId: number;
	// the total amount of chunks that the original message
	// was split to
	totalChunks: number;
	// the length of the data split into chunks
	dataLength: number;
	// the data of the chunks (can be converted back to an ArrayBuffer)
	data: number[];
}

Data chunks

Data chunks are the simplest chunks sent right after the init chunk has been sent.

interface DataChunk {
	type: "data";
	index: number;
	chunkCollectionId: number;
	data: number[];
}

End chunks

The last chunk to send, it tells the recipient to end the collection of the chunks for this chunkCollectionId.

interface EndChunk {
	type: "end";
	// this should be totalChunks - 1
	index: number;
	chunkCollectionId: number;
	data: number[];
}

Chunked messaging flow process

Here is an sketch of how this would work between a content-script and a background service worker.

What is the purpose of this change/feature? Why?

In my testings, the maximum allowed size of messages is 67108862 bytes or ~67 Mbs. This means that if you're trying to send a larger message or bytes of a bigger file, you can't use the browser's messaging API.
The solution to this is chunked messaging, where developers don't need to worry about limits, because if they're handling larger data, the messaging library automatically splits their data into chunks, sends it to the message target and reassembles it on the other side. It would be also nice to allow developers to optionally customize max chunk sizes or otherwise use a fixed limit.

Personally, we deal with large data transmissions in the ArConnect extension, sometimes even gigabytes, so we need a permanent solution to send these datas.

Example implementations

A primitive and basic implementation of this can be found at the ArConnect crypto wallet repo:

• Chunks handler
• Data builder (from chunks)

Code of Conduct

• I agree to follow this project's Code of Conduct
• I checked the current issues for duplicate problems.

[martonlederer]

I've been thinking a lot about making this feature optional or specific to a type of messaging API. I've found that the best option we have is to implement this for long lived connection ports only.
The main reason for this is that the basic message flow is for smaller messages anyway and ports are for bigger data / longer data transfer time. It is also much easier to implement this with ports and I'm afraid the service worker could shut down after some time if we did this with the normal chrome.runtime.sendMessage flow.

[spookyuser]

Running into this problem now, another solution might be - although prob a different RFC - but using Transferables with postMessage like over here https://groups.google.com/a/chromium.org/g/chromium-extensions/c/UyNHEHQKlJA/m/it_eC58DAwAJ