Find a way to do less copying when sending large typed data

[alexmarkov]

When sending a large typed data through a message port, there is an optimization to convert it to an external typed data (

sdk/runtime/vm/raw_object_snapshot.cc

Lines 1945 to 1961 in f3c51a4

	if ((kind == Snapshot::kMessage) &&
	(bytes >= kExternalizeTypedDataThreshold)) {
	// Write as external.
	writer->WriteIndexedObject(external_cid);
	writer->WriteTags(writer->GetObjectTags(this));
	writer->Write<RawObject*>(ptr()->length_);
	uint8_t* data = reinterpret_cast<uint8_t*>(ptr()->data());
	void* passed_data = malloc(bytes);
	if (passed_data == NULL) {
	OUT_OF_MEMORY();
	}
	memmove(passed_data, data, bytes);
	static_cast<MessageWriter*>(writer)->finalizable_data()->Put(
	bytes,
	passed_data, // data
	passed_data, // peer,
	IsolateMessageTypedDataFinalizer);

) to avoid copying on the receiver side. This morphs object from one type to another, which causes problems flutter/flutter#22796, #34752.

As a workaround to these problems, the optimization of converting typed data to external will be put under a flag and disabled by default. However, we still need to find a way to do less copying when sending large amount of data through message ports, preferably without morphing object types.

[a-siva]

The original issue that triggered this optimization is here #31959

[hooluupog]

If Dart isolate support transferable Data. #4149

[mraleph]

Note that transferrables on the Web are implemented by externalizing typed arrays behind the scenes. This is the kind of implementation we want to avoid, because it introduces the problem that type of the object changes dynamically: you take a normal typed array and send it to another worker - it suddenly becomes an empty typed array on the sender side.

[alexmarkov]

Maybe we can introduce transferable typed data as separate classes (they could be private in dart:typed_data), created with separate public factory constructors (e.g. Int8List.transferable(int length)). Unlike other typed data classes, transferable could have a mutable length (if needed), or an extra flag indicating that contents was moved to another isolate and should not be accessed anymore. This way it will be possible to avoid both extra copying and dynamic changes of object types.

The disadvantage of this approach is that users will need to change their code and use the new API to create transferable typed data.

/cc @aam

[yjbanov]

you take a normal typed array and send it to another worker - it suddenly becomes an empty typed array on the sender side

Why is this a problem?

[mraleph]

Because right now we can treat type array length as a constant value - and optimize based on that, independent of whether it escape anywhere or not. You are loosing this property if you start changing array length when you send the data over. e.g. right now

class Matrix3D {
  final storage = new Float32Array(9)
}

Matrix3D smth;

smth.storage[1];  // No bounds check needed. Static guarantees of length

[matanlurey]

You could just "0-out" the array, instead of clearing all elements, right?

[yjbanov]

we can treat type array length as a constant value

Ah, nice! Didn't know you did that.

You could just "0-out" the array, instead of clearing all elements, right?

That's is as bad as copying :)

[matanlurey]

FWIW, in AOT mode, it might be reasonable to segfault if you attempt to access a typed array that has been transferred. In JIT mode you could throw an exception? I don't know if that makes sense, but I imagine the number of folks that will be using transferable arrays is rather small, and it is probably OK to give them undefined behavior if you transfer the array and then try to use it.

[yjbanov]

it might be reasonable to segfault if you attempt to access a typed array that has been transferred

That means inserting a check before every access to a typed array. Since typed arrays tend to be used in performance-critical parts of the code, I'd try to avoid adding checks.

[matanlurey]

Oh, I mean don't insert a check, and let the program just crash :)

[yjbanov]

Oh, I mean don't insert a check, and let the program just crash :)

The thing is, it won't crash. Because the memory is not copied/moved anywhere, it's still there. So the crash is not guaranteed to happen predictably.

[matanlurey]

Hmm. Could the pointer to the array be pointed to a null location, or something? 💥

[yjbanov]

Could the pointer to the array be pointed to a null location, or something?

That's a @mraleph question. Your isolate could have multiple pointers to the array. You'd need some indirection for that to work, which, again, adds overhead.

[rmacnak-google]

TransferableTypedData was introduced.