RFC: Java layer EValue serialization

[kirklandsign]

🚀 The feature, motivation and pitch

Context: #6470 (comment)

We need to design a way to serialize an EValue (and its embedding tensor), for some IPC use case in AOSP.

This won’t be the official serialization across ET. ET uses fbs for serialization. This is only for the Java frameworks layer for AOSP.

Basic layout

Tag (1 byte)
Bytes_of_payload (8 bytes?)
<
<
<
<
<
<
<
Payload (var)
…
…

Where payload can be one of them

None (0 byte): No value or absence of a value.
Tensor (var): A multi-dimensional array used for numerical computations.
String/uint8_array (var): A sequence of characters, often used to represent text.
Double (8): A 64-bit floating-point number, used for decimal arithmetic.
Int (4 or 8?): A 32-bit integer, used for whole numbers.
Bool (1): A boolean value, either true or false.
ListBool (var): A list of boolean values.
ListDouble (var): A list of double-precision floating-point numbers.
ListInt (var): A list of integers.
ListTensor (var): A list of tensors.
ListScalar (var): A list of scalar values (e.g., numbers).
ListOptionalTensor (var): A list of optional tensors, where each element may be present or absent.

Per Jacob, we don’t care about List types (6-11). They are internal to ET runtime.

For those without variant length (0, 3, 4, 5), we just serialize the value directly.

For 2 String, let’s assume that it’s uint8_t[]. We just serialize the array directly.

So we will focus on the tensor type.

Tensor type

Scalar_type (1 byte)
Num_dim (1)
Sizes (var)
…
Dim_order (var)
…
Data (var)
…
…
…
…
…

Questions

Do we really need a field for Bytes_of_payload?
Do we need Dim order?
Do we need TensorShapeDynamism?

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

[qiaoli31]

The proposal looks good to me. I saw you mentioned ET uses fbs serialization, we can consider it as an option as well. You could provide conversion java EValue <-> fbs and fbs definition in ET.

[kirklandsign]

The proposal looks good to me. I saw you mentioned ET uses fbs serialization, we can consider it as an option as well. You could provide conversion java EValue <-> fbs and fbs definition in ET.

fbs is only for runtime. It's over complicated for java serialization use case. Contains other runtime details. You can see the schema here

executorch/schema/program.fbs

Lines 71 to 144 in b07be36

	scalar_type:ScalarType;
	// Offset in scalar_type elements (e.g., multiples of 4 bytes for an int
	// scalar type) from the beginning of the tensor buffer to the beginning of
	// the actual data. Currently, the runtime only supports a value of zero.
	storage_offset:int;
	sizes:[int];
	// Specifies in what order the dimensions are laid out in memory (from outer
	// to inner).
	//
	// For example, given a rank 3 Tensor of size (3, 5, 2). If we name
	// dimensions: [row, column, batch], then a dim_order of:
	// - (2, 0, 1) represents a [batch, row, column] ordering where "column" is
	// the innermost dimension, then comes "row", and the outermost dimension is
	// "batch".
	// - (0, 2, 1) represents a [row, batch, column] ordering where "column" is
	// the innermost dimension, then comes "batch", and the outermost dimension
	// is "row".
	dim_order:[ubyte];
	// out of scope M1
	requires_grad:bool;
	// Overall, a Tensor is either constant or mutable. At method load time
	// constant tensors receive a dataptr into the serialized program. Mutable
	// tensors can either receive a pointer from the heirarchical allocator or a
	// nullptr if they will receive a data pointer at execution time (inputs
	// and control flow placeholders can be like this). Mutable tensors may or
	// may not also have an initial value in the serialized program.
	//
	// In summary:
	// data_buffer_idx > 0, allocation_info = Null: Tensor is a constant.
	// data_buffer_idx = 0, allocation_info = Non Null: Tensor is mutable and
	// will receive a dataptr at method load time.
	// data_buffer_idx = 0, allocation_info = Null: Tensor is mutable and
	// will receive a dataptr at input time or during execution.
	// data_buffer_idx > 0, allocation_info = Non Null: Tensor is mutable and
	// will receive a dataptr at method load time, and has an initial state.
	//
	// Tensor data is stored inline if program.constant_buffer is null. Otherwise
	// it is in a segment. If this tensor's allocation_info is null then the
	// tensor data location is specified by program.constant_segment. If the
	// allocation_info is non_null then the data is somewhere in
	// program.mutable_data_segments. If tensor_info is Null, then the data is
	// in program.mutable_data_segments[0] otherwise if tensor_info is non-null
	// then the mutable_data_segment index is specified by
	// tensor_info.mutable_data_segments_index.
	data_buffer_idx:uint;
	// [Optional] preallocation details for non-constants (null otherwise).
	allocation_info:AllocationDetails;
	// May not be needed.
	layout:byte;
	// Determines the type of the tensor's shape, from the point of view of its
	// dynamic or not behavior, and consequently how the allocation of the
	// underlying memory is handled, and also how to interpret the sizes and
	// strides fields.
	// 1. dynamism == STATIC: sizes field represents the static shape of
	// the tensor.
	// 2. dynamism == DYNAMIC_BOUND: sizes field represents the upper bound shape
	// of the tensor. Each dimension of the tensor at runtime should never
	// exceed the corresponding dimension of the upper bound shape.
	//
	// 3. dynamism == DYNAMIC_UNBOUND: the stored sizes field can be ignored since
	// shape is fully dynamic.
	shape_dynamism:TensorShapeDynamism;
	// [Optional] Additional information about the Tensor that is not applicable
	// to most tensors.
	extra_tensor_info:ExtraTensorInfo;

[qiaoli31]

I see. it makes sense.

For training, I propose we reuse this serialized EValue as well. The workflow is as below.

1. Partner implements onTrainingExample to generate a list of TrainingExampleRecord which has byte[] holding training data. In TFLite, this byte[] is serialized tf.example proto. For executorch, byte[] is serialized EValue[].
2. We need a custom dataset/dataloader that initial calls to onTrainingExample to read training data one by one (IPC has size limit). In TFLite, model graph has external_dataset custom op did this. The conversion of byte[] -> tf.example proto -> tf.tensor is written in model graph as well.

I found pytorch can define custom dataset/dataloader and IterableDataset + Dataloader looks similar to what external_dataset done.

1. Is dataset/dataloader available in executorch? Does it have c++ API interface?
2. can dataset write into executorch model graph? or we need write c++ code in ODP to connect it?
3. if step (1) above looks good, we will keep existing byte[] for training API.

I'm quite new to pytorch, feel free to suggest other options.

[qiaoli31]

the comment above is only to start discussion and low priority. Do we have ETA for this serialization format?

[kirklandsign]

Hi @qiaoli31 for serialization, we are working on it. My target is before mid November.

[kirklandsign]

I see. it makes sense.

For training, I propose we reuse this serialized EValue as well. The workflow is as below.

1. Partner implements onTrainingExample to generate a list of TrainingExampleRecord which has byte[] holding training data. In TFLite, this byte[] is serialized tf.example proto. For executorch, byte[] is serialized EValue[].
2. We need a custom dataset/dataloader that initial calls to onTrainingExample to read training data one by one (IPC has size limit). In TFLite, model graph has external_dataset custom op did this. The conversion of byte[] -> tf.example proto -> tf.tensor is written in model graph as well.

I found pytorch can define custom dataset/dataloader and IterableDataset + Dataloader looks similar to what external_dataset done.

1. Is dataset/dataloader available in executorch? Does it have c++ API interface?
2. can dataset write into executorch model graph? or we need write c++ code in ODP to connect it?
3. if step (1) above looks good, we will keep existing byte[] for training API.

I'm quite new to pytorch, feel free to suggest other options.

cc @JacobSzwejbka on this. I'm not familiar with training part