diff --git a/docs/docs/02-hooks/03-executorch-bindings/useExecutorchModule.md b/docs/docs/02-hooks/03-executorch-bindings/useExecutorchModule.md index 55d5d8e2b8..137b19d92a 100644 --- a/docs/docs/02-hooks/03-executorch-bindings/useExecutorchModule.md +++ b/docs/docs/02-hooks/03-executorch-bindings/useExecutorchModule.md @@ -2,7 +2,7 @@ title: useExecutorchModule --- -ExecuTorch bindings provide streamlined interface to access the [Module API](https://pytorch.org/executorch/stable/extension-module.html) directly from Javascript. +useExecutorchModule provides React Native bindings to the ExecuTorch [Module API](https://pytorch.org/executorch/stable/extension-module.html) directly from JavaScript. :::caution These bindings are primarily intended for custom model integration where no dedicated hook exists. If you are considering using a provided model, first verify whether a dedicated hook is available. Dedicated hooks simplify the implementation process by managing necessary pre and post-processing automatically. Utilizing these can save you effort and reduce complexity, ensuring you do not implement additional handling that is already covered. @@ -32,33 +32,74 @@ For more information on loading resources, take a look at [loading models](../.. ### Returns -| Field | Type | Description | -| :----------------: | :--------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | -| `error` | string | null | Contains the error message if the model failed to load. | -| `isGenerating` | `boolean` | Indicates whether the model is currently processing an inference. | -| `isReady` | `boolean` | Indicates whether the model has successfully loaded and is ready for inference. | -| `loadMethod` | `(methodName: string) => Promise` | Loads resources specific to `methodName` into memory before execution. | -| `loadForward` | `() => Promise` | Loads resources specific to `forward` method into memory before execution. Uses `loadMethod` under the hood. | -| `forward` | `(input: ETInput, shape: number[]) => Promise` | Executes the model's forward pass, where `input` is a Javascript typed array and `shape` is an array of integers representing input Tensor shape. The output is a Tensor - raw result of inference. | -| `downloadProgress` | `number` | Represents the download progress as a value between 0 and 1. | +| Field | Type | Description | +| :----------------: | :--------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------: | +| `error` | string | null | Contains the error message if the model failed to load. | +| `isGenerating` | `boolean` | Indicates whether the model is currently processing an inference. | +| `isReady` | `boolean` | Indicates whether the model has successfully loaded and is ready for inference. | +| `forward` | `(input: TensorPtr[]) => Promise` | Executes the model's forward pass, where `input` is an array of TensorPtr objects. If the inference is successful, an array of tensor pointers is returned. | +| `downloadProgress` | `number` | Represents the download progress as a value between 0 and 1. | -## ETInput +## TensorPtr -The `ETInput` type defines the typed arrays that can be used as inputs in the `forward` method: +TensorPtr is a JS representation of the underlying tensor, which is then passed to the model. You can read more about creating tensors [here](https://docs.pytorch.org/executorch/stable/extension-tensor.html). On JS side, the TensorPtr holds the following information: -- Int8Array -- Int32Array -- BigInt64Array -- Float32Array -- Float64Array +
+Type definitions -## Errors +```typescript +interface TensorPtr { + dataPtr: TensorBuffer; + sizes: number[]; + scalarType: ScalarType; +} -All functions provided by the `useExecutorchModule` hook are asynchronous and may throw an error. The `ETError` enum includes errors [defined by the ExecuTorch team](https://github.com/pytorch/executorch/blob/release/0.7/runtime/core/error.h) and additional errors specified by our library. +type TensorBuffer = + | ArrayBuffer + | Float32Array + | Float64Array + | Int8Array + | Int16Array + | Int32Array + | Uint8Array + | Uint16Array + | Uint32Array + | BigInt64Array + | BigUint64Array; + +enum ScalarType { + BYTE = 0, + CHAR = 1, + SHORT = 2, + INT = 3, + LONG = 4, + HALF = 5, + FLOAT = 6, + DOUBLE = 7, + BOOL = 11, + QINT8 = 12, + QUINT8 = 13, + QINT32 = 14, + QUINT4X2 = 16, + QUINT2X4 = 17, + BITS16 = 22, + FLOAT8E5M2 = 23, + FLOAT8E4M3FN = 24, + FLOAT8E5M2FNUZ = 25, + FLOAT8E4M3FNUZ = 26, + UINT16 = 27, + UINT32 = 28, + UINT64 = 29, +} +``` -## Performing inference +
-To run model with ExecuTorch Bindings it's essential to specify the shape of the input tensor. However, there's no need to explicitly define the input type, as it will automatically be inferred from the array you pass to `forward` method. However you will still need to explicitly provide shape for the tensor. Outputs from the model, such as classification probabilities, are returned in raw format. +`dataPtr` - Represents a data buffer that will be used to create a tensor on the native side. This can be either an `ArrayBuffer` or a `TypedArray`. If your model takes in a datatype which is not covered by any of the `TypedArray` types, just pass an `ArrayBuffer` here. + +`sizes` - Represents a shape of a given tensor, i.e. for a 640x640 RGB image with a batch size of 1, you would need to pass `[1, 3, 640, 640]` here. + +`scalarType` - An enum resembling the ExecuTorch's `ScalarType`. For example, if your model was exported with float32 as an input, you will need to pass `ScalarType.FLOAT` here. ## End to end example @@ -72,6 +113,7 @@ First, import the necessary functions from the `react-native-executorch` package import { useExecutorchModule, STYLE_TRANSFER_CANDY, + ScalarType, } from 'react-native-executorch'; // Initialize the executorch module with the predefined style transfer model. @@ -82,22 +124,26 @@ const executorchModule = useExecutorchModule({ ### Setting up input parameters -To prepare the input for the model, define the shape of the input tensor. This shape depends on the model's requirements. For the `STYLE_TRANSFER_CANDY` model, we need a tensor of shape `[1, 3, 640, 640]`, corresponding to a batch size of 1, 3 color channels (RGB), and dimensions of 640x640 pixels. +To prepare the model input, define the tensor shape according to your model's requirements (defined by the model export process). For example, the STYLE_TRANSFER_CANDY model expects a tensor with shape `[1, 3, 640, 640]` — representing a batch size of 1, 3 color channels (RGB), and 640×640 pixel dimensions. ```typescript -const shape = [1, 3, 640, 640]; -// Create a Float32Array to hold the pixel data of the image, -// which should be preprocessed according to the model's specific needs. -const input = new Float32Array(1 * 3 * 640 * 640); // fill this array with your image data +const inputTensor = { + dataPtr: new Float32Array(1 * 3 * 640 * 640), // or other TypedArray / ArrayBuffer + sizes: [1, 3, 640, 640], + scalarType: ScalarType.FLOAT, +}; ``` ### Performing inference +After passing input to the forward function, you'll receive an array of TensorPtr objects. Each TensorPtr contains its `dataPtr` as an ArrayBuffer. Since ArrayBuffer represents raw binary data, you'll need to interpret it according to the tensor's underlying data type (e.g., creating a Float32Array view for float32 tensors, Int32Array for int32 tensors, etc.). + ```typescript try { // Perform the forward operation and receive the stylized image output. - const output = await executorchModule.forward(input, shape); - console.log('Stylization successful. Output Shape:', output.length); + const output = await executorchModule.forward([inputTensor]); + // Interpret the output ArrayBuffer + // foo(output[0].dataPtr); } catch (error) { // Log any errors that occur during the forward pass. console.error('Error during model execution:', error); diff --git a/docs/docs/03-typescript-api/03-executorch-bindings/ExecutorchModule.md b/docs/docs/03-typescript-api/03-executorch-bindings/ExecutorchModule.md index d7e9f2678a..58b09c9be2 100644 --- a/docs/docs/03-typescript-api/03-executorch-bindings/ExecutorchModule.md +++ b/docs/docs/03-typescript-api/03-executorch-bindings/ExecutorchModule.md @@ -2,7 +2,11 @@ title: ExecutorchModule --- -TypeScript API implementation of the [useExecutorchModule](../../02-hooks/03-executorch-bindings/useExecutorchModule.md) hook. +ExecutorchModule provides TypeScript bindings for the underlying ExecuTorch [Module API](https://pytorch.org/executorch/stable/extension-module.html). + +:::tip +For React applications, consider using the [`useExecutorchModule`](../../02-hooks/03-executorch-bindings/useExecutorchModule.md) hook instead, which provides automatic state management, loading progress tracking, and cleanup on unmount. +::: ## Reference @@ -10,60 +14,150 @@ TypeScript API implementation of the [useExecutorchModule](../../02-hooks/03-exe import { ExecutorchModule, STYLE_TRANSFER_CANDY, + ScalarType, } from 'react-native-executorch'; // Creating the input array -const shape = [1, 3, 640, 640]; -const input = new Float32Array(1 * 3 * 640 * 640); +const inputTensor = { + dataPtr: new Float32Array(1 * 3 * 640 * 640), + sizes: [1, 3, 640, 640], + scalarType: ScalarType.FLOAT, +}; + +// Creating an instance +const model = new ExecutorchModule(); // Loading the model -await ExecutorchModule.load(STYLE_TRANSFER_CANDY); +await model.load(STYLE_TRANSFER_CANDY); -// Running the model -const output = await ExecutorchModule.forward(input, shape); +// Running the forward method +const output = await model.forward([inputTensor]); ``` ### Methods -| Method | Type | Description | -| -------------------- | ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `load` | `(modelSource: ResourceSource): Promise` | Loads the model, where `modelSource` is a string that specifies the location of the model binary. | -| `forward` | `(input: ETInput, shape: number[]): Promise` | Executes the model's forward pass, where `input` is a JavaScript typed array and `shape` is an array of integers representing input Tensor shape. The output is a Tensor - raw result of inference. | -| `loadMethod` | `(methodName: string): Promise` | Loads resources specific to `methodName` into memory before execution. | -| `loadForward` | `(): Promise` | Loads resources specific to `forward` method into memory before execution. Uses `loadMethod` under the hood. | -| `onDownloadProgress` | `(callback: (downloadProgress: number) => void): any` | Subscribe to the download progress event. | +| Method | Type | Description | +| --------------- | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `load` | `(modelSource: ResourceSource, onDownloadProgressCallback?: (progress: number) => void): Promise` | Loads the model, where `modelSource` is a string, number, or object that specifies the location of the model binary. Optionally accepts a download progress callback. | +| `forward` | `(inputTensor: TensorPtr[]): Promise` | Executes the model's forward pass, where input is an array of TensorPtr objects. If the inference is successful, an array of tensor pointers is returned. | +| `getInputShape` | `(methodName: string, index: number): Promise` | Returns the expected input shape for a specific method and input index. | +| `delete` | `(): void` | Unloads the model and releases resources. | + +## TensorPtr + +TensorPtr is a JS representation of the underlying tensor, which is then passed to the model. You can read more about creating tensors [here](https://docs.pytorch.org/executorch/stable/extension-tensor.html). On JS side, the TensorPtr holds the following information:
Type definitions ```typescript -type ResourceSource = string | number | object; - -export type ETInput = +interface TensorPtr { + dataPtr: TensorBuffer; + sizes: number[]; + scalarType: ScalarType; +} + +type TensorBuffer = + | ArrayBuffer + | Float32Array + | Float64Array | Int8Array + | Int16Array | Int32Array + | Uint8Array + | Uint16Array + | Uint32Array | BigInt64Array - | Float32Array - | Float64Array; + | BigUint64Array; + +enum ScalarType { + BYTE = 0, + CHAR = 1, + SHORT = 2, + INT = 3, + LONG = 4, + HALF = 5, + FLOAT = 6, + DOUBLE = 7, + BOOL = 11, + QINT8 = 12, + QUINT8 = 13, + QINT32 = 14, + QUINT4X2 = 16, + QUINT2X4 = 17, + BITS16 = 22, + FLOAT8E5M2 = 23, + FLOAT8E4M3FN = 24, + FLOAT8E5M2FNUZ = 25, + FLOAT8E4M3FNUZ = 26, + UINT16 = 27, + UINT32 = 28, + UINT64 = 29, +} ```
-## Loading the model +`dataPtr` - Represents a data buffer that will be used to create a tensor on the native side. This can be either an `ArrayBuffer` or a `TypedArray`. If your model takes in a datatype which is not covered by any of the `TypedArray` types, just pass an `ArrayBuffer` here. + +`sizes` - Represents the shape of a given tensor, i.e. for a 640x640 RGB image with a batch size of 1, you would need to pass `[1, 3, 640, 640]` here. + +`scalarType` - An enum resembling the ExecuTorch's `ScalarType`. For example, if your model was exported with float32 as an input, you will need to pass `ScalarType.FLOAT` here. + +## End to end example + +This example demonstrates the integration and usage of the ExecuTorch bindings with a [style transfer model](../../02-hooks/02-computer-vision/useStyleTransfer.md). Specifically, we'll be using the `STYLE_TRANSFER_CANDY` model, which applies artistic style transfer to an input image. + +### Importing the Module and loading the model -To load the model, use the `load` method. It accepts the `modelSource` which is a string that specifies the location of the model binary. For more information, take a look at [loading models](../../01-fundamentals/02-loading-models.md) page. This method returns a promise, which can resolve to an error or void. +First, import the necessary functions from the `react-native-executorch` package and initialize the ExecuTorch module with the specified style transfer model. -## Running the model +```typescript +import { + ExecutorchModule, + STYLE_TRANSFER_CANDY, + ScalarType, +} from 'react-native-executorch'; -To run the model use the `forward` method. It accepts two arguments: `input` and `shape`. The `input` is a JavaScript typed array, and `shape` is an array of integers representing the input tensor shape. There's no need to explicitly define the input type, as it will automatically be inferred from the typed array you pass to forward method. Outputs from the model, such as classification probabilities, are returned in raw format. +// Initialize the executorch module +const executorchModule = new ExecutorchModule(); -## Loading methods +// Load the model with optional download progress callback +await executorchModule.load(STYLE_TRANSFER_CANDY, (progress) => { + console.log(`Download progress: ${progress}%`); +}); +``` -Loads resources specific to methodName into memory before execution. +### Setting up input parameters -## Loading forward +To prepare the model input, define the tensor shape according to your model's requirements (defined by the model export process). For example, the STYLE_TRANSFER_CANDY model expects a tensor with shape `[1, 3, 640, 640]` — representing a batch size of 1, 3 color channels (RGB), and 640×640 pixel dimensions. -Loads resources specific to `forward` method into memory before execution. Uses loadMethod under the hood. +```typescript +const inputTensor = { + dataPtr: new Float32Array(1 * 3 * 640 * 640), // or other TypedArray / ArrayBuffer + sizes: [1, 3, 640, 640], + scalarType: ScalarType.FLOAT, +}; +``` + +### Performing inference + +After passing input to the forward function, you'll receive an array of TensorPtr objects. Each TensorPtr contains its `dataPtr` field as an ArrayBuffer. Since ArrayBuffer represents raw binary data, you'll need to interpret it according to the tensor's underlying data type (e.g., creating a Float32Array view for float32 tensors, Int32Array for int32 tensors, etc.). + +```typescript +try { + // Perform the forward operation and receive the stylized image output. + const output = await executorchModule.forward([inputTensor]); + // Interpret the output ArrayBuffer + // foo(output[0].dataPtr); +} catch (error) { + // Log any errors that occur during the forward pass. + console.error('Error during model execution:', error); +} + +// Clean up resources when done +executorchModule.delete(); +``` :::info This code assumes that you have handled preprocessing of the input image (scaling, normalization) and postprocessing of the output (interpreting the raw output data) according to the model's requirements. Make sure to adjust these parts depending on your specific data and model outputs. diff --git a/packages/react-native-executorch/common/rnexecutorch/host_objects/JSTensorViewOut.h b/packages/react-native-executorch/common/rnexecutorch/host_objects/JSTensorViewOut.h index 4ea4fefc1c..5d2f885021 100644 --- a/packages/react-native-executorch/common/rnexecutorch/host_objects/JSTensorViewOut.h +++ b/packages/react-native-executorch/common/rnexecutorch/host_objects/JSTensorViewOut.h @@ -16,7 +16,7 @@ struct JSTensorViewOut { JSTensorViewOut(std::vector sizes, ScalarType scalarType, std::shared_ptr dataPtr) - : sizes(std::move(sizes)), scalarType(scalarType), - dataPtr(std::move(dataPtr)) {} + : dataPtr(std::move(dataPtr)), sizes(std::move(sizes)), + scalarType(scalarType) {} }; } // namespace rnexecutorch diff --git a/packages/react-native-executorch/src/types/common.ts b/packages/react-native-executorch/src/types/common.ts index b4567b2b0e..165e8fe7f5 100644 --- a/packages/react-native-executorch/src/types/common.ts +++ b/packages/react-native-executorch/src/types/common.ts @@ -1,12 +1,5 @@ export type ResourceSource = string | number | object; -export type ETInput = - | Int8Array - | Int32Array - | BigInt64Array - | Float32Array - | Float64Array; - export enum ScalarType { BYTE = 0, CHAR = 1,