From d2351edf3b2a43e3455a723e4044c48b4c114b35 Mon Sep 17 00:00:00 2001 From: Itamar Haber Date: Wed, 8 Apr 2020 17:14:42 +0300 Subject: [PATCH 01/45] WIP --- docs/commands.md | 487 ++++++++++++++++++++++++++---------------- docs/requirements.txt | 3 + mkdocs.yml | 14 +- 3 files changed, 319 insertions(+), 185 deletions(-) create mode 100644 docs/requirements.txt diff --git a/docs/commands.md b/docs/commands.md index 6b421eb51..ff58e72d4 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -1,305 +1,432 @@ # RedisAI Commands +RedisAI is a Redis module, and as such it implements several data types and the respective commands to use them. -## AI.TENSORSET +All of RedisAI's commands are begin with the `AI.` prefix. The following sections describe these commands. -Set a tensor. +**Syntax Conventions** -Stores a tensor of defined type with shape given by shape1..shapeN. +The following conventions are used for describing the RedisAI Redis API: -```sql -AI.TENSORSET tensor_key data_type shape1 shape2 ... [BLOB data | VALUES val1 val2 ...] +* `COMMAND`: a command or an argument name +* ``: a mandatory argument +* `[optional]`: an optional argument +* `"`: a literal double quote character +* `|`: an exclusive logical or operator +* `...`: more of the same as before + +## AI.TENSORSET +The **`AI.TENSORSET`** command stores a tensor as the value of a key. + +**Redis API** + +``` +AI.TENSORSET + [shape ...] [BLOB | VALUES [val ...]] ``` -* tensor_key - Key for storing the tensor -* data_type - Numeric data type of tensor elements, one of FLOAT, DOUBLE, INT8, INT16, INT32, INT64, UINT8, UINT16 -* shape - Shape of the tensor, that is how many elements for each axis +_Arguments_ -Optional args: +* **key**: the tensor's key name +* **type**: the tensor's data type can be one of: 'FLOAT', 'DOUBLE', 'INT8', 'INT16', 'INT32', 'INT64', 'UINT8' or 'UINT16' +* **shape**: one or more dimensions, or the number of elements per axis, for the tensor +* **BLOB**: indicates that data is in binary format and is provided via the subsequent `data` argument +* **VALUES**: indicates that data is numeric and is provided by one or more subsequent `val` arguments -* BLOB data - provide tensor content as a binary buffer -* VALUES val1 val2 - provide tensor content as individual values +_Return_ -> If no BLOB or VALUES are specified, the tensor is allocated but not initialized to any value. +A simple 'OK' string or an error. -### TENSORSET Example +**Examples** -> Set a 2x2 tensor at `foo` -> 1 2 -> 3 4 +Given the following: $\begin{equation*} A = \begin{bmatrix} 1 & 2 \\ 3 & 4 \\ \end{bmatrix} \end{equation*}$ + +This will set the key 'mytensor' to the 2x2 RedisAI tensor: -```sql -AI.TENSORSET foo FLOAT 2 2 VALUES 1 2 3 4 ``` +redis> AI.TENSORSET mytensor FLOAT 2 2 VALUES 1 2 3 4 +OK +``` + +!!! note "Uninitialized Tensor Values" + As both `BLOB` and `VALUES` are optional arguments, it is possible to use the `AI.TENSORSET` to create an uninitialized tensor. + TBD: why is that useful? -!!! warning "Overhead of `AI.TENSORSET` with the optional arg VALUES" - - It is possible to set a tensor by specifying each individual value (VALUES ... ) or the entire tensor content as a binary buffer (BLOB ...). You should always try to use the `BLOB` option since it removes the overhead of parsing each individual value and does not require serialization/deserialization of the tensor, thus reducing the overall command latency an improving the maximum attainable performance of the model server. ---- +!!! important "Using `BLOB` is preferable to `VALUES`" + While it is possible to set the tensor using binary data or numerical values, it is recommended that you use the `BLOB` option. It requires less resources and performs better compared to specifying the values discretely. ## AI.TENSORGET +The **`AI.TENSORGET`** command returns a tensor stored as key's value. -Get a tensor. +**Redis API** -```sql -AI.TENSORGET tensor_key [BLOB | VALUES | META] +``` +AI.TENSORGET ``` -* tensor_key - Key for the tensor -* BLOB - Return tensor content as a binary buffer -* VALUES - Return tensor content as a list of values -* META - Only return tensor meta data (datat type and shape) +_Arguments_ -### TENSORGET Example +* **key**: the tensor's key name +* **format**: the reply's format can be on of the following: + * **BLOB**: returns the binary representation of the tensor's data + * **VALUES**: returns the numerical representation of the tensor's data + * **META**: returns the tensor's meta data exclusively -Get binary data for tensor at `foo`. Meta data is also returned. +_Return_ + +Array containing the tensor's data. + +The returned array consists of the following elements: + +1. The tensor's data type as a String +1. The tensor's shape as an Array consisting of an item per dimension +1. The tensor's data when called with the `BLOB` or `VALUES` arguments + +The reply's third element type, that is the tensor's data, depends on the given argument: + +* **BLOB**: the tensor's binary data as a String +* **VALUES**: the tensor's values an Array +* **META**: when used no data is returned + +**Examples** + +Given a tensor value stored at the 'mytensor' key: -```sql -AI.TENSORGET foo BLOB +``` +redis> AI.TENSORSET mytensor FLOAT 2 2 VALUES 1 2 3 4 +OK +``` + +The following shows how the tensor's binary data is read: + +``` +redis> AI.TENSORGET mytensor BLOB +1) FLOAT +2) 1) (integer) 2 + 2) (integer) 2 +3) "\x00\x00\x80?\x00\x00\x00@\x00\x00@@\x00\x00\x80@" ``` -!!! warning "Overhead of `AI.TENSORGET` with the optional arg VALUES" - - It is possible to receive a tensor as a list of each individual value (VALUES ... ) or the entire tensor content as a binary buffer (BLOB ...). You should always try to use the `BLOB` option since it removes the overhead of replying each individual value and does not require serialization/deserialization of the tensor, thus reducing the overall command latency an improving the maximum attainable performance of the model server. ---- +!!! important "Using `BLOB` is preferable to `VALUES`" + While it is possible to get the tensor as binary data or numerical values, it is recommended that you use the `BLOB` option. It requires less resources and performs better compared to returning the values discretely. ## AI.MODELSET +The **`AI.MODELSET`** commands stores a model as the value of a key. -Set a model. +**Redis API** -```sql -AI.MODELSET model_key backend device [INPUTS name1 name2 ... OUTPUTS name1 name2 ...] model_blob +``` +AI.MODELSET + [INPUTS ...] [OUTPUTS name ...] ``` -* model_key - Key for storing the model -* backend - The backend corresponding to the model being set. Allowed values: `TF`, `TORCH`, `ONNX`. -* device - Device where the model is loaded and where the computation will run. Allowed values: `CPU`, `GPU`. -* INPUTS name1 name2 ... - Name of the nodes in the provided graph corresponding to inputs [`TF` backend only] -* OUTPUTS name1 name2 ... - Name of the nodes in the provided graph corresponding to outputs [`TF` backend only] -* model_blob - Binary buffer containing the model protobuf saved from a supported backend +_Arguments_ -### MODELSET Example +* **key**: the model's key name +* **backend**: the backend for the model can be one of: + * **TF**: a TensorFlow backend + * **TORCH**: a PyTorch backend + * **ONNX**: a ONNX backend +* **device**: the device that will execute the model can be of: + * **CPU**: a CPU device + * **GPU**: a GPU device +* **INPUTS**: one or more names of the model's input nodes (applicable only for TensorFlow models) +* **OUTPUTS**: one or more names of the model's output nodes (applicable only for TensorFlow models) +* **model**: the Protobuf-serialized model -```sql -AI.MODELSET resnet18 TORCH GPU < foo.pt -``` +_Return_ -```sql -AI.MODELSET resnet18 TF CPU INPUTS in1 OUTPUTS linear4 < foo.pb -``` +A simple 'OK' string or an error. -```sql -AI.MODELSET mnist_net ONNX CPU < mnist.onnx +**Examples** + +This example shows to set a model 'mymodel' key using the contents of a local file with [`redis-cli`](https://redis.io/topics/cli). Refer to the [Clients Page](clients.md) for additional client choice that are native to your programming language: + +``` +$ cat mobilenet_v2_1.4_224_frozen.pb | redis-cli -x AI.MODELSET mymodel TF CPU INPUTS input OUTPUTS MobilenetV2/Predictions/Reshape_1 +OK ``` ## AI.MODELGET +The **`AI.MODELGET`** command returns a model stored as a key's value. -Get a model. +**Redis API** -```sql -AI.MODELGET model_key ``` +AI.MODELGET +``` + +_Arguments + +* **key**: the model's key name -* model_key - Key for the model +_Return_ -The command returns the model as serialized by the backend, that is a string containing a protobuf. +A Bulk String that is the Protobuf-serialized representation of the model by the backend. +**Examples** + +Assuming that your model is stored under the 'mymodel' key, you can use the following read and save it to local file 'model.ext' with [`redis-cli`](https://redis.io/topics/cli): + +``` +$ redis-cli --raw AI.MODELGET mymodel > model.ext +``` ## AI.MODELDEL +The **`AI.MODELDEL`** deletes a model stored as a key's value. -Removes a model at a specified key. +**Redis API** -```sql -AI.MODELDEL model_key +``` +AI.MODELDEL ``` -* model_key - Key for the model +_Arguments_ -Currently, the command is fully equivalent to calling `DEL` on `model_key`. +* **key**: the model's key name +_Return_ -## AI.MODELRUN +A simple 'OK' string or an error. -Run a model. +**Examples** -```sql -AI.MODELRUN model_key INPUTS input_key1 ... OUTPUTS output_key1 ... +Assuming that your model is stored under the 'mymodel' key, you can delete it like this: + +``` +redis> AI.MODELDEL mymodel +OK ``` -* model_key - Key for the model -* INPUTS input_key1 ... - Keys for tensors to use as inputs -* OUTPUTS output_key2 ... - Keys for storing output tensors +!!! note "The `AI.MODELDEL` vis a vis the `DEL` command" + The `AI.MODELDEL` is equivalent to the [Redis `DEL` command](https://redis.io/commands/del) and should be used in its stead. This ensures compatibility with all deployment options (i.e., stand-alone vs. cluster, OSS vs. Enterprise). -The request is queued and evaded asynchronously from a separate thread. The client blocks until the computation finishes. +## AI.MODELRUN +The **`AI.MODELRUN`** command runs a model stored as a key's value using its specified backend and device. It accepts one or more input tensors and store output tensors. -If needed, input tensors are copied to the device specified in `AI.MODELSET` before execution. +The run request is put in a queue and is executed asynchronously by a worker thread. The client that had issued the run request is blocked until the model's run is completed. When needed, tensors' data is automatically copied to the device prior to execution. -### MODELRUN Example +!!! warning "Intermediate memory overhead" + The execution of models will generate intermediate tensors that are not allocated by the Redis allocator, but by whatever allocator is used in the backends (which may act on main memory or GPU memory, depending on the device), thus not being limited by `maxmemory` configuration settings of Redis. + +**Redis API** -```sql -AI.MODELRUN resnet18 INPUTS image12 OUTPUTS label12 ``` +AI.MODELRUN INPUTS [input ...] OUTPUTS [output ...] +``` + +_Arguments_ + +* **key**: the model's key name +* **INPUTS**: denotes the beginning of the input tensors keys' list, followed by one or more key names +* **OUTPUTS**: denotes the beginning of the output tensors keys' list, followed by one or more key names + +_Return_ -!!! warning "Intermediate tensors memory overhead when issuing `AI.MODELRUN` and `AI.SCRIPTRUN`" - - The execution of models will generate intermediate tensors that are not allocated by the Redis allocator, but by whatever allocator is used in the backends (which may act on main memory or GPU memory, depending on the device), thus not being limited by maxmemory settings on Redis. ---- +A simple 'OK' string or an error. +**Examples** +Assuming that running the model that's stored at 'mymodel' with the tensor 'mytensor' as input outputs two tensors - 'classes' and 'predictions', the following command does that: + +``` +redis> AI.MODELRUN mymodel INPUTS mytensor OUTPUTS classes predictions +OK +``` ## AI.SCRIPTSET +The **`AI.SCRIPTSET`** command stores a [TorchScript](https://pytorch.org/docs/stable/jit.html) as the value of a key. -Set a script. +**Redis API** -```sql -AI.SCRIPTSET script_key device script_source ``` +AI.SCRIPTSET "