Skip to content

Latest commit

 

History

History
611 lines (534 loc) · 25 KB

README.md

File metadata and controls

611 lines (534 loc) · 25 KB
Raw

Samples based on tensorflow-serving-api package

This document contains examples to run GetModelStatus, GetModelMetadata, Predict functions over gRPC API and REST API. Samples are based on tensorflow-serving-api package.

Note : tensorflow-serving-api package is heavy as it includes tensorflow as its dependency. For a lightweight alternative, see ovmsclient package (along with the samples)

It covers following topics:

Requirement

Note: Provided examples and their dependencies are updated and validated for Python 3.7+ version. For older versions of Python, dependencies versions adjustment might be required.

Clone the repository and enter directory:

git clone https://github.com/openvinotoolkit/model_server.git
cd model_server/client/python/tensorflow-serving-api/samples

Install client dependencies:

pip3 install -r requirements.txt

In the examples listed below, OVMS can be started using a command:

wget -N https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.{xml,bin} -P models/resnet50/1
docker run -d -u $(id -u) -v $(pwd)/models:/models -p 8000:8000 -p 9000:9000 openvino/model_server:latest --model_name resnet --model_path /models/resnet50 --port 9000 --rest_port 8000

gRPC API Client Examples

Model Status API

  • Command
python grpc_get_model_status.py --help
usage: grpc_get_model_status.py [-h] [--grpc_address GRPC_ADDRESS]
                           [--grpc_port GRPC_PORT]
                           [--model_name MODEL_NAME]
                           [--model_version MODEL_VERSION]
  • Arguments
Argument Description
-h, --help Show help message and exit
--grpc_address GRPC_ADDRESS Specify url to grpc service. Default: localhost
--grpc_port GRPC_PORT Specify port to grpc service. Default: 9000
--model_name MODEL_NAME Model name to query. Default: resnet
--model_version MODEL_VERSION Model version to query. Lists all versions if not specified
  • Usage Example
python grpc_get_model_status.py --grpc_port 9000 --model_name resnet

Getting model status for model: resnet

Model version: 1
State AVAILABLE
Error code:  0
Error message:  OK

Model Metadata API

  • Command
python grpc_get_model_metadata.py --help
usage: grpc_get_model_metadata.py [-h] [--grpc_address GRPC_ADDRESS]
                           [--grpc_port GRPC_PORT]
                           [--model_name MODEL_NAME]
                           [--model_version MODEL_VERSION]
  • Arguments
Argument Description
-h, --help Show this help message and exit
--grpc_address GRPC_ADDRESS Specify url to grpc service. Default:localhost
--grpc_port GRPC_PORT Specify port to grpc service. Default: 9000
--model_name MODEL_NAME Define model name, must be same as is in service. Default: resnet
--model_version MODEL_VERSION Define model version - must be numerical
  • Usage Example
python grpc_get_model_metadata.py --grpc_port 9000 --model_name resnet --model_version 1

Getting model metadata for model: resnet
Inputs metadata:
	Input name: 0; shape: [1, 3, 224, 224]; dtype: DT_FLOAT
Outputs metadata:
	Output name: 1463; shape: [1, 1000]; dtype: DT_FLOAT

Predict API

Submitting gRPC requests based on a dataset from numpy files:

  • Command
python grpc_predict_resnet.py --help
usage: grpc_predict_resnet.py [-h] --images_numpy_path IMAGES_NUMPY_PATH
                              [--labels_numpy_path LABELS_NUMPY_PATH]
                              [--grpc_address GRPC_ADDRESS]
                              [--grpc_port GRPC_PORT]
                              [--input_name INPUT_NAME]
                              [--output_name OUTPUT_NAME]
                              [--transpose_input {False,True}]
                              [--transpose_method {nchw2nhwc,nhwc2nchw}]
                              [--iterations ITERATIONS]
                              [--batchsize BATCHSIZE]
                              [--model_name MODEL_NAME]
                              [--pipeline_name PIPELINE_NAME] 
                              [--dag-batch-size-auto] [--tls]
                              [--server_cert SERVER_CERT]
                              [--client_cert CLIENT_CERT]
                              [--client_key CLIENT_KEY]
  • Arguments
Argument Description
-h,--help Show help message and exit
--images_numpy_path Numpy in shape [n,w,h,c] or [n,c,h,w]
--labels_numpy_path Numpy in shape [n,1] - can be used to check model accuracy
--grpc_address GRPC_ADDRESS Specify url to grpc service. Default:localhost
--grpc_port GRPC_PORT Specify port to grpc service. Default: 9000
--input_name Specify input tensor name. Default: input
--output_name Specify output name. Default: resnet_v1_50/predictions/Reshape_1
--transpose_input {False,True} Set to False to skip NHWC>NCHW or NCHW>NHWC input transposing. Default: True
--transpose_method {nchw2nhwc,nhwc2nchw} How the input transposition should be executed: nhwc2nchw or nhwc2nchw. Default nhwc2nchw
--iterations Number of requests iterations, as default use number of images in numpy memmap. Default: 0 (consume all frames)
--batchsize Number of images in a single request. Default: 1
--model_name Define model name, must be same as is in service. Default: resnet
--pipeline_name Define pipeline name, must be same as is in service
--dag-batch-size-auto Add demultiplexer dimension at front
--tls enables TLS communication with gRPC endpoint
--server_cert SERVER_CERT Path to the server certificate, used only with TLS communication
--client_cert CLIENT_CERT Path to the client certificate, used only with TLS communication
--client_key CLIENT_KEY Path to the client key, used only with TLS communication
  • Usage example
python grpc_predict_resnet.py --grpc_port 9000 --images_numpy_path ../../imgs.npy --input_name 0 --output_name 1463 --transpose_input False --labels_numpy_path ../../lbs.npy

Image data range: 0.0 : 255.0
Start processing:
        Model name: resnet
        Iterations: 10
        Images numpy path: ../../imgs.npy
        Numpy file shape: (10, 3, 224, 224)

Iteration 1; Processing time: 29.75 ms; speed 33.62 fps
imagenet top results in a single batch:
         0 airliner 404 ; Correct match.
Iteration 2; Processing time: 27.06 ms; speed 36.96 fps
imagenet top results in a single batch:
         0 Arctic fox, white fox, Alopex lagopus 279 ; Correct match.
Iteration 3; Processing time: 26.81 ms; speed 37.30 fps
imagenet top results in a single batch:
         0 bee 309 ; Correct match.
Iteration 4; Processing time: 27.25 ms; speed 36.70 fps
imagenet top results in a single batch:
         0 golden retriever 207 ; Correct match.
Iteration 5; Processing time: 25.64 ms; speed 39.00 fps
imagenet top results in a single batch:
         0 gorilla, Gorilla gorilla 366 ; Correct match.
Iteration 6; Processing time: 34.41 ms; speed 29.06 fps
imagenet top results in a single batch:
         0 magnetic compass 635 ; Correct match.
Iteration 7; Processing time: 23.39 ms; speed 42.75 fps
imagenet top results in a single batch:
         0 peacock 84 ; Correct match.
Iteration 8; Processing time: 23.82 ms; speed 41.98 fps
imagenet top results in a single batch:
         0 pelican 144 ; Correct match.
Iteration 9; Processing time: 25.66 ms; speed 38.97 fps
imagenet top results in a single batch:
         0 snail 113 ; Correct match.
Iteration 10; Processing time: 22.44 ms; speed 44.57 fps
imagenet top results in a single batch:
         0 zebra 340 ; Correct match.

processing time for all iterations
average time: 26.10 ms; average speed: 38.31 fps
median time: 25.50 ms; median speed: 39.22 fps
max time: 34.00 ms; min speed: 29.41 fps
min time: 22.00 ms; max speed: 45.45 fps
time percentile 90: 29.50 ms; speed percentile 90: 33.90 fps
time percentile 50: 25.50 ms; speed percentile 50: 39.22 fps
time standard deviation: 3.33
time variance: 11.09
Classification accuracy: 100.00

Submitting gRPC requests with data in binary format:

Using binary inputs feature requires loading model with layout set to --layout NHWC:NCHW:

wget -N https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.{xml,bin} -P models/resnet50/1
docker run -d -u $(id -u) -v $(pwd)/models:/models -p 8000:8000 -p 9000:9000 openvino/model_server:latest --model_name resnet --model_path /models/resnet50 --port 9000 --rest_port 8000 --layout NHWC:NCHW
python grpc_predict_binary_resnet.py --help
usage: grpc_predict_binary_resnet.py [-h] [--images_list IMAGES_LIST]
                                     [--grpc_address GRPC_ADDRESS]
                                     [--grpc_port GRPC_PORT]
                                     [--input_name INPUT_NAME]
                                     [--output_name OUTPUT_NAME]
                                     [--model_name MODEL_NAME]
                                     [--batchsize BATCHSIZE]
  • Arguments
Argument Description
-h, --help show this help message and exit
--images_list IMAGES_LIST path to a file with a list of labeled images
--grpc_address GRPC_ADDRESS Specify url to grpc service. default:localhost
--grpc_port GRPC_PORT Specify port to grpc service. default: 9000
--input_name INPUT_NAME Specify input tensor name. default: image_bytes
--output_name OUTPUT_NAME Specify output name. default: probabilities
--model_name MODEL_NAME Define model name, must be same as is in service default: resnet
--batchsize BATCHSIZE Number of images in a single request. default: 1
  • Usage example
python grpc_predict_binary_resnet.py --grpc_address localhost --model_name resnet --input_name 0 --output_name 1463 --grpc_port 9000 --images ../../resnet_input_images.txt

Start processing:
        Model name: resnet
        Images list file: ../../resnet_input_images.txt
Batch: 0; Processing time: 31.59 ms; speed 31.65 fps
         1 airliner 404 ; Correct match.
Batch: 1; Processing time: 30.08 ms; speed 33.24 fps
         2 Arctic fox, white fox, Alopex lagopus 279 ; Correct match.
Batch: 2; Processing time: 28.80 ms; speed 34.72 fps
         3 bee 309 ; Correct match.
Batch: 3; Processing time: 29.39 ms; speed 34.03 fps
         4 golden retriever 207 ; Correct match.
Batch: 4; Processing time: 29.91 ms; speed 33.43 fps
         5 gorilla, Gorilla gorilla 366 ; Correct match.
Batch: 5; Processing time: 28.52 ms; speed 35.07 fps
         6 magnetic compass 635 ; Correct match.
Batch: 6; Processing time: 28.69 ms; speed 34.85 fps
         7 peacock 84 ; Correct match.
Batch: 7; Processing time: 26.02 ms; speed 38.43 fps
         8 pelican 144 ; Correct match.
Batch: 8; Processing time: 24.28 ms; speed 41.19 fps
         9 snail 113 ; Correct match.
Batch: 9; Processing time: 29.93 ms; speed 33.41 fps
         10 zebra 340 ; Correct match.
Overall accuracy= 100.0 %
Average latency= 28.2 ms

REST API Client Examples

Access to Google Cloud Storage might require proper configuration of https_proxy in the docker engine or in the docker container. In the examples listed below, OVMS can be started using a command:

wget -N https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.{xml,bin} -P models/resnet50/1
docker run -d -u $(id -u) -v $(pwd)/models:/models -p 8000:8000 -p 9000:9000 openvino/model_server:latest --model_name resnet --model_path /models/resnet50 --port 9000 --rest_port 8000

Model Status API

  • Command
python rest_get_model_status.py --help
usage: rest_get_model_status.py [-h] [--rest_url REST_URL]
                                [--rest_port REST_PORT]
                                [--model_name MODEL_NAME]
                                [--model_version MODEL_VERSION]
                                [--client_cert CLIENT_CERT]
                                [--client_key CLIENT_KEY]
                                [--ignore_server_verification]
                                [--server_cert SERVER_CERT]
  • Arguments
Argument Description
-h, --help Show help message and exit
--rest_url REST_URL Specify url to REST API service. Default:http://localhost
--rest_port REST_PORT Specify port to REST API service. Default: 8000
--model_name MODEL_NAME Model name to query, must be same as is in service. Default : resnet
--model_version MODEL_VERSION Model version to query - must be numerical. List all version if omitted
--client_cert CLIENT_CERT Specify mTLS client certificate file. Default: None.
--client_key CLIENT_KEY Specify mTLS client key file. Default: None.
--ignore_server_verification Skip TLS host verification. Do not use in production. Default: False.
--server_cert SERVER_CERT Path to a custom directory containing trusted CA certificates, server certificate, or a CA_BUNDLE file. Default: None, will use default system CA cert store.
  • Usage Example
python rest_get_model_status.py --rest_port 8000 --model_version 1
{
 "model_version_status": [
  {
   "version": "1",
   "state": "AVAILABLE",
   "status": {
    "error_code": "OK",
    "error_message": "OK"
   }
  }
 ]
}

Model Metadata API

  • Command
python rest_get_model_metadata.py --help
usage: rest_get_model_metadata.py [-h] [--rest_url REST_URL]
                                  [--rest_port REST_PORT]
                                  [--model_name MODEL_NAME]
                                  [--model_version MODEL_VERSION]
                                  [--client_cert CLIENT_CERT]
                                  [--client_key CLIENT_KEY]
                                  [--ignore_server_verification]
                                  [--server_cert SERVER_CERT]
  • Arguments
Argument Description
-h, --help show this help message and exit
--rest_url REST_URL Specify url to REST API service. default: http://localhost
--rest_port REST_PORT Specify port to REST API service. default: 8000
--model_name MODEL_NAME Define model name, must be same as is in service. default: resnet
--model_version MODEL_VERSION Define model version - must be numerical
--client_cert CLIENT_CERT Specify mTLS client certificate file. Default: None.
--client_key CLIENT_KEY Specify mTLS client key file. Default: None.
--ignore_server_verification Skip TLS host verification. Do not use in production. Default: False.
--server_cert SERVER_CERT Path to a custom directory containing trusted CA certificates, server certificate, or a CA_BUNDLE file.
  • Usage Example
python rest_get_model_metadata.py --rest_port 8000
{
 "modelSpec": {
  "name": "resnet",
  "signatureName": "",
  "version": "1"
 },
 "metadata": {
  "signature_def": {
   "@type": "type.googleapis.com/tensorflow.serving.SignatureDefMap",
   "signatureDef": {
    "serving_default": {
     "inputs": {
      "0": {
       "dtype": "DT_FLOAT",
       "tensorShape": {
        "dim": [
         {
          "size": "1",
          "name": ""
         },
         {
          "size": "3",
          "name": ""
         },
         {
          "size": "224",
          "name": ""
         },
         {
          "size": "224",
          "name": ""
         }
        ],
        "unknownRank": false
       },
       "name": "0"
      }
     },
     "outputs": {
      "1463": {
       "dtype": "DT_FLOAT",
       "tensorShape": {
        "dim": [
         {
          "size": "1",
          "name": ""
         },
         {
          "size": "1000",
          "name": ""
         }
        ],
        "unknownRank": false
       },
       "name": "1463"
      }
     },
     "methodName": "",
     "defaults": {}
    }
   }
  }
 }
}

Predict API

  • Command :
python rest_predict_resnet.py --help
usage: rest_predict_resnet.py [-h] --images_numpy_path IMAGES_NUMPY_PATH
                              [--labels_numpy_path LABELS_NUMPY_PATH]
                              [--rest_url REST_URL] [--rest_port REST_PORT]
                              [--input_name INPUT_NAME]
                              [--output_name OUTPUT_NAME]
                              [--transpose_input {False,True}]
                              [--transpose_method {nchw2nhwc,nhwc2nchw}]
                              [--iterations ITERATIONS]
                              [--batchsize BATCHSIZE]
                              [--model_name MODEL_NAME]
                              [--request_format {row_noname,row_name,column_noname,column_name}]
                              [--model_version MODEL_VERSION]
                              [--client_cert CLIENT_CERT]
                              [--client_key CLIENT_KEY]
                              [--ignore_server_verification]
                              [--server_cert SERVER_CERT]
  • Arguments :
Argument Description
-h, --help Show help message and exit
--images_numpy_path IMAGES_NUMPY_PATH Numpy in shape [n,w,h,c] or [n,c,h,w]
--labels_numpy_path LABELS_NUMPY_PATH Numpy in shape [n,1] - can be used to check model accuracy
--rest_url REST_URL Specify url to REST API service. Default: http://localhost
--rest_port REST_PORT Specify port to REST API service. Default: 8000
--input_name INPUT_NAME Specify input tensor name. Default: input
--output_name OUTPUT_NAME Specify output name. Default: resnet_v1_50/predictions/Reshape_1
--transpose_input {False,True} Set to False to skip NHWC>NCHW or NCHW>NHWC input transposing. Default: True
--transpose_method {nchw2nhwc,nhwc2nchw} How the input transposition should be executed: nhwc2nchw or nhwc2nchw
--iterations ITERATIONS Number of requests iterations, as default use number of images in numpy memmap. Default: 0 (consume all frames)
--batchsize BATCHSIZE Number of images in a single request. Default: 1
--model_name MODEL_NAME Define model name, must be same as is in service. Default: resnet
--request_format {row_noname,row_name,column_noname,column_name} Request format according to TF Serving API:row_noname,row_name,column_noname,column_name
--model_version MODEL_VERSION Model version to be used. Default: LATEST
--client_cert CLIENT_CERT Specify mTLS client certificate file. Default: None
--client_key CLIENT_KEY Specify mTLS client key file. Default: None
--ignore_server_verification Skip TLS host verification. Do not use in production. Default: False
--server_cert SERVER_CERT Path to a custom directory containing trusted CA certificates, server certificate, or a CA_BUNDLE file. Default: None, will use default system CA cert store
  • Usage Example
python rest_predict_resnet.py --images_numpy_path ../../imgs.npy --labels_numpy_path ../../lbs.npy --input_name 0 --output_name 1463 --rest_port 8000 --transpose_input False

Image data range: 0 : 255
Start processing:
        Model name: resnet
        Iterations: 10
        Images numpy path: ../../imgs.npy
        Images in shape: (10, 3, 224, 224)

output shape: (1, 1000)
Iteration 1; Processing time: 54.98 ms; speed 18.19 fps
imagenet top results in a single batch:
         0 airliner 404 ; Correct match.
output shape: (1, 1000)
Iteration 2; Processing time: 46.54 ms; speed 21.49 fps
imagenet top results in a single batch:
         0 Arctic fox, white fox, Alopex lagopus 279 ; Correct match.
output shape: (1, 1000)
Iteration 3; Processing time: 50.70 ms; speed 19.73 fps
imagenet top results in a single batch:
         0 bee 309 ; Correct match.
output shape: (1, 1000)
Iteration 4; Processing time: 46.89 ms; speed 21.33 fps
imagenet top results in a single batch:
         0 golden retriever 207 ; Correct match.
output shape: (1, 1000)
Iteration 5; Processing time: 45.78 ms; speed 21.84 fps
imagenet top results in a single batch:
         0 gorilla, Gorilla gorilla 366 ; Correct match.
output shape: (1, 1000)
Iteration 6; Processing time: 48.72 ms; speed 20.53 fps
imagenet top results in a single batch:
         0 magnetic compass 635 ; Correct match.
output shape: (1, 1000)
Iteration 7; Processing time: 45.20 ms; speed 22.12 fps
imagenet top results in a single batch:
         0 peacock 84 ; Correct match.
output shape: (1, 1000)
Iteration 8; Processing time: 45.50 ms; speed 21.98 fps
imagenet top results in a single batch:
         0 pelican 144 ; Correct match.
output shape: (1, 1000)
Iteration 9; Processing time: 45.30 ms; speed 22.08 fps
imagenet top results in a single batch:
         0 snail 113 ; Correct match.
output shape: (1, 1000)
Iteration 10; Processing time: 44.19 ms; speed 22.63 fps
imagenet top results in a single batch:
         0 zebra 340 ; Correct match.

processing time for all iterations
average time: 46.80 ms; average speed: 21.37 fps
median time: 45.50 ms; median speed: 21.98 fps
max time: 54.00 ms; min speed: 18.52 fps
min time: 44.00 ms; max speed: 22.73 fps
time percentile 90: 50.40 ms; speed percentile 90: 19.84 fps
time percentile 50: 45.50 ms; speed percentile 50: 21.98 fps
time standard deviation: 2.93
time variance: 8.56
Classification accuracy: 100.00

Submitting REST requests with data in binary format:

Using binary inputs feature requires loading model with layout set to --layout NHWC:NCHW:

wget -N https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.{xml,bin} -P models/resnet50/1
docker run -d -u $(id -u) -v $(pwd)/models:/models -p 8000:8000 -p 9000:9000 openvino/model_server:latest --model_name resnet --model_path /models/resnet50 --port 9000 --rest_port 8000 --layout NHWC:NCHW
python rest_predict_binary_resnet.py --help
usage: rest_predict_binary_resnet.py [-h] [--images_list IMAGES_LIST]
                                     [--rest_url REST_URL]
                                     [--input_name INPUT_NAME]
                                     [--output_name OUTPUT_NAME]
                                     [--model_name MODEL_NAME]
                                     [--request_format {row_noname,row_name,column_noname,column_name}]
                                     [--batchsize BATCHSIZE]
  • Arguments
Argument Description
-h, --help show this help message and exit
--images_list IMAGES_LIST path to a file with a list of labeled images
--rest_url REST_URL Specify url to REST API service. default: http://localhost:8000
--input_name INPUT_NAME Specify input tensor name. default: image_bytes
--output_name OUTPUT_NAME Specify output name. default: probabilities
--model_name MODEL_NAME Define model name, must be same as is in service default: resnet
--request_format {row_noname,row_name,column_noname,column_name} Request format according to TF Serving API: row_noname,row_name,column_noname,column_name
--batchsize BATCHSIZE Number of images in a single request. default: 1
  • Usage example
python rest_predict_binary_resnet.py --rest_url http://localhost:8000 --model_name resnet --input_name 0 --output_name 1463  --images ../../resnet_input_images.txt

Start processing:
        Model name: resnet
        Images list file: ../../resnet_input_images.txt
Batch: 0; Processing time: 21.47 ms; speed 46.57 fps
output shape: (1, 1000)
         1 airliner 404 ; Correct match.
Batch: 1; Processing time: 22.43 ms; speed 44.58 fps
output shape: (1, 1000)
         2 Arctic fox, white fox, Alopex lagopus 279 ; Correct match.
Batch: 2; Processing time: 24.57 ms; speed 40.71 fps
output shape: (1, 1000)
         3 bee 309 ; Correct match.
Batch: 3; Processing time: 30.92 ms; speed 32.34 fps
output shape: (1, 1000)
         4 golden retriever 207 ; Correct match.
Batch: 4; Processing time: 30.70 ms; speed 32.58 fps
output shape: (1, 1000)
         5 gorilla, Gorilla gorilla 366 ; Correct match.
Batch: 5; Processing time: 30.26 ms; speed 33.05 fps
output shape: (1, 1000)
         6 magnetic compass 635 ; Correct match.
Batch: 6; Processing time: 31.26 ms; speed 31.99 fps
output shape: (1, 1000)
         7 peacock 84 ; Correct match.
Batch: 7; Processing time: 29.17 ms; speed 34.28 fps
output shape: (1, 1000)
         8 pelican 144 ; Correct match.
Batch: 8; Processing time: 25.83 ms; speed 38.71 fps
output shape: (1, 1000)
         9 snail 113 ; Correct match.
Batch: 9; Processing time: 32.50 ms; speed 30.77 fps
output shape: (1, 1000)
         10 zebra 340 ; Correct match.
Overall accuracy= 100.0 %
Average latency= 27.4 ms