Current date and time : 2023-11-09 10:06:29 ## In directory /home/model-server/serve | Executing command pip install --force-reinstall . Processing /home/model-server/serve Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting Pillow (from torchserve==0.9.0b20231109) Using cached Pillow-10.1.0-cp39-cp39-manylinux_2_28_x86_64.whl.metadata (9.5 kB) Collecting psutil (from torchserve==0.9.0b20231109) Using cached psutil-5.9.6-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB) Collecting packaging (from torchserve==0.9.0b20231109) Using cached packaging-23.2-py3-none-any.whl.metadata (3.2 kB) Collecting wheel (from torchserve==0.9.0b20231109) Using cached wheel-0.41.3-py3-none-any.whl.metadata (2.2 kB) Using cached packaging-23.2-py3-none-any.whl (53 kB) Using cached Pillow-10.1.0-cp39-cp39-manylinux_2_28_x86_64.whl (3.6 MB) Using cached psutil-5.9.6-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB) Using cached wheel-0.41.3-py3-none-any.whl (65 kB) Building wheels for collected packages: torchserve Building wheel for torchserve (setup.py): started Building wheel for torchserve (setup.py): finished with status 'done' Created wheel for torchserve: filename=torchserve-0.9.0b20231109-py3-none-any.whl size=24037463 sha256=a6f623e37a57a1ae94da8cf936f1459b331c0acff6f5402979f0fedff7f38dd8 Stored in directory: /home/model-server/tmp/pip-ephem-wheel-cache-2nu4xvm7/wheels/48/b6/68/1e3c687231d78684724780843d490beab1851a8659471ab609 Successfully built torchserve Installing collected packages: wheel, psutil, Pillow, packaging, torchserve Attempting uninstall: wheel Found existing installation: wheel 0.41.3 Uninstalling wheel-0.41.3: Successfully uninstalled wheel-0.41.3 Attempting uninstall: psutil Found existing installation: psutil 5.9.6 Uninstalling psutil-5.9.6: Successfully uninstalled psutil-5.9.6 Attempting uninstall: Pillow Found existing installation: Pillow 10.1.0 Uninstalling Pillow-10.1.0: Successfully uninstalled Pillow-10.1.0 Attempting uninstall: packaging Found existing installation: packaging 23.2 Uninstalling packaging-23.2: Successfully uninstalled packaging-23.2 Attempting uninstall: torchserve Found existing installation: torchserve 0.9.0b20231109 Uninstalling torchserve-0.9.0b20231109: Successfully uninstalled torchserve-0.9.0b20231109 Successfully installed Pillow-10.1.0 packaging-23.2 psutil-5.9.6 torchserve-0.9.0b20231109 wheel-0.41.3 ## In directory /home/model-server/serve | Executing command pip install --force-reinstall model-archiver/. Processing ./model-archiver Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting enum-compat (from torch-model-archiver==0.9.0b20231109) Using cached enum_compat-0.0.3-py3-none-any.whl (1.3 kB) Building wheels for collected packages: torch-model-archiver Building wheel for torch-model-archiver (setup.py): started Building wheel for torch-model-archiver (setup.py): finished with status 'done' Created wheel for torch-model-archiver: filename=torch_model_archiver-0.9.0b20231109-py3-none-any.whl size=15939 sha256=a8d098647a1b0d7d158f422051e531d1530668149389cad56e56b37c4906feb4 Stored in directory: /root/.cache/pip/wheels/e1/f3/79/282a8d907249f2cfbdc3c6e9faac38b8badcf7ffea2030084b Successfully built torch-model-archiver Installing collected packages: enum-compat, torch-model-archiver Attempting uninstall: enum-compat Found existing installation: enum-compat 0.0.3 Uninstalling enum-compat-0.0.3: Successfully uninstalled enum-compat-0.0.3 Attempting uninstall: torch-model-archiver Found existing installation: torch-model-archiver 0.9.0b20231109 Uninstalling torch-model-archiver-0.9.0b20231109: Successfully uninstalled torch-model-archiver-0.9.0b20231109 Successfully installed enum-compat-0.0.3 torch-model-archiver-0.9.0b20231109 ## In directory /home/model-server/serve | Executing command pip install --force-reinstall workflow-archiver/. Processing ./workflow-archiver Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Building wheels for collected packages: torch-workflow-archiver Building wheel for torch-workflow-archiver (setup.py): started Building wheel for torch-workflow-archiver (setup.py): finished with status 'done' Created wheel for torch-workflow-archiver: filename=torch_workflow_archiver-0.2.11b20231109-py3-none-any.whl size=12737 sha256=4efd8d24da6ddd0be71df7732bac9b90931130c4accc97f559e59d56463a83f6 Stored in directory: /root/.cache/pip/wheels/29/4e/0a/3c3afed4ca0474ba9c8d76af57cd3350933f1e0556fec83324 Successfully built torch-workflow-archiver Installing collected packages: torch-workflow-archiver Attempting uninstall: torch-workflow-archiver Found existing installation: torch-workflow-archiver 0.2.11b20231109 Uninstalling torch-workflow-archiver-0.2.11b20231109: Successfully uninstalled torch-workflow-archiver-0.2.11b20231109 Successfully installed torch-workflow-archiver-0.2.11b20231109 ## Starting generate_mars, mar_config:/home/model-server/serve/ts_scripts/../ts_scripts/mar_config.json, model_store_dir:/home/model-server/serve/ts_scripts/../model_store_gen ## In directory: /home/model-server/serve | Executing command: torch-model-archiver --model-name fastrcnn --version 1.0 --model-file examples/object_detector/fast-rcnn/model.py --serialized-file /home/model-server/serve/ts_scripts/../model_store_gen/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --archive-format zip-store --export-path /home/model-server/serve/ts_scripts/../model_store_gen --force ## fastrcnn.mar is generated. ## In directory: /home/model-server/serve | Executing command: torch-model-archiver --model-name alexnet --version 1.0 --model-file examples/image_classifier/alexnet/model.py --serialized-file /home/model-server/serve/ts_scripts/../model_store_gen/alexnet-owt-7be5be79.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /home/model-server/serve/ts_scripts/../model_store_gen --force ## alexnet.mar is generated. ## In directory: /home/model-server/serve | Executing command: torch-model-archiver --model-name densenet161 --version 1.0 --model-file examples/image_classifier/densenet_161/model.py --serialized-file /home/model-server/serve/ts_scripts/../model_store_gen/densenet161-8d451a50.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /home/model-server/serve/ts_scripts/../model_store_gen --force ## densenet161.mar is generated. ## In directory: /home/model-server/serve | Executing command: torch-model-archiver --model-name mnist --version 1.0 --model-file examples/image_classifier/mnist/mnist.py --serialized-file examples/image_classifier/mnist/mnist_cnn.pt --handler examples/image_classifier/mnist/mnist_handler.py --archive-format zip-store --export-path /home/model-server/serve/ts_scripts/../model_store_gen --force ## mnist.mar is generated. ## In directory: /home/model-server/serve | Executing command: torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file /home/model-server/serve/ts_scripts/../model_store_gen/resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /home/model-server/serve/ts_scripts/../model_store_gen --force ## resnet-152-batch.mar is generated. ## In directory: /home/model-server/serve | Executing command: torch-model-archiver --model-name resnet-18 --version 1.0 --model-file examples/image_classifier/resnet_18/model.py --serialized-file /home/model-server/serve/ts_scripts/../model_store_gen/resnet18-f37072fd.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /home/model-server/serve/ts_scripts/../model_store_gen --force ## resnet-18.mar is generated. ## In directory: /home/model-server/serve | Executing command: torch-model-archiver --model-name squeezenet1_1 --version 1.0 --model-file examples/image_classifier/squeezenet/model.py --serialized-file /home/model-server/serve/ts_scripts/../model_store_gen/squeezenet1_1-b8a52dc0.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /home/model-server/serve/ts_scripts/../model_store_gen --force ## squeezenet1_1.mar is generated. ## In directory: /home/model-server/serve | Executing command: torch-model-archiver --model-name vgg16 --version 1.0 --model-file examples/image_classifier/vgg_16/model.py --serialized-file /home/model-server/serve/ts_scripts/../model_store_gen/vgg16-397923af.pth --handler examples/image_classifier/vgg_16/vgg_handler.py --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /home/model-server/serve/ts_scripts/../model_store_gen --force ## vgg16.mar is generated. ## In directory: /home/model-server/serve | Executing command: torch-model-archiver --model-name deeplabv3_resnet_101_eager --version 1.0 --model-file examples/image_segmenter/deeplabv3/model.py --serialized-file /home/model-server/serve/ts_scripts/../model_store_gen/deeplabv3_resnet101_coco-586e9e4e.pth --handler image_segmenter --extra-files examples/image_segmenter/deeplabv3/deeplabv3.py,examples/image_segmenter/deeplabv3/intermediate_layer_getter.py,examples/image_segmenter/deeplabv3/fcn.py --archive-format zip-store --export-path /home/model-server/serve/ts_scripts/../model_store_gen --force ## deeplabv3_resnet_101_eager.mar is generated. ## In directory: /home/model-server/serve | Executing command: torch-model-archiver --model-name fcn_resnet_101 --version 1.0 --model-file examples/image_segmenter/fcn/model.py --serialized-file /home/model-server/serve/ts_scripts/../model_store_gen/fcn_resnet101_coco-7ecb50ca.pth --handler image_segmenter --extra-files examples/image_segmenter/fcn/fcn.py,examples/image_segmenter/fcn/intermediate_layer_getter.py --archive-format zip-store --export-path /home/model-server/serve/ts_scripts/../model_store_gen --force ## fcn_resnet_101.mar is generated. ## In directory: /home/model-server/serve | Executing command: torch-model-archiver --model-name maskrcnn --version 1.0 --model-file examples/object_detector/maskrcnn/model.py --serialized-file /home/model-server/serve/ts_scripts/../model_store_gen/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --archive-format zip-store --export-path /home/model-server/serve/ts_scripts/../model_store_gen --force ## maskrcnn.mar is generated. ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/model-server/serve/test | Executing command: torchserve --start --model-store=model_store --ncs ## Successfully started TorchServe newman management_api_collection Iteration 1/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 210ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 38ms 5ms 185µs 636µs 195ms 7ms 436µs 248ms ✓ Successful request Iteration 2/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist 200 OK ★ 43ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 583µs (cache) (cache) 38ms 3ms 172µs 45ms ✓ Successful request Iteration 3/82 → management request POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161 200 OK ★ 584ms time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 0 initial workers. Use scale workers API to │ add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 334µs (cache) (cache) 580ms 2ms 50µs 585ms ✓ Successful request Iteration 4/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161 500 Internal Server Error ★ 9ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Model file already exists densenet161.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 276µs (cache) (cache) 5ms 2ms 50µs 10ms ✓ Successful request Iteration 5/82 → management request DELETE http://localhost:8081/models/densenet161 200 OK ★ 45ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 550µs 24µs 167µs 42ms 1ms 54µs 46ms ✓ Successful request Iteration 6/82 → management request POST http://localhost:8081/models 400 Bad Request ★ 4ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Parameter url is required." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 256µs (cache) (cache) 1ms 1ms 30µs 5ms ✓ Successful request Iteration 7/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 17ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 403µs 16µs 140µs 14ms 1ms 30µs 17ms ✓ Successful request Iteration 8/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 29ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 316µs (cache) (cache) 26ms 1ms 32µs 30ms ✓ Successful request Iteration 9/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 7ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 444µs (cache) (cache) 5ms 1ms 84µs 8ms ✓ Successful request Iteration 10/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 28ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 231µs (cache) (cache) 25ms 1ms 30µs 28ms ✓ Successful request Iteration 11/82 → management request PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true 500 Internal Server Error ★ 1431ms time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model mnist │ version: null" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 230µs (cache) (cache) 1428ms 2ms 43µs 1431ms ✓ Successful request Iteration 12/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 18ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 443µs 17µs 130µs 15ms 1ms 30µs 18ms ✓ Successful request Iteration 13/82 → management request GET http://localhost:8081/models/squeezenet1_1/all 200 OK ★ 7ms time ★ 250B↑ 628B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 361B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 231µs (cache) (cache) 4ms 1ms 38µs 7ms ✓ Successful request Iteration 14/82 → management request GET http://localhost:8081/models/squeezenet1_1/1.0 200 OK ★ 4ms time ★ 250B↑ 628B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 361B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 224µs (cache) (cache) 1ms 1ms 27µs 4ms ✓ Successful request Iteration 15/82 → management request GET http://localhost:8081/models/squeezenet1_1 200 OK ★ 3ms time ★ 246B↑ 628B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 361B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 260µs (cache) (cache) 1ms 1ms 26µs 5ms ✓ Successful request Iteration 16/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 217µs (cache) (cache) 3ms 1ms 27µs 5ms ✓ Successful request Iteration 17/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4 400 Bad Request ★ 3ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Invalid RuntimeType value: python4" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 226µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 18/82 → management request GET http://localhost:8081/models?limit=&next_page_token= 200 OK ★ 16ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 426µs 16µs 118µs 13ms 1ms 29µs 17ms ✓ Successful request Iteration 19/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 29ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 232µs (cache) (cache) 26ms 1ms 30µs 29ms ✓ Successful request Iteration 20/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1 202 Accepted ★ 5ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 232µs (cache) (cache) 2ms 1ms 38µs 5ms ✓ Successful request Iteration 21/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true 200 OK ★ 4ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 345µs (cache) (cache) 2ms 1ms 27µs 5ms ✓ Successful request Iteration 22/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true 200 OK ★ 6ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1, version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 4ms 329µs (cache) (cache) 3ms 1ms 40µs 9ms ✓ Successful request Iteration 23/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true 404 Not Found ★ 4ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version: 0.0 does not exist for mo │ del: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 265µs (cache) (cache) 1ms 1ms 29µs 4ms ✓ Successful request Iteration 24/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1 202 Accepted ★ 15ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 389µs 16µs 122µs 12ms 1ms 30µs 15ms ✓ Successful request Iteration 25/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default 200 OK ★ 4ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B │ { │ "status": "Default vesion succsesfully updated for m │ odel \"squeezenet1_1\" to \"1.0\"" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 243µs (cache) (cache) 1ms 1ms 27µs 4ms ✓ Successful request Iteration 26/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default 404 Not Found ★ 3ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version 0.0 does not exist for mod │ el squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 27/82 → management request PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default 404 Not Found ★ 15ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet0_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 401µs 16µs 124µs 12ms 1ms 29µs 15ms ✓ Successful request Iteration 28/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 13ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 398µs 16µs 141µs 10ms 1ms 41µs 13ms ✓ Successful request Iteration 29/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle 200 OK ★ 29ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 241µs (cache) (cache) 26ms 1ms 29µs 29ms ✓ Successful request Iteration 30/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 234µs (cache) (cache) 3ms 1ms 27µs 5ms ✓ Successful request Iteration 31/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0 500 Internal Server Error ★ 1855ms time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model squeez │ enet1_1 version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 229µs (cache) (cache) 1851ms 2ms 45µs 1855ms ✓ Successful request Iteration 32/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0 200 OK ★ 29ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 406µs 16µs 130µs 26ms 1ms 28µs 29ms ✓ Successful request Iteration 33/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 234µs (cache) (cache) 3ms 1ms 27µs 5ms ✓ Successful request Iteration 34/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2 200 OK ★ 1092ms time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 228µs (cache) (cache) 1089ms 1ms 32µs 1092ms ✓ Successful request Iteration 35/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 38ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 258µs (cache) (cache) 36ms 1ms 30µs 38ms ✓ Successful request Iteration 36/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1 200 OK ★ 6.6s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 245µs (cache) (cache) 6.6s 1ms 35µs 6.6s ✓ Successful request Iteration 37/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 67ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 343µs (cache) (cache) 62ms 4ms 32µs 68ms ✓ Successful request Iteration 38/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk 200 OK ★ 5.6s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 251µs (cache) (cache) 5.6s 3ms 96µs 5.6s ✓ Successful request Iteration 39/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 54ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 308µs (cache) (cache) 51ms 1ms 32µs 54ms ✓ Successful request Iteration 40/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1 200 OK ★ 39ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 267µs (cache) (cache) 36ms 1ms 30µs 39ms ✓ Successful request Iteration 41/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 254µs (cache) (cache) 2ms 1ms 26µs 5ms ✓ Successful request Iteration 42/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true 200 OK ★ 216ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 230µs (cache) (cache) 214ms 1ms 38µs 217ms ✓ Successful request Iteration 43/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 11ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 249µs (cache) (cache) 8ms 1ms 29µs 11ms ✓ Successful request Iteration 44/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1 200 OK ★ 217ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 249µs (cache) (cache) 215ms 1ms 32µs 218ms ✓ Successful request Iteration 45/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 11ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 263µs (cache) (cache) 9ms 1ms 28µs 12ms ✓ Successful request Iteration 46/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false 200 OK ★ 218ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 245µs (cache) (cache) 215ms 1ms 33µs 218ms ✓ Successful request Iteration 47/82 → management request GET http://localhost:8081/models?limit=1 200 OK ★ 3ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 260µs (cache) (cache) 1ms 1ms 30µs 3ms ✓ Successful request Iteration 48/82 → management request GET http://localhost:8081/models?limit=-1 200 OK ★ 3ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B │ { │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 231µs (cache) (cache) 1ms 1ms 26µs 3ms ✓ Successful request Iteration 49/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=1 200 OK ★ 3ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 223µs (cache) (cache) 756µs 1ms 25µs 3ms ✓ Successful request Iteration 50/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=-1 200 OK ★ 3ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 998µs 215µs (cache) (cache) 945µs 1ms 24µs 3ms ✓ Successful request Iteration 51/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=10 202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 991µs 223µs (cache) (cache) 1ms 1ms 36µs 3ms ✓ Successful request Iteration 52/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=-1 202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 603µs (cache) (cache) 1ms 1ms 26µs 3ms ✓ Successful request Iteration 53/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 4ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B │ { │ "status": "Workers scaled to 1 for model: resnet-18" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 237µs (cache) (cache) 1ms 1ms 38µs 4ms ✓ Successful request Iteration 54/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 5ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 319µs (cache) (cache) 2ms 1ms 27µs 5ms ✓ Successful request Iteration 55/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=-1 202 Accepted ★ 3ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 232µs (cache) (cache) 1ms 1ms 33µs 3ms ✓ Successful request Iteration 56/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=0 202 Accepted ★ 3ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 246µs (cache) (cache) 1ms 1ms 26µs 3ms ✓ Successful request Iteration 57/82 → management request POST http://localhost:8081/models?url=&model_name=resnet-18 404 Not Found ★ 3ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "empty url" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 244µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 58/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18 400 Bad Request ★ 734ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 872µs 32µs 178µs 730ms 1ms 49µs 735ms ✓ Successful request Iteration 59/82 → management request GET http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 4ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 440µs 17µs 136µs 1ms 1ms 30µs 4ms ✓ Successful request Iteration 60/82 → management request GET http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 4ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 522µs 22µs 160µs 1ms 1ms 28µs 5ms ✓ Successful request Iteration 61/82 → management request GET http://localhost:8081/models?next_page_token=12 200 OK ★ 4ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 387µs 16µs 119µs 1ms 1ms 36µs 4ms ✓ Successful request Iteration 62/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan 202 Accepted ★ 4ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 343µs (cache) (cache) 1ms 1ms 42µs 5ms ✓ Successful request Iteration 63/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan 202 Accepted ★ 4ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 343µs (cache) (cache) 1ms 1ms 37µs 5ms ✓ Successful request Iteration 64/82 → management request PUT http://localhost:8081/models/resnet-18 202 Accepted ★ 3ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 329µs (cache) (cache) 1ms 1ms 26µs 4ms ✓ Successful request Iteration 65/82 → management request PUT http://localhost:8081/models/resnet181?min_worker=1 404 Not Found ★ 6ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: resnet181" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 235µs (cache) (cache) 1ms 4ms 38µs 6ms ✓ Successful request Iteration 66/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1 400 Bad Request ★ 5ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 598µs 26µs 184µs 1ms 1ms 45µs 6ms ✓ Successful request Iteration 67/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1 202 Accepted ★ 4ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 382µs 15µs 116µs 1ms 1ms 34µs 4ms ✓ Successful request Iteration 68/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=0 202 Accepted ★ 10ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 4ms 312µs (cache) (cache) 6ms 1ms 38µs 13ms ✓ Successful request Iteration 69/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=-1 500 Internal Server Error ★ 4ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 500, │ "type": "IndexOutOfBoundsException", │ "message": "Index -1 out of bounds for length 0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 244µs (cache) (cache) 1ms 1ms 30µs 4ms ✓ Successful request Iteration 70/82 → management request PUT http://localhost:8081/models/resnet-18?max_worker=-1 400 Bad Request ★ 3ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 403µs 16µs 117µs 1ms 1ms 29µs 3ms ✓ Successful request Iteration 71/82 → management request PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default 404 Not Found ★ 3ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 384µs 15µs 115µs 1ms 1ms 28µs 4ms ✓ Successful request Iteration 72/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 11ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 975µs 371µs 16µs 112µs 9ms 1ms 26µs 11ms ✓ Successful request Iteration 73/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 3ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 977µs 230µs (cache) (cache) 1ms 1ms 30µs 3ms ✓ Successful request Iteration 74/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 47ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 996µs 380µs 15µs 118µs 44ms 2ms 42µs 47ms ✓ Successful request Iteration 75/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true 200 OK ★ 11ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 357µs (cache) (cache) 4ms 6ms 40µs 12ms ✓ Successful request Iteration 76/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 31ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 254µs (cache) (cache) 29ms 1ms 30µs 32ms ✓ Successful request Iteration 77/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan 200 OK ★ 5ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 238µs (cache) (cache) 2ms 1ms 29µs 5ms ✓ Successful request Iteration 78/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 35ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 235µs (cache) (cache) 32ms 1ms 43µs 36ms ✓ Successful request Iteration 79/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true 200 OK ★ 5ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 310µs (cache) (cache) 3ms 988µs 26µs 6ms ✓ Successful request Iteration 80/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 27ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 227µs (cache) (cache) 25ms 1ms 32µs 27ms ✓ Successful request Iteration 81/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1 200 OK ★ 5ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 237µs (cache) (cache) 2ms 1ms 41µs 5ms ✓ Successful request Iteration 82/82 → management request DELETE http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 3ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 265µs (cache) (cache) 1ms 992µs 26µs 3ms ✓ Successful request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 82 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 21.3s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 8.02kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 241ms [min: 3ms, max: 6.6s, s.d.: 983ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 91µs [min: 15µs, max: 185µs, s.d.: 82µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 238ms [min: 756µs, max: 6.6s, s.d.: 983ms] │ └─────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/model-server/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/model-server/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties ## Successfully started TorchServe newman inference Iteration 1/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_v4.mar&model_name=my_text_classifier&initial_workers=1&synchronous=true 200 OK ★ 9.7s time ★ 388B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "status": "Model \"my_text_classifier\" Version: 1.0 │ registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 54ms 6ms 193µs 9ms 9.6s 7ms 414µs 9.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/my_text_classifier 200 OK ★ 671ms time ★ 353B↑ 373B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 138B │ { │ "World": 0.02911965735256672, │ "Sports": 2.9431601433316246e-05, │ "Business": 0.9074352383613586, │ "Sci/Tec": 0.06341567635536194 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 7ms 1ms 42µs 693µs 666ms 2ms 84µs 678ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/my_text_classifier 200 OK ★ 90ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B │ { │ "status": "Model \"my_text_classifier\" unregistered │ " │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 374µs (cache) (cache) 87ms 1ms 95µs 91ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 26ms time ★ 233B↑ 4.07kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.78kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 2.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 100.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="76489874bf5c",} 639.66 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="my_text_classifier",Level="M │ odel",Hostname="76489874bf5c",} 639.91 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="76489874bf5c",} 131.6 │ 4044570922852 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilization gauge │ GPUMemoryUtilization{Level="Host",DeviceId="0",Hostnam │ e="76489874bf5c",} 0.0 │ # HELP ts_queue_latency_microseconds Torchserve promet │ heus counter metric with unit: Microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{model_name="my_text_clas │ sifier",model_version="default",hostname="76489874bf5c │ ",} 150.511 │ # HELP WorkerLoadTime Torchserve prometheus gauge metr │ ic with unit: Milliseconds │ # TYPE WorkerLoadTime gauge │ WorkerLoadTime{WorkerName="W-9000-my_text_classifier_1 │ .0",Level="Host",Hostname="76489874bf5c",} 5054.0 │ # HELP DiskUtilization Torchserve prometheus gauge met │ ric with unit: Perc │ (showing 2.05kB/3.78kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 504µs 16µs 139µs 23ms 1ms 43µs 27ms ✓ Successful GET request Iteration 2/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_scripted_v3.mar&model_name=my_text_classifier_scripted&initial_workers=1&synchronous=true 200 OK ★ 7.8s time ★ 406B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 103B │ { │ "status": "Model \"my_text_classifier_scripted\" Ver │ sion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 288µs (cache) (cache) 7.8s 3ms 69µs 7.8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/my_text_classifier_scripted 200 OK ★ 700ms time ★ 362B↑ 372B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 137B │ { │ "World": 0.04559721797704697, │ "Sports": 0.0003771769697777927, │ "Business": 0.08623101562261581, │ "Sci/Tec": 0.8677946329116821 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 455µs (cache) (cache) 697ms 1ms 73µs 702ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/my_text_classifier_scripted 200 OK ★ 76ms time ★ 263B↑ 335B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 69B │ { │ "status": "Model \"my_text_classifier_scripted\" unr │ egistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 361µs (cache) (cache) 73ms 1ms 75µs 77ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 6ms time ★ 233B↑ 4.77kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.48kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 2.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 100.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel="Model",Hostname="76489874bf5c",} 693.54 │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="76489874bf5c",} 639.66 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="my_text_classifier_scripted" │ ,Level="Model",Hostname="76489874bf5c",} 693.74 │ PredictionTime{ModelName="my_text_classifier",Level="M │ odel",Hostname="76489874bf5c",} 639.91 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="76489874bf5c",} 131.6 │ 4044570922852 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilization gauge │ GPUMemoryUtilization{Level="Host",DeviceId="0",Hostnam │ e="76489874bf5c",} 0.0 │ # HELP ts_queue_latency_microseconds Torchserve promet │ heus counter metric with unit: Microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{model_name="my_text_clas │ sifier_scripted │ (showing 2.05kB/4.48kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 312µs (cache) (cache) 3ms 1ms 73µs 7ms ✓ Successful GET request Iteration 3/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true 200 OK ★ 3.2s time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 347µs (cache) (cache) 3.2s 1ms 89µs 3.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/squeezenet1_1 200 OK ★ 1163ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.2785071134567261, │ "lynx": 0.25299131870269775, │ "tiger_cat": 0.24496451020240784, │ "Egyptian_cat": 0.2172253131866455, │ "cougar": 0.0022175421472638845 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 330µs (cache) (cache) 1159ms 2ms 60µs 1164ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 9ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 272µs (cache) (cache) 6ms 1ms 38µs 9ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 6ms time ★ 233B↑ 5.39kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.1kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 1.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 100.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel="Model",Hostname="76489874bf5c",} 693.54 │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="76489874bf5c",} 639.66 │ HandlerTime{ModelName="squeezenet1_1",Level="Model",Ho │ stname="76489874bf5c",} 1152.68 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="my_text_classifier_scripted" │ ,Level="Model",Hostname="76489874bf5c",} 693.74 │ PredictionTime{ModelName="my_text_classifier",Level="M │ odel",Hostname="76489874bf5c",} 639.91 │ PredictionTime{ModelName="squeezenet1_1",Level="Model" │ ,Hostname="76489874bf5c",} 1152.88 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="76489874bf5c",} 131.6 │ 4044570922852 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilizati │ (showing 2.05kB/5.1kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 259µs (cache) (cache) 3ms 1ms 52µs 8ms ✓ Successful GET request Iteration 4/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/squeezenet1_1_scripted.mar&model_name=squeezenet1_1_scripted&initial_workers=1&synchronous=true 200 OK ★ 3.3s time ★ 393B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 98B │ { │ "status": "Model \"squeezenet1_1_scripted\" Version: │ 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 279µs (cache) (cache) 3.3s 1ms 40µs 3.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/squeezenet1_1_scripted 200 OK ★ 1213ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.2785071134567261, │ "lynx": 0.25299131870269775, │ "tiger_cat": 0.24496451020240784, │ "Egyptian_cat": 0.2172253131866455, │ "cougar": 0.0022175421472638845 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 306µs (cache) (cache) 1211ms 1ms 43µs 1214ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/squeezenet1_1_scripted 200 OK ★ 11ms time ★ 258B↑ 330B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 64B │ { │ "status": "Model \"squeezenet1_1_scripted\" unregist │ ered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 266µs (cache) (cache) 8ms 1ms 36µs 11ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 6ms time ★ 233B↑ 6.06kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.77kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 2.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 100.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="squeezenet1_1_scripted",Level=" │ Model",Hostname="76489874bf5c",} 1205.37 │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel="Model",Hostname="76489874bf5c",} 693.54 │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="76489874bf5c",} 639.66 │ HandlerTime{ModelName="squeezenet1_1",Level="Model",Ho │ stname="76489874bf5c",} 1152.68 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="squeezenet1_1_scripted",Leve │ l="Model",Hostname="76489874bf5c",} 1205.58 │ PredictionTime{ModelName="my_text_classifier_scripted" │ ,Level="Model",Hostname="76489874bf5c",} 693.74 │ PredictionTime{ModelName="my_text_classifier",Level="M │ odel",Hostname="76489874bf5c",} 639.91 │ PredictionTime{ModelName="squeezene │ (showing 2.05kB/5.77kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 256µs (cache) (cache) 3ms 1ms 53µs 7ms ✓ Successful GET request Iteration 5/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161&initial_workers=1&synchronous=true 200 OK ★ 4.3s time ★ 330B↑ 353B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 87B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 349µs (cache) (cache) 4.3s 1ms 42µs 4.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/densenet161 200 OK ★ 1219ms time ★ 111.24kB↑ 415B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 180B │ { │ "tabby": 0.4666191339492798, │ "tiger_cat": 0.46449020504951477, │ "Egyptian_cat": 0.06614047288894653, │ "lynx": 0.001292437082156539, │ "plastic_bag": 0.0002290968259330839 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 295µs (cache) (cache) 1216ms 1ms 43µs 1220ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/densenet161 200 OK ★ 30ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 277µs (cache) (cache) 27ms 1ms 38µs 30ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 6.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.38kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 1.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 100.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="squeezenet1_1_scripted",Level=" │ Model",Hostname="76489874bf5c",} 1205.37 │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel="Model",Hostname="76489874bf5c",} 693.54 │ HandlerTime{ModelName="densenet161",Level="Model",Host │ name="76489874bf5c",} 1211.44 │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="76489874bf5c",} 639.66 │ HandlerTime{ModelName="squeezenet1_1",Level="Model",Ho │ stname="76489874bf5c",} 1152.68 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="squeezenet1_1_scripted",Leve │ l="Model",Hostname="76489874bf5c",} 1205.58 │ PredictionTime{ModelName= │ (showing 2.05kB/6.38kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 263µs (cache) (cache) 2ms 2ms 72µs 6ms ✓ Successful GET request Iteration 6/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=alexnet.mar&model_name=alexnet&initial_workers=1&synchronous=true 200 OK ★ 4.9s time ★ 322B↑ 349B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 83B │ { │ "status": "Model \"alexnet\" Version: 1.0 registered │ with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 308µs (cache) (cache) 4.9s 2ms 54µs 4.9s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/alexnet 200 OK ★ 1092ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.31847354769706726, │ "tiger_cat": 0.25793972611427307, │ "Egyptian_cat": 0.24254822731018066, │ "lynx": 0.16879358887672424, │ "tiger": 0.006487949751317501 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 394µs (cache) (cache) 1089ms 1ms 44µs 1095ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/alexnet 200 OK ★ 52ms time ★ 243B↑ 315B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 49B │ { │ "status": "Model \"alexnet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 278µs (cache) (cache) 50ms 1ms 43µs 53ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 7.26kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.96kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 0.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 100.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="alexnet",Level="Model",Hostname │ ="76489874bf5c",} 1083.76 │ HandlerTime{ModelName="squeezenet1_1_scripted",Level=" │ Model",Hostname="76489874bf5c",} 1205.37 │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel="Model",Hostname="76489874bf5c",} 693.54 │ HandlerTime{ModelName="densenet161",Level="Model",Host │ name="76489874bf5c",} 1211.44 │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="76489874bf5c",} 639.66 │ HandlerTime{ModelName="squeezenet1_1",Level="Model",Ho │ stname="76489874bf5c",} 1152.68 │ # HELP PredictionTime Tor │ (showing 2.05kB/6.96kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 297µs (cache) (cache) 2ms 2ms 50µs 5ms ✓ Successful GET request Iteration 7/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/alexnet_scripted.mar&model_name=alexnet_scripted&initial_workers=1&synchronous=true 200 OK ★ 7.6s time ★ 381B↑ 358B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 92B │ { │ "status": "Model \"alexnet_scripted\" Version: 1.0 r │ egistered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 274µs (cache) (cache) 7.6s 1ms 41µs 7.6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/alexnet_scripted 200 OK ★ 1147ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.31847354769706726, │ "tiger_cat": 0.25793972611427307, │ "Egyptian_cat": 0.24254822731018066, │ "lynx": 0.16879358887672424, │ "tiger": 0.006487949751317501 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 316µs (cache) (cache) 1144ms 1ms 42µs 1148ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/alexnet_scripted 200 OK ★ 115ms time ★ 252B↑ 324B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58B │ { │ "status": "Model \"alexnet_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 281µs (cache) (cache) 112ms 1ms 42µs 115ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 7.89kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 7.6kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="76489874bf │ 5c",} 1140204.441 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 2.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 100.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="alexnet",Level="Model",Hostname │ ="76489874bf5c",} 1083.76 │ HandlerTime{ModelName="squeezenet1_1_scripted",Level=" │ Model",Hostname="76489874bf5c",} 1205.37 │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel="Model",Hostname="76489874bf5c",} 693.54 │ HandlerTime{ModelName="densenet161",Level="Model",Host │ name="76489874bf5c",} 1211.44 │ HandlerTime{ModelName="my_text_classifier",Level="Mode │ l",Hostname="76489874 │ (showing 2.05kB/7.6kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 277µs (cache) (cache) 2ms 1ms 47µs 5ms ✓ Successful GET request Iteration 8/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&initial_workers=1&synchronous=true 200 OK ★ 3.5s time ★ 326B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 277µs (cache) (cache) 3.5s 1ms 43µs 3.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-18 200 OK ★ 1072ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.4096633791923523, │ "tiger_cat": 0.3467046618461609, │ "Egyptian_cat": 0.13002873957157135, │ "lynx": 0.023919539526104927, │ "bucket": 0.011532176285982132 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 292µs (cache) (cache) 1070ms 1ms 42µs 1073ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-18 200 OK ★ 20ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 266µs (cache) (cache) 17ms 1ms 37µs 20ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 7ms time ★ 233B↑ 8.49kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.2kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="76489874bf │ 5c",} 1140204.441 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 1.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 100.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="alexnet",Level="Model",Hostname │ ="76489874bf5c",} 1083.76 │ HandlerTime{ModelName="squeezenet1_1_scripted",Level=" │ Model",Hostname="76489874bf5c",} 1205.37 │ HandlerTime{ModelName="resnet-18",Level="Model",Hostna │ me="76489874bf5c",} 1064.56 │ HandlerTime{ModelName="my_text_classifier_scripted",Le │ vel │ (showing 2.05kB/8.2kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 338µs (cache) (cache) 2ms 3ms 43µs 7ms ✓ Successful GET request Iteration 9/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-18_scripted.mar&model_name=resnet-18_scripted&initial_workers=1&synchronous=true 200 OK ★ 4s time ★ 385B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "status": "Model \"resnet-18_scripted\" Version: 1.0 │ registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 400µs (cache) (cache) 4s 1ms 41µs 4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-18_scripted 200 OK ★ 1269ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.4096633791923523, │ "tiger_cat": 0.3467046618461609, │ "Egyptian_cat": 0.13002873957157135, │ "lynx": 0.023919539526104927, │ "bucket": 0.011532176285982132 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 268µs (cache) (cache) 1267ms 1ms 42µs 1270ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-18_scripted 200 OK ★ 25ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B │ { │ "status": "Model \"resnet-18_scripted\" unregistered │ " │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 272µs (cache) (cache) 22ms 2ms 48µs 26ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 11ms time ★ 233B↑ 9.15kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.86kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="76489874bf │ 5c",} 1140204.441 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 2.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="alexnet",Level="Model",Hostname │ ="76489874bf5c",} 1083.76 │ HandlerTime{ModelName="squeezenet1_1_scripted",Level=" │ Model",Hostname="76489874bf5c",} 1205.37 │ HandlerTime{Mo │ (showing 2.05kB/8.86kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 343µs (cache) (cache) 2ms 7ms 50µs 12ms ✓ Successful GET request Iteration 10/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=vgg16.mar&model_name=vgg16&initial_workers=1&synchronous=true 200 OK ★ 6.7s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"vgg16\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 334µs (cache) (cache) 6.7s 1ms 41µs 6.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/vgg16 200 OK ★ 1086ms time ★ 111.24kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tiger_cat": 0.44697248935699463, │ "tabby": 0.4408800005912781, │ "Egyptian_cat": 0.059045590460300446, │ "tiger": 0.020596392452716827, │ "lynx": 0.009934596717357635 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 315µs (cache) (cache) 1083ms 1ms 42µs 1087ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/vgg16 200 OK ★ 136ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"vgg16\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 287µs (cache) (cache) 133ms 2ms 56µs 137ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 6ms time ★ 233B↑ 9.72kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.43kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="76489874bf5c",} 10805 │ 55.301 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="76489874bf │ 5c",} 1140204.441 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 1.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="alexnet",Level="Model",Hostname │ ="76489874bf5c",} 10 │ (showing 2.05kB/9.43kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 330µs (cache) (cache) 3ms 1ms 56µs 7ms ✓ Successful GET request Iteration 11/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/vgg16_scripted.mar&model_name=vgg16_scripted&initial_workers=1&synchronous=true 200 OK ★ 12.8s time ★ 377B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"vgg16_scripted\" Version: 1.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 274µs (cache) (cache) 12.8s 1ms 41µs 12.8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/vgg16_scripted 200 OK ★ 1150ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tiger_cat": 0.44697248935699463, │ "tabby": 0.4408800005912781, │ "Egyptian_cat": 0.059045590460300446, │ "tiger": 0.020596392452716827, │ "lynx": 0.009934596717357635 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 552µs (cache) (cache) 1146ms 2ms 61µs 1151ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/vgg16_scripted 200 OK ★ 171ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"vgg16_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 280µs (cache) (cache) 169ms 1ms 41µs 172ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 10.35kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 10.06kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="76489874bf5c",} 10805 │ 55.301 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="76489874bf │ 5c",} 1140204.441 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 0.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus ga │ (showing 2.05kB/10.06kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 270µs (cache) (cache) 2ms 1ms 40µs 5ms ✓ Successful GET request Iteration 12/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_v2.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.2s time ★ 362B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 2.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 260µs (cache) (cache) 3.2s 1ms 40µs 3.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/mnist 200 OK ★ 1069ms time ★ 537B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B │ 0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 279µs (cache) (cache) 1066ms 1ms 39µs 1069ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 10ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 262µs (cache) (cache) 7ms 1ms 43µs 10ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 10.92kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 10.63kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10649 │ 72.799 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="76489874bf5c",} 10805 │ 55.301 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="76489874bf │ 5c",} 1140204.441 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 0.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TY │ (showing 2.05kB/10.63kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 318µs (cache) (cache) 2ms 1ms 38µs 5ms ✓ Successful GET request Iteration 13/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_scripted_v2.mar&model_name=mnist_scripted&initial_workers=1&synchronous=true 200 OK ★ 3.1s time ★ 380B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"mnist_scripted\" Version: 2.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 362µs (cache) (cache) 3.1s 2ms 40µs 3.1s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/mnist_scripted 200 OK ★ 1134ms time ★ 546B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B │ 0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 286µs (cache) (cache) 1131ms 1ms 41µs 1135ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist_scripted 200 OK ★ 10ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"mnist_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 273µs (cache) (cache) 8ms 1ms 35µs 10ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 11.55kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 11.25kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1130346.166 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10649 │ 72.799 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="76489874bf5c",} 10805 │ 55.301 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="76489874bf │ 5c",} 1140204.441 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 1.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization │ (showing 2.05kB/11.25kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 255µs (cache) (cache) 2ms 1ms 48µs 5ms ✓ Successful GET request Iteration 14/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=fastrcnn.mar&model_name=fastrcnn&initial_workers=1&synchronous=true 200 OK ★ 5s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B │ { │ "status": "Model \"fastrcnn\" Version: 1.0 registere │ d with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 275µs (cache) (cache) 5s 1ms 40µs 5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fastrcnn 200 OK ★ 1185ms time ★ 289.21kB↑ 2.99kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.76kB │ [ │ { │ "person": [ │ 167.39584350585938, │ 57.20302200317383, │ 301.3600158691406, │ 436.7975158691406 │ ], │ "score": 0.999519944190979 │ }, │ { │ "person": [ │ 89.54701232910156, │ 64.83567810058594, │ 191.42428588867188, │ 446.7607727050781 │ ], │ "score": 0.9994966983795166 │ }, │ { │ "person": [ │ 362.3782653808594, │ 161.84133911132812, │ 515.5023193359375, │ 385.2898254394531 │ ], │ "score": 0.997706413269043 │ }, │ { │ "handbag": [ │ 67.3762435913086, │ 277.5755615234375, │ 111.67390441894531, │ 400.206787109375 │ ], │ "score": 0.992499053478241 │ }, │ { │ "handbag": [ │ 228.68252563476562, │ 146.0069580078125, │ 303.55120849609375, │ 231.0885009765625 │ ], │ "score": 0.9922404289245605 │ }, │ { │ "handbag": [ │ 379.411376953125, │ 259.957763671875, │ 419.0797424316406, │ 317.9610290527344 │ ], │ "score": 0.9898613691329956 │ }, │ { │ "person": [ │ 518.4950561523438, │ 149.73016357421875, │ 636.6343383789062, │ 365.4129638671875 │ ], │ "score": 0.9821107983589172 │ }, │ { │ "bench": [ │ 269.08184814453125, │ 217.342041015625, │ 423.77099609375, │ 390.3786315917969 │ ], │ "score": 0.9573412537574768 │ }, │ { │ "person": [ │ 539.6298217773438, │ 157.75868225097656, │ 616.1533813476562, │ 253.1112823486328 │ ], │ "score": 0.8995409607887268 │ }, │ { │ "person": [ │ 477.06231689453125, │ 147.80885314941406, │ 610.777587890625, │ 296.83734130859375 │ ], │ "score": 0.8751543164253235 │ }, │ { │ "bench": [ │ 286.08746337890625, │ 216.53285217285156, │ 550.7698364257812, │ 383.1822509765625 │ ], │ "score": 0.8436443209648132 │ }, │ { │ "person": [ │ 627.4741821289062, │ 177.05838012695312, │ 640.0, │ 247.72152709960938 │ ], │ "score": 0.8257318139076233 │ }, │ { │ "bench": [ │ 88.7638168334961, │ 226.1649627685547, │ 563.6144409179688, │ │ (showing 2.05kB/2.76kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 288µs (cache) (cache) 1182ms 1ms 42µs 1186ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fastrcnn 200 OK ★ 50ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B │ { │ "status": "Model \"fastrcnn\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 274µs (cache) (cache) 47ms 1ms 43µs 50ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 12.14kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 11.84kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1130346.166 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10649 │ 72.799 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 76775.992 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="76489874bf5c",} 10805 │ 55.301 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="76489874bf │ 5c",} 1140204.441 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 0.0 │ (showing 2.05kB/11.84kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 257µs (cache) (cache) 2ms 1ms 44µs 5ms ✓ Successful GET request Iteration 15/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=maskrcnn.mar&model_name=maskrcnn&initial_workers=1&synchronous=true 200 OK ★ 4.5s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B │ { │ "status": "Model \"maskrcnn\" Version: 1.0 registere │ d with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 258µs (cache) (cache) 4.5s 1ms 41µs 4.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/maskrcnn 200 OK ★ 1195ms time ★ 289.21kB↑ 2.99kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.75kB │ [ │ { │ "person": [ │ 169.50636291503906, │ 49.98552322387695, │ 300.8945617675781, │ 442.4545593261719 │ ], │ "score": 0.999543309211731 │ }, │ { │ "person": [ │ 90.4118423461914, │ 66.79048919677734, │ 194.19305419921875, │ 437.2776794433594 │ ], │ "score": 0.9993956089019775 │ }, │ { │ "person": [ │ 362.3172912597656, │ 158.173583984375, │ 521.2589721679688, │ 385.7226867675781 │ ], │ "score": 0.9952280521392822 │ }, │ { │ "handbag": [ │ 68.57540893554688, │ 279.31817626953125, │ 111.15328979492188, │ 400.9166259765625 │ ], │ "score": 0.9938817024230957 │ }, │ { │ "person": [ │ 474.0157165527344, │ 147.3479461669922, │ 638.1209716796875, │ 364.6508483886719 │ ], │ "score": 0.9897466897964478 │ }, │ { │ "handbag": [ │ 225.59584045410156, │ 142.90003967285156, │ 302.48638916015625, │ 230.32838439941406 │ ], │ "score": 0.9891216158866882 │ }, │ { │ "handbag": [ │ 380.2604675292969, │ 259.2012023925781, │ 419.5366516113281, │ 318.27728271484375 │ ], │ "score": 0.9688038229942322 │ }, │ { │ "bench": [ │ 273.48565673828125, │ 217.4883575439453, │ 441.06536865234375, │ 396.2416687011719 │ ], │ "score": 0.9617544412612915 │ }, │ { │ "person": [ │ 541.2896728515625, │ 156.6611785888672, │ 619.9386596679688, │ 249.45326232910156 │ ], │ "score": 0.8177021145820618 │ }, │ { │ "person": [ │ 362.9620361328125, │ 163.89923095703125, │ 500.7698059082031, │ 293.91229248046875 │ ], │ "score": 0.8016975522041321 │ }, │ { │ "chair": [ │ 455.20849609375, │ 207.54010009765625, │ 491.08526611328125, │ 274.6475830078125 │ ], │ "score": 0.7758331298828125 │ }, │ { │ "person": [ │ 549.1538696289062, │ 177.42056274414062, │ 640.0, │ 364.5394592285156 │ ], │ "score": 0.7176421880722046 │ }, │ { │ "person": [ │ 626.230712890625, │ 178.6534423828125, │ 640.0, │ 2 │ (showing 2.05kB/2.75kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 259µs (cache) (cache) 1193ms 1ms 42µs 1196ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/maskrcnn 200 OK ★ 52ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B │ { │ "status": "Model \"maskrcnn\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 270µs (cache) (cache) 49ms 1ms 40µs 52ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 12.73kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 12.43kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1130346.166 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10649 │ 72.799 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 76775.992 │ ts_inference_latency_microseconds{model_name="maskrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 88788.655 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="76489874bf5c",} 10805 │ 55.301 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="76489874bf │ 5c",} 1140204.441 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ t │ (showing 2.05kB/12.43kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 257µs (cache) (cache) 2ms 1ms 52µs 5ms ✓ Successful GET request Iteration 16/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=fcn_resnet_101.mar&model_name=fcn_resnet_101&initial_workers=1&synchronous=true 200 OK ★ 5.7s time ★ 336B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"fcn_resnet_101\" Version: 1.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 340µs (cache) (cache) 5.7s 1ms 42µs 5.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fcn_resnet_101 200 OK ★ 1270ms time ★ 289.22kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993864297866821 │ ], │ [ │ 0.0, │ 0.9993854761123657 │ ], │ [ │ 0.0, │ 0.9993811845779419 │ ], │ [ │ 0.0, │ 0.9993742108345032 │ ], │ [ │ 0.0, │ 0.9993641972541809 │ ], │ [ │ 0.0, │ 0.9993515610694885 │ ], │ [ │ 0.0, │ 0.9993364214897156 │ ], │ [ │ 0.0, │ 0.9993187189102173 │ ], │ [ │ 0.0, │ 0.9992934465408325 │ ], │ [ │ 0.0, │ 0.9992607235908508 │ ], │ [ │ 0.0, │ 0.9992249011993408 │ ], │ [ │ 0.0, │ 0.9991866946220398 │ ], │ [ │ 0.0, │ 0.9991452693939209 │ ], │ [ │ 0.0, │ 0.9991005659103394 │ ], │ [ │ 0.0, │ 0.9990523457527161 │ ], │ [ │ 0.0, │ 0.9990004897117615 │ ], │ [ │ 0.0, │ 0.998968243598938 │ ], │ [ │ 0.0, │ 0.9989573955535889 │ ], │ [ │ 0.0, │ 0.9989456534385681 │ ], │ [ │ 0.0, │ 0.9989331364631653 │ ], │ [ │ 0.0, │ 0.9989200830459595 │ ], │ [ │ 0.0, │ 0.9989060163497925 │ ], │ [ │ 0.0, │ 0.998891294002533 │ ], │ [ │ 0.0, │ 0.9988754391670227 │ ], │ [ │ 0.0, │ 0.9988815188407898 │ ], │ [ │ 0.0, │ 0.9989089965820312 │ ], │ [ │ 0.0, │ 0.9989352822303772 │ ], │ [ │ 0.0, │ 0.9989607334136963 │ ], │ [ │ 0.0, │ 0.9989848732948303 │ ], │ [ │ 0.0, │ 0.9990076422691345 │ ], │ [ │ 0.0, │ 0.9990293979644775 │ ], │ [ │ 0.0, │ 0.999049723148346 │ ], │ [ │ 0.0, │ 0.999051034450531 │ ], │ [ │ 0.0, │ 0.9990313053131104 │ ], │ [ │ 0.0, │ 0.9990096092224121 │ ], │ [ │ 0.0, │ 0.9989851117134094 │ ], │ [ │ 0.0, │ 0.9989587068557739 │ ], │ [ │ 0.0, │ 0.998929798603 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 305µs (cache) (cache) 1265ms 3ms 505µs 1271ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fcn_resnet_101 200 OK ★ 51ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"fcn_resnet_101\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 420µs (cache) (cache) 47ms 2ms 54µs 52ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 13.35kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.06kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1130346.166 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10649 │ 72.799 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="76489874bf5c │ ",} 1260521.179 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 76775.992 │ ts_inference_latency_microseconds{model_name="maskrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 88788.655 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="76489874bf5c",} 10805 │ 55.301 │ ts_inference_latency_microseconds{model_name="alexnet_ │ scripted",model_version="default",hostname="76489874bf │ 5c",} 1140204.441 │ ts_inference_latency_microseconds{model_name="squeezen │ (showing 2.05kB/13.06kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 320µs (cache) (cache) 2ms 1ms 52µs 6ms ✓ Successful GET request Iteration 17/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/fcn_resnet_101_scripted.mar&model_name=fcn_resnet_101_scripted&initial_workers=1&synchronous=true 200 OK ★ 7.2s time ★ 395B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 99B │ { │ "status": "Model \"fcn_resnet_101_scripted\" Version │ : 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 445µs (cache) (cache) 7.2s 1ms 41µs 7.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fcn_resnet_101_scripted 200 OK ★ 1697ms time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993864297866821 │ ], │ [ │ 0.0, │ 0.9993854761123657 │ ], │ [ │ 0.0, │ 0.9993811845779419 │ ], │ [ │ 0.0, │ 0.9993742108345032 │ ], │ [ │ 0.0, │ 0.9993641972541809 │ ], │ [ │ 0.0, │ 0.9993515610694885 │ ], │ [ │ 0.0, │ 0.9993364214897156 │ ], │ [ │ 0.0, │ 0.9993187189102173 │ ], │ [ │ 0.0, │ 0.9992934465408325 │ ], │ [ │ 0.0, │ 0.9992607235908508 │ ], │ [ │ 0.0, │ 0.9992249011993408 │ ], │ [ │ 0.0, │ 0.9991866946220398 │ ], │ [ │ 0.0, │ 0.9991452693939209 │ ], │ [ │ 0.0, │ 0.9991005659103394 │ ], │ [ │ 0.0, │ 0.9990523457527161 │ ], │ [ │ 0.0, │ 0.9990004897117615 │ ], │ [ │ 0.0, │ 0.998968243598938 │ ], │ [ │ 0.0, │ 0.9989573955535889 │ ], │ [ │ 0.0, │ 0.9989456534385681 │ ], │ [ │ 0.0, │ 0.9989331364631653 │ ], │ [ │ 0.0, │ 0.9989200830459595 │ ], │ [ │ 0.0, │ 0.9989060163497925 │ ], │ [ │ 0.0, │ 0.998891294002533 │ ], │ [ │ 0.0, │ 0.9988754391670227 │ ], │ [ │ 0.0, │ 0.9988815188407898 │ ], │ [ │ 0.0, │ 0.9989089965820312 │ ], │ [ │ 0.0, │ 0.9989352822303772 │ ], │ [ │ 0.0, │ 0.9989607334136963 │ ], │ [ │ 0.0, │ 0.9989848732948303 │ ], │ [ │ 0.0, │ 0.9990076422691345 │ ], │ [ │ 0.0, │ 0.9990293979644775 │ ], │ [ │ 0.0, │ 0.999049723148346 │ ], │ [ │ 0.0, │ 0.999051034450531 │ ], │ [ │ 0.0, │ 0.9990313053131104 │ ], │ [ │ 0.0, │ 0.9990096092224121 │ ], │ [ │ 0.0, │ 0.9989851117134094 │ ], │ [ │ 0.0, │ 0.9989587068557739 │ ], │ [ │ 0.0, │ 0.998929798603 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 419µs (cache) (cache) 1693ms 2ms 503µs 1699ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fcn_resnet_101_scripted 200 OK ★ 86ms time ★ 259B↑ 331B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 65B │ { │ "status": "Model \"fcn_resnet_101_scripted\" unregis │ tered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 370µs (cache) (cache) 82ms 2ms 54µs 87ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 14.04kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.74kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="764 │ 89874bf5c",} 1688748.453 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1130346.166 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10649 │ 72.799 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="76489874bf5c │ ",} 1260521.179 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 76775.992 │ ts_inference_latency_microseconds{model_name="maskrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 88788.655 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="76489874bf5c",} 10805 │ 55.301 │ ts_inference_latency_microseconds{model_name="a │ (showing 2.05kB/13.74kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 374µs (cache) (cache) 2ms 1ms 43µs 5ms ✓ Successful GET request Iteration 18/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=deeplabv3_resnet_101_eager.mar&model_name=deeplabv3_resnet_101_eager&initial_workers=1&synchronous=true 200 OK ★ 5.2s time ★ 360B↑ 369B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "status": "Model \"deeplabv3_resnet_101_eager\" Vers │ ion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 293µs (cache) (cache) 5.2s 3ms 60µs 5.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/deeplabv3_resnet_101_eager 200 OK ★ 1273ms time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988669157028198 │ ], │ [ │ 0.0, │ 0.998843789100647 │ ], │ [ │ 0.0, │ 0.9988172650337219 │ ], │ [ │ 0.0, │ 0.9987861514091492 │ ], │ [ │ 0.0, │ 0.9987504482269287 │ ], │ [ │ 0.0, │ 0.9987117052078247 │ ], │ [ │ 0.0, │ 0.9986679553985596 │ ], │ [ │ 0.0, │ 0.9986202716827393 │ ], │ [ │ 0.0, │ 0.9985180497169495 │ ], │ [ │ 0.0, │ 0.9983478784561157 │ ], │ [ │ 0.0, │ 0.9981531500816345 │ ], │ [ │ 0.0, │ 0.9979324340820312 │ ], │ [ │ 0.0, │ 0.9976813793182373 │ ], │ [ │ 0.0, │ 0.9973964691162109 │ ], │ [ │ 0.0, │ 0.9970728158950806 │ ], │ [ │ 0.0, │ 0.9967058300971985 │ ], │ [ │ 0.0, │ 0.9964045286178589 │ ], │ [ │ 0.0, │ 0.9961955547332764 │ ], │ [ │ 0.0, │ 0.9959736466407776 │ ], │ [ │ 0.0, │ 0.9957382678985596 │ ], │ [ │ 0.0, │ 0.9954885840415955 │ ], │ [ │ 0.0, │ 0.9952237010002136 │ ], │ [ │ 0.0, │ 0.9949430823326111 │ ], │ [ │ 0.0, │ 0.9946452379226685 │ ], │ [ │ 0.0, │ 0.9945149421691895 │ ], │ [ │ 0.0, │ 0.9945646524429321 │ ], │ [ │ 0.0, │ 0.9946145415306091 │ ], │ [ │ 0.0, │ 0.9946632385253906 │ ], │ [ │ 0.0, │ 0.9947116374969482 │ ], │ [ │ 0.0, │ 0.9947587847709656 │ ], │ [ │ 0.0, │ 0.9948055148124695 │ ], │ [ │ 0.0, │ 0.9948518872261047 │ ], │ [ │ 0.0, │ 0.9948964715003967 │ ], │ [ │ 0.0, │ 0.9949400424957275 │ ], │ [ │ 0.0, │ 0.9949837923049927 │ ], │ [ │ 0.0, │ 0.9950267672538757 │ ], │ [ │ 0.0, │ 0.9950693845748901 │ ], │ [ │ 0.0, │ 0.995111525 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 363µs (cache) (cache) 1269ms 3ms 493µs 1275ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/deeplabv3_resnet_101_eager 200 OK ★ 55ms time ★ 262B↑ 334B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 68B │ { │ "status": "Model \"deeplabv3_resnet_101_eager\" unre │ gistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 479µs (cache) (cache) 52ms 1ms 59µs 57ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 14.73kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 14.44kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_eager",model_version="default",hostname=" │ 76489874bf5c",} 1264835.762 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="764 │ 89874bf5c",} 1688748.453 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1130346.166 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10649 │ 72.799 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="76489874bf5c │ ",} 1260521.179 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 76775.992 │ ts_inference_latency_microseconds{model_name="maskrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 88788.655 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_micro │ (showing 2.05kB/14.44kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 296µs (cache) (cache) 1ms 1ms 41µs 4ms ✓ Successful GET request Iteration 19/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.s3.amazonaws.com/mar_files/deeplabv3_resnet_101_scripted.mar&model_name=deeplabv3_resnet_101_scripted&initial_workers=1&synchronous=true 200 OK ★ 16.7s time ★ 412B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B │ { │ "status": "Model \"deeplabv3_resnet_101_scripted\" V │ ersion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 252µs (cache) (cache) 16.7s 1ms 742µs 16.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/deeplabv3_resnet_101_scripted 200 OK ★ 1515ms time ★ 289.24kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988669157028198 │ ], │ [ │ 0.0, │ 0.998843789100647 │ ], │ [ │ 0.0, │ 0.9988172650337219 │ ], │ [ │ 0.0, │ 0.9987861514091492 │ ], │ [ │ 0.0, │ 0.9987504482269287 │ ], │ [ │ 0.0, │ 0.9987117052078247 │ ], │ [ │ 0.0, │ 0.9986679553985596 │ ], │ [ │ 0.0, │ 0.9986202716827393 │ ], │ [ │ 0.0, │ 0.9985180497169495 │ ], │ [ │ 0.0, │ 0.9983478784561157 │ ], │ [ │ 0.0, │ 0.9981531500816345 │ ], │ [ │ 0.0, │ 0.9979324340820312 │ ], │ [ │ 0.0, │ 0.9976813793182373 │ ], │ [ │ 0.0, │ 0.9973964691162109 │ ], │ [ │ 0.0, │ 0.9970728158950806 │ ], │ [ │ 0.0, │ 0.9967058300971985 │ ], │ [ │ 0.0, │ 0.9964045286178589 │ ], │ [ │ 0.0, │ 0.9961955547332764 │ ], │ [ │ 0.0, │ 0.9959736466407776 │ ], │ [ │ 0.0, │ 0.9957382678985596 │ ], │ [ │ 0.0, │ 0.9954885840415955 │ ], │ [ │ 0.0, │ 0.9952237010002136 │ ], │ [ │ 0.0, │ 0.9949430823326111 │ ], │ [ │ 0.0, │ 0.9946452379226685 │ ], │ [ │ 0.0, │ 0.9945149421691895 │ ], │ [ │ 0.0, │ 0.9945646524429321 │ ], │ [ │ 0.0, │ 0.9946145415306091 │ ], │ [ │ 0.0, │ 0.9946632385253906 │ ], │ [ │ 0.0, │ 0.9947116374969482 │ ], │ [ │ 0.0, │ 0.9947587847709656 │ ], │ [ │ 0.0, │ 0.9948055148124695 │ ], │ [ │ 0.0, │ 0.9948518872261047 │ ], │ [ │ 0.0, │ 0.9948964715003967 │ ], │ [ │ 0.0, │ 0.9949400424957275 │ ], │ [ │ 0.0, │ 0.9949837923049927 │ ], │ [ │ 0.0, │ 0.9950267672538757 │ ], │ [ │ 0.0, │ 0.9950693845748901 │ ], │ [ │ 0.0, │ 0.995111525 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 307µs (cache) (cache) 1510ms 3ms 538µs 1516ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/deeplabv3_resnet_101_scripted 200 OK ★ 98ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B │ { │ "status": "Model \"deeplabv3_resnet_101_scripted\" u │ nregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 383µs (cache) (cache) 94ms 1ms 58µs 99ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 15.45kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 15.16kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_eager",model_version="default",hostname=" │ 76489874bf5c",} 1264835.762 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="764 │ 89874bf5c",} 1688748.453 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1130346.166 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10649 │ 72.799 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="76489874bf5c │ ",} 1260521.179 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 76775.992 │ ts_inference_latency_microseconds{model_name="maskrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 88788.655 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_micro │ (showing 2.05kB/15.16kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 271µs (cache) (cache) 2ms 1ms 57µs 5ms ✓ Successful GET request Iteration 20/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&initial_workers=1&synchronous=true 200 OK ★ 5.3s time ★ 333B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 257µs (cache) (cache) 5.3s 1ms 41µs 5.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet152 200 OK ★ 1057ms time ★ 111.24kB↑ 412B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 177B │ { │ "tiger_cat": 0.5798618793487549, │ "tabby": 0.38344117999076843, │ "Egyptian_cat": 0.034211479127407074, │ "lynx": 0.0005819834186695516, │ "quilt": 0.00027332050376571715 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 302µs (cache) (cache) 1055ms 1ms 41µs 1058ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet152 200 OK ★ 67ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 265µs (cache) (cache) 64ms 1ms 52µs 67ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 6ms time ★ 233B↑ 16.05kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 15.75kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_eager",model_version="default",hostname=" │ 76489874bf5c",} 1264835.762 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="764 │ 89874bf5c",} 1688748.453 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1130346.166 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10649 │ 72.799 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="76489874bf5c │ ",} 1260521.179 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 76775.992 │ ts_inference_latency_microseconds{model_name="maskrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 88788.655 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier_scripted",model_version="default",hostname= │ "76489874bf5c",} 695738.085 │ ts_inference_latency_micro │ (showing 2.05kB/15.75kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 316µs (cache) (cache) 3ms 1ms 49µs 7ms ✓ Successful GET request Iteration 21/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-152-batch_scripted.mar&model_name=resnet-152-batch_scripted&initial_workers=1&synchronous=true 200 OK ★ 7.6s time ★ 399B↑ 368B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "status": "Model \"resnet-152-batch_scripted\" Versi │ on: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 341µs (cache) (cache) 7.6s 1ms 40µs 7.6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-152-batch_scripted 200 OK ★ 1502ms time ★ 111.26kB↑ 412B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 177B │ { │ "tiger_cat": 0.5798618793487549, │ "tabby": 0.38344117999076843, │ "Egyptian_cat": 0.034211479127407074, │ "lynx": 0.0005819834186695516, │ "quilt": 0.00027332050376571715 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 290µs (cache) (cache) 1499ms 1ms 40µs 1503ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-152-batch_scripted 200 OK ★ 90ms time ★ 261B↑ 333B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 67B │ { │ "status": "Model \"resnet-152-batch_scripted\" unreg │ istered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 258µs (cache) (cache) 88ms 1ms 40µs 91ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 16.74kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 16.44kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_eager",model_version="default",hostname=" │ 76489874bf5c",} 1264835.762 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="764 │ 89874bf5c",} 1688748.453 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1130346.166 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10649 │ 72.799 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="76489874bf5c │ ",} 1260521.179 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 76775.992 │ ts_inference_latency_microseconds{model_name="maskrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 88788.655 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 52-batch_scripted",model_version="default",hostname="7 │ 6489874bf5c",} 1497895.872 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_micros │ (showing 2.05kB/16.44kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 258µs (cache) (cache) 2ms 1ms 40µs 4ms ✓ Successful GET request Iteration 22/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/distill_bert_qa_eager.mar&model_name=distill_bert_qa_eager&initial_workers=1&synchronous=true 200 OK ★ 10.5s time ★ 391B↑ 363B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "status": "Model \"distill_bert_qa_eager\" Version: │ 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 261µs (cache) (cache) 10.5s 2ms 50µs 10.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/distill_bert_qa_eager 200 OK ★ 795ms time ★ 359B↑ 247B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 79B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13B │ a nice puppet └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 379µs (cache) (cache) 792ms 1ms 40µs 797ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/distill_bert_qa_eager 200 OK ★ 104ms time ★ 257B↑ 329B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Model \"distill_bert_qa_eager\" unregiste │ red" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 324µs (cache) (cache) 101ms 1ms 40µs 105ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 17.4kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 17.11kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_eager",model_version="default",hostname=" │ 76489874bf5c",} 1264835.762 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="764 │ 89874bf5c",} 1688748.453 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1130346.166 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10649 │ 72.799 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="76489874bf5c │ ",} 1260521.179 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 76775.992 │ ts_inference_latency_microseconds{model_name="maskrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 88788.655 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 52-batch_scripted",model_version="default",hostname="7 │ 6489874bf5c",} 1497895.872 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_micros │ (showing 2.05kB/17.11kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 364µs (cache) (cache) 2ms 1ms 41µs 5ms ✓ Successful GET request Iteration 23/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_token_classification_no_torchscript.mar&model_name=bert_token_classification_no_torchscript&initial_workers=1&synchronous=true 200 OK ★ 13.3s time ★ 429B↑ 383B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 116B │ { │ "status": "Model \"bert_token_classification_no_torc │ hscript\" Version: 1.0 registered with 1 initial worke │ rs" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 269µs (cache) (cache) 13.3s 2ms 48µs 13.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/bert_token_classification_no_torchscript 200 OK ★ 807ms time ★ 375B↑ 5.45kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.22kB │ [ │ [ │ "[CLS]", │ "B-LOC" │ ], │ [ │ "bloomberg", │ "O" │ ], │ [ │ "has", │ "I-ORG" │ ], │ [ │ "decided", │ "B-PER" │ ], │ [ │ "to", │ "O" │ ], │ [ │ "publish", │ "I-PER" │ ], │ [ │ "a", │ "I-MISC" │ ], │ [ │ "new", │ "B-PER" │ ], │ [ │ "report", │ "O" │ ], │ [ │ "on", │ "O" │ ], │ [ │ "global", │ "B-PER" │ ], │ [ │ "economic", │ "O" │ ], │ [ │ "situation", │ "B-LOC" │ ], │ [ │ ".", │ "B-PER" │ ], │ [ │ "[SEP]", │ "B-PER" │ ], │ [ │ "[PAD]", │ "I-ORG" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ (showing 2.05kB/5.22kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 377µs (cache) (cache) 804ms 1ms 50µs 808ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/bert_token_classification_no_torchscript 200 OK ★ 124ms time ★ 276B↑ 348B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 82B │ { │ "status": "Model \"bert_token_classification_no_torc │ hscript\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 316µs (cache) (cache) 121ms 1ms 41µs 125ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 18.16kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 17.87kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_eager",model_version="default",hostname=" │ 76489874bf5c",} 1264835.762 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="764 │ 89874bf5c",} 1688748.453 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8_scripted",model_version="default",hostname="76489874 │ bf5c",} 1263796.89 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 8",model_version="default",hostname="76489874bf5c",} 1 │ 066497.515 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1130346.166 │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10649 │ 72.799 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="76489874bf5c │ ",} 1260521.179 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 76775.992 │ ts_inference_latency_microseconds{model_name="maskrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 88788.655 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 52-batch_scripted",model_version="default",hostname="7 │ 6489874bf5c",} 1497895.872 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1_scripted",model_version="default",hostname="7648 │ 9874bf5c",} 1207692.952 │ ts_inference_latency_microseconds{model_name="alexnet" │ ,model_version="default",hostname="76489874bf5c",} 108 │ 5750.139 │ ts_inference_latency_micros │ (showing 2.05kB/17.87kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 277µs (cache) (cache) 2ms 1ms 39µs 5ms ✓ Successful GET request Iteration 24/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_seqc_without_torchscript.mar&model_name=bert_seqc_without_torchscript&initial_workers=1&synchronous=true 200 OK ★ 12.5s time ★ 407B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B │ { │ "status": "Model \"bert_seqc_without_torchscript\" V │ ersion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 245µs (cache) (cache) 12.5s 1ms 38µs 12.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/bert_seqc_without_torchscript 200 OK ★ 800ms time ★ 364B↑ 246B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 12B │ Not Accepted └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 312µs (cache) (cache) 796ms 1ms 41µs 801ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/bert_seqc_without_torchscript 200 OK ★ 137ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B │ { │ "status": "Model \"bert_seqc_without_torchscript\" u │ nregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 283µs (cache) (cache) 134ms 1ms 40µs 137ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 18.87kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 18.58kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="vgg16_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1143675.454 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101_scripted",model_version="default",hostname="764 │ 89874bf5c",} 1688748.453 │ ts_inference_latency_microseconds{model_name="densenet │ 161",model_version="default",hostname="76489874bf5c",} │ 1213462.103 │ ts_inference_latency_microseconds{model_name="mnist_sc │ ripted",model_version="default",hostname="76489874bf5c │ ",} 1130346.166 │ ts_inference_latency_microseconds{model_name="fcn_resn │ et_101",model_version="default",hostname="76489874bf5c │ ",} 1260521.179 │ ts_inference_latency_microseconds{model_name="my_text_ │ classifier",model_version="default",hostname="76489874 │ bf5c",} 643034.335 │ ts_inference_latency_microseconds{model_name="fastrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 76775.992 │ ts_inference_latency_microseconds{model_name="maskrcnn │ ",model_version="default",hostname="76489874bf5c",} 11 │ 88788.655 │ ts_inference_latency_microseconds{model_name="resnet-1 │ 52-batch_scripted",model_version="default",hostname="7 │ 6489874bf5c",} 1497895.872 │ ts_inference_latency_microseconds{model_name="vgg16",m │ odel_version="default",hostname="76489874bf5c",} 10805 │ 55.301 │ ts_inference_latency_microseconds{model_name="bert_seq │ c_without_torchscript",model_version="default",hostnam │ e="76489874bf5c",} 796040.105 │ ts_inference_latency_microseconds{model_name="resnet15 │ 2",model_version="default",hostname="76489874bf5c",} 1 │ 053566.656 │ ts_inference_latency_microseconds{model_name="distill_ │ bert_qa_eager",model_version="default",hostname="76489 │ 874bf5c",} 791502.378 │ ts_inference_latency_microseconds{model_name="squeezen │ et1_1",model_version="default",hostname="76489874bf5c" │ ,} 1155593.518 │ ts_inference_latency_microseconds{model_name="deeplabv │ 3_resnet_101_eager",model_version="default",hostname=" │ 76489874bf5c",} 1264835.762 │ ts_inference │ (showing 2.05kB/18.58kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 268µs (cache) (cache) 1ms 1ms 42µs 4ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬───────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ iterations │ 24 │ 0 │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ requests │ 96 │ 0 │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ test-scripts │ 96 │ 0 │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ assertions │ 113 │ 0 │ ├─────────────────────────┴─────────────────────┴───────────────────┤ │ total run duration: 3m 20.9s │ ├───────────────────────────────────────────────────────────────────┤ │ total data received: 10.18MB (approx) │ ├───────────────────────────────────────────────────────────────────┤ │ average response time: 2s [min: 3ms, max: 16.7s, s.d.: 3.4s] │ ├───────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 92µs [min: 16µs, max: 193µs, s.d.: 78µs] │ ├───────────────────────────────────────────────────────────────────┤ │ average first byte time: 2s [min: 1ms, max: 16.7s, s.d.: 3.4s] │ └───────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/model-server/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/model-server/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties ## Successfully started TorchServe newman inference → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161_scripted.mar&model_name=densenet161_scripted&initial_workers=1&synchronous=true 200 OK ★ 6.8s time ★ 389B↑ 362B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 96B │ { │ "status": "Model \"densenet161_scripted\" Version: 1 │ .0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 41ms 5ms 172µs 860µs 6.8s 7ms 388µs 6.9s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/densenet161_scripted 200 OK ★ 1996ms time ★ 111.25kB↑ 415B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 180B │ { │ "tabby": 0.4666191339492798, │ "tiger_cat": 0.46449020504951477, │ "Egyptian_cat": 0.06614047288894653, │ "lynx": 0.001292437082156539, │ "plastic_bag": 0.0002290968259330839 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 7ms 1ms 80µs 564µs 1991ms 2ms 81µs 2s ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/densenet161_scripted 200 OK ★ 78ms time ★ 256B↑ 328B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 62B │ { │ "status": "Model \"densenet161_scripted\" unregister │ ed" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 372µs (cache) (cache) 74ms 2ms 87µs 79ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 30ms time ★ 233B↑ 4.08kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.79kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="densenet │ 161_scripted",model_version="default",hostname="764898 │ 74bf5c",} 1970467.311 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 3.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 100.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="densenet161_scripted",Level="Mo │ del",Hostname="76489874bf5c",} 1967.13 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="densenet161_scripted",Level= │ "Model",Hostname="76489874bf5c",} 1967.35 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="76489874bf5c",} 131.9 │ 216651916504 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilization gauge │ GPUMemoryUtilization{Level="Host",DeviceId="0",Hostnam │ e="76489874bf5c",} 0.0 │ # HELP ts_queue_latency_microseconds Torchserve promet │ heus counter metric with unit: Microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{model_name="densenet161_ │ scripted",model_version="default",hostname="76489874bf │ 5c",} 139.177 │ # HELP WorkerLoadTime Torchserve prometheus gauge metr │ ic with unit: Milliseconds │ # TYPE WorkerLoadTime gauge │ WorkerLoadTime{WorkerName="W-9000-densenet161_scripted │ _1.0",Level="Host",Hostname="76489874bf5c",} 3661.0 │ # HELP DiskUtilization Torchserve prometheus gauge met │ ric wit │ (showing 2.05kB/3.79kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 498µs 19µs 136µs 27ms 1ms 43µs 31ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬───────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ requests │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ test-scripts │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼───────────────────┤ │ assertions │ 5 │ 0 │ ├─────────────────────────┴─────────────────────┴───────────────────┤ │ total run duration: 9.1s │ ├───────────────────────────────────────────────────────────────────┤ │ total data received: 4.13kB (approx) │ ├───────────────────────────────────────────────────────────────────┤ │ average response time: 2.2s [min: 30ms, max: 6.8s, s.d.: 2.7s] │ ├───────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 99µs [min: 19µs, max: 172µs, s.d.: 61µs] │ ├───────────────────────────────────────────────────────────────────┤ │ average first byte time: 2.2s [min: 27ms, max: 6.8s, s.d.: 2.7s] │ └───────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/model-server/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/model-server/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config.properties ## Successfully started TorchServe newman https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 220ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 44ms 14ms 178µs 671µs 143ms 49ms 11ms 598µs 264ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 68ms time ★ 230B↑ 60.36kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60.09kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/60.09kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 1ms 26µs 250µs 34ms 29ms 2ms 99µs 69ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 49ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 724µs 17µs 231µs 41ms 4ms 1ms 37µs 49ms ✓ Status code is 200 → HTTPS Register Model - SqueezeNet POST https://localhost:8444/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true 200 OK ★ 3.1s time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 394µs (cache) (cache) (cache) 3.1s 1ms 84µs 3.1s ✓ Successful POST request → HTTPS Get SqueezeNet Model Description GET https://localhost:8444/models/squeezenet1_1 200 OK ★ 42ms time ★ 246B↑ 926B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 659B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-11-09T10:11:25.694Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 4433, │ "gpu": true, │ "gpuUsage": "gpuId::0 utilization.gpu [%]::0 % │ utilization.memory [%]::0 % memory.used [MiB]::203 Mi │ B" │ } │ ], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 395µs (cache) (cache) (cache) 40ms 1ms 46µs 43ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 4ms time ★ 308B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 297µs (cache) (cache) (cache) 2ms 1ms 31µs 5ms ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 5ms time ★ 309B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 277µs (cache) (cache) (cache) 1ms 2ms 38µs 6ms ✓ Successful PUT request → HTTPS - Inference - SqueezeNet POST https://localhost:8443/predictions/squeezenet1_1 200 OK ★ 1110ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.2785071134567261, │ "lynx": 0.25299131870269775, │ "tiger_cat": 0.24496451020240784, │ "Egyptian_cat": 0.2172253131866455, │ "cougar": 0.0022175421472638845 │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 6ms 797µs (cache) (cache) (cache) 1107ms 1ms 42µs 1115ms ✓ Status code is 200 → HTTPS UnRegister Model SqueezeNet DELETE https://localhost:8444/models/squeezenet1_1 200 OK ★ 20ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 290µs (cache) (cache) (cache) 17ms 1ms 33µs 20ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 9 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 5s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 87.13kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 520ms [min: 4ms, max: 3.1s, s.d.: 991ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 74µs [min: 17µs, max: 178µs, s.d.: 73µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 490ms [min: 1ms, max: 3.1s, s.d.: 1002ms] │ └────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/model-server/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/model-server/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties ## Successfully started TorchServe newman management_api_collection Iteration 1/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 207ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 39ms 5ms 171µs 845µs 188ms 11ms 609µs 246ms ✓ Successful request Iteration 2/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist 200 OK ★ 33ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 560µs (cache) (cache) 29ms 2ms 100µs 34ms ✓ Successful request Iteration 3/82 → management request POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161 200 OK ★ 503ms time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 0 initial workers. Use scale workers API to │ add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 268µs (cache) (cache) 499ms 2ms 58µs 503ms ✓ Successful request Iteration 4/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161 500 Internal Server Error ★ 8ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Model file already exists densenet161.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 301µs (cache) (cache) 4ms 2ms 47µs 8ms ✓ Successful request Iteration 5/82 → management request DELETE http://localhost:8081/models/densenet161 200 OK ★ 44ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 590µs 23µs 150µs 41ms 1ms 54µs 45ms ✓ Successful request Iteration 6/82 → management request POST http://localhost:8081/models 400 Bad Request ★ 4ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Parameter url is required." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 252µs (cache) (cache) 1ms 1ms 31µs 4ms ✓ Successful request Iteration 7/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 17ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 403µs 16µs 117µs 15ms 1ms 33µs 18ms ✓ Successful request Iteration 8/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 28ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 240µs (cache) (cache) 25ms 1ms 31µs 29ms ✓ Successful request Iteration 9/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 7ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 437µs (cache) (cache) 3ms 2ms 29µs 7ms ✓ Successful request Iteration 10/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 25ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 236µs (cache) (cache) 23ms 1ms 31µs 25ms ✓ Successful request Iteration 11/82 → management request PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true 500 Internal Server Error ★ 1405ms time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model mnist │ version: null" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 237µs (cache) (cache) 1401ms 2ms 36µs 1405ms ✓ Successful request Iteration 12/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 17ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 419µs 19µs 149µs 14ms 1ms 32µs 17ms ✓ Successful request Iteration 13/82 → management request GET http://localhost:8081/models/squeezenet1_1/all 200 OK ★ 6ms time ★ 250B↑ 628B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 361B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 238µs (cache) (cache) 3ms 1ms 29µs 6ms ✓ Successful request Iteration 14/82 → management request GET http://localhost:8081/models/squeezenet1_1/1.0 200 OK ★ 3ms time ★ 250B↑ 628B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 361B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 242µs (cache) (cache) 1ms 1ms 29µs 3ms ✓ Successful request Iteration 15/82 → management request GET http://localhost:8081/models/squeezenet1_1 200 OK ★ 3ms time ★ 246B↑ 628B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 361B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 250µs (cache) (cache) 1ms 1ms 29µs 5ms ✓ Successful request Iteration 16/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 245µs (cache) (cache) 3ms 1ms 29µs 5ms ✓ Successful request Iteration 17/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4 400 Bad Request ★ 3ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Invalid RuntimeType value: python4" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 238µs (cache) (cache) 1ms 1ms 29µs 3ms ✓ Successful request Iteration 18/82 → management request GET http://localhost:8081/models?limit=&next_page_token= 200 OK ★ 15ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 418µs 17µs 140µs 12ms 1ms 32µs 15ms ✓ Successful request Iteration 19/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 26ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 234µs (cache) (cache) 24ms 1ms 30µs 26ms ✓ Successful request Iteration 20/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1 202 Accepted ★ 5ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 241µs (cache) (cache) 2ms 1ms 39µs 5ms ✓ Successful request Iteration 21/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true 200 OK ★ 4ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 333µs (cache) (cache) 2ms 1ms 28µs 7ms ✓ Successful request Iteration 22/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true 200 OK ★ 4ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1, version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 233µs (cache) (cache) 1ms 1ms 38µs 4ms ✓ Successful request Iteration 23/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true 404 Not Found ★ 3ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version: 0.0 does not exist for mo │ del: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 236µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 24/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1 202 Accepted ★ 14ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 392µs 16µs 139µs 12ms 1ms 31µs 14ms ✓ Successful request Iteration 25/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default 200 OK ★ 3ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B │ { │ "status": "Default vesion succsesfully updated for m │ odel \"squeezenet1_1\" to \"1.0\"" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 250µs (cache) (cache) 1ms 1ms 28µs 4ms ✓ Successful request Iteration 26/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default 404 Not Found ★ 3ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version 0.0 does not exist for mod │ el squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 242µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 27/82 → management request PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default 404 Not Found ★ 15ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet0_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 399µs 16µs 143µs 12ms 1ms 30µs 15ms ✓ Successful request Iteration 28/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 12ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 533µs 22µs 164µs 9ms 1ms 29µs 12ms ✓ Successful request Iteration 29/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle 200 OK ★ 26ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 234µs (cache) (cache) 24ms 1ms 30µs 27ms ✓ Successful request Iteration 30/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 240µs (cache) (cache) 3ms 1ms 30µs 5ms ✓ Successful request Iteration 31/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0 500 Internal Server Error ★ 1756ms time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model squeez │ enet1_1 version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 229µs (cache) (cache) 1753ms 1ms 34µs 1756ms ✓ Successful request Iteration 32/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0 200 OK ★ 27ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 423µs 16µs 135µs 24ms 1ms 30µs 27ms ✓ Successful request Iteration 33/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 251µs (cache) (cache) 3ms 1ms 30µs 5ms ✓ Successful request Iteration 34/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2 200 OK ★ 996ms time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 242µs (cache) (cache) 993ms 1ms 32µs 996ms ✓ Successful request Iteration 35/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 38ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 242µs (cache) (cache) 36ms 1ms 29µs 38ms ✓ Successful request Iteration 36/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1 200 OK ★ 5.3s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 247µs (cache) (cache) 5.3s 1ms 32µs 5.3s ✓ Successful request Iteration 37/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 53ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 254µs (cache) (cache) 48ms 3ms 33µs 53ms ✓ Successful request Iteration 38/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk 200 OK ★ 5.3s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 265µs (cache) (cache) 5.3s 2ms 67µs 5.3s ✓ Successful request Iteration 39/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 51ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 314µs (cache) (cache) 48ms 1ms 33µs 52ms ✓ Successful request Iteration 40/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1 200 OK ★ 29ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 277µs (cache) (cache) 25ms 2ms 35µs 29ms ✓ Successful request Iteration 41/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 240µs (cache) (cache) 3ms 1ms 26µs 6ms ✓ Successful request Iteration 42/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true 200 OK ★ 198ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 221µs (cache) (cache) 195ms 1ms 32µs 198ms ✓ Successful request Iteration 43/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 12ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 243µs (cache) (cache) 10ms 1ms 28µs 12ms ✓ Successful request Iteration 44/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1 200 OK ★ 198ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 235µs (cache) (cache) 195ms 1ms 33µs 198ms ✓ Successful request Iteration 45/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 11ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 255µs (cache) (cache) 9ms 1ms 27µs 12ms ✓ Successful request Iteration 46/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false 200 OK ★ 198ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 239µs (cache) (cache) 195ms 2ms 45µs 199ms ✓ Successful request Iteration 47/82 → management request GET http://localhost:8081/models?limit=1 200 OK ★ 3ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 332µs (cache) (cache) 1ms 1ms 26µs 4ms ✓ Successful request Iteration 48/82 → management request GET http://localhost:8081/models?limit=-1 200 OK ★ 3ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B │ { │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 969µs 223µs (cache) (cache) 1ms 969µs 25µs 3ms ✓ Successful request Iteration 49/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=1 200 OK ★ 3ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 967µs 214µs (cache) (cache) 731µs 1ms 24µs 3ms ✓ Successful request Iteration 50/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=-1 200 OK ★ 3ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 984µs 211µs (cache) (cache) 1ms 959µs 24µs 3ms ✓ Successful request Iteration 51/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=10 202 Accepted ★ 4ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 953µs 209µs (cache) (cache) 1ms 1ms 31µs 4ms ✓ Successful request Iteration 52/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=-1 202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 309µs (cache) (cache) 1ms 1ms 26µs 4ms ✓ Successful request Iteration 53/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 4ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B │ { │ "status": "Workers scaled to 1 for model: resnet-18" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 225µs (cache) (cache) 1ms 1ms 33µs 4ms ✓ Successful request Iteration 54/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 3ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 257µs (cache) (cache) 1ms 1ms 26µs 3ms ✓ Successful request Iteration 55/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=-1 202 Accepted ★ 4ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 226µs (cache) (cache) 1ms 1ms 35µs 4ms ✓ Successful request Iteration 56/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=0 202 Accepted ★ 4ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 969µs 231µs (cache) (cache) 1ms 1ms 37µs 4ms ✓ Successful request Iteration 57/82 → management request POST http://localhost:8081/models?url=&model_name=resnet-18 404 Not Found ★ 4ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "empty url" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 307µs (cache) (cache) 1ms 1ms 33µs 4ms ✓ Successful request Iteration 58/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18 400 Bad Request ★ 661ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 867µs 31µs 170µs 658ms 1ms 46µs 663ms ✓ Successful request Iteration 59/82 → management request GET http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 4ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 415µs 15µs 128µs 1ms 1ms 27µs 4ms ✓ Successful request Iteration 60/82 → management request GET http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 5ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 389µs 15µs 127µs 1ms 2ms 37µs 5ms ✓ Successful request Iteration 61/82 → management request GET http://localhost:8081/models?next_page_token=12 200 OK ★ 4ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 404µs 15µs 124µs 1ms 1ms 26µs 5ms ✓ Successful request Iteration 62/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan 202 Accepted ★ 4ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 249µs (cache) (cache) 1ms 1ms 29µs 4ms ✓ Successful request Iteration 63/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan 202 Accepted ★ 4ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 248µs (cache) (cache) 1ms 1ms 27µs 4ms ✓ Successful request Iteration 64/82 → management request PUT http://localhost:8081/models/resnet-18 202 Accepted ★ 4ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 243µs (cache) (cache) 1ms 1ms 27µs 4ms ✓ Successful request Iteration 65/82 → management request PUT http://localhost:8081/models/resnet181?min_worker=1 404 Not Found ★ 6ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: resnet181" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 235µs (cache) (cache) 1ms 3ms 36µs 6ms ✓ Successful request Iteration 66/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1 400 Bad Request ★ 4ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 416µs 17µs 135µs 1ms 1ms 27µs 4ms ✓ Successful request Iteration 67/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1 202 Accepted ★ 4ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 383µs 15µs 113µs 1ms 1ms 25µs 4ms ✓ Successful request Iteration 68/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=0 202 Accepted ★ 8ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 974µs 229µs (cache) (cache) 6ms 1ms 34µs 8ms ✓ Successful request Iteration 69/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=-1 500 Internal Server Error ★ 5ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 500, │ "type": "IndexOutOfBoundsException", │ "message": "Index -1 out of bounds for length 0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 233µs (cache) (cache) 2ms 1ms 37µs 5ms ✓ Successful request Iteration 70/82 → management request PUT http://localhost:8081/models/resnet-18?max_worker=-1 400 Bad Request ★ 3ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 427µs 15µs 119µs 1ms 980µs 27µs 4ms ✓ Successful request Iteration 71/82 → management request PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default 404 Not Found ★ 3ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 379µs 16µs 135µs 1ms 961µs 25µs 4ms ✓ Successful request Iteration 72/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 12ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 958µs 356µs 15µs 134µs 9ms 1ms 34µs 12ms ✓ Successful request Iteration 73/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 3ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 222µs (cache) (cache) 884µs 1ms 35µs 4ms ✓ Successful request Iteration 74/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 44ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 991µs 380µs 16µs 110µs 40ms 1ms 45µs 44ms ✓ Successful request Iteration 75/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true 200 OK ★ 10ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 324µs (cache) (cache) 3ms 5ms 38µs 11ms ✓ Successful request Iteration 76/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 37ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 308µs (cache) (cache) 35ms 1ms 31µs 39ms ✓ Successful request Iteration 77/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan 200 OK ★ 5ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 234µs (cache) (cache) 2ms 1ms 28µs 5ms ✓ Successful request Iteration 78/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 25ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 230µs (cache) (cache) 22ms 1ms 43µs 25ms ✓ Successful request Iteration 79/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true 200 OK ★ 5ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 230µs (cache) (cache) 2ms 1ms 39µs 6ms ✓ Successful request Iteration 80/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 25ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 976µs 226µs (cache) (cache) 22ms 1ms 28µs 25ms ✓ Successful request Iteration 81/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1 200 OK ★ 5ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 235µs (cache) (cache) 2ms 1ms 29µs 6ms ✓ Successful request Iteration 82/82 → management request DELETE http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 3ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 228µs (cache) (cache) 963µs 1ms 26µs 3ms ✓ Successful request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 82 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 19.2s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 8.02kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 215ms [min: 3ms, max: 5.3s, s.d.: 860ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 85µs [min: 15µs, max: 171µs, s.d.: 75µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 213ms [min: 731µs, max: 5.3s, s.d.: 860ms] │ └─────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/model-server/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/model-server/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties ## Successfully started TorchServe newman kf_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 42ms 5ms 167µs 894µs 3.2s 7ms 426µs 3.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/v1/models/mnist:predict 200 OK ★ 1085ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B │ { │ "predictions": [ │ 2 │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 7ms 1ms 42µs 696µs 1080ms 2ms 79µs 1091ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Explanations Model POST http://localhost:8080/v1/models/mnist:explain 200 OK ★ 49ms time ★ 680B↑ 26.03kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.8kB │ { │ "explanations": [ │ [ │ [ │ [ │ 0.00411491156406366, │ 0.006386819636653406, │ 0.007885973099344936, │ 0.00978349953260172, │ 0.006857994354999078, │ 0.007661511086234281, │ 0.00724303323753992, │ 0.004503332773062601, │ 0.005925100364745234, │ 0.00425683594155513, │ 0.004948218280827045, │ 0.0052734607708874014, │ 0.00552379977584875, │ 0.007789356714326144, │ 0.00875936298179764, │ 0.004304804488403438, │ 0.010970579450370315, │ 0.003248439029377017, │ 0.005998033014745172, │ 0.0037543657538744346, │ 0.002613859439212664, │ 0.003848187994672177, │ 0.001622227790519026, │ 0.003381873981926139, │ 0.0018886652311849064, │ 0.0016343575236780036, │ -0.00020177349913281484, │ -0.00041332174148868504 │ ], │ [ │ 0.0013083149003370088, │ 0.004350892223350875, │ 0.01243385037121854, │ 0.009710436220655011, │ 0.015587524379727208, │ 0.008045015079536989, │ 0.009372996819792045, │ 0.010047291718101946, │ 0.006854140347229055, │ 0.010055773522627364, │ 0.004764408272483784, │ 0.007292147830859448, │ 0.0117979995890296, │ 0.006462684391740702, │ 0.00338321394993708, │ 0.00922505833719805, │ 0.0016750685986131253, │ 0.0073623054269427324, │ 0.005270057745873351, │ 0.005453597595786016, │ 0.004055587881788062, │ 0.005824795889695814, │ 0.009513341843060968, │ 0.010290186352043448, │ 0.007989786432421646, │ 0.004176460129843604, │ 0.002310402405866635, │ 0.0010978400588839846 │ ], │ [ │ 0.010249213261901957, │ 0.009570611591614923, │ 0.022740227162370678, │ 0.01757456849777241, │ 0.021561972077376767, │ │ (showing 2.05kB/25.8kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 428µs (cache) (cache) 44ms 3ms 88µs 51ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 21ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 301µs (cache) (cache) 19ms 1ms 42µs 22ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 24ms time ★ 233B↑ 4kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.71kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 11024 │ 03.883 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 2.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 0.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="mnist",Level="Model",Hostname=" │ 76489874bf5c",} 37.84 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="mnist",Level="Model",Hostnam │ e="76489874bf5c",} 38.03 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="76489874bf5c",} 131.9 │ 2378616333008 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilization gauge │ GPUMemoryUtilization{Level="Host",DeviceId="0",Hostnam │ e="76489874bf5c",} 0.0 │ # HELP ts_queue_latency_microseconds Torchserve promet │ heus counter metric with unit: Microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{model_name="mnist",model │ _version="default",hostname="76489874bf5c",} 239.95499 │ 999999998 │ # HELP WorkerLoadTime Torchserve prometheus gauge metr │ ic with unit: Milliseconds │ # TYPE WorkerLoadTime gauge │ WorkerLoadTime{WorkerName="W-9000-mnist_1.0",Level="Ho │ st",Hostname="76489874bf5c",} 3087.0 │ # HELP DiskUtilization Torchserve prometheus gauge met │ ric with unit: Percent │ # TYPE DiskUtilization gauge │ DiskUtilization{Level="H │ (showing 2.05kB/3.71kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 473µs 15µs 135µs 21ms 1ms 37µs 24ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 7 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 4.7s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 29.66kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 892ms [min: 21ms, max: 3.2s, s.d.: 1263ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 87µs [min: 15µs, max: 167µs, s.d.: 66µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 886ms [min: 19ms, max: 3.2s, s.d.: 1258ms] │ └─────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/model-server/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/model-server/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kf.properties ## Successfully started TorchServe newman kf_https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 205ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 44ms 6ms 171µs 940µs 138ms 46ms 12ms 648µs 249ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 71ms time ★ 230B↑ 60.36kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60.09kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/60.09kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 3ms 1ms 40µs 371µs 41ms 22ms 3ms 163µs 73ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 43ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 1ms 23µs 276µs 34ms 4ms 2ms 46µs 44ms ✓ Status code is 200 → HTTPS Register Model - Mnist POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.1s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 364µs (cache) (cache) (cache) 3.1s 1ms 94µs 3.1s ✓ Successful POST request → HTTPS Get Mnist Model Description GET https://localhost:8444/models/mnist 200 OK ★ 46ms time ★ 238B↑ 910B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 643B │ [ │ { │ "modelName": "mnist", │ "modelVersion": "1.0", │ "modelUrl": "mnist.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-11-09T10:12:03.756Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 4863, │ "gpu": true, │ "gpuUsage": "gpuId::0 utilization.gpu [%]::8 % │ utilization.memory [%]::0 % memory.used [MiB]::201 Mi │ B" │ } │ ], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 423µs (cache) (cache) (cache) 43ms 1ms 46µs 46ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true 200 OK ★ 4.9s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Workers scaled to 5 for model: mnist" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 300µs (cache) (cache) (cache) 4.9s 1ms 36µs 4.9s ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false 202 Accepted ★ 8ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 358µs (cache) (cache) (cache) 3ms 3ms 31µs 8ms ✓ Successful PUT request → HTTPS - Inference - Mnist_KF POST https://localhost:8443/v1/models/mnist:predict 200 OK ★ 1168ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B │ { │ "predictions": [ │ 2 │ ] │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 6ms 817µs (cache) (cache) (cache) 1164ms 2ms 49µs 1174ms ✓ Status code is 200 → HTTPS - Explanations - Mnist_KF POST https://localhost:8443/v1/models/mnist:explain 200 OK ★ 1148ms time ★ 680B↑ 26.03kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.8kB │ { │ "explanations": [ │ [ │ [ │ [ │ 0.00411491156406366, │ 0.006386819636653406, │ 0.007885973099344936, │ 0.00978349953260172, │ 0.006857994354999078, │ 0.007661511086234281, │ 0.00724303323753992, │ 0.004503332773062601, │ 0.005925100364745234, │ 0.00425683594155513, │ 0.004948218280827045, │ 0.0052734607708874014, │ 0.00552379977584875, │ 0.007789356714326144, │ 0.00875936298179764, │ 0.004304804488403438, │ 0.010970579450370315, │ 0.003248439029377017, │ 0.005998033014745172, │ 0.0037543657538744346, │ 0.002613859439212664, │ 0.003848187994672177, │ 0.001622227790519026, │ 0.003381873981926139, │ 0.0018886652311849064, │ 0.0016343575236780036, │ -0.00020177349913281484, │ -0.00041332174148868504 │ ], │ [ │ 0.0013083149003370088, │ 0.004350892223350875, │ 0.01243385037121854, │ 0.009710436220655011, │ 0.015587524379727208, │ 0.008045015079536989, │ 0.009372996819792045, │ 0.010047291718101946, │ 0.006854140347229055, │ 0.010055773522627364, │ 0.004764408272483784, │ 0.007292147830859448, │ 0.0117979995890296, │ 0.006462684391740702, │ 0.00338321394993708, │ 0.00922505833719805, │ 0.0016750685986131253, │ 0.0073623054269427324, │ 0.005270057745873351, │ 0.005453597595786016, │ 0.004055587881788062, │ 0.005824795889695814, │ 0.009513341843060968, │ 0.010290186352043448, │ 0.007989786432421646, │ 0.004176460129843604, │ 0.002310402405866635, │ 0.0010978400588839846 │ ], │ [ │ 0.010249213261901957, │ 0.009570611591614923, │ 0.022740227162370678, │ 0.01757456849777241, │ 0.021561972077376767, │ │ (showing 2.05kB/25.8kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 408µs (cache) (cache) (cache) 1144ms 2ms 61µs 1149ms ✓ Status code is 200 → HTTPS UnRegister Model Mnist DELETE https://localhost:8444/models/mnist 200 OK ★ 53ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 314µs (cache) (cache) (cache) 50ms 2ms 45µs 54ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 10 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 11.3s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 112.75kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 1083ms [min: 8ms, max: 4.9s, s.d.: 1588ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 78µs [min: 23µs, max: 171µs, s.d.: 65µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 1057ms [min: 3ms, max: 4.9s, s.d.: 1603ms] │ └─────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/model-server/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/model-server/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties ## Successfully started TorchServe newman kfv2_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.3s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 44ms 5ms 173µs 672µs 3.3s 8ms 391µs 3.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/v2/models/mnist/infer 200 OK ★ 1084ms time ★ 7kB↑ 490B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 255B │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "input-0", │ "datatype": "INT64", │ "data": [ │ 1 │ ], │ "shape": [ │ 1 │ ] │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 7ms 1ms 34µs 683µs 1078ms 2ms 81µs 1090ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Explanations Model POST http://localhost:8080/v2/models/mnist/explain 200 OK ★ 20ms time ★ 7kB↑ 14.05kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.81kB │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "input-0", │ "datatype": "FP64", │ "data": [ │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.004054752905330159, │ -0.00022612877238111284, │ -0.00012734140041416805, │ 0.005648369169530469, │ 0.008904784127020225, │ 0.0026385363549800635, │ (showing 2.05kB/13.81kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 432µs (cache) (cache) 17ms 2ms 87µs 22ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 21ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 304µs (cache) (cache) 17ms 1ms 53µs 21ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 29ms time ★ 233B↑ 3.99kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.7kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10735 │ 81.146 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 2.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 100.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="mnist",Level="Model",Hostname=" │ 76489874bf5c",} 11.45 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="mnist",Level="Model",Hostnam │ e="76489874bf5c",} 12.06 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="76489874bf5c",} 131.9 │ 244613647461 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilization gauge │ GPUMemoryUtilization{Level="Host",DeviceId="0",Hostnam │ e="76489874bf5c",} 0.0 │ # HELP ts_queue_latency_microseconds Torchserve promet │ heus counter metric with unit: Microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{model_name="mnist",model │ _version="default",hostname="76489874bf5c",} 228.082 │ # HELP WorkerLoadTime Torchserve prometheus gauge metr │ ic with unit: Milliseconds │ # TYPE WorkerLoadTime gauge │ WorkerLoadTime{WorkerName="W-9000-mnist_1.0",Level="Ho │ st",Hostname="76489874bf5c",} 3126.0 │ # HELP DiskUtilization Torchserve prometheus gauge met │ ric with unit: Percent │ # TYPE DiskUtilization gauge │ DiskUtilization{Level="Host",Hostn │ (showing 2.05kB/3.7kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 595µs 22µs 163µs 25ms 2ms 41µs 30ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 7 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 4.7s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 17.89kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 895ms [min: 20ms, max: 3.3s, s.d.: 1281ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 87µs [min: 22µs, max: 173µs, s.d.: 70µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 889ms [min: 17ms, max: 3.3s, s.d.: 1276ms] │ └─────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/model-server/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/model-server/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kfv2.properties ## Successfully started TorchServe newman kfv2_https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 227ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 50ms 6ms 171µs 678µs 156ms 50ms 12ms 704µs 277ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 68ms time ★ 230B↑ 60.36kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60.09kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/60.09kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 1ms 26µs 337µs 38ms 23ms 3ms 149µs 70ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 40ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.9.0" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 705µs 15µs 242µs 32ms 4ms 1ms 39µs 40ms ✓ Status code is 200 → HTTPS Register Model - Mnist POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.1s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 383µs (cache) (cache) (cache) 3.1s 2ms 86µs 3.1s ✓ Successful POST request → HTTPS Get Mnist Model Description GET https://localhost:8444/models/mnist 200 OK ★ 43ms time ★ 238B↑ 910B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 643B │ [ │ { │ "modelName": "mnist", │ "modelVersion": "1.0", │ "modelUrl": "mnist.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-11-09T10:12:26.020Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 5123, │ "gpu": true, │ "gpuUsage": "gpuId::0 utilization.gpu [%]::0 % │ utilization.memory [%]::0 % memory.used [MiB]::201 Mi │ B" │ } │ ], │ "jobQueueStatus": { │ "remainingCapacity": 100, │ "pendingRequests": 0 │ } │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 376µs (cache) (cache) (cache) 40ms 1ms 47µs 43ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true 200 OK ★ 4.8s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Workers scaled to 5 for model: mnist" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 288µs (cache) (cache) (cache) 4.8s 1ms 39µs 4.8s ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false 202 Accepted ★ 7ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 363µs (cache) (cache) (cache) 2ms 3ms 31µs 8ms ✓ Successful PUT request → HTTPS - Inference - Mnist_KF POST https://localhost:8443/v2/models/mnist/infer 200 OK ★ 1130ms time ★ 7kB↑ 490B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 255B │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "input-0", │ "datatype": "INT64", │ "data": [ │ 1 │ ], │ "shape": [ │ 1 │ ] │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 6ms 780µs (cache) (cache) (cache) 1126ms 1ms 40µs 1136ms ✓ Status code is 200 → HTTPS - Explanations - Mnist_KF POST https://localhost:8443/v2/models/mnist/explain 200 OK ★ 1094ms time ★ 7kB↑ 14.05kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.81kB │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "input-0", │ "datatype": "FP64", │ "data": [ │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.004054752905330159, │ -0.00022612877238111284, │ -0.00012734140041416805, │ 0.005648369169530469, │ 0.008904784127020225, │ 0.0026385363549800635, │ (showing 2.05kB/13.81kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 441µs (cache) (cache) (cache) 1090ms 2ms 62µs 1095ms ✓ Status code is 200 → HTTPS UnRegister Model Mnist DELETE https://localhost:8444/models/mnist 200 OK ★ 47ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 685µs (cache) (cache) (cache) 43ms 2ms 47µs 47ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 10 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 11.1s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 100.98kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 1074ms [min: 7ms, max: 4.8s, s.d.: 1585ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 71µs [min: 15µs, max: 171µs, s.d.: 71µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 1046ms [min: 2ms, max: 4.8s, s.d.: 1600ms] │ └─────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/model-server/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/model-server/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties ## Successfully started TorchServe newman explanation_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.3s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 51ms 5ms 173µs 644µs 3.3s 8ms 408µs 3.3s ✓ Successful POST request → Model Zoo - Explanations Model POST http://localhost:8080/explanations/mnist 200 OK ★ 1093ms time ★ 538B↑ 23.49kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 23.25kB │ [ │ [ │ [ │ 6.648591045533238e-05, │ -0.00010804429890033218, │ 8.721354818396032e-05, │ -0.00012106717671442712, │ 2.503638883157563e-05, │ -4.222534696849854e-05, │ -5.234481622174935e-05, │ -0.0001129952996887743, │ -1.3404519060012e-05, │ -7.235943908282604e-05, │ 1.2628218559708836e-05, │ -3.151635076493018e-05, │ -7.058359189778393e-05, │ -6.872767542522039e-05, │ -7.877427476501056e-05, │ -0.00013302445643979515, │ -3.925594657159814e-05, │ -0.00022929556817503425, │ 1.7131093240407097e-05, │ -0.00038463532735411607, │ -0.00044930379485277715, │ -0.00036369334007626896, │ -0.0005100106054392341, │ -0.0003102968189014311, │ -0.0003109183438172911, │ -3.881059264629927e-05, │ -2.4037940229934885e-05, │ -9.439808465169235e-06 │ ], │ [ │ -0.0004905373189552229, │ -0.00030237292447135014, │ -0.0011733355781587747, │ -0.000620188604123592, │ -0.001111946662367573, │ -0.000625542348208264, │ -0.0006076063364562596, │ -0.0004149972485453505, │ -0.00032000955613699085, │ -0.0001822901762720663, │ -0.0005491939996314846, │ 4.5198536803641234e-05, │ -0.0004397756519046575, │ -0.00032800120357852277, │ -0.0003846251109144256, │ -0.0007280315618350451, │ -0.00038077003190172145, │ -0.0006797483504938083, │ -0.0009410585165934732, │ -0.0005321377201548544, │ -0.0010344589935200924, │ -0.000887104064347332, │ -0.0008278138365755973, │ -0.0007645866442520586, │ -0.0005408324540174419, │ -0.00029662005086652496, │ -0.00014873050464471097, │ 1.6982954746205134e-05 │ ], │ [ │ -0.0012253710100524357, │ -0.0007436613362088342, │ -0.0020355413547651696, │ -0.0007993709223737693, │ -0.0014981657010739102, │ -0.0010168153868082954, │ -0.0010618055219676179, │ -0.0011912340923378774, │ -0.0006529843062293307, │ -0.001023770038827213, │ -0.0009184820075279481 │ (showing 2.05kB/23.25kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 7ms 1ms 42µs 532µs 1088ms 2ms 92µs 1099ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 24ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 374µs (cache) (cache) 21ms 1ms 79µs 24ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 24ms time ★ 233B↑ 3.99kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.7kB │ # HELP ts_inference_latency_microseconds Torchserve pr │ ometheus counter metric with unit: Microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{model_name="mnist",m │ odel_version="default",hostname="76489874bf5c",} 10690 │ 21.913 │ # HELP WorkerThreadTime Torchserve prometheus gauge me │ tric with unit: Milliseconds │ # TYPE WorkerThreadTime gauge │ WorkerThreadTime{Level="Host",Hostname="76489874bf5c", │ } 3.0 │ # HELP CPUUtilization Torchserve prometheus gauge metr │ ic with unit: Percent │ # TYPE CPUUtilization gauge │ CPUUtilization{Level="Host",Hostname="76489874bf5c",} │ 100.0 │ # HELP QueueTime Torchserve prometheus gauge metric wi │ th unit: Milliseconds │ # TYPE QueueTime gauge │ QueueTime{Level="Host",Hostname="76489874bf5c",} 0.0 │ # HELP HandlerTime Torchserve prometheus gauge metric │ with unit: ms │ # TYPE HandlerTime gauge │ HandlerTime{ModelName="mnist",Level="Model",Hostname=" │ 76489874bf5c",} 1063.94 │ # HELP PredictionTime Torchserve prometheus gauge metr │ ic with unit: ms │ # TYPE PredictionTime gauge │ PredictionTime{ModelName="mnist",Level="Model",Hostnam │ e="76489874bf5c",} 1064.14 │ # HELP DiskUsage Torchserve prometheus gauge metric wi │ th unit: Gigabytes │ # TYPE DiskUsage gauge │ DiskUsage{Level="Host",Hostname="76489874bf5c",} 131.9 │ 2512893676758 │ # HELP GPUMemoryUtilization Torchserve prometheus gaug │ e metric with unit: Percent │ # TYPE GPUMemoryUtilization gauge │ GPUMemoryUtilization{Level="Host",DeviceId="0",Hostnam │ e="76489874bf5c",} 0.0 │ # HELP ts_queue_latency_microseconds Torchserve promet │ heus counter metric with unit: Microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{model_name="mnist",model │ _version="default",hostname="76489874bf5c",} 154.632 │ # HELP WorkerLoadTime Torchserve prometheus gauge metr │ ic with unit: Milliseconds │ # TYPE WorkerLoadTime gauge │ WorkerLoadTime{WorkerName="W-9000-mnist_1.0",Level="Ho │ st",Hostname="76489874bf5c",} 3131.0 │ # HELP DiskUtilization Torchserve prometheus gauge met │ ric with unit: Percent │ # TYPE DiskUtilization gauge │ DiskUtilization{Level="Host", │ (showing 2.05kB/3.7kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 472µs 16µs 178µs 21ms 1ms 34µs 25ms ✓ Successful GET request ┌─────────────────────────┬──────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ requests │ 4 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ test-scripts │ 4 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ assertions │ 5 │ 0 │ ├─────────────────────────┴──────────────────────┴─────────────────────┤ │ total run duration: 4.6s │ ├──────────────────────────────────────────────────────────────────────┤ │ total data received: 27.08kB (approx) │ ├──────────────────────────────────────────────────────────────────────┤ │ average response time: 1115ms [min: 24ms, max: 3.3s, s.d.: 1345ms] │ ├──────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 85µs [min: 16µs, max: 173µs, s.d.: 69µs] │ ├──────────────────────────────────────────────────────────────────────┤ │ average first byte time: 1108ms [min: 21ms, max: 3.3s, s.d.: 1340ms] │ └──────────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/model-server/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /home/model-server/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /home/model-server/serve/test | Executing command: torchserve --start --model-store=model_store --workflow-store=model_store --ncs ## Successfully started TorchServe newman management_api_collection Iteration 1/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 200 OK ★ 7.8s time ★ 321B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 46ms 5ms 168µs 631µs 7.8s 9ms 384µs 7.8s ✓ Successful request Iteration 2/11 → workflow management request GET http://localhost:8081/workflows 200 OK ★ 8ms time ★ 235B↑ 423B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 156B │ { │ "workflows": [ │ { │ "workflowName": "densenet", │ "workflowUrl": "https://torchserve.s3.amazonaws. │ com/war_files/densenet_wf.war" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 550µs (cache) (cache) 4ms 1ms 104µs 9ms ✓ Successful request Iteration 3/11 → workflow management request GET http://localhost:8081/workflows/densenet 200 OK ★ 6ms time ★ 244B↑ 559B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 292B │ [ │ { │ "workflowName": "densenet", │ "workflowUrl": "https://torchserve.s3.amazonaws.co │ m/war_files/densenet_wf.war", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 50, │ "workflowDag": "{pre_processing=[densenet], densen │ et=[post_processing]}" │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 269µs (cache) (cache) 3ms 1ms 36µs 7ms ✓ Successful request Iteration 4/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 500 Internal Server Error ★ 6ms time ★ 321B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "code": 500, │ "type": "FileAlreadyExistsException", │ "message": "densenet_wf.war" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 263µs (cache) (cache) 3ms 1ms 35µs 6ms ✓ Successful request Iteration 5/11 → workflow management request DELETE http://localhost:8081/workflows/densenet 200 OK ★ 113ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 553µs 22µs 207µs 109ms 1ms 58µs 113ms ✓ Successful request Iteration 6/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/does_not_exist.war 400 Bad Request ★ 298ms time ★ 324B↑ 441B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 170B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.s3.amazonaws.com/war_files/does_not_exist. │ war" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 351µs (cache) (cache) 294ms 2ms 50µs 299ms ✓ Successful request Iteration 7/11 → workflow management request GET http://localhost:8081/workflows/does_not_exist 404 Not Found ★ 14ms time ★ 250B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found: does_not_exist" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 405µs 16µs 144µs 11ms 1ms 33µs 14ms ✓ Successful request Iteration 8/11 → workflow management request DELETE http://localhost:8081/workflows/does_not_exist 404 Not Found ★ 23ms time ★ 253B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found: does_not_exist" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 441µs 16µs 123µs 20ms 1ms 37µs 24ms ✓ Successful request Iteration 9/11 → workflow management request POST http://localhost:8081/workflows?url=malformed_url,? 404 Not Found ★ 17ms time ★ 275B↑ 396B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 127B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found in workflow store: ma │ lformed_url,?" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 398µs 15µs 138µs 12ms 2ms 31µs 18ms ✓ Successful request Iteration 10/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/custom_python_dep.war 500 Internal Server Error ★ 1704ms time ★ 327B↑ 531B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 250B │ { │ "code": 500, │ "type": "WorkflowException", │ "message": "Workflow custom_python_dep has failed to │ register. Failures: [Workflow Node custom_python_dep_ │ _custom_python_dep failed to register. Details: Model │ not found at: custom_python_dep.mar]" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 390µs 15µs 130µs 1700ms 2ms 42µs 1704ms ✓ Successful request Iteration 11/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/loading-memory-error.war 500 Internal Server Error ★ 1509ms time ★ 330B↑ 543B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 262B │ { │ "code": 500, │ "type": "WorkflowException", │ "message": "Workflow loading-memory-error has failed │ to register. Failures: [Workflow Node loading-memory- │ error__loading-memory-error failed to register. Detail │ s: Model not found at: loading-memory-error.mar]" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 441µs 16µs 146µs 1505ms 2ms 43µs 1509ms ✓ Successful request ┌─────────────────────────┬────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ iterations │ 11 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ requests │ 11 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ test-scripts │ 11 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼────────────────────┼────────────────────┤ │ assertions │ 11 │ 0 │ ├─────────────────────────┴────────────────────┴────────────────────┤ │ total run duration: 11.8s │ ├───────────────────────────────────────────────────────────────────┤ │ total data received: 1.7kB (approx) │ ├───────────────────────────────────────────────────────────────────┤ │ average response time: 1049ms [min: 6ms, max: 7.8s, s.d.: 2.2s] │ ├───────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 83µs [min: 15µs, max: 168µs, s.d.: 74µs] │ ├───────────────────────────────────────────────────────────────────┤ │ average first byte time: 1045ms [min: 3ms, max: 7.8s, s.d.: 2.2s] │ └───────────────────────────────────────────────────────────────────┘ ## Stopping TorchServe ## In directory: /home/model-server/serve/test | Executing command: ['torchserve', '--stop'] ## Successfully stopped TorchServe ## Started regression tests ## Started densenet mar creation ## In directory: /home/model-server/tmp/workspace/model_store | Executing command: torch-model-archiver --model-name densenet161_v1 --version 1.1 --model-file /home/model-server/serve/ts_scripts/../examples/image_classifier/densenet_161/model.py --serialized-file /home/model-server/tmp/workspace/model_store/densenet161-8d451a50.pth --extra-files /home/model-server/serve/ts_scripts/../examples/image_classifier/index_to_name.json --handler image_classifier --force ## Started regression pytests ## In directory: /home/model-server/serve/test/pytest | Executing command: python -m pytest -v ./ ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-7.3.1, pluggy-1.3.0 -- /home/venv/bin/python cachedir: .pytest_cache rootdir: /home/model-server/serve plugins: mock-3.12.0, cov-4.1.0 collecting ... collected 114 items test_auto_recover.py::test_tp_inference 2023-11-09T10:13:20,028 [INFO ] W-9000-tp_model_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 24 2023-11-09T10:13:20,029 [DEBUG] W-9000-tp_model_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-tp_model_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED 2023-11-09T10:13:20,029 [INFO ] W-9000-tp_model_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:5920.0|#WorkerName:W-9000-tp_model_1.0,Level:Host|#hostname:76489874bf5c,timestamp:1699524800 2023-11-09T10:13:20,029 [INFO ] W-9000-tp_model_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery succeeded, reset recoveryStartTS 2023-11-09T10:13:20,029 [INFO ] W-9000-tp_model_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:2.0|#Level:Host|#hostname:76489874bf5c,timestamp:1699524800 PASSED [ 0%] test_continuous_batching.py::test_echo_stream_inference 2023-11-09T10:13:36,093 [DEBUG] W-9000-streaming_handler_1.0 org.pytorch.serve.job.RestJob - Waiting time ns: 320759733, Backend time ns: 14175644 2023-11-09T10:13:36,093 [INFO ] W-9000-streaming_handler_1.0 TS_METRICS - QueueTime.Milliseconds:320.0|#Level:Host|#hostname:76489874bf5c,timestamp:1699524816 2023-11-09T10:13:36,093 [INFO ] W-9000-streaming_handler_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 14 2023-11-09T10:13:36,093 [INFO ] W-9000-streaming_handler_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:0.0|#Level:Host|#hostname:76489874bf5c,timestamp:1699524816 PASSED [ 1%] test_continuous_batching.py::test_decoding_stage PASSED [ 2%] test_continuous_batching.py::test_closed_connection 2023-11-09T10:13:41,534 [INFO ] W-9000-streaming_handler_1.0-stdout MODEL_METRICS - HandlerTime.ms:11.72|#ModelName:streaming_handler,Level:Model|#hostname:76489874bf5c,requestID:794432f9-22cf-4619-ab63-a11dfbd9951f,timestamp:1699524821 2023-11-09T10:13:41,534 [INFO ] W-9000-streaming_handler_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - result=[METRICS]PredictionTime.Milliseconds:11.84|#ModelName:streaming_handler,Level:Model|#hostname:76489874bf5c,1699524821,794432f9-22cf-4619-ab63-a11dfbd9951f, pattern=[METRICS] 2023-11-09T10:13:41,535 [INFO ] W-9000-streaming_handler_1.0 org.pytorch.serve.wlm.ContinuousBatching - Connection to client got closed; Removing job: 794432f9-22cf-4619-ab63-a11dfbd9951f 2023-11-09T10:13:41,535 [INFO ] W-9000-streaming_handler_1.0-stdout MODEL_METRICS - PredictionTime.ms:11.84|#ModelName:streaming_handler,Level:Model|#hostname:76489874bf5c,requestID:794432f9-22cf-4619-ab63-a11dfbd9951f,timestamp:1699524821PASSED [ 3%]2023-11-09T10:13:41,535 [INFO ] W-9000-streaming_handler_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 12 test_distributed_inference_handler.py::test_large_model_inference SKIPPED [ 4%] test_example_dcgan.py::test_model_archive_creation PASSED [ 5%] test_example_dcgan.py::test_model_register_unregister PASSED [ 6%] test_example_dcgan.py::test_image_generation_without_any_input_constraints PASSED [ 7%] test_example_dcgan.py::test_image_generation_with_input_constraints PASSED [ 7%] test_example_intel_extension_for_pytorch.py::test_single_worker_affinity SKIPPED [ 8%] test_example_intel_extension_for_pytorch.py::test_multi_worker_affinity SKIPPED [ 9%] test_example_intel_extension_for_pytorch.py::test_worker_scale_up_affinity SKIPPED [ 10%] test_example_intel_extension_for_pytorch.py::test_worker_scale_down_affinity SKIPPED [ 11%] test_example_micro_batching.py::test_single_example_inference[yaml_config] PASSED [ 12%] test_example_micro_batching.py::test_multi_example_inference[4-yaml_config] PASSED [ 13%] test_example_micro_batching.py::test_multi_example_inference[4-no_config] PASSED [ 14%] test_example_micro_batching.py::test_single_example_inference[no_config] 2023-11-09T10:14:37,051 [INFO ] W-9001-image_classifier_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:1.0|#Level:Host|#hostname:76489874bf5c,timestamp:1699524877 PASSED [ 14%] test_example_micro_batching.py::test_multi_example_inference[16-no_config] PASSED [ 15%] test_example_micro_batching.py::test_multi_example_inference[16-yaml_config] PASSED [ 16%] test_example_scriptable_tokenzier.py::test_handler PASSED [ 17%] test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_sample_text PASSED [ 18%] test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_empty_string 2023-11-09T10:15:21,582 [INFO ] W-9001-scriptable_tokenizer_untrained_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:2.0|#Level:Host|#hostname:76489874bf5c,timestamp:1699524921 2023-11-09T10:15:21,582 [INFO ] epollEventLoopGroup-3-4 ACCESS_LOG - /127.0.0.1:50858 "POST /models?model_name=scriptable_tokenizer_untrained&url=scriptable_tokenizer_untrained.mar&initial_workers=1&synchronous=true HTTP/1.1" 200 7535 2023-11-09T10:15:21,582 [INFO ] epollEventLoopGroup-3-4 TS_METRICS - Requests2XX.Count:1.0|#Level:Host|#hostname:76489874bf5c,timestamp:1699524921 PASSED [ 19%] test_example_scriptable_tokenzier.py::test_inference_with_pretrained_model PASSED [ 20%] test_gRPC_inference_api.py::test_inference_apis PASSED [ 21%] test_gRPC_inference_api.py::test_inference_stream_apis PASSED [ 21%] test_gRPC_inference_api.py::test_inference_stream2_apis PASSED [ 22%] test_gRPC_management_apis.py::test_management_apis PASSED [ 23%] test_handler.py::test_mnist_model_register_and_inference_on_valid_model PASSED [ 24%] test_handler.py::test_mnist_model_register_using_non_existent_handler_with_nonzero_workers PASSED [ 25%] test_handler.py::test_mnist_model_register_scale_inference_with_non_existent_handler PASSED [ 26%] test_handler.py::test_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 27%] test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model PASSED [ 28%] test_handler.py::test_kserve_mnist_model_register_scale_inference_with_non_existent_handler PASSED [ 28%] test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 29%] test_handler.py::test_huggingface_bert_batch_inference PASSED [ 30%] test_handler.py::test_MMF_activity_recognition_model_register_and_inference_on_valid_model SKIPPED [ 31%] test_handler.py::test_huggingface_bert_model_parallel_inference PASSED [ 32%] test_metrics.py::test_logs_created PASSED [ 33%] test_metrics.py::test_logs_startup_cfg_created_snapshot_enabled PASSED [ 34%] test_metrics.py::test_logs_startup_cfg_created_snapshot_disabled PASSED [ 35%] test_metrics.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 35%] test_metrics.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 36%] test_metrics.py::test_log_location_var_snapshot_disabled PASSED [ 37%] test_metrics.py::test_log_location_var_snapshot_enabled PASSED [ 38%] test_metrics.py::test_async_logging PASSED [ 39%] test_metrics.py::test_async_logging_non_boolean PASSED [ 40%] test_metrics.py::test_metrics_location_var_snapshot_disabled PASSED [ 41%] test_metrics.py::test_metrics_location_var_snapshot_enabled PASSED [ 42%] test_metrics.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 42%] test_metrics.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 43%] test_metrics.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 44%] test_metrics.py::test_metrics_log_mode PASSED [ 45%] test_metrics.py::test_metrics_prometheus_mode PASSED [ 46%] test_metrics.py::test_collect_system_metrics_when_not_disabled PASSED [ 47%] test_metrics.py::test_disable_system_metrics_using_config_properties PASSED [ 48%] test_metrics.py::test_disable_system_metrics_using_environment_variable PASSED [ 49%] test_metrics_kf.py::test_logs_created PASSED [ 50%] test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_enabled PASSED [ 50%] test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_disabled PASSED [ 51%] test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 52%] test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 53%] test_metrics_kf.py::test_log_location_var_snapshot_disabled PASSED [ 54%] test_metrics_kf.py::test_log_location_var_snapshot_enabled PASSED [ 55%] test_metrics_kf.py::test_async_logging PASSED [ 56%] test_metrics_kf.py::test_async_logging_non_boolean PASSED [ 57%] test_metrics_kf.py::test_metrics_location_var_snapshot_disabled PASSED [ 57%] test_metrics_kf.py::test_metrics_location_var_snapshot_enabled PASSED [ 58%] test_metrics_kf.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 59%] test_metrics_kf.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 60%] test_metrics_kf.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 61%] test_model_archiver.py::test_multiple_model_versions_registration PASSED [ 62%] test_model_archiver.py::test_duplicate_model_registration_using_local_url_followed_by_http_url PASSED [ 63%] test_model_archiver.py::test_duplicate_model_registration_using_http_url_followed_by_local_url PASSED [ 64%] test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_without_force PASSED [ 64%] test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_with_force PASSED [ 65%] test_model_archiver.py::test_model_archiver_without_handler_flag PASSED [ 66%] test_model_archiver.py::test_model_archiver_without_model_name_flag PASSED [ 67%] test_model_archiver.py::test_model_archiver_without_model_file_flag PASSED [ 68%] test_model_archiver.py::test_model_archiver_without_serialized_flag PASSED [ 69%] test_onnx.py::test_convert_to_onnx PASSED [ 70%] test_onnx.py::test_model_packaging_and_start PASSED [ 71%] test_onnx.py::test_model_start PASSED [ 71%] test_onnx.py::test_inference PASSED [ 72%] test_onnx.py::test_stop PASSED [ 73%] test_parallelism.py::test_tp_inference PASSED [ 74%] test_pytorch_profiler.py::test_profiler_default_and_custom_handler[/home/model-server/serve/test/pytest/profiler_utils/resnet_custom.py] PASSED [ 75%] test_pytorch_profiler.py::test_profiler_default_and_custom_handler[image_classifier] PASSED [ 76%] test_pytorch_profiler.py::test_profiler_arguments_override[/home/model-server/serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 77%] test_pytorch_profiler.py::test_batch_input[/home/model-server/serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 78%] test_sm_mme_requirements.py::test_no_model_loaded PASSED [ 78%] test_sm_mme_requirements.py::test_oom_on_model_load SKIPPED (Logic n...) [ 79%] test_sm_mme_requirements.py::test_oom_on_invoke SKIPPED (Logic needs...) [ 80%] test_snapshot.py::test_snapshot_created_on_start_and_stop PASSED [ 81%] test_snapshot.py::test_snapshot_created_on_management_api_invoke PASSED [ 82%] test_snapshot.py::test_start_from_snapshot PASSED [ 83%] test_snapshot.py::test_start_from_latest PASSED [ 84%] test_snapshot.py::test_start_from_read_only_snapshot PASSED [ 85%] test_snapshot.py::test_no_config_snapshots_cli_option PASSED [ 85%] test_snapshot.py::test_start_from_default PASSED [ 86%] test_snapshot.py::test_start_from_non_existing_snapshot PASSED [ 87%] test_snapshot.py::test_torchserve_init_with_non_existent_model_store PASSED [ 88%] test_snapshot.py::test_restart_torchserve_with_last_snapshot_with_model_mar_removed PASSED [ 89%] test_snapshot.py::test_replace_mar_file_with_dummy PASSED [ 90%] test_snapshot.py::test_restart_torchserve_with_one_of_model_mar_removed PASSED [ 91%] test_torch_compile.py::TestTorchCompile::test_archive_model_artifacts PASSED [ 92%] test_torch_compile.py::TestTorchCompile::test_start_torchserve PASSED [ 92%] test_torch_compile.py::TestTorchCompile::test_server_status PASSED [ 93%] test_torch_compile.py::TestTorchCompile::test_registered_model PASSED [ 94%] test_torch_compile.py::TestTorchCompile::test_serve_inference SKIPPED [ 95%] test_torch_xla.py::TestTorchXLA::test_archive_model_artifacts SKIPPED [ 96%] test_torch_xla.py::TestTorchXLA::test_start_torchserve SKIPPED (PyTo...) [ 97%] test_torch_xla.py::TestTorchXLA::test_server_status SKIPPED (PyTorch...) [ 98%] test_torch_xla.py::TestTorchXLA::test_registered_model SKIPPED (PyTo...) [ 99%] test_torch_xla.py::TestTorchXLA::test_serve_inference SKIPPED (PyTor...) [100%] =============================== warnings summary =============================== ../../../../venv/lib/python3.9/site-packages/ts/torch_handler/base_handler.py:13 /home/venv/lib/python3.9/site-packages/ts/torch_handler/base_handler.py:13: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html from pkg_resources import packaging ../../../../venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 /home/venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../../../venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 /home/venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.logging')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../../../venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2350 ../../../../venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2350 /home/venv/lib/python3.9/site-packages/pkg_resources/__init__.py:2350: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(parent) ../../../../venv/lib/python3.9/site-packages/google/rpc/__init__.py:20 /home/venv/lib/python3.9/site-packages/google/rpc/__init__.py:20: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.rpc')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages pkg_resources.declare_namespace(__name__) test/pytest/test_example_scriptable_tokenzier.py::test_handler /home/venv/lib/python3.9/site-packages/torch/jit/_recursive.py:266: UserWarning: 'batch_first' was found in ScriptModule constants, but was not actually set in __init__. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/pytest/test_example_scriptable_tokenzier.py::test_handler /home/venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1501: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. (Triggered internally at ../aten/src/ATen/NestedTensorImpl.cpp:177.) return forward_call(*args, **kwargs) test/pytest/test_example_scriptable_tokenzier.py::test_handler /home/model-server/serve/test/pytest/../../examples/text_classification_with_scriptable_tokenizer/handler.py:97: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument. data = F.softmax(data) test/pytest/test_gRPC_inference_api.py::test_inference_stream2_apis /home/venv/lib/python3.9/site-packages/_pytest/threadexception.py:73: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-16 Traceback (most recent call last): File "/usr/lib/python3.9/threading.py", line 980, in _bootstrap_inner self.run() File "/usr/lib/python3.9/threading.py", line 917, in run self._target(*self._args, **self._kwargs) File "/home/model-server/serve/test/pytest/test_gRPC_inference_api.py", line 174, in __infer_stream2 inference_pb2.PredictionsRequest( File "/usr/lib/python3.9/_collections_abc.py", line 941, in update self[key] = other[key] TypeError: expected bytes, int found warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ========== 100 passed, 14 skipped, 10 warnings in 1276.10s (0:21:16) =========== Removing file : management_pb2_grpc.py Removing file : inference_pb2_grpc.py Removing file : management_pb2.py Removing file : inference_pb2.py ## Deleting model_store_gen_dir: /home/model-server/serve/ts_scripts/../model_store_gen