Processing /Volumes/workplace/python_env/serve
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting Pillow (from torchserve==0.8.1b20230814)
  Obtaining dependency information for Pillow from https://files.pythonhosted.org/packages/5a/29/aa1678cae507a480a6d75453c1de98940e5eb6bd8f0e8e8347ec29a4dfc0/Pillow-10.0.0-cp38-cp38-macosx_10_10_x86_64.whl.metadata
  Using cached Pillow-10.0.0-cp38-cp38-macosx_10_10_x86_64.whl.metadata (9.5 kB)
Collecting psutil (from torchserve==0.8.1b20230814)
  Using cached psutil-5.9.5-cp36-abi3-macosx_10_9_x86_64.whl (245 kB)
Collecting packaging (from torchserve==0.8.1b20230814)
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting wheel (from torchserve==0.8.1b20230814)
  Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/28/f5/6955d7b3a5d71ce6bac104f9cf98c1b0513ad656cdaca8ea7d579196f771/wheel-0.41.1-py3-none-any.whl.metadata
  Using cached wheel-0.41.1-py3-none-any.whl.metadata (2.2 kB)
Using cached Pillow-10.0.0-cp38-cp38-macosx_10_10_x86_64.whl (3.4 MB)
Using cached wheel-0.41.1-py3-none-any.whl (64 kB)
Building wheels for collected packages: torchserve
  Building wheel for torchserve (setup.py): started
  Building wheel for torchserve (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [103 lines of output]
      -- torchserve building version: 0.8.1b20230814
      running bdist_wheel
      running build
      running build_py
      running build_frontend
      Starting a Gradle Daemon (subsequent builds will be faster)
      > Task :archive:clean
      
      > Task :server:killServer
      No server running!
      
      > Task :server:clean
      > Task :archive:compileJava
      > Task :archive:processResources NO-SOURCE
      > Task :archive:classes
      > Task :archive:jar
      > Task :archive:assemble
      > Task :server:extractIncludeProto
      > Task :server:extractProto
      > Task :server:generateProto UP-TO-DATE
      
      > Task :server:compileJava
      /Volumes/workplace/python_env/serve/frontend/server/src/main/java/org/pytorch/serve/util/ApiUtils.java:421: error: cannot find symbol
              jobQueueStatus.setRemainingCapacity(model.getJobQueueRemainingCapacity());
                                                       ^
        symbol:   method getJobQueueRemainingCapacity()
        location: variable model of type Model
      /Volumes/workplace/python_env/serve/frontend/server/src/main/java/org/pytorch/serve/util/ApiUtils.java:422: error: cannot find symbol
              jobQueueStatus.setPendingRequests(model.getPendingRequestsInJobQueue());
                                                     ^
        symbol:   method getPendingRequestsInJobQueue()
        location: variable model of type Model
      2 errors
      
      > Task :server:compileJava FAILED
      
      FAILURE: Build failed with an exception.
      
      * What went wrong:
      Execution failed for task ':server:compileJava'.
      > Compilation failed; see the compiler error output for details.
      
      * Try:
      > Run with --stacktrace option to get the stack trace.
      > Run with --info or --debug option to get more log output.
      > Run with --scan to get full insights.
      
      * Get more help at https://help.gradle.org
      
      Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
      
      You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
      
      See https://docs.gradle.org/7.6/userguide/command_line_interface.html#sec:command_line_warnings
      
      BUILD FAILED in 12s
      9 actionable tasks: 8 executed, 1 up-to-date
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/Volumes/workplace/python_env/serve/setup.py", line 163, in <module>
          setup(
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 349, in run
          self.run_command("build")
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/Volumes/workplace/python_env/serve/setup.py", line 110, in run
          self.run_command("build_frontend")
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/Volumes/workplace/python_env/serve/setup.py", line 97, in run
          subprocess.check_call(build_frontend_command[platform.system()], shell=True)
        File "/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/subprocess.py", line 364, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command 'frontend/gradlew -p frontend clean assemble' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for torchserve
  Running setup.py clean for torchserve
Failed to build torchserve
ERROR: Could not build wheels for torchserve, which is required to install pyproject.toml-based projects
Processing ./model-archiver
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting enum-compat (from torch-model-archiver==0.8.1b20230814)
  Using cached enum_compat-0.0.3-py3-none-any.whl (1.3 kB)
Building wheels for collected packages: torch-model-archiver
  Building wheel for torch-model-archiver (setup.py): started
  Building wheel for torch-model-archiver (setup.py): finished with status 'done'
  Created wheel for torch-model-archiver: filename=torch_model_archiver-0.8.1b20230814-py3-none-any.whl size=14828 sha256=be3a96e1c1067563e8a9de5f0bfbffe5aea4e028f7cb98b67146ef2168752239
  Stored in directory: /Users/lninga/Library/Caches/pip/wheels/40/d0/f5/5019474ce9b424781ec8c156be2dabd8d7526bfba45adc4d1e
Successfully built torch-model-archiver
Installing collected packages: enum-compat, torch-model-archiver
  Attempting uninstall: enum-compat
    Found existing installation: enum-compat 0.0.3
    Uninstalling enum-compat-0.0.3:
      Successfully uninstalled enum-compat-0.0.3
  Attempting uninstall: torch-model-archiver
    Found existing installation: torch-model-archiver 0.8.1b20230814
    Uninstalling torch-model-archiver-0.8.1b20230814:
      Successfully uninstalled torch-model-archiver-0.8.1b20230814
Successfully installed enum-compat-0.0.3 torch-model-archiver-0.8.1b20230814
Processing ./workflow-archiver
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: torch-workflow-archiver
  Building wheel for torch-workflow-archiver (setup.py): started
  Building wheel for torch-workflow-archiver (setup.py): finished with status 'done'
  Created wheel for torch-workflow-archiver: filename=torch_workflow_archiver-0.2.9b20230814-py3-none-any.whl size=12727 sha256=19de85ee02997d1faf2f3a2f65405a8a60bfb5cb3731c84330d86b92ba6b1d26
  Stored in directory: /Users/lninga/Library/Caches/pip/wheels/6a/08/97/e89e9bf08db09591cf7092e9dbfc31fa105e0423bc3c8da6c1
Successfully built torch-workflow-archiver
Installing collected packages: torch-workflow-archiver
  Attempting uninstall: torch-workflow-archiver
    Found existing installation: torch-workflow-archiver 0.2.9b20230814
    Uninstalling torch-workflow-archiver-0.2.9b20230814:
      Successfully uninstalled torch-workflow-archiver-0.2.9b20230814
Successfully installed torch-workflow-archiver-0.2.9b20230814
WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar ...
WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar ...
WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar ...
WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar ...
WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar ...
WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar ...
WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar ...
WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar ...
WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar ...
WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar ...
WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar ...
newman: could not find "htmlextra" reporter
  ensure that the reporter is installed in the same directory as newman
  please install reporter using npm

newman

management_api_collection

Iteration 1/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1  
  200 OK ★ 228ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total 
  38ms      6ms    1ms          434µs           211ms            7ms        405µs     266ms 

  ✓  Successful  request

Iteration 2/82

→ management request
  POST http://localhost:8081/models?url=mnist.mar&model_name=mnist  
  200 OK ★ 99ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B
  │ {
  │   "status": "Model \"mnist\" Version: 1.0 registered w
  │ ith 0 initial workers. Use scale workers API to add wo
  │ rkers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       783µs   (cache)      (cache)         95ms             2ms        77µs      100ms 

  ✓  Successful  request

Iteration 3/82

→ management request
  POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161  
  200 OK ★ 878ms time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B
  │ {
  │   "status": "Model \"densenet161\" Version: 1.0 regist
  │ ered with 0 initial workers. Use scale workers API to 
  │ add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       283µs   (cache)      (cache)         875ms            1ms        75µs      879ms 

  ✓  Successful  request

Iteration 4/82

→ management request
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161  
  500 Internal Server Error ★ 7ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B
  │ {
  │   "code": 500,
  │   "type": "InternalServerException",
  │   "message": "Model file already exists densenet161.ma
  │ r"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       275µs   (cache)      (cache)         4ms              1ms        51µs      7ms   

  ✓  Successful  request

Iteration 5/82

→ management request
  DELETE http://localhost:8081/models/densenet161  
  200 OK ★ 24ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B
  │ {
  │   "status": "Model \"densenet161\" unregistered"
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       1ms    261µs        277µs           20ms             1ms        74µs      26ms  

  ✓  Successful  request

Iteration 6/82

→ management request
  POST http://localhost:8081/models  
  400 Bad Request ★ 6ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B
  │ {
  │   "code": 400,
  │   "type": "BadRequestException",
  │   "message": "Parameter url is required."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       268µs   (cache)      (cache)         3ms              2ms        105µs     7ms   

  ✓  Successful  request

Iteration 7/82

→ management request
  DELETE http://localhost:8081/models/mnist  
  200 OK ★ 25ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Model \"mnist\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       696µs   197µs        209µs           21ms             1ms        54µs      26ms  

  ✓  Successful  request

Iteration 8/82

→ management request
  POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler  
  200 OK ★ 53ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B
  │ {
  │   "status": "Model \"mnist\" Version: 1.0 registered w
  │ ith 0 initial workers. Use scale workers API to add wo
  │ rkers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       266µs   (cache)      (cache)         50ms             1ms        37µs      53ms  

  ✓  Successful  request

Iteration 9/82

→ management request
  DELETE http://localhost:8081/models/mnist  
  200 OK ★ 8ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Model \"mnist\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       287µs   (cache)      (cache)         5ms              1ms        38µs      9ms   

  ✓  Successful  request

Iteration 10/82

→ management request
  POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler  
  200 OK ★ 55ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B
  │ {
  │   "status": "Model \"mnist\" Version: 1.0 registered w
  │ ith 0 initial workers. Use scale workers API to add wo
  │ rkers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       299µs   (cache)      (cache)         52ms             1ms        37µs      56ms  

  ✓  Successful  request

Iteration 11/82

→ management request
  PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true  
  500 Internal Server Error ★ 2s time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B
  │ {
  │   "code": 500,
  │   "type": "InternalServerException",
  │   "message": "Failed to start workers for model mnist 
  │ version: null"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       243µs   (cache)      (cache)         2s               1ms        37µs      2s    

  ✓  Successful  request

Iteration 12/82

→ management request
  DELETE http://localhost:8081/models/mnist  
  200 OK ★ 17ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Model \"mnist\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       687µs   172µs        229µs           14ms             1ms        32µs      18ms  

  ✓  Successful  request

Iteration 13/82

→ management request
  GET http://localhost:8081/models/squeezenet1_1/all  
  200 OK ★ 7ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B
  │ [
  │   {
  │     "modelName": "squeezenet1_1",
  │     "modelVersion": "1.0",
  │     "modelUrl": "squeezenet1_1.mar",
  │     "runtime": "python",
  │     "minWorkers": 0,
  │     "maxWorkers": 0,
  │     "batchSize": 1,
  │     "maxBatchDelay": 100,
  │     "loadedAtStartup": false,
  │     "workers": []
  │   }
  │ ]
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       235µs   (cache)      (cache)         4ms              1ms        37µs      7ms   

  ✓  Successful  request

Iteration 14/82

→ management request
  GET http://localhost:8081/models/squeezenet1_1/1.0  
  200 OK ★ 3ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B
  │ [
  │   {
  │     "modelName": "squeezenet1_1",
  │     "modelVersion": "1.0",
  │     "modelUrl": "squeezenet1_1.mar",
  │     "runtime": "python",
  │     "minWorkers": 0,
  │     "maxWorkers": 0,
  │     "batchSize": 1,
  │     "maxBatchDelay": 100,
  │     "loadedAtStartup": false,
  │     "workers": []
  │   }
  │ ]
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       192µs   (cache)      (cache)         1ms              1ms        36µs      3ms   

  ✓  Successful  request

Iteration 15/82

→ management request
  GET http://localhost:8081/models/squeezenet1_1  
  200 OK ★ 4ms time ★ 246B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B
  │ [
  │   {
  │     "modelName": "squeezenet1_1",
  │     "modelVersion": "1.0",
  │     "modelUrl": "squeezenet1_1.mar",
  │     "runtime": "python",
  │     "minWorkers": 0,
  │     "maxWorkers": 0,
  │     "batchSize": 1,
  │     "maxBatchDelay": 100,
  │     "loadedAtStartup": false,
  │     "workers": []
  │   }
  │ ]
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       251µs   (cache)      (cache)         1ms              1ms        34µs      5ms   

  ✓  Successful  request

Iteration 16/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1  
  200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       198µs   (cache)      (cache)         4ms              1ms        28µs      6ms   

  ✓  Successful  request

Iteration 17/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4  
  400 Bad Request ★ 4ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B
  │ {
  │   "code": 400,
  │   "type": "BadRequestException",
  │   "message": "Invalid RuntimeType value: python4"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       245µs   (cache)      (cache)         1ms              1ms        29µs      4ms   

  ✓  Successful  request

Iteration 18/82

→ management request
  GET http://localhost:8081/models?limit=&next_page_token=  
  200 OK ★ 14ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B
  │ {
  │   "models": []
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       664µs   163µs        252µs           11ms             1ms        42µs      14ms  

  ✓  Successful  request

Iteration 19/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1  
  200 OK ★ 41ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       245µs   (cache)      (cache)         38ms             1ms        25µs      41ms  

  ✓  Successful  request

Iteration 20/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1?min_worker=1  
  202 Accepted ★ 4ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       187µs   (cache)      (cache)         2ms              1ms        30µs      4ms   

  ✓  Successful  request

Iteration 21/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true  
  200 OK ★ 12ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B
  │ {
  │   "status": "Workers scaled to 1 for model: squeezenet
  │ 1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       160µs   (cache)      (cache)         10ms             1ms        31µs      12ms  

  ✓  Successful  request

Iteration 22/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true  
  200 OK ★ 5ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B
  │ {
  │   "status": "Workers scaled to 1 for model: squeezenet
  │ 1_1, version: 1.0"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       209µs   (cache)      (cache)         2ms              2ms        38µs      5ms   

  ✓  Successful  request

Iteration 23/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true  
  404 Not Found ★ 3ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B
  │ {
  │   "code": 404,
  │   "type": "ModelVersionNotFoundException",
  │   "message": "Model version: 0.0 does not exist for mo
  │ del: squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       197µs   (cache)      (cache)         1ms              1ms        28µs      4ms   

  ✓  Successful  request

Iteration 24/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1  
  202 Accepted ★ 16ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       552µs   152µs        222µs           13ms             1ms        35µs      17ms  

  ✓  Successful  request

Iteration 25/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default  
  200 OK ★ 5ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B
  │ {
  │   "status": "Default vesion succsesfully updated for m
  │ odel \"squeezenet1_1\" to \"1.0\""
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       377µs   (cache)      (cache)         2ms              1ms        29µs      6ms   

  ✓  Successful  request

Iteration 26/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default  
  404 Not Found ★ 4ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B
  │ {
  │   "code": 404,
  │   "type": "ModelVersionNotFoundException",
  │   "message": "Model version 0.0 does not exist for mod
  │ el squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       218µs   (cache)      (cache)         1ms              1ms        28µs      4ms   

  ✓  Successful  request

Iteration 27/82

→ management request
  PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default  
  404 Not Found ★ 17ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: squeezenet0_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       614µs   200µs        223µs           13ms             1ms        28µs      17ms  

  ✓  Successful  request

Iteration 28/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1  
  200 OK ★ 31ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  854µs     451µs   116µs        188µs           28ms             1ms        33µs      31ms  

  ✓  Successful  request

Iteration 29/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle  
  200 OK ★ 46ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       191µs   (cache)      (cache)         43ms             1ms        27µs      46ms  

  ✓  Successful  request

Iteration 30/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1  
  200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  992µs     162µs   (cache)      (cache)         4ms              1ms        31µs      7ms   

  ✓  Successful  request

Iteration 31/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0  
  500 Internal Server Error ★ 1866ms time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B
  │ {
  │   "code": 500,
  │   "type": "InternalServerException",
  │   "message": "Failed to start workers for model squeez
  │ enet1_1 version: 1.0"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total  
  1ms       204µs   (cache)      (cache)         1864ms           1ms        31µs      1867ms 

  ✓  Successful  request

Iteration 32/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0  
  200 OK ★ 52ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       562µs   149µs        236µs           49ms             1ms        26µs      53ms  

  ✓  Successful  request

Iteration 33/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1  
  200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       193µs   (cache)      (cache)         4ms              1ms        30µs      6ms   

  ✓  Successful  request

Iteration 34/82

→ management request
  POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2  
  200 OK ★ 1479ms time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B
  │ {
  │   "status": "Model \"resnet152\" Version: 1.0 register
  │ ed with 0 initial workers. Use scale workers API to ad
  │ d workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total  
  1ms       282µs   (cache)      (cache)         1477ms           1ms        26µs      1480ms 

  ✓  Successful  request

Iteration 35/82

→ management request
  DELETE http://localhost:8081/models/resnet152  
  200 OK ★ 7ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet152\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       281µs   (cache)      (cache)         4ms              1ms        73µs      8ms   

  ✓  Successful  request

Iteration 36/82

→ management request
  POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1  
  200 OK ★ 6.2s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B
  │ {
  │   "status": "Model \"resnet152\" Version: 1.0 register
  │ ed with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       259µs   (cache)      (cache)         6.2s             1ms        29µs      6.2s  

  ✓  Successful  request

Iteration 37/82

→ management request
  DELETE http://localhost:8081/models/resnet152  
  200 OK ★ 19ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet152\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       183µs   (cache)      (cache)         17ms             1ms        30µs      19ms  

  ✓  Successful  request

Iteration 38/82

→ management request
  POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk  
  200 OK ★ 4.8s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B
  │ {
  │   "status": "Model \"resnet152\" Version: 1.0 register
  │ ed with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       185µs   (cache)      (cache)         4.8s             3ms        58µs      4.8s  

  ✓  Successful  request

Iteration 39/82

→ management request
  DELETE http://localhost:8081/models/resnet152  
  200 OK ★ 40ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet152\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       354µs   (cache)      (cache)         38ms             1ms        36µs      41ms  

  ✓  Successful  request

Iteration 40/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1  
  200 OK ★ 41ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       253µs   (cache)      (cache)         38ms             1ms        29µs      41ms  

  ✓  Successful  request

Iteration 41/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1  
  200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  979µs     186µs   (cache)      (cache)         3ms              1ms        26µs      6ms   

  ✓  Successful  request

Iteration 42/82

→ management request
  POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true  
  200 OK ★ 288ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B
  │ {
  │   "status": "Model \"resnet-18\" Version: 1.0 register
  │ ed with 0 initial workers. Use scale workers API to ad
  │ d workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       170µs   (cache)      (cache)         285ms            1ms        29µs      288ms 

  ✓  Successful  request

Iteration 43/82

→ management request
  DELETE http://localhost:8081/models/resnet-18  
  200 OK ★ 6ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet-18\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       176µs   (cache)      (cache)         3ms              1ms        33µs      6ms   

  ✓  Successful  request

Iteration 44/82

→ management request
  POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1  
  200 OK ★ 319ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B
  │ {
  │   "status": "Model \"resnet-18\" Version: 1.0 register
  │ ed with 0 initial workers. Use scale workers API to ad
  │ d workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       198µs   (cache)      (cache)         317ms            1ms        27µs      319ms 

  ✓  Successful  request

Iteration 45/82

→ management request
  DELETE http://localhost:8081/models/resnet-18  
  200 OK ★ 6ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet-18\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       528µs   (cache)      (cache)         4ms              1ms        32µs      6ms   

  ✓  Successful  request

Iteration 46/82

→ management request
  POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false  
  200 OK ★ 393ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B
  │ {
  │   "status": "Model \"resnet-18\" Version: 1.0 register
  │ ed with 0 initial workers. Use scale workers API to ad
  │ d workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       173µs   (cache)      (cache)         390ms            989µs      30µs      393ms 

  ✓  Successful  request

Iteration 47/82

→ management request
  GET http://localhost:8081/models?limit=1  
  200 OK ★ 3ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B
  │ {
  │   "nextPageToken": "1",
  │   "models": [
  │     {
  │       "modelName": "resnet-18",
  │       "modelUrl": "resnet-18.mar"
  │     }
  │   ]
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  857µs     161µs   (cache)      (cache)         1ms              895µs      24µs      3ms   

  ✓  Successful  request

Iteration 48/82

→ management request
  GET http://localhost:8081/models?limit=-1  
  200 OK ★ 3ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B
  │ {
  │   "models": [
  │     {
  │       "modelName": "resnet-18",
  │       "modelUrl": "resnet-18.mar"
  │     }
  │   ]
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       228µs   (cache)      (cache)         1ms              1ms        29µs      3ms   

  ✓  Successful  request

Iteration 49/82

→ management request
  GET http://localhost:8081/models?limit=1&next_page_token=1  
  200 OK ★ 3ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B
  │ {
  │   "models": []
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       171µs   (cache)      (cache)         1ms              1ms        33µs      3ms   

  ✓  Successful  request

Iteration 50/82

→ management request
  GET http://localhost:8081/models?limit=1&next_page_token=-1  
  200 OK ★ 4ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B
  │ {
  │   "nextPageToken": "1",
  │   "models": [
  │     {
  │       "modelName": "resnet-18",
  │       "modelUrl": "resnet-18.mar"
  │     }
  │   ]
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       183µs   (cache)      (cache)         1ms              1ms        32µs      4ms   

  ✓  Successful  request

Iteration 51/82

→ management request
  PUT http://localhost:8081/models/resnet-18?number_gpu=10  
  202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       207µs   (cache)      (cache)         1ms              1ms        28µs      3ms   

  ✓  Successful  request

Iteration 52/82

→ management request
  PUT http://localhost:8081/models/resnet-18?number_gpu=-1  
  202 Accepted ★ 4ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       174µs   (cache)      (cache)         1ms              1ms        34µs      4ms   

  ✓  Successful  request

Iteration 53/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true  
  200 OK ★ 4ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B
  │ {
  │   "status": "Workers scaled to 1 for model: resnet-18"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       200µs   (cache)      (cache)         1ms              1ms        39µs      4ms   

  ✓  Successful  request

Iteration 54/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false  
  202 Accepted ★ 3ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       175µs   (cache)      (cache)         1ms              1ms        28µs      3ms   

  ✓  Successful  request

Iteration 55/82

→ management request
  PUT http://localhost:8081/models/resnet-18?timeout=-1  
  202 Accepted ★ 3ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       200µs   (cache)      (cache)         1ms              1ms        30µs      3ms   

  ✓  Successful  request

Iteration 56/82

→ management request
  PUT http://localhost:8081/models/resnet-18?timeout=0  
  202 Accepted ★ 3ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       178µs   (cache)      (cache)         1ms              1ms        36µs      4ms   

  ✓  Successful  request

Iteration 57/82

→ management request
  POST http://localhost:8081/models?url=&model_name=resnet-18  
  404 Not Found ★ 3ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "empty url"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       190µs   (cache)      (cache)         1ms              1ms        27µs      3ms   

  ✓  Successful  request

Iteration 58/82

→ management request
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18  
  400 Bad Request ★ 845ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B
  │ {
  │   "code": 400,
  │   "type": "DownloadArchiveException",
  │   "message": "Failed to download archive from: https:/
  │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma
  │ r"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       567µs   187µs        222µs           842ms            1ms        30µs      845ms 

  ✓  Successful  request

Iteration 59/82

→ management request
  GET http://localhost:8081/models/invalid_squeezenet1_1  
  404 Not Found ★ 13ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: invalid_squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  989µs     487µs   621µs        285µs           10ms             1ms        27µs      13ms  

  ✓  Successful  request

Iteration 60/82

→ management request
  GET http://localhost:8081/models/squeezenet1_1/0.0  
  404 Not Found ★ 14ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       425µs   130µs        218µs           11ms             1ms        28µs      14ms  

  ✓  Successful  request

Iteration 61/82

→ management request
  GET http://localhost:8081/models?next_page_token=12  
  200 OK ★ 14ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B
  │ {
  │   "models": []
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       533µs   154µs        262µs           11ms             1ms        32µs      14ms  

  ✓  Successful  request

Iteration 62/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan  
  202 Accepted ★ 3ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       179µs   (cache)      (cache)         1ms              1ms        30µs      3ms   

  ✓  Successful  request

Iteration 63/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan  
  202 Accepted ★ 3ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       188µs   (cache)      (cache)         1ms              1ms        38µs      4ms   

  ✓  Successful  request

Iteration 64/82

→ management request
  PUT http://localhost:8081/models/resnet-18  
  202 Accepted ★ 3ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       171µs   (cache)      (cache)         1ms              1ms        30µs      3ms   

  ✓  Successful  request

Iteration 65/82

→ management request
  PUT http://localhost:8081/models/resnet181?min_worker=1  
  404 Not Found ★ 3ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: resnet181"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       195µs   (cache)      (cache)         1ms              1ms        33µs      4ms   

  ✓  Successful  request

Iteration 66/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1  
  400 Bad Request ★ 13ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B
  │ {
  │   "code": 400,
  │   "type": "BadRequestException",
  │   "message": "max_worker cannot be less than min_worke
  │ r."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       473µs   151µs        225µs           10ms             1ms        28µs      15ms  

  ✓  Successful  request

Iteration 67/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=1  
  202 Accepted ★ 13ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       443µs   136µs        227µs           10ms             1ms        31µs      14ms  

  ✓  Successful  request

Iteration 68/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=0  
  202 Accepted ★ 18ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       186µs   (cache)      (cache)         15ms             1ms        35µs      18ms  

  ✓  Successful  request

Iteration 69/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=-1  
  500 Internal Server Error ★ 4ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B
  │ {
  │   "code": 500,
  │   "type": "IndexOutOfBoundsException",
  │   "message": "Index -1 out of bounds for length 0"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       171µs   (cache)      (cache)         2ms              1ms        34µs      4ms   

  ✓  Successful  request

Iteration 70/82

→ management request
  PUT http://localhost:8081/models/resnet-18?max_worker=-1  
  400 Bad Request ★ 13ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B
  │ {
  │   "code": 400,
  │   "type": "BadRequestException",
  │   "message": "max_worker cannot be less than min_worke
  │ r."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       492µs   145µs        197µs           10ms             1ms        31µs      13ms  

  ✓  Successful  request

Iteration 71/82

→ management request
  PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default  
  404 Not Found ★ 14ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: invalid_squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       511µs   188µs        244µs           11ms             1ms        30µs      14ms  

  ✓  Successful  request

Iteration 72/82

→ management request
  DELETE http://localhost:8081/models/resnet-18  
  200 OK ★ 17ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet-18\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       501µs   142µs        200µs           14ms             1ms        29µs      17ms  

  ✓  Successful  request

Iteration 73/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1/0.0  
  404 Not Found ★ 4ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       190µs   (cache)      (cache)         2ms              1ms        29µs      5ms   

  ✓  Successful  request

Iteration 74/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1  
  200 OK ★ 44ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       467µs   138µs        212µs           42ms             1ms        25µs      44ms  

  ✓  Successful  request

Iteration 75/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true  
  200 OK ★ 6ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  968µs     155µs   (cache)      (cache)         3ms              1ms        27µs      6ms   

  ✓  Successful  request

Iteration 76/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1  
  200 OK ★ 42ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       207µs   (cache)      (cache)         40ms             1ms        28µs      42ms  

  ✓  Successful  request

Iteration 77/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan  
  200 OK ★ 6ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       170µs   (cache)      (cache)         3ms              1ms        27µs      6ms   

  ✓  Successful  request

Iteration 78/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1  
  200 OK ★ 40ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       160µs   (cache)      (cache)         38ms             1ms        26µs      40ms  

  ✓  Successful  request

Iteration 79/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true  
  200 OK ★ 6ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       167µs   (cache)      (cache)         3ms              1ms        32µs      6ms   

  ✓  Successful  request

Iteration 80/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1  
  200 OK ★ 44ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       179µs   (cache)      (cache)         42ms             1ms        33µs      44ms  

  ✓  Successful  request

Iteration 81/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1  
  200 OK ★ 8ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       216µs   (cache)      (cache)         3ms              3ms        30µs      8ms   

  ✓  Successful  request

Iteration 82/82

→ management request
  DELETE http://localhost:8081/models/invalid_squeezenet1_1  
  404 Not Found ★ 3ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: invalid_squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       166µs   (cache)      (cache)         1ms              1ms        31µs      3ms   

  ✓  Successful  request

┌─────────────────────────┬─────────────────────┬────────────────────┐
│                         │            executed │             failed │
├─────────────────────────┼─────────────────────┼────────────────────┤
│              iterations │                  82 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│                requests │                  82 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│            test-scripts │                  82 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│      prerequest-scripts │                   0 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│              assertions │                  82 │                  0 │
├─────────────────────────┴─────────────────────┴────────────────────┤
│ total run duration: 22.1s                                          │
├────────────────────────────────────────────────────────────────────┤
│ total data received: 7.75kB (approx)                               │
├────────────────────────────────────────────────────────────────────┤
│ average response time: 250ms [min: 3ms, max: 6.2s, s.d.: 921ms]    │
├────────────────────────────────────────────────────────────────────┤
│ average DNS lookup time: 941µs [min: 116µs, max: 1ms, s.d.: 843µs] │
├────────────────────────────────────────────────────────────────────┤
│ average first byte time: 248ms [min: 1ms, max: 6.2s, s.d.: 921ms]  │
└────────────────────────────────────────────────────────────────────┘
newman: could not find "htmlextra" reporter
  ensure that the reporter is installed in the same directory as newman
  please install reporter using npm

newman

inference

Iteration 1/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_v4.mar&model_name=my_text_classifier&initial_workers=1&synchronous=true  
  200 OK ★ 1m 41.8s time ★ 388B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B
  │ {
  │   "status": "Model \"my_text_classifier\" Version: 1.0
  │  registered with 1 initial workers"
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total    
  33ms      5ms    1ms          323µs           1m 41.8s         7ms        454µs     1m 41.8s 

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/my_text_classifier  
  200 OK ★ 66ms time ★ 353B↑ 372B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 76B
  │ 
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 137B
  │ {
  │   "World": 0.02911965176463127,
  │   "Sports": 2.943157232948579e-05,
  │   "Business": 0.9074352383613586,
  │   "Sci/Tec": 0.06341569125652313
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total 
  8ms       1ms    329µs        479µs           61ms             2ms        49µs      74ms  

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/my_text_classifier  
  200 OK ★ 32ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B
  │ {
  │   "status": "Model \"my_text_classifier\" unregistered
  │ "
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       615µs   (cache)      (cache)         28ms             1ms        584µs     33ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 21ms time ★ 233B↑ 4.08kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.79kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 61.7
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com",} 8.439990997314453
  │ # HELP WorkerLoadTime Torchserve prometheus gauge metr
  │ ic with unit: Milliseconds
  │ # TYPE WorkerLoadTime gauge
  │ WorkerLoadTime{WorkerName="W-9000-my_text_classifier_1
  │ .0",Level="Host",Hostname="147dda19895c.ant.amazon.com
  │ ",} 4347.0
  │ # HELP MemoryUsed Torchserve prometheus gauge metric w
  │ ith unit: Megabytes
  │ # TYPE MemoryUsed gauge
  │ MemoryUsed{Level="Host",Hostname="147dda19895c.ant.ama
  │ zon.com",} 9492.06640625
  │ # HELP PredictionTime Torchserve prometheus gauge metr
  │ ic with unit: ms
  │ # TYPE PredictionTime gauge
  │ PredictionTime{ModelName="my_text_classifier",Level="M
  │ odel",Hostname="147dda19895c.ant.amazon.com",} 39.99
  │ # HELP MemoryAvailable Torchserve prometheus gauge met
  │ ric with unit: Megabytes
  │ # TYPE MemoryAvailable gauge
  │ MemoryAvailable{Level="Host",Hostname="147dda19895c.an
  │ t.amazon.com",} 6267.28125
  │ # HELP Requests2XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests2XX counter
  │ Requests2XX{Level="Host",Hostname="147dda19895c.ant.am
  │ azon.com",} 3.0
  │ # HELP GPUMemoryUsed Torchserve prometheus gauge metri
  │ c with unit: Megabytes
  │ # TYPE GPUMemoryUsed gauge
  │ # HELP DiskAvailable Torchserve 
  │ (showing 2.05kB/3.79kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       618µs   178µs        221µs           18ms             1ms        78µs      23ms  

  ✓  Successful GET request

Iteration 2/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_scripted_v3.mar&model_name=my_text_classifier_scripted&initial_workers=1&synchronous=true  
  200 OK ★ 1m 54.7s time ★ 406B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 103B
  │ {
  │   "status": "Model \"my_text_classifier_scripted\" Ver
  │ sion: 1.0 registered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total    
  1ms       258µs   (cache)      (cache)         1m 54.7s         1ms        54µs      1m 54.7s 

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/my_text_classifier_scripted  
  200 OK ★ 76ms time ★ 362B↑ 372B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 76B
  │ 
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 137B
  │ {
  │   "World": 0.04559723287820816,
  │   "Sports": 0.0003771767660509795,
  │   "Business": 0.08623101562261581,
  │   "Sci/Tec": 0.8677946329116821
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       347µs   (cache)      (cache)         72ms             2ms        45µs      77ms  

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/my_text_classifier_scripted  
  200 OK ★ 22ms time ★ 263B↑ 335B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 69B
  │ {
  │   "status": "Model \"my_text_classifier_scripted\" unr
  │ egistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       256µs   (cache)      (cache)         19ms             1ms        49µs      23ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 6ms time ★ 233B↑ 4.87kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.58kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 62.5
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com",} 8.439990997314453
  │ # HELP WorkerLoadTime Torchserve prometheus gauge metr
  │ ic with unit: Milliseconds
  │ # TYPE WorkerLoadTime gauge
  │ WorkerLoadTime{WorkerName="W-9000-my_text_classifier_1
  │ .0",Level="Host",Hostname="147dda19895c.ant.amazon.com
  │ ",} 4347.0
  │ WorkerLoadTime{WorkerName="W-9001-my_text_classifier_s
  │ cripted_1.0",Level="Host",Hostname="147dda19895c.ant.a
  │ mazon.com",} 3871.0
  │ # HELP MemoryUsed Torchserve prometheus gauge metric w
  │ ith unit: Megabytes
  │ # TYPE MemoryUsed gauge
  │ MemoryUsed{Level="Host",Hostname="147dda19895c.ant.ama
  │ zon.com",} 9339.265625
  │ # HELP PredictionTime Torchserve prometheus gauge metr
  │ ic with unit: ms
  │ # TYPE PredictionTime gauge
  │ PredictionTime{ModelName="my_text_classifier",Level="M
  │ odel",Hostname="147dda19895c.ant.amazon.com",} 39.99
  │ PredictionTime{ModelName="my_text_classifier_scripted"
  │ ,Level="Model",Hostname="147dda19895c.ant.amazon.com",
  │ } 69.21
  │ # HELP MemoryAvailable Torchserve prometheus gauge met
  │ ric with unit: Megabytes
  │ # TYPE MemoryAvailable gauge
  │ MemoryA
  │ (showing 2.05kB/4.58kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       296µs   (cache)      (cache)         2ms              2ms        48µs      6ms   

  ✓  Successful GET request

Iteration 3/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true  
  200 OK ★ 2.7s time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       237µs   (cache)      (cache)         2.7s             1ms        36µs      2.7s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/squeezenet1_1  
  200 OK ★ 133ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B
  │ {
  │   "tabby": 0.27850738167762756,
  │   "lynx": 0.25299158692359924,
  │   "tiger_cat": 0.24496474862098694,
  │   "Egyptian_cat": 0.21722468733787537,
  │   "cougar": 0.002217544475570321
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       364µs   (cache)      (cache)         130ms            1ms        31µs      135ms 

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/squeezenet1_1  
  200 OK ★ 19ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       276µs   (cache)      (cache)         16ms             1ms        34µs      19ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 5ms time ★ 233B↑ 5.58kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.29kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1"
  │ ,model_version="default",hostname="147dda19895c.ant.am
  │ azon.com",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 62.5
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com",} 8.439990997314453
  │ # HELP WorkerLoadTime Torchserve prometheus gauge metr
  │ ic with unit: Milliseconds
  │ # TYPE WorkerLoadTime gauge
  │ WorkerLoadTime{WorkerName="W-9000-my_text_classifier_1
  │ .0",Level="Host",Hostname="147dda19895c.ant.amazon.com
  │ ",} 4347.0
  │ WorkerLoadTime{WorkerName="W-9001-my_text_classifier_s
  │ cripted_1.0",Level="Host",Hostname="147dda19895c.ant.a
  │ mazon.com",} 3871.0
  │ WorkerLoadTime{WorkerName="W-9002-squeezenet1_1_1.0",L
  │ evel="Host",Hostname="147dda19895c.ant.amazon.com",} 2
  │ 684.0
  │ # HELP MemoryUsed Torchserve prometheus gauge metric w
  │ ith unit: Megabytes
  │ # TYPE MemoryUsed gauge
  │ MemoryUsed{Level="Host",Hostname="147dda19895c.ant.ama
  │ zon.com",} 9339.265625
  │ # HELP PredictionTime Torchserve prometheus gauge metr
  │ ic with unit: ms
  │ # TYPE PredictionTime gauge
  │ PredictionTime{ModelName="my_text_classifier",Level="M
  │ odel",Hostname="147dda19895c.ant.amazon.com",}
  │ (showing 2.05kB/5.29kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       218µs   (cache)      (cache)         2ms              1ms        35µs      5ms   

  ✓  Successful GET request

Iteration 4/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/squeezenet1_1_scripted.mar&model_name=squeezenet1_1_scripted&initial_workers=1&synchronous=true  
  200 OK ★ 7.5s time ★ 393B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 98B
  │ {
  │   "status": "Model \"squeezenet1_1_scripted\" Version:
  │  1.0 registered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       228µs   (cache)      (cache)         7.5s             1ms        26µs      7.5s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/squeezenet1_1_scripted  
  200 OK ★ 124ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B
  │ {
  │   "tabby": 0.27850738167762756,
  │   "lynx": 0.25299158692359924,
  │   "tiger_cat": 0.24496474862098694,
  │   "Egyptian_cat": 0.21722468733787537,
  │   "cougar": 0.002217544475570321
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       235µs   (cache)      (cache)         121ms            1ms        31µs      125ms 

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/squeezenet1_1_scripted  
  200 OK ★ 19ms time ★ 258B↑ 330B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 64B
  │ {
  │   "status": "Model \"squeezenet1_1_scripted\" unregist
  │ ered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       275µs   (cache)      (cache)         16ms             1ms        34µs      19ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 5ms time ★ 233B↑ 6.34kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.05kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1"
  │ ,model_version="default",hostname="147dda19895c.ant.am
  │ azon.com",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 62.5
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com",} 8.439990997314453
  │ # HELP WorkerLoadTime Torchserve prometheus gauge metr
  │ ic with unit: Milliseconds
  │ # TYPE WorkerLoadTime gauge
  │ WorkerLoadTime{WorkerName="W-9003-squeezenet1_1_script
  │ ed_1.0",Level="Host",Hostname="147dda19895c.ant.amazon
  │ .com",} 2784.0
  │ WorkerLoadTime{WorkerName="W-9000-my_text_classifier_1
  │ .0",Level="Host",Hostname="147dda19895c.ant.amazon.com
  │ ",} 4347.0
  │ WorkerLoadTime{WorkerName="W-9001-my_text_classifier_s
  │ cripted_1.0",Level="Host",Hostname="147dda19895c.ant.a
  │ mazon.com",} 3871.0
  │ WorkerLoadTime{WorkerName="W-9002-squeezenet1_1_1.0",L
  │ evel="Host",Hostname="147dda19895c.ant.amazon.com",} 2
  │ 684.0
  │ # HELP MemoryUsed Torchserve prometheus gauge metric w
  │ ith unit: Megabytes
  │ # TYPE MemoryUsed gauge
  │ MemoryUsed{Level="Ho
  │ (showing 2.05kB/6.05kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       644µs   (cache)      (cache)         2ms              1ms        38µs      5ms   

  ✓  Successful GET request

Iteration 5/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161&initial_workers=1&synchronous=true  
  200 OK ★ 3.8s time ★ 330B↑ 353B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 87B
  │ {
  │   "status": "Model \"densenet161\" Version: 1.0 regist
  │ ered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       222µs   (cache)      (cache)         3.8s             1ms        31µs      3.8s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/densenet161  
  200 OK ★ 455ms time ★ 111.24kB↑ 418B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 183B
  │ {
  │   "tabby": 0.46661895513534546,
  │   "tiger_cat": 0.46449047327041626,
  │   "Egyptian_cat": 0.06614057719707489,
  │   "lynx": 0.0012924439506605268,
  │   "plastic_bag": 0.00022909804829396307
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       252µs   (cache)      (cache)         452ms            1ms        28µs      456ms 

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/densenet161  
  200 OK ★ 20ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B
  │ {
  │   "status": "Model \"densenet161\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       211µs   (cache)      (cache)         17ms             1ms        37µs      20ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 4ms time ★ 233B↑ 7.03kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.74kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1"
  │ ,model_version="default",hostname="147dda19895c.ant.am
  │ azon.com",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 62.5
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com",} 8.439990997314453
  │ # HELP WorkerLoadTime Torchserve prometheus gauge metr
  │ ic with unit: Milliseconds
  │ # TYPE WorkerLoadTime gauge
  │ WorkerLoadTime{WorkerName="W-9003-squeezenet1_1_script
  │ ed_1.0",Level="Host",Hostname="147dda19895c.ant.amazon
  │ .com",} 2784.0
  │ WorkerLoadTime{WorkerName="W-9000-my_text_classifier_1
  │ .0",Level="Host",Hostname="147dda19895c.ant.amazon.com
  │ ",} 4347.0
  │ WorkerLoadTime{WorkerName="W-9001-my_text_classifier_s
  │ cripted_1.0",Level="Host",Hostname="147dda19895c.ant.a
  │ mazon.com",} 3871.0
  │ WorkerLoadTime{WorkerName="W-9002-squeezenet1_1_1.0",L
  │ evel="Host",Hostname="147dda19895c.ant.amazon.com",} 2
  │ 68
  │ (showing 2.05kB/6.74kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       230µs   (cache)      (cache)         2ms              1ms        41µs      5ms   

  ✓  Successful GET request

Iteration 6/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=alexnet.mar&model_name=alexnet&initial_workers=1&synchronous=true  
  200 OK ★ 4.9s time ★ 322B↑ 349B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 83B
  │ {
  │   "status": "Model \"alexnet\" Version: 1.0 registered
  │  with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       293µs   (cache)      (cache)         4.9s             1ms        34µs      4.9s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/alexnet  
  200 OK ★ 105ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B
  │ {
  │   "tabby": 0.31847333908081055,
  │   "tiger_cat": 0.25793954730033875,
  │   "Egyptian_cat": 0.24254852533340454,
  │   "lynx": 0.16879378259181976,
  │   "tiger": 0.006487946026027203
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       222µs   (cache)      (cache)         102ms            1ms        32µs      106ms 

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/alexnet  
  200 OK ★ 23ms time ★ 243B↑ 315B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 49B
  │ {
  │   "status": "Model \"alexnet\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       240µs   (cache)      (cache)         20ms             1ms        36µs      23ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 4ms time ★ 233B↑ 7.7kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 7.41kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1"
  │ ,model_version="default",hostname="147dda19895c.ant.am
  │ azon.com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 62.5
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com",} 8.439990997314453
  │ # HELP WorkerLoadTime Torchserve prometheus gauge metr
  │ ic with unit: Milliseconds
  │ # TYPE WorkerLoadTime gauge
  │ WorkerLoadTime{WorkerName="W-9003-squeezenet1_1_script
  │ ed_1.0",Level="Host",Hostname="147dda19895c.ant.amazon
  │ .com",} 2784.0
  │ WorkerLoadTime{WorkerName="W-9000-my_text_classifier_1
  │ .0",Level="Host",Hostname="147dda19895c.ant.amazon.com
  │ ",} 4347.0
  │ WorkerLoadTime{WorkerName="W-9001-my_text_classifier_s
  │ cripted_1.0",Level="Host",Hostname="147dda19895c.ant.a
  │ mazon.com",}
  │ (showing 2.05kB/7.41kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       222µs   (cache)      (cache)         1ms              1ms        39µs      5ms   

  ✓  Successful GET request

Iteration 7/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/alexnet_scripted.mar&model_name=alexnet_scripted&initial_workers=1&synchronous=true  
  200 OK ★ 2m 33.9s time ★ 381B↑ 358B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 92B
  │ {
  │   "status": "Model \"alexnet_scripted\" Version: 1.0 r
  │ egistered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total    
  1ms       251µs   (cache)      (cache)         2m 33.9s         3ms        41µs      2m 33.9s 

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/alexnet_scripted  
  200 OK ★ 163ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B
  │ {
  │   "tabby": 0.31847333908081055,
  │   "tiger_cat": 0.25793954730033875,
  │   "Egyptian_cat": 0.24254852533340454,
  │   "lynx": 0.16879378259181976,
  │   "tiger": 0.006487946026027203
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       283µs   (cache)      (cache)         160ms            1ms        33µs      164ms 

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/alexnet_scripted  
  200 OK ★ 26ms time ★ 252B↑ 324B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58B
  │ {
  │   "status": "Model \"alexnet_scripted\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       309µs   (cache)      (cache)         24ms             1ms        35µs      28ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 7ms time ★ 233B↑ 8.43kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.14kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1"
  │ ,model_version="default",hostname="147dda19895c.ant.am
  │ azon.com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 69.4
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com",} 8.439990997314453
  │ # HELP WorkerLoadTime Torchserve prometheus gauge metr
  │ ic with unit: Milliseconds
  │ # TYPE WorkerLoadTime gauge
  │ WorkerLoadTime{WorkerName="W-9003-squeezenet1_1_script
  │ ed_1.0",Level="Host",Hostname="147dda19895c.ant.amazon
  │ .com",} 2784.0
  │ WorkerLoadTime{WorkerName="W-9000-my_text_classifier_1
  │ .0",Level="Host",Hostname="147dda19895c.ant.amazon.com
  │ ",} 
  │ (showing 2.05kB/8.14kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       268µs   (cache)      (cache)         4ms              1ms        39µs      7ms   

  ✓  Successful GET request

Iteration 8/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&initial_workers=1&synchronous=true  
  200 OK ★ 3.1s time ★ 326B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B
  │ {
  │   "status": "Model \"resnet-18\" Version: 1.0 register
  │ ed with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       237µs   (cache)      (cache)         3.1s             1ms        31µs      3.1s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/resnet-18  
  200 OK ★ 107ms time ★ 111.24kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B
  │ {
  │   "tabby": 0.40966302156448364,
  │   "tiger_cat": 0.3467046916484833,
  │   "Egyptian_cat": 0.13002900779247284,
  │   "lynx": 0.023919543251395226,
  │   "bucket": 0.011532166972756386
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       216µs   (cache)      (cache)         105ms            1ms        27µs      109ms 

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/resnet-18  
  200 OK ★ 18ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet-18\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       235µs   (cache)      (cache)         16ms             1ms        35µs      19ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 5ms time ★ 233B↑ 9.11kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.82kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1"
  │ ,model_version="default",hostname="147dda19895c.ant.am
  │ azon.com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 69.4
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com",} 8.439990997314453
  │ # HELP WorkerLoadTime Torchserve prometheus gauge metr
  │ ic with unit: Milliseconds
  │ # TYPE WorkerLoadTime gauge
  │ WorkerLoadTime{WorkerName="W-9003-squeezenet1_1_script
  │ ed_1.0",Level="Host",Hostname="147dda19895c.ant.amazon
  │ .com",}
  │ (showing 2.05kB/8.82kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       241µs   (cache)      (cache)         2ms              1ms        36µs      5ms   

  ✓  Successful GET request

Iteration 9/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-18_scripted.mar&model_name=resnet-18_scripted&initial_workers=1&synchronous=true  
  200 OK ★ 22.7s time ★ 385B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B
  │ {
  │   "status": "Model \"resnet-18_scripted\" Version: 1.0
  │  registered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       203µs   (cache)      (cache)         22.7s            1ms        26µs      22.7s 

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/resnet-18_scripted  
  200 OK ★ 193ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B
  │ {
  │   "tabby": 0.40966302156448364,
  │   "tiger_cat": 0.3467046916484833,
  │   "Egyptian_cat": 0.13002900779247284,
  │   "lynx": 0.023919543251395226,
  │   "bucket": 0.011532166972756386
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       243µs   (cache)      (cache)         190ms            1ms        33µs      194ms 

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/resnet-18_scripted  
  200 OK ★ 19ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B
  │ {
  │   "status": "Model \"resnet-18_scripted\" unregistered
  │ "
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       206µs   (cache)      (cache)         16ms             2ms        42µs      19ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 6ms time ★ 233B↑ 9.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.55kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1"
  │ ,model_version="default",hostname="147dda19895c.ant.am
  │ azon.com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 69.4
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com",} 8.439990997314453
  │ # HELP WorkerLoadTime Torchserve prometheus gauge metr
  │ ic with unit: Milliseconds
  │ # TYPE WorkerL
  │ (showing 2.05kB/9.55kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       216µs   (cache)      (cache)         3ms              1ms        49µs      8ms   

  ✓  Successful GET request

Iteration 10/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=vgg16.mar&model_name=vgg16&initial_workers=1&synchronous=true  
  200 OK ★ 7.9s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B
  │ {
  │   "status": "Model \"vgg16\" Version: 1.0 registered w
  │ ith 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       244µs   (cache)      (cache)         7.9s             1ms        34µs      7.9s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/vgg16  
  200 OK ★ 417ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B
  │ {
  │   "tiger_cat": 0.44697263836860657,
  │   "tabby": 0.4408797323703766,
  │   "Egyptian_cat": 0.05904562026262283,
  │   "tiger": 0.020596399903297424,
  │   "lynx": 0.009934580884873867
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       256µs   (cache)      (cache)         414ms            1ms        29µs      418ms 

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/vgg16  
  200 OK ★ 20ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Model \"vgg16\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       243µs   (cache)      (cache)         18ms             1ms        32µs      21ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 7ms time ★ 233B↑ 10.5kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 10.21kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1"
  │ ,model_version="default",hostname="147dda19895c.ant.am
  │ azon.com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 71.4
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com"
  │ (showing 2.05kB/10.21kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       255µs   (cache)      (cache)         2ms              3ms        42µs      8ms   

  ✓  Successful GET request

Iteration 11/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/vgg16_scripted.mar&model_name=vgg16_scripted&initial_workers=1&synchronous=true  
  200 OK ★ 3m 24.3s time ★ 377B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B
  │ {
  │   "status": "Model \"vgg16_scripted\" Version: 1.0 reg
  │ istered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total    
  1ms       239µs   (cache)      (cache)         3m 24.3s         1ms        32µs      3m 24.3s 

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/vgg16_scripted  
  200 OK ★ 512ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B
  │ {
  │   "tiger_cat": 0.44697263836860657,
  │   "tabby": 0.4408797323703766,
  │   "Egyptian_cat": 0.05904562026262283,
  │   "tiger": 0.020596399903297424,
  │   "lynx": 0.009934580884873867
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       197µs   (cache)      (cache)         510ms            1ms        35µs      513ms 

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/vgg16_scripted  
  200 OK ★ 22ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B
  │ {
  │   "status": "Model \"vgg16_scripted\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       203µs   (cache)      (cache)         19ms             1ms        37µs      22ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 4ms time ★ 233B↑ 11.22kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 10.92kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="vgg16_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1"
  │ ,model_version="default",hostname="147dda19895c.ant.am
  │ azon.com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 50.2
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prom
  │ (showing 2.05kB/10.92kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       418µs   (cache)      (cache)         1ms              1ms        43µs      5ms   

  ✓  Successful GET request

Iteration 12/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_v2.mar&model_name=mnist&initial_workers=1&synchronous=true  
  200 OK ★ 5.6s time ★ 362B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B
  │ {
  │   "status": "Model \"mnist\" Version: 2.0 registered w
  │ ith 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       228µs   (cache)      (cache)         5.6s             1ms        33µs      5.6s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/mnist  
  200 OK ★ 25ms time ★ 537B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 272B
  │ 
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B
  │ 0
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  3ms       203µs   (cache)      (cache)         23ms             1ms        29µs      27ms  

  ✓  Successful POST request
  ✓  Test expected TEXT response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/mnist  
  200 OK ★ 20ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Model \"mnist\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       209µs   (cache)      (cache)         18ms             1ms        32µs      21ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 4ms time ★ 233B↑ 11.87kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 11.58kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="vgg16_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1"
  │ ,model_version="default",hostname="147dda19895c.ant.am
  │ azon.com",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 50.2
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5
  │ (showing 2.05kB/11.58kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       190µs   (cache)      (cache)         1ms              1ms        37µs      4ms   

  ✓  Successful GET request

Iteration 13/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_scripted_v2.mar&model_name=mnist_scripted&initial_workers=1&synchronous=true  
  200 OK ★ 4.7s time ★ 380B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B
  │ {
  │   "status": "Model \"mnist_scripted\" Version: 2.0 reg
  │ istered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       200µs   (cache)      (cache)         4.7s             1ms        30µs      4.7s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/mnist_scripted  
  200 OK ★ 76ms time ★ 546B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 272B
  │ 
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B
  │ 0
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       196µs   (cache)      (cache)         73ms             1ms        31µs      77ms  

  ✓  Successful POST request
  ✓  Test expected TEXT response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/mnist_scripted  
  200 OK ★ 27ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B
  │ {
  │   "status": "Model \"mnist_scripted\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       198µs   (cache)      (cache)         23ms             2ms        38µs      28ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 5ms time ★ 233B↑ 12.58kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 12.29kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="mnist_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="vgg16_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1"
  │ ,model_version="default",hostname="147dda19895c.ant.am
  │ azon.com",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com
  │ (showing 2.05kB/12.29kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       268µs   (cache)      (cache)         2ms              1ms        41µs      5ms   

  ✓  Successful GET request

Iteration 14/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=fastrcnn.mar&model_name=fastrcnn&initial_workers=1&synchronous=true  
  200 OK ★ 4.5s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B
  │ {
  │   "status": "Model \"fastrcnn\" Version: 1.0 registere
  │ d with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       180µs   (cache)      (cache)         4.5s             1ms        32µs      4.5s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/fastrcnn  
  200 OK ★ 5s time ★ 289.21kB↑ 3.01kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 288.94kB
  │ (showing 2.05kB/288.94kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.77kB
  │ [
  │   {
  │     "person": [
  │       167.39581298828125,
  │       57.203060150146484,
  │       301.3599853515625,
  │       436.7974548339844
  │     ],
  │     "score": 0.9995203018188477
  │   },
  │   {
  │     "person": [
  │       89.54701232910156,
  │       64.83567810058594,
  │       191.42428588867188,
  │       446.7608337402344
  │     ],
  │     "score": 0.9994966983795166
  │   },
  │   {
  │     "person": [
  │       362.3782653808594,
  │       161.84133911132812,
  │       515.5023193359375,
  │       385.28985595703125
  │     ],
  │     "score": 0.9977061748504639
  │   },
  │   {
  │     "handbag": [
  │       67.37623596191406,
  │       277.5755920410156,
  │       111.67391204833984,
  │       400.2067565917969
  │     ],
  │     "score": 0.9924989342689514
  │   },
  │   {
  │     "handbag": [
  │       228.6824951171875,
  │       146.0069580078125,
  │       303.55120849609375,
  │       231.0884552001953
  │     ],
  │     "score": 0.9922404289245605
  │   },
  │   {
  │     "handbag": [
  │       379.4114074707031,
  │       259.9577941894531,
  │       419.07977294921875,
  │       317.9610290527344
  │     ],
  │     "score": 0.9898614883422852
  │   },
  │   {
  │     "person": [
  │       518.4952392578125,
  │       149.730224609375,
  │       636.6343994140625,
  │       365.4129333496094
  │     ],
  │     "score": 0.9821106791496277
  │   },
  │   {
  │     "bench": [
  │       269.08184814453125,
  │       217.342041015625,
  │       423.77105712890625,
  │       390.37860107421875
  │     ],
  │     "score": 0.9573412537574768
  │   },
  │   {
  │     "person": [
  │       539.6298217773438,
  │       157.75868225097656,
  │       616.1533813476562,
  │       253.1112823486328
  │     ],
  │     "score": 0.8995411396026611
  │   },
  │   {
  │     "person": [
  │       477.0623779296875,
  │       147.80885314941406,
  │       610.7775268554688,
  │       296.837158203125
  │     ],
  │     "score": 0.8751554489135742
  │   },
  │   {
  │     "bench": [
  │       286.0874938964844,
  │       216.53285217285156,
  │       550.7698974609375,
  │       383.1822814941406
  │     ],
  │     "score": 0.843645453453064
  │   },
  │   {
  │     "person": [
  │       627.4741821289062,
  │       177.05838012695312,
  │       640.0,
  │       247.72152709960938
  │     ],
  │     "score": 0.8257319331169128
  │   },
  │   {
  │     "bench": [
  │       88.76394653320312,
  │       226.1649169921875,
  │       563.6145019531
  │ (showing 2.05kB/2.77kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       213µs   (cache)      (cache)         5s               1ms        30µs      5s    

  ✓  Successful POST request

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/fastrcnn  
  200 OK ★ 20ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B
  │ {
  │   "status": "Model \"fastrcnn\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       188µs   (cache)      (cache)         17ms             2ms        43µs      20ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 5ms time ★ 233B↑ 13.26kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 12.97kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="mnist_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="fastrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="vgg16_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1"
  │ ,model_version="default",hostname="147dda19895c.ant.am
  │ azon.com",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric
  │ (showing 2.05kB/12.97kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       220µs   (cache)      (cache)         2ms              1ms        49µs      5ms   

  ✓  Successful GET request

Iteration 15/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=maskrcnn.mar&model_name=maskrcnn&initial_workers=1&synchronous=true  
  200 OK ★ 4.1s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B
  │ {
  │   "status": "Model \"maskrcnn\" Version: 1.0 registere
  │ d with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       220µs   (cache)      (cache)         4.1s             1ms        30µs      4.1s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/maskrcnn  
  200 OK ★ 5.7s time ★ 289.21kB↑ 2.98kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 288.94kB
  │ (showing 2.05kB/288.94kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.75kB
  │ [
  │   {
  │     "person": [
  │       169.50636291503906,
  │       49.98552322387695,
  │       300.8945617675781,
  │       442.4544982910156
  │     ],
  │     "score": 0.9995434880256653
  │   },
  │   {
  │     "person": [
  │       90.4118423461914,
  │       66.79046630859375,
  │       194.19305419921875,
  │       437.2776794433594
  │     ],
  │     "score": 0.9993956089019775
  │   },
  │   {
  │     "person": [
  │       362.3172912597656,
  │       158.17352294921875,
  │       521.2589721679688,
  │       385.72265625
  │     ],
  │     "score": 0.9952278137207031
  │   },
  │   {
  │     "handbag": [
  │       68.57540893554688,
  │       279.31817626953125,
  │       111.15328979492188,
  │       400.9165954589844
  │     ],
  │     "score": 0.9938815832138062
  │   },
  │   {
  │     "person": [
  │       474.0157470703125,
  │       147.34793090820312,
  │       638.1209716796875,
  │       364.65087890625
  │     ],
  │     "score": 0.9897463321685791
  │   },
  │   {
  │     "handbag": [
  │       225.59584045410156,
  │       142.90000915527344,
  │       302.48638916015625,
  │       230.3284149169922
  │     ],
  │     "score": 0.9891214966773987
  │   },
  │   {
  │     "handbag": [
  │       380.2604675292969,
  │       259.2012023925781,
  │       419.5366516113281,
  │       318.27728271484375
  │     ],
  │     "score": 0.9688039422035217
  │   },
  │   {
  │     "bench": [
  │       273.4856872558594,
  │       217.48834228515625,
  │       441.0653991699219,
  │       396.24169921875
  │     ],
  │     "score": 0.961754560470581
  │   },
  │   {
  │     "person": [
  │       541.2896728515625,
  │       156.6611785888672,
  │       619.9386596679688,
  │       249.45326232910156
  │     ],
  │     "score": 0.8177028298377991
  │   },
  │   {
  │     "person": [
  │       362.9620361328125,
  │       163.89923095703125,
  │       500.7697448730469,
  │       293.91229248046875
  │     ],
  │     "score": 0.8016964793205261
  │   },
  │   {
  │     "chair": [
  │       455.20849609375,
  │       207.5401153564453,
  │       491.08526611328125,
  │       274.6475524902344
  │     ],
  │     "score": 0.7758325338363647
  │   },
  │   {
  │     "person": [
  │       549.1538696289062,
  │       177.42063903808594,
  │       640.0,
  │       364.5394592285156
  │     ],
  │     "score": 0.7176413536071777
  │   },
  │   {
  │     "person": [
  │       626.230712890625,
  │       178.6534423828125,
  │       640.0,
  │       246.146606
  │ (showing 2.05kB/2.75kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       179µs   (cache)      (cache)         5.7s             1ms        38µs      5.7s  

  ✓  Successful POST request

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/maskrcnn  
  200 OK ★ 19ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B
  │ {
  │   "status": "Model \"maskrcnn\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       206µs   (cache)      (cache)         17ms             1ms        34µs      20ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 4ms time ★ 233B↑ 13.94kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.64kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="maskrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="mnist_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="fastrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="vgg16_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1"
  │ ,model_ver
  │ (showing 2.05kB/13.64kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       188µs   (cache)      (cache)         2ms              1ms        45µs      4ms   

  ✓  Successful GET request

Iteration 16/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=fcn_resnet_101.mar&model_name=fcn_resnet_101&initial_workers=1&synchronous=true  
  200 OK ★ 4.6s time ★ 336B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B
  │ {
  │   "status": "Model \"fcn_resnet_101\" Version: 1.0 reg
  │ istered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       197µs   (cache)      (cache)         4.6s             1ms        32µs      4.6s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/fcn_resnet_101  
  200 OK ★ 1289ms time ★ 289.22kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 288.94kB
  │ (showing 2.05kB/288.94kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB
  │ [
  │   [
  │     [
  │       0.0,
  │       0.9993855953216553
  │     ],
  │     [
  │       0.0,
  │       0.9993855953216553
  │     ],
  │     [
  │       0.0,
  │       0.9993855953216553
  │     ],
  │     [
  │       0.0,
  │       0.9993855953216553
  │     ],
  │     [
  │       0.0,
  │       0.9993864297866821
  │     ],
  │     [
  │       0.0,
  │       0.9993854761123657
  │     ],
  │     [
  │       0.0,
  │       0.9993811845779419
  │     ],
  │     [
  │       0.0,
  │       0.9993742108345032
  │     ],
  │     [
  │       0.0,
  │       0.9993641972541809
  │     ],
  │     [
  │       0.0,
  │       0.9993515610694885
  │     ],
  │     [
  │       0.0,
  │       0.9993364214897156
  │     ],
  │     [
  │       0.0,
  │       0.9993187189102173
  │     ],
  │     [
  │       0.0,
  │       0.9992934465408325
  │     ],
  │     [
  │       0.0,
  │       0.9992607235908508
  │     ],
  │     [
  │       0.0,
  │       0.9992249011993408
  │     ],
  │     [
  │       0.0,
  │       0.9991866946220398
  │     ],
  │     [
  │       0.0,
  │       0.9991452693939209
  │     ],
  │     [
  │       0.0,
  │       0.9991005659103394
  │     ],
  │     [
  │       0.0,
  │       0.9990523457527161
  │     ],
  │     [
  │       0.0,
  │       0.9990004897117615
  │     ],
  │     [
  │       0.0,
  │       0.998968243598938
  │     ],
  │     [
  │       0.0,
  │       0.9989572763442993
  │     ],
  │     [
  │       0.0,
  │       0.9989456534385681
  │     ],
  │     [
  │       0.0,
  │       0.9989331364631653
  │     ],
  │     [
  │       0.0,
  │       0.9989200830459595
  │     ],
  │     [
  │       0.0,
  │       0.9989060163497925
  │     ],
  │     [
  │       0.0,
  │       0.998891294002533
  │     ],
  │     [
  │       0.0,
  │       0.9988754391670227
  │     ],
  │     [
  │       0.0,
  │       0.9988815188407898
  │     ],
  │     [
  │       0.0,
  │       0.9989089965820312
  │     ],
  │     [
  │       0.0,
  │       0.9989352822303772
  │     ],
  │     [
  │       0.0,
  │       0.9989607334136963
  │     ],
  │     [
  │       0.0,
  │       0.9989848732948303
  │     ],
  │     [
  │       0.0,
  │       0.9990076422691345
  │     ],
  │     [
  │       0.0,
  │       0.9990293979644775
  │     ],
  │     [
  │       0.0,
  │       0.9990498423576355
  │     ],
  │     [
  │       0.0,
  │       0.999051034450531
  │     ],
  │     [
  │       0.0,
  │       0.9990313053131104
  │     ],
  │     [
  │       0.0,
  │       0.9990096092224121
  │     ],
  │     [
  │       0.0,
  │       0.9989851117134094
  │     ],
  │     [
  │       0.0,
  │       0.9989587068557739
  │     ],
  │     [
  │       0.0,
  │       0.99892979860
  │ (showing 2.05kB/2.47MB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total  
  2ms       223µs   (cache)      (cache)         1285ms           3ms        1ms       1293ms 

  ✓  Successful POST request

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/fcn_resnet_101  
  200 OK ★ 21ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B
  │ {
  │   "status": "Model \"fcn_resnet_101\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  3ms       268µs   (cache)      (cache)         19ms             1ms        59µs      23ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 4ms time ★ 233B↑ 14.65kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 14.36kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="maskrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="fcn_resnet_101
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="mnist_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="fastrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="vgg16_scripted
  │ ",model_ve
  │ (showing 2.05kB/14.36kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       217µs   (cache)      (cache)         1ms              1ms        36µs      4ms   

  ✓  Successful GET request

Iteration 17/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/fcn_resnet_101_scripted.mar&model_name=fcn_resnet_101_scripted&initial_workers=1&synchronous=true  
  200 OK ★ 2m 16.9s time ★ 395B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 99B
  │ {
  │   "status": "Model \"fcn_resnet_101_scripted\" Version
  │ : 1.0 registered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total    
  1ms       208µs   (cache)      (cache)         2m 16.8s         4ms        44µs      2m 16.9s 

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/fcn_resnet_101_scripted  
  200 OK ★ 1596ms time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 288.94kB
  │ (showing 2.05kB/288.94kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB
  │ [
  │   [
  │     [
  │       0.0,
  │       0.9993855953216553
  │     ],
  │     [
  │       0.0,
  │       0.9993855953216553
  │     ],
  │     [
  │       0.0,
  │       0.9993855953216553
  │     ],
  │     [
  │       0.0,
  │       0.9993855953216553
  │     ],
  │     [
  │       0.0,
  │       0.9993864297866821
  │     ],
  │     [
  │       0.0,
  │       0.9993854761123657
  │     ],
  │     [
  │       0.0,
  │       0.9993811845779419
  │     ],
  │     [
  │       0.0,
  │       0.9993742108345032
  │     ],
  │     [
  │       0.0,
  │       0.9993641972541809
  │     ],
  │     [
  │       0.0,
  │       0.9993515610694885
  │     ],
  │     [
  │       0.0,
  │       0.9993364214897156
  │     ],
  │     [
  │       0.0,
  │       0.9993187189102173
  │     ],
  │     [
  │       0.0,
  │       0.9992934465408325
  │     ],
  │     [
  │       0.0,
  │       0.9992607235908508
  │     ],
  │     [
  │       0.0,
  │       0.9992249011993408
  │     ],
  │     [
  │       0.0,
  │       0.9991866946220398
  │     ],
  │     [
  │       0.0,
  │       0.9991452693939209
  │     ],
  │     [
  │       0.0,
  │       0.9991005659103394
  │     ],
  │     [
  │       0.0,
  │       0.9990523457527161
  │     ],
  │     [
  │       0.0,
  │       0.9990004897117615
  │     ],
  │     [
  │       0.0,
  │       0.998968243598938
  │     ],
  │     [
  │       0.0,
  │       0.9989572763442993
  │     ],
  │     [
  │       0.0,
  │       0.9989456534385681
  │     ],
  │     [
  │       0.0,
  │       0.9989331364631653
  │     ],
  │     [
  │       0.0,
  │       0.9989200830459595
  │     ],
  │     [
  │       0.0,
  │       0.9989060163497925
  │     ],
  │     [
  │       0.0,
  │       0.998891294002533
  │     ],
  │     [
  │       0.0,
  │       0.9988754391670227
  │     ],
  │     [
  │       0.0,
  │       0.9988815188407898
  │     ],
  │     [
  │       0.0,
  │       0.9989089965820312
  │     ],
  │     [
  │       0.0,
  │       0.9989352822303772
  │     ],
  │     [
  │       0.0,
  │       0.9989607334136963
  │     ],
  │     [
  │       0.0,
  │       0.9989848732948303
  │     ],
  │     [
  │       0.0,
  │       0.9990076422691345
  │     ],
  │     [
  │       0.0,
  │       0.9990293979644775
  │     ],
  │     [
  │       0.0,
  │       0.9990498423576355
  │     ],
  │     [
  │       0.0,
  │       0.999051034450531
  │     ],
  │     [
  │       0.0,
  │       0.9990313053131104
  │     ],
  │     [
  │       0.0,
  │       0.9990096092224121
  │     ],
  │     [
  │       0.0,
  │       0.9989851117134094
  │     ],
  │     [
  │       0.0,
  │       0.9989587068557739
  │     ],
  │     [
  │       0.0,
  │       0.99892979860
  │ (showing 2.05kB/2.47MB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total  
  6ms       556µs   (cache)      (cache)         1590ms           3ms        1ms       1603ms 

  ✓  Successful POST request

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/fcn_resnet_101_scripted  
  200 OK ★ 17ms time ★ 259B↑ 331B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 65B
  │ {
  │   "status": "Model \"fcn_resnet_101_scripted\" unregis
  │ tered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       328µs   (cache)      (cache)         15ms             1ms        66µs      19ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 4ms time ★ 233B↑ 15.43kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 15.13kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="maskrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="fcn_resnet_101
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="mnist_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="fastrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="fcn_resnet_101
  │ _scripted"
  │ (showing 2.05kB/15.13kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       200µs   (cache)      (cache)         2ms              1ms        38µs      4ms   

  ✓  Successful GET request

Iteration 18/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=deeplabv3_resnet_101_eager.mar&model_name=deeplabv3_resnet_101_eager&initial_workers=1&synchronous=true  
  200 OK ★ 4.8s time ★ 360B↑ 369B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B
  │ {
  │   "status": "Model \"deeplabv3_resnet_101_eager\" Vers
  │ ion: 1.0 registered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       214µs   (cache)      (cache)         4.8s             1ms        31µs      4.8s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/deeplabv3_resnet_101_eager  
  200 OK ★ 1432ms time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 288.94kB
  │ (showing 2.05kB/288.94kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB
  │ [
  │   [
  │     [
  │       0.0,
  │       0.9988765120506287
  │     ],
  │     [
  │       0.0,
  │       0.9988765120506287
  │     ],
  │     [
  │       0.0,
  │       0.9988765120506287
  │     ],
  │     [
  │       0.0,
  │       0.9988765120506287
  │     ],
  │     [
  │       0.0,
  │       0.9988669157028198
  │     ],
  │     [
  │       0.0,
  │       0.998843789100647
  │     ],
  │     [
  │       0.0,
  │       0.9988172650337219
  │     ],
  │     [
  │       0.0,
  │       0.9987861514091492
  │     ],
  │     [
  │       0.0,
  │       0.9987504482269287
  │     ],
  │     [
  │       0.0,
  │       0.9987117052078247
  │     ],
  │     [
  │       0.0,
  │       0.9986679553985596
  │     ],
  │     [
  │       0.0,
  │       0.9986202716827393
  │     ],
  │     [
  │       0.0,
  │       0.9985180497169495
  │     ],
  │     [
  │       0.0,
  │       0.9983478784561157
  │     ],
  │     [
  │       0.0,
  │       0.9981531500816345
  │     ],
  │     [
  │       0.0,
  │       0.9979324340820312
  │     ],
  │     [
  │       0.0,
  │       0.9976813793182373
  │     ],
  │     [
  │       0.0,
  │       0.9973964691162109
  │     ],
  │     [
  │       0.0,
  │       0.9970728158950806
  │     ],
  │     [
  │       0.0,
  │       0.9967058300971985
  │     ],
  │     [
  │       0.0,
  │       0.9964045286178589
  │     ],
  │     [
  │       0.0,
  │       0.9961955547332764
  │     ],
  │     [
  │       0.0,
  │       0.9959736466407776
  │     ],
  │     [
  │       0.0,
  │       0.9957382678985596
  │     ],
  │     [
  │       0.0,
  │       0.9954884648323059
  │     ],
  │     [
  │       0.0,
  │       0.9952237010002136
  │     ],
  │     [
  │       0.0,
  │       0.9949429631233215
  │     ],
  │     [
  │       0.0,
  │       0.9946452379226685
  │     ],
  │     [
  │       0.0,
  │       0.9945149421691895
  │     ],
  │     [
  │       0.0,
  │       0.9945646524429321
  │     ],
  │     [
  │       0.0,
  │       0.9946145415306091
  │     ],
  │     [
  │       0.0,
  │       0.9946632385253906
  │     ],
  │     [
  │       0.0,
  │       0.9947116374969482
  │     ],
  │     [
  │       0.0,
  │       0.9947587847709656
  │     ],
  │     [
  │       0.0,
  │       0.9948055148124695
  │     ],
  │     [
  │       0.0,
  │       0.9948518872261047
  │     ],
  │     [
  │       0.0,
  │       0.9948964715003967
  │     ],
  │     [
  │       0.0,
  │       0.9949400424957275
  │     ],
  │     [
  │       0.0,
  │       0.9949837923049927
  │     ],
  │     [
  │       0.0,
  │       0.9950267672538757
  │     ],
  │     [
  │       0.0,
  │       0.9950693845748901
  │     ],
  │     [
  │       0.0,
  │       0.995111525
  │ (showing 2.05kB/2.47MB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total  
  2ms       230µs   (cache)      (cache)         1428ms           3ms        1ms       1435ms 

  ✓  Successful POST request

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/deeplabv3_resnet_101_eager  
  200 OK ★ 21ms time ★ 262B↑ 334B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 68B
  │ {
  │   "status": "Model \"deeplabv3_resnet_101_eager\" unre
  │ gistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       296µs   (cache)      (cache)         19ms             1ms        61µs      23ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 5ms time ★ 233B↑ 16.21kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 15.92kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="maskrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="fcn_resnet_101
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="mnist_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="fastrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="deeplabv3_resn
  │ et_101_eag
  │ (showing 2.05kB/15.92kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       222µs   (cache)      (cache)         2ms              1ms        46µs      6ms   

  ✓  Successful GET request

Iteration 19/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.s3.amazonaws.com/mar_files/deeplabv3_resnet_101_scripted.mar&model_name=deeplabv3_resnet_101_scripted&initial_workers=1&synchronous=true  
  200 OK ★ 2m 9.6s time ★ 412B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B
  │ {
  │   "status": "Model \"deeplabv3_resnet_101_scripted\" V
  │ ersion: 1.0 registered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total   
  1ms       198µs   (cache)      (cache)         2m 9.6s          2ms        47µs      2m 9.6s 

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/deeplabv3_resnet_101_scripted  
  200 OK ★ 1843ms time ★ 289.24kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 288.94kB
  │ (showing 2.05kB/288.94kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB
  │ [
  │   [
  │     [
  │       0.0,
  │       0.9988765120506287
  │     ],
  │     [
  │       0.0,
  │       0.9988765120506287
  │     ],
  │     [
  │       0.0,
  │       0.9988765120506287
  │     ],
  │     [
  │       0.0,
  │       0.9988765120506287
  │     ],
  │     [
  │       0.0,
  │       0.9988669157028198
  │     ],
  │     [
  │       0.0,
  │       0.998843789100647
  │     ],
  │     [
  │       0.0,
  │       0.9988172650337219
  │     ],
  │     [
  │       0.0,
  │       0.9987861514091492
  │     ],
  │     [
  │       0.0,
  │       0.9987504482269287
  │     ],
  │     [
  │       0.0,
  │       0.9987117052078247
  │     ],
  │     [
  │       0.0,
  │       0.9986679553985596
  │     ],
  │     [
  │       0.0,
  │       0.9986202716827393
  │     ],
  │     [
  │       0.0,
  │       0.9985180497169495
  │     ],
  │     [
  │       0.0,
  │       0.9983478784561157
  │     ],
  │     [
  │       0.0,
  │       0.9981531500816345
  │     ],
  │     [
  │       0.0,
  │       0.9979324340820312
  │     ],
  │     [
  │       0.0,
  │       0.9976813793182373
  │     ],
  │     [
  │       0.0,
  │       0.9973964691162109
  │     ],
  │     [
  │       0.0,
  │       0.9970728158950806
  │     ],
  │     [
  │       0.0,
  │       0.9967058300971985
  │     ],
  │     [
  │       0.0,
  │       0.9964045286178589
  │     ],
  │     [
  │       0.0,
  │       0.9961955547332764
  │     ],
  │     [
  │       0.0,
  │       0.9959736466407776
  │     ],
  │     [
  │       0.0,
  │       0.9957382678985596
  │     ],
  │     [
  │       0.0,
  │       0.9954884648323059
  │     ],
  │     [
  │       0.0,
  │       0.9952237010002136
  │     ],
  │     [
  │       0.0,
  │       0.9949429631233215
  │     ],
  │     [
  │       0.0,
  │       0.9946452379226685
  │     ],
  │     [
  │       0.0,
  │       0.9945149421691895
  │     ],
  │     [
  │       0.0,
  │       0.9945646524429321
  │     ],
  │     [
  │       0.0,
  │       0.9946145415306091
  │     ],
  │     [
  │       0.0,
  │       0.9946632385253906
  │     ],
  │     [
  │       0.0,
  │       0.9947116374969482
  │     ],
  │     [
  │       0.0,
  │       0.9947587847709656
  │     ],
  │     [
  │       0.0,
  │       0.9948055148124695
  │     ],
  │     [
  │       0.0,
  │       0.9948518872261047
  │     ],
  │     [
  │       0.0,
  │       0.9948964715003967
  │     ],
  │     [
  │       0.0,
  │       0.9949400424957275
  │     ],
  │     [
  │       0.0,
  │       0.9949837923049927
  │     ],
  │     [
  │       0.0,
  │       0.9950267672538757
  │     ],
  │     [
  │       0.0,
  │       0.9950693845748901
  │     ],
  │     [
  │       0.0,
  │       0.995111525
  │ (showing 2.05kB/2.47MB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total  
  3ms       247µs   (cache)      (cache)         1837ms           3ms        1ms       1846ms 

  ✓  Successful POST request

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/deeplabv3_resnet_101_scripted  
  200 OK ★ 19ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B
  │ {
  │   "status": "Model \"deeplabv3_resnet_101_scripted\" u
  │ nregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       254µs   (cache)      (cache)         16ms             1ms        65µs      20ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 5ms time ★ 233B↑ 17.02kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 16.72kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="maskrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="fcn_resnet_101
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="mnist_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="fastrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="deeplabv3_resn
  │ et_101_eag
  │ (showing 2.05kB/16.72kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       209µs   (cache)      (cache)         2ms              1ms        44µs      5ms   

  ✓  Successful GET request

Iteration 20/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&initial_workers=1&synchronous=true  
  200 OK ★ 4.9s time ★ 333B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B
  │ {
  │   "status": "Model \"resnet152\" Version: 1.0 register
  │ ed with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       191µs   (cache)      (cache)         4.9s             1ms        32µs      4.9s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/resnet152  
  200 OK ★ 407ms time ★ 111.24kB↑ 411B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 176B
  │ {
  │   "tiger_cat": 0.5798617601394653,
  │   "tabby": 0.38344141840934753,
  │   "Egyptian_cat": 0.03421143442392349,
  │   "lynx": 0.0005819805082865059,
  │   "quilt": 0.00027331963065080345
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       190µs   (cache)      (cache)         405ms            1ms        136µs     409ms 

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/resnet152  
  200 OK ★ 21ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet152\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       208µs   (cache)      (cache)         18ms             1ms        40µs      21ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 4ms time ★ 233B↑ 17.7kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 17.41kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="maskrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="fcn_resnet_101
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="mnist_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="fastrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="deeplabv3_resn
  │ et_101_eag
  │ (showing 2.05kB/17.41kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       210µs   (cache)      (cache)         2ms              1ms        39µs      4ms   

  ✓  Successful GET request

Iteration 21/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-152-batch_scripted.mar&model_name=resnet-152-batch_scripted&initial_workers=1&synchronous=true  
  200 OK ★ 1m 35.7s time ★ 399B↑ 368B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B
  │ {
  │   "status": "Model \"resnet-152-batch_scripted\" Versi
  │ on: 1.0 registered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total    
  1ms       203µs   (cache)      (cache)         1m 35.7s         2ms        92µs      1m 35.7s 

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/resnet-152-batch_scripted  
  200 OK ★ 797ms time ★ 111.26kB↑ 411B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 176B
  │ {
  │   "tiger_cat": 0.5798617601394653,
  │   "tabby": 0.38344141840934753,
  │   "Egyptian_cat": 0.03421143442392349,
  │   "lynx": 0.0005819805082865059,
  │   "quilt": 0.00027331963065080345
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  3ms       292µs   (cache)      (cache)         794ms            1ms        32µs      799ms 

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/resnet-152-batch_scripted  
  200 OK ★ 33ms time ★ 261B↑ 333B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 67B
  │ {
  │   "status": "Model \"resnet-152-batch_scripted\" unreg
  │ istered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       427µs   (cache)      (cache)         30ms             1ms        41µs      34ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 10ms time ★ 233B↑ 18.48kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 18.19kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="maskrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-152-bat
  │ ch_scripted",model_version="default",hostname="147dda1
  │ 9895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="fcn_resnet_101
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="mnist_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="fastrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet16
  │ (showing 2.05kB/18.19kB)
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total 
  7ms       2ms    (cache)      (cache)         5ms              2ms        63µs      17ms  

  ✓  Successful GET request

Iteration 22/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/distill_bert_qa_eager.mar&model_name=distill_bert_qa_eager&initial_workers=1&synchronous=true  
  200 OK ★ 1m 32.1s time ★ 391B↑ 363B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B
  │ {
  │   "status": "Model \"distill_bert_qa_eager\" Version: 
  │ 1.0 registered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total    
  4ms       456µs   (cache)      (cache)         1m 32.1s         1ms        32µs      1m 32.1s 

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/distill_bert_qa_eager  
  200 OK ★ 257ms time ★ 359B↑ 247B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 79B
  │ 
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13B
  │ a nice puppet
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       214µs   (cache)      (cache)         255ms            1ms        30µs      259ms 

  ✓  Successful POST request

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/distill_bert_qa_eager  
  200 OK ★ 20ms time ★ 257B↑ 329B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B
  │ {
  │   "status": "Model \"distill_bert_qa_eager\" unregiste
  │ red"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       212µs   (cache)      (cache)         17ms             1ms        37µs      21ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 6ms time ★ 233B↑ 19.24kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 18.94kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="maskrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-152-bat
  │ ch_scripted",model_version="default",hostname="147dda1
  │ 9895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="fcn_resnet_101
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="mnist_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="fastrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="distill_be
  │ (showing 2.05kB/18.94kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       260µs   (cache)      (cache)         2ms              2ms        38µs      7ms   

  ✓  Successful GET request

Iteration 23/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_token_classification_no_torchscript.mar&model_name=bert_token_classification_no_torchscript&initial_workers=1&synchronous=true  
  200 OK ★ 2m 39.8s time ★ 429B↑ 383B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 116B
  │ {
  │   "status": "Model \"bert_token_classification_no_torc
  │ hscript\" Version: 1.0 registered with 1 initial worke
  │ rs"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total    
  1ms       248µs   (cache)      (cache)         2m 39.8s         2ms        32µs      2m 39.8s 

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/bert_token_classification_no_torchscript  
  200 OK ★ 379ms time ★ 375B↑ 5.45kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 76B
  │ 
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.22kB
  │ [
  │   [
  │     "[CLS]",
  │     "B-LOC"
  │   ],
  │   [
  │     "bloomberg",
  │     "O"
  │   ],
  │   [
  │     "has",
  │     "I-ORG"
  │   ],
  │   [
  │     "decided",
  │     "B-PER"
  │   ],
  │   [
  │     "to",
  │     "O"
  │   ],
  │   [
  │     "publish",
  │     "I-PER"
  │   ],
  │   [
  │     "a",
  │     "I-MISC"
  │   ],
  │   [
  │     "new",
  │     "B-PER"
  │   ],
  │   [
  │     "report",
  │     "O"
  │   ],
  │   [
  │     "on",
  │     "O"
  │   ],
  │   [
  │     "global",
  │     "B-PER"
  │   ],
  │   [
  │     "economic",
  │     "O"
  │   ],
  │   [
  │     "situation",
  │     "B-LOC"
  │   ],
  │   [
  │     ".",
  │     "B-PER"
  │   ],
  │   [
  │     "[SEP]",
  │     "B-PER"
  │   ],
  │   [
  │     "[PAD]",
  │     "I-ORG"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │     "B-MISC"
  │   ],
  │   [
  │     "[PAD]",
  │ (showing 2.05kB/5.22kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  3ms       253µs   (cache)      (cache)         377ms            1ms        36µs      381ms 

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/bert_token_classification_no_torchscript  
  200 OK ★ 22ms time ★ 276B↑ 348B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 82B
  │ {
  │   "status": "Model \"bert_token_classification_no_torc
  │ hscript\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       203µs   (cache)      (cache)         19ms             1ms        39µs      23ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 11ms time ★ 233B↑ 20.1kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 19.81kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="maskrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-152-bat
  │ ch_scripted",model_version="default",hostname="147dda1
  │ 9895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.amazon.c
  │ om",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18_scri
  │ pted",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="fcn_resnet_101
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-18",mod
  │ el_version="default",hostname="147dda19895c.ant.amazon
  │ .com",} 1.0
  │ ts_inference_requests_total{model_name="mnist_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="fastrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier",model_version="default",hostname="147dda19895c.a
  │ nt.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="distill_be
  │ (showing 2.05kB/19.81kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       219µs   (cache)      (cache)         9ms              1ms        42µs      12ms  

  ✓  Successful GET request

Iteration 24/24

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_seqc_without_torchscript.mar&model_name=bert_seqc_without_torchscript&initial_workers=1&synchronous=true  
  200 OK ★ 2m 49.9s time ★ 407B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B
  │ {
  │   "status": "Model \"bert_seqc_without_torchscript\" V
  │ ersion: 1.0 registered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total    
  1ms       190µs   (cache)      (cache)         2m 49.9s         1ms        40µs      2m 49.9s 

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/bert_seqc_without_torchscript  
  200 OK ★ 464ms time ★ 364B↑ 246B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 76B
  │ 
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 12B
  │ Not Accepted
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  4ms       248µs   (cache)      (cache)         462ms            1ms        33µs      468ms 

  ✓  Successful POST request
  ✓  Test expected TEXT response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/bert_seqc_without_torchscript  
  200 OK ★ 25ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B
  │ {
  │   "status": "Model \"bert_seqc_without_torchscript\" u
  │ nregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       213µs   (cache)      (cache)         22ms             1ms        40µs      25ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 37ms time ★ 233B↑ 20.91kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 20.61kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="alexnet_script
  │ ed",model_version="default",hostname="147dda19895c.ant
  │ .amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="maskrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="resnet-152-bat
  │ ch_scripted",model_version="default",hostname="147dda1
  │ 9895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="vgg16",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ ts_inference_requests_total{model_name="fastrcnn",mode
  │ l_version="default",hostname="147dda19895c.ant.amazon.
  │ com",} 1.0
  │ ts_inference_requests_total{model_name="squeezenet1_1_
  │ scripted",model_version="default",hostname="147dda1989
  │ 5c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="my_text_classi
  │ fier_scripted",model_version="default",hostname="147dd
  │ a19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="densenet161",m
  │ odel_version="default",hostname="147dda19895c.ant.amaz
  │ on.com",} 1.0
  │ ts_inference_requests_total{model_name="fcn_resnet_101
  │ _scripted",model_version="default",hostname="147dda198
  │ 95c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="vgg16_scripted
  │ ",model_version="default",hostname="147dda19895c.ant.a
  │ mazon.com",} 1.0
  │ ts_inference_requests_total{model_name="deeplabv3_resn
  │ et_101_scripted",model_version="default",hostname="147
  │ dda19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="bert_token_cla
  │ ssification_no_torchscript",model_version="default",ho
  │ stname="147dda19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="bert_seqc_with
  │ out_torchscript",model_version="default",hostname="147
  │ dda19895c.ant.amazon.com",} 1.0
  │ ts_inference_requests_total{model_name="alexnet",model
  │ _version="default",hostname="147dda19895c.ant.
  │ (showing 2.05kB/20.61kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       301µs   (cache)      (cache)         34ms             1ms        50µs      38ms  

  ✓  Successful GET request

┌─────────────────────────┬──────────────────────┬──────────────────────┐
│                         │             executed │               failed │
├─────────────────────────┼──────────────────────┼──────────────────────┤
│              iterations │                   24 │                    0 │
├─────────────────────────┼──────────────────────┼──────────────────────┤
│                requests │                   96 │                    0 │
├─────────────────────────┼──────────────────────┼──────────────────────┤
│            test-scripts │                   96 │                    0 │
├─────────────────────────┼──────────────────────┼──────────────────────┤
│      prerequest-scripts │                    0 │                    0 │
├─────────────────────────┼──────────────────────┼──────────────────────┤
│              assertions │                  113 │                    0 │
├─────────────────────────┴──────────────────────┴──────────────────────┤
│ total run duration: 24m 31.2s                                         │
├───────────────────────────────────────────────────────────────────────┤
│ total data received: 10.2MB (approx)                                  │
├───────────────────────────────────────────────────────────────────────┤
│ average response time: 15.2s [min: 4ms, max: 3m 24.3s, s.d.: 42.7s]   │
├───────────────────────────────────────────────────────────────────────┤
│ average DNS lookup time: 731µs [min: 178µs, max: 1ms, s.d.: 599µs]    │
├───────────────────────────────────────────────────────────────────────┤
│ average first byte time: 15.2s [min: 1ms, max: 3m 24.3s, s.d.: 42.7s] │
└───────────────────────────────────────────────────────────────────────┘
newman: could not find "htmlextra" reporter
  ensure that the reporter is installed in the same directory as newman
  please install reporter using npm

newman

inference

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161_scripted.mar&model_name=densenet161_scripted&initial_workers=1&synchronous=true  
  200 OK ★ 43.9s time ★ 389B↑ 362B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 96B
  │ {
  │   "status": "Model \"densenet161_scripted\" Version: 1
  │ .0 registered with 1 initial workers"
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total 
  36ms      5ms    1ms          302µs           43.9s            7ms        408µs     43.9s 

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/predictions/densenet161_scripted  
  200 OK ★ 1205ms time ★ 111.25kB↑ 418B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 183B
  │ {
  │   "tabby": 0.46661895513534546,
  │   "tiger_cat": 0.46449047327041626,
  │   "Egyptian_cat": 0.06614057719707489,
  │   "lynx": 0.0012924439506605268,
  │   "plastic_bag": 0.00022909804829396307
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total  
  5ms       1ms    329µs        208µs           1199ms           2ms        66µs      1209ms 

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/densenet161_scripted  
  200 OK ★ 24ms time ★ 256B↑ 328B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 62B
  │ {
  │   "status": "Model \"densenet161_scripted\" unregister
  │ ed"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       494µs   (cache)      (cache)         20ms             1ms        90µs      24ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 23ms time ★ 233B↑ 4.1kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.81kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="densenet161_sc
  │ ripted",model_version="default",hostname="147dda19895c
  │ .ant.amazon.com",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 66.1
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com",} 8.439990997314453
  │ # HELP WorkerLoadTime Torchserve prometheus gauge metr
  │ ic with unit: Milliseconds
  │ # TYPE WorkerLoadTime gauge
  │ WorkerLoadTime{WorkerName="W-9000-densenet161_scripted
  │ _1.0",Level="Host",Hostname="147dda19895c.ant.amazon.c
  │ om",} 3351.0
  │ # HELP MemoryUsed Torchserve prometheus gauge metric w
  │ ith unit: Megabytes
  │ # TYPE MemoryUsed gauge
  │ MemoryUsed{Level="Host",Hostname="147dda19895c.ant.ama
  │ zon.com",} 7868.00390625
  │ # HELP PredictionTime Torchserve prometheus gauge metr
  │ ic with unit: ms
  │ # TYPE PredictionTime gauge
  │ PredictionTime{ModelName="densenet161_scripted",Level=
  │ "Model",Hostname="147dda19895c.ant.amazon.com",} 1177.
  │ 16
  │ # HELP MemoryAvailable Torchserve prometheus gauge met
  │ ric with unit: Megabytes
  │ # TYPE MemoryAvailable gauge
  │ MemoryAvailable{Level="Host",Hostname="147dda19895c.an
  │ t.amazon.com",} 5548.8203125
  │ # HELP Requests2XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests2XX counter
  │ Requests2XX{Level="Host",Hostname="147dda19895c.ant.am
  │ azon.com",} 3.0
  │ # HELP GPUMemoryUsed Torchserve prometheus gauge metri
  │ c with unit: Megabytes
  │ # TYPE GPUMemoryUsed gauge
  │ # HELP DiskAvailable T
  │ (showing 2.05kB/3.81kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       690µs   199µs        284µs           19ms             1ms        80µs      24ms  

  ✓  Successful GET request

┌─────────────────────────┬─────────────────────┬─────────────────────┐
│                         │            executed │              failed │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│              iterations │                   1 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│                requests │                   4 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│            test-scripts │                   4 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│      prerequest-scripts │                   0 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│              assertions │                   5 │                   0 │
├─────────────────────────┴─────────────────────┴─────────────────────┤
│ total run duration: 45.3s                                           │
├─────────────────────────────────────────────────────────────────────┤
│ total data received: 4.15kB (approx)                                │
├─────────────────────────────────────────────────────────────────────┤
│ average response time: 11.3s [min: 23ms, max: 43.9s, s.d.: 18.8s]   │
├─────────────────────────────────────────────────────────────────────┤
│ average DNS lookup time: 734µs [min: 199µs, max: 1ms, s.d.: 592µs]  │
├─────────────────────────────────────────────────────────────────────┤
│ average first byte time: 11.2s [min: 19ms, max: 43.9s, s.d.: 18.8s] │
└─────────────────────────────────────────────────────────────────────┘
newman: could not find "htmlextra" reporter
  ensure that the reporter is installed in the same directory as newman
  please install reporter using npm

newman

https_test_collection

→ HTTPS Inference API  Description
  OPTIONS https://localhost:8443  
  200 OK ★ 151ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB
  │ {
  │   "openapi": "3.0.1",
  │   "info": {
  │     "title": "TorchServe APIs",
  │     "description": "TorchServe is a flexible and easy 
  │ to use tool for serving deep learning models",
  │     "version": "0.8.1"
  │   },
  │   "paths": {
  │     "/": {
  │       "options": {
  │         "description": "Get openapi description.",
  │         "operationId": "apiDescription",
  │         "parameters": [],
  │         "responses": {
  │           "200": {
  │             "description": "A openapi 3.0.1 descriptor
  │ ",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "openapi",
  │                     "info",
  │                     "paths"
  │                   ],
  │                   "properties": {
  │                     "openapi": {
  │                       "type": "string"
  │                     },
  │                     "info": {
  │                       "type": "object"
  │                     },
  │                     "paths": {
  │                       "type": "object"
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           },
  │           "500": {
  │             "description": "Internal Server Error",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "code",
  │                     "type",
  │                     "message"
  │                   ],
  │                   "properties": {
  │                     "code": {
  │                       "type": "integer",
  │                       "description": "Error code."
  │                     },
  │                     "type": {
  │                       "type": "string",
  │                       "description": "Error type."
  │                     },
  │                     "message": {
  │                       "type": "string",
  │                       "description": "Error message."
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           }
  │         }
  │       }
  │     },
  │     "/ping": {
  │      
  │ (showing 2.05kB/23.41kB)
  └
  prepare   wait   dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  28ms      8ms    1ms          2ms             103ms           28ms             6ms        356µs     178ms 

  ✓  Status code is 200

→ HTTPS Management API Description
  OPTIONS https://localhost:8444  
  200 OK ★ 57ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB
  │ {
  │   "openapi": "3.0.1",
  │   "info": {
  │     "title": "TorchServe APIs",
  │     "description": "TorchServe is a flexible and easy 
  │ to use tool for serving deep learning models",
  │     "version": "0.8.1"
  │   },
  │   "paths": {
  │     "/": {
  │       "options": {
  │         "description": "Get openapi description.",
  │         "operationId": "apiDescription",
  │         "parameters": [],
  │         "responses": {
  │           "200": {
  │             "description": "A openapi 3.0.1 descriptor
  │ ",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "openapi",
  │                     "info",
  │                     "paths"
  │                   ],
  │                   "properties": {
  │                     "openapi": {
  │                       "type": "string"
  │                     },
  │                     "info": {
  │                       "type": "object"
  │                     },
  │                     "paths": {
  │                       "type": "object"
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           },
  │           "500": {
  │             "description": "Internal Server Error",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "code",
  │                     "type",
  │                     "message"
  │                   ],
  │                   "properties": {
  │                     "code": {
  │                       "type": "integer",
  │                       "description": "Error code."
  │                     },
  │                     "type": {
  │                       "type": "string",
  │                       "description": "Error type."
  │                     },
  │                     "message": {
  │                       "type": "string",
  │                       "description": "Error message."
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           }
  │         }
  │       }
  │     },
  │     "/models": {
  │    
  │ (showing 2.05kB/58.58kB)
  └
  prepare   wait   dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       1ms    179µs        348µs           38ms            13ms             1ms        71µs      57ms  

  ✓  Status code is 200

→ HTTPS Metrics API Description
  OPTIONS https://localhost:8445  
  200 OK ★ 40ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB
  │ {
  │   "openapi": "3.0.1",
  │   "info": {
  │     "title": "TorchServe APIs",
  │     "description": "TorchServe is a flexible and easy 
  │ to use tool for serving deep learning models",
  │     "version": "0.8.1"
  │   },
  │   "paths": {
  │     "/metrics": {
  │       "get": {
  │         "description": "Get TorchServe application met
  │ rics in prometheus format.",
  │         "operationId": "metrics",
  │         "parameters": [
  │           {
  │             "in": "query",
  │             "name": "name[]",
  │             "description": "Names of metrics to filter
  │ ",
  │             "required": false,
  │             "schema": {
  │               "type": "string"
  │             }
  │           }
  │         ],
  │         "responses": {
  │           "200": {
  │             "description": "TorchServe application met
  │ rics",
  │             "content": {
  │               "text/plain; version=0.0.4; charset=utf-
  │ 8": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "# HELP",
  │                     "# TYPE",
  │                     "metric"
  │                   ],
  │                   "properties": {
  │                     "# HELP": {
  │                       "type": "string",
  │                       "description": "Help text for To
  │ rchServe metric."
  │                     },
  │                     "# TYPE": {
  │                       "type": "string",
  │                       "description": "Type of TorchSer
  │ ve metric."
  │                     },
  │                     "metric": {
  │                       "type": "string",
  │                       "description": "TorchServe appli
  │ cation metric."
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           },
  │           "500": {
  │             "description": "Internal Server Error",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "code",
  │                     "type",
  │                     "message"
  │                   ],
  │                   "properties": {
  │                     "code": {
  │                
  │ (showing 2.05kB/2.55kB)
  └
  prepare   wait   dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  2ms       1ms    234µs        442µs           31ms            4ms              1ms        64µs      41ms  

  ✓  Status code is 200

→ HTTPS Register Model - SqueezeNet
  POST https://localhost:8444/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true  
  200 OK ★ 2.9s time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  2ms       509µs   (cache)      (cache)         (cache)         2.9s             1ms        78µs      2.9s  

  ✓  Successful POST request

→ HTTPS Get SqueezeNet Model Description
  GET https://localhost:8444/models/squeezenet1_1  
  200 OK ★ 6ms time ★ 246B↑ 753B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 486B
  │ [
  │   {
  │     "modelName": "squeezenet1_1",
  │     "modelVersion": "1.0",
  │     "modelUrl": "squeezenet1_1.mar",
  │     "runtime": "python",
  │     "minWorkers": 1,
  │     "maxWorkers": 1,
  │     "batchSize": 1,
  │     "maxBatchDelay": 100,
  │     "loadedAtStartup": false,
  │     "workers": [
  │       {
  │         "id": "9000",
  │         "startTime": "2023-08-14T22:56:30.166Z",
  │         "status": "READY",
  │         "memoryUsage": 0,
  │         "pid": 18486,
  │         "gpu": false,
  │         "gpuUsage": "N/A"
  │       }
  │     ]
  │   }
  │ ]
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       265µs   (cache)      (cache)         (cache)         4ms              1ms        49µs      6ms   

  ✓  Successful GET request

→ HTTPS Scale up Workers -  Synchronous
  PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=true  
  200 OK ★ 4ms time ★ 308B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B
  │ {
  │   "status": "Workers scaled to 1 for model: squeezenet
  │ 1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       247µs   (cache)      (cache)         (cache)         2ms              1ms        50µs      4ms   

  ✓  Successful PUT request

→ HTTPS Scale up Workers - Asynchronous
  PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=false  
  202 Accepted ★ 4ms time ★ 309B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       255µs   (cache)      (cache)         (cache)         2ms              1ms        40µs      5ms   

  ✓  Successful PUT request

→ HTTPS - Inference - SqueezeNet
  POST https://localhost:8443/predictions/squeezenet1_1  
  200 OK ★ 55ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B
  │ {
  │   "tabby": 0.27850738167762756,
  │   "lynx": 0.25299158692359924,
  │   "tiger_cat": 0.24496474862098694,
  │   "Egyptian_cat": 0.21722468733787537,
  │   "cougar": 0.002217544475570321
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  5ms       694µs   (cache)      (cache)         (cache)         53ms             1ms        33µs      60ms  

  ✓  Status code is 200

→ HTTPS UnRegister Model SqueezeNet
  DELETE https://localhost:8444/models/squeezenet1_1  
  200 OK ★ 20ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       223µs   (cache)      (cache)         (cache)         17ms             1ms        33µs      20ms  

  ✓  Successful DELETE request

┌─────────────────────────┬─────────────────────┬────────────────────┐
│                         │            executed │             failed │
├─────────────────────────┼─────────────────────┼────────────────────┤
│              iterations │                   1 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│                requests │                   9 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│            test-scripts │                   9 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│      prerequest-scripts │                   0 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│              assertions │                   9 │                  0 │
├─────────────────────────┴─────────────────────┴────────────────────┤
│ total run duration: 3.5s                                           │
├────────────────────────────────────────────────────────────────────┤
│ total data received: 85.45kB (approx)                              │
├────────────────────────────────────────────────────────────────────┤
│ average response time: 363ms [min: 4ms, max: 2.9s, s.d.: 909ms]    │
├────────────────────────────────────────────────────────────────────┤
│ average DNS lookup time: 496µs [min: 179µs, max: 1ms, s.d.: 410µs] │
├────────────────────────────────────────────────────────────────────┤
│ average first byte time: 339ms [min: 2ms, max: 2.9s, s.d.: 916ms]  │
└────────────────────────────────────────────────────────────────────┘
newman: could not find "htmlextra" reporter
  ensure that the reporter is installed in the same directory as newman
  please install reporter using npm

newman

management_api_collection

Iteration 1/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1  
  200 OK ★ 153ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total 
  28ms      4ms    1ms          339µs           140ms            6ms        349µs     181ms 

  ✓  Successful  request

Iteration 2/82

→ management request
  POST http://localhost:8081/models?url=mnist.mar&model_name=mnist  
  200 OK ★ 96ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B
  │ {
  │   "status": "Model \"mnist\" Version: 1.0 registered w
  │ ith 0 initial workers. Use scale workers API to add wo
  │ rkers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       639µs   (cache)      (cache)         93ms             1ms        61µs      97ms  

  ✓  Successful  request

Iteration 3/82

→ management request
  POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161  
  200 OK ★ 833ms time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B
  │ {
  │   "status": "Model \"densenet161\" Version: 1.0 regist
  │ ered with 0 initial workers. Use scale workers API to 
  │ add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       303µs   (cache)      (cache)         830ms            1ms        64µs      833ms 

  ✓  Successful  request

Iteration 4/82

→ management request
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161  
  500 Internal Server Error ★ 7ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B
  │ {
  │   "code": 500,
  │   "type": "InternalServerException",
  │   "message": "Model file already exists densenet161.ma
  │ r"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       277µs   (cache)      (cache)         3ms              1ms        56µs      7ms   

  ✓  Successful  request

Iteration 5/82

→ management request
  DELETE http://localhost:8081/models/densenet161  
  200 OK ★ 18ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B
  │ {
  │   "status": "Model \"densenet161\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       651µs   184µs        295µs           14ms             1ms        66µs      18ms  

  ✓  Successful  request

Iteration 6/82

→ management request
  POST http://localhost:8081/models  
  400 Bad Request ★ 4ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B
  │ {
  │   "code": 400,
  │   "type": "BadRequestException",
  │   "message": "Parameter url is required."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       218µs   (cache)      (cache)         1ms              1ms        53µs      4ms   

  ✓  Successful  request

Iteration 7/82

→ management request
  DELETE http://localhost:8081/models/mnist  
  200 OK ★ 18ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Model \"mnist\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       616µs   193µs        261µs           14ms             1ms        46µs      18ms  

  ✓  Successful  request

Iteration 8/82

→ management request
  POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler  
  200 OK ★ 45ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B
  │ {
  │   "status": "Model \"mnist\" Version: 1.0 registered w
  │ ith 0 initial workers. Use scale workers API to add wo
  │ rkers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       277µs   (cache)      (cache)         42ms             1ms        41µs      45ms  

  ✓  Successful  request

Iteration 9/82

→ management request
  DELETE http://localhost:8081/models/mnist  
  200 OK ★ 5ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Model \"mnist\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       178µs   (cache)      (cache)         3ms              1ms        28µs      5ms   

  ✓  Successful  request

Iteration 10/82

→ management request
  POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler  
  200 OK ★ 43ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B
  │ {
  │   "status": "Model \"mnist\" Version: 1.0 registered w
  │ ith 0 initial workers. Use scale workers API to add wo
  │ rkers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       205µs   (cache)      (cache)         40ms             1ms        32µs      43ms  

  ✓  Successful  request

Iteration 11/82

→ management request
  PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true  
  500 Internal Server Error ★ 1398ms time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B
  │ {
  │   "code": 500,
  │   "type": "InternalServerException",
  │   "message": "Failed to start workers for model mnist 
  │ version: null"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total  
  997µs     178µs   (cache)      (cache)         1395ms           1ms        35µs      1398ms 

  ✓  Successful  request

Iteration 12/82

→ management request
  DELETE http://localhost:8081/models/mnist  
  200 OK ★ 17ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Model \"mnist\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       614µs   191µs        261µs           14ms             1ms        36µs      17ms  

  ✓  Successful  request

Iteration 13/82

→ management request
  GET http://localhost:8081/models/squeezenet1_1/all  
  200 OK ★ 5ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B
  │ [
  │   {
  │     "modelName": "squeezenet1_1",
  │     "modelVersion": "1.0",
  │     "modelUrl": "squeezenet1_1.mar",
  │     "runtime": "python",
  │     "minWorkers": 0,
  │     "maxWorkers": 0,
  │     "batchSize": 1,
  │     "maxBatchDelay": 100,
  │     "loadedAtStartup": false,
  │     "workers": []
  │   }
  │ ]
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  941µs     186µs   (cache)      (cache)         3ms              1ms        28µs      5ms   

  ✓  Successful  request

Iteration 14/82

→ management request
  GET http://localhost:8081/models/squeezenet1_1/1.0  
  200 OK ★ 3ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B
  │ [
  │   {
  │     "modelName": "squeezenet1_1",
  │     "modelVersion": "1.0",
  │     "modelUrl": "squeezenet1_1.mar",
  │     "runtime": "python",
  │     "minWorkers": 0,
  │     "maxWorkers": 0,
  │     "batchSize": 1,
  │     "maxBatchDelay": 100,
  │     "loadedAtStartup": false,
  │     "workers": []
  │   }
  │ ]
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  917µs     171µs   (cache)      (cache)         1ms              902µs      30µs      3ms   

  ✓  Successful  request

Iteration 15/82

→ management request
  GET http://localhost:8081/models/squeezenet1_1  
  200 OK ★ 3ms time ★ 246B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B
  │ [
  │   {
  │     "modelName": "squeezenet1_1",
  │     "modelVersion": "1.0",
  │     "modelUrl": "squeezenet1_1.mar",
  │     "runtime": "python",
  │     "minWorkers": 0,
  │     "maxWorkers": 0,
  │     "batchSize": 1,
  │     "maxBatchDelay": 100,
  │     "loadedAtStartup": false,
  │     "workers": []
  │   }
  │ ]
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       189µs   (cache)      (cache)         1ms              1ms        27µs      3ms   

  ✓  Successful  request

Iteration 16/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1  
  200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       176µs   (cache)      (cache)         3ms              1ms        37µs      6ms   

  ✓  Successful  request

Iteration 17/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4  
  400 Bad Request ★ 3ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B
  │ {
  │   "code": 400,
  │   "type": "BadRequestException",
  │   "message": "Invalid RuntimeType value: python4"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       217µs   (cache)      (cache)         1ms              1ms        26µs      4ms   

  ✓  Successful  request

Iteration 18/82

→ management request
  GET http://localhost:8081/models?limit=&next_page_token=  
  200 OK ★ 12ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B
  │ {
  │   "models": []
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  970µs     505µs   131µs        193µs           10ms             1ms        26µs      12ms  

  ✓  Successful  request

Iteration 19/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1  
  200 OK ★ 37ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  943µs     184µs   (cache)      (cache)         35ms             1ms        26µs      37ms  

  ✓  Successful  request

Iteration 20/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1?min_worker=1  
  202 Accepted ★ 4ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  933µs     162µs   (cache)      (cache)         1ms              1ms        38µs      4ms   

  ✓  Successful  request

Iteration 21/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true  
  200 OK ★ 4ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B
  │ {
  │   "status": "Workers scaled to 1 for model: squeezenet
  │ 1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       243µs   (cache)      (cache)         2ms              1ms        44µs      4ms   

  ✓  Successful  request

Iteration 22/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true  
  200 OK ★ 4ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B
  │ {
  │   "status": "Workers scaled to 1 for model: squeezenet
  │ 1_1, version: 1.0"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  988µs     199µs   (cache)      (cache)         1ms              1ms        27µs      4ms   

  ✓  Successful  request

Iteration 23/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true  
  404 Not Found ★ 3ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B
  │ {
  │   "code": 404,
  │   "type": "ModelVersionNotFoundException",
  │   "message": "Model version: 0.0 does not exist for mo
  │ del: squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       179µs   (cache)      (cache)         1ms              973µs      24µs      3ms   

  ✓  Successful  request

Iteration 24/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1  
  202 Accepted ★ 13ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  919µs     450µs   122µs        214µs           11ms             983µs      25µs      13ms  

  ✓  Successful  request

Iteration 25/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default  
  200 OK ★ 14ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B
  │ {
  │   "status": "Default vesion succsesfully updated for m
  │ odel \"squeezenet1_1\" to \"1.0\""
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  903µs     177µs   (cache)      (cache)         12ms             1ms        33µs      14ms  

  ✓  Successful  request

Iteration 26/82

→ management request
  PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default  
  404 Not Found ★ 4ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B
  │ {
  │   "code": 404,
  │   "type": "ModelVersionNotFoundException",
  │   "message": "Model version 0.0 does not exist for mod
  │ el squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       191µs   (cache)      (cache)         1ms              1ms        27µs      4ms   

  ✓  Successful  request

Iteration 27/82

→ management request
  PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default  
  404 Not Found ★ 12ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: squeezenet0_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  874µs     448µs   121µs        185µs           9ms              957µs      23µs      12ms  

  ✓  Successful  request

Iteration 28/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1  
  200 OK ★ 24ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  866µs     440µs   122µs        173µs           21ms             942µs      25µs      24ms  

  ✓  Successful  request

Iteration 29/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle  
  200 OK ★ 35ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  918µs     166µs   (cache)      (cache)         33ms             1ms        26µs      35ms  

  ✓  Successful  request

Iteration 30/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1  
  200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  896µs     157µs   (cache)      (cache)         3ms              1ms        29µs      5ms   

  ✓  Successful  request

Iteration 31/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0  
  500 Internal Server Error ★ 2s time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B
  │ {
  │   "code": 500,
  │   "type": "InternalServerException",
  │   "message": "Failed to start workers for model squeez
  │ enet1_1 version: 1.0"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  994µs     178µs   (cache)      (cache)         2s               1ms        31µs      2s    

  ✓  Successful  request

Iteration 32/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0  
  200 OK ★ 62ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       645µs   166µs        207µs           59ms             1ms        27µs      63ms  

  ✓  Successful  request

Iteration 33/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1  
  200 OK ★ 7ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       226µs   (cache)      (cache)         4ms              1ms        29µs      7ms   

  ✓  Successful  request

Iteration 34/82

→ management request
  POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2  
  200 OK ★ 1611ms time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B
  │ {
  │   "status": "Model \"resnet152\" Version: 1.0 register
  │ ed with 0 initial workers. Use scale workers API to ad
  │ d workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total  
  1ms       226µs   (cache)      (cache)         1609ms           967µs      25µs      1611ms 

  ✓  Successful  request

Iteration 35/82

→ management request
  DELETE http://localhost:8081/models/resnet152  
  200 OK ★ 6ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet152\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  974µs     213µs   (cache)      (cache)         3ms              1ms        33µs      6ms   

  ✓  Successful  request

Iteration 36/82

→ management request
  POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1  
  200 OK ★ 5s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B
  │ {
  │   "status": "Model \"resnet152\" Version: 1.0 register
  │ ed with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       307µs   (cache)      (cache)         5s               3ms        68µs      5s    

  ✓  Successful  request

Iteration 37/82

→ management request
  DELETE http://localhost:8081/models/resnet152  
  200 OK ★ 22ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet152\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       395µs   (cache)      (cache)         19ms             1ms        35µs      22ms  

  ✓  Successful  request

Iteration 38/82

→ management request
  POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk  
  200 OK ★ 4.8s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B
  │ {
  │   "status": "Model \"resnet152\" Version: 1.0 register
  │ ed with 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       238µs   (cache)      (cache)         4.8s             1ms        32µs      4.8s  

  ✓  Successful  request

Iteration 39/82

→ management request
  DELETE http://localhost:8081/models/resnet152  
  200 OK ★ 20ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet152\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       203µs   (cache)      (cache)         18ms             1ms        33µs      21ms  

  ✓  Successful  request

Iteration 40/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1  
  200 OK ★ 43ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       209µs   (cache)      (cache)         41ms             1ms        28µs      44ms  

  ✓  Successful  request

Iteration 41/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1  
  200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  991µs     159µs   (cache)      (cache)         3ms              1ms        31µs      6ms   

  ✓  Successful  request

Iteration 42/82

→ management request
  POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true  
  200 OK ★ 280ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B
  │ {
  │   "status": "Model \"resnet-18\" Version: 1.0 register
  │ ed with 0 initial workers. Use scale workers API to ad
  │ d workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       183µs   (cache)      (cache)         278ms            1ms        23µs      280ms 

  ✓  Successful  request

Iteration 43/82

→ management request
  DELETE http://localhost:8081/models/resnet-18  
  200 OK ★ 5ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet-18\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       181µs   (cache)      (cache)         3ms              1ms        27µs      6ms   

  ✓  Successful  request

Iteration 44/82

→ management request
  POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1  
  200 OK ★ 276ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B
  │ {
  │   "status": "Model \"resnet-18\" Version: 1.0 register
  │ ed with 0 initial workers. Use scale workers API to ad
  │ d workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       189µs   (cache)      (cache)         274ms            915µs      24µs      277ms 

  ✓  Successful  request

Iteration 45/82

→ management request
  DELETE http://localhost:8081/models/resnet-18  
  200 OK ★ 5ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet-18\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  925µs     181µs   (cache)      (cache)         3ms              1ms        27µs      5ms   

  ✓  Successful  request

Iteration 46/82

→ management request
  POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false  
  200 OK ★ 279ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B
  │ {
  │   "status": "Model \"resnet-18\" Version: 1.0 register
  │ ed with 0 initial workers. Use scale workers API to ad
  │ d workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       215µs   (cache)      (cache)         277ms            943µs      28µs      280ms 

  ✓  Successful  request

Iteration 47/82

→ management request
  GET http://localhost:8081/models?limit=1  
  200 OK ★ 4ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B
  │ {
  │   "nextPageToken": "1",
  │   "models": [
  │     {
  │       "modelName": "resnet-18",
  │       "modelUrl": "resnet-18.mar"
  │     }
  │   ]
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       240µs   (cache)      (cache)         1ms              1ms        37µs      4ms   

  ✓  Successful  request

Iteration 48/82

→ management request
  GET http://localhost:8081/models?limit=-1  
  200 OK ★ 4ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B
  │ {
  │   "models": [
  │     {
  │       "modelName": "resnet-18",
  │       "modelUrl": "resnet-18.mar"
  │     }
  │   ]
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       207µs   (cache)      (cache)         1ms              1ms        34µs      4ms   

  ✓  Successful  request

Iteration 49/82

→ management request
  GET http://localhost:8081/models?limit=1&next_page_token=1  
  200 OK ★ 3ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B
  │ {
  │   "models": []
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       171µs   (cache)      (cache)         1ms              1ms        29µs      3ms   

  ✓  Successful  request

Iteration 50/82

→ management request
  GET http://localhost:8081/models?limit=1&next_page_token=-1  
  200 OK ★ 3ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B
  │ {
  │   "nextPageToken": "1",
  │   "models": [
  │     {
  │       "modelName": "resnet-18",
  │       "modelUrl": "resnet-18.mar"
  │     }
  │   ]
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       176µs   (cache)      (cache)         1ms              1ms        29µs      3ms   

  ✓  Successful  request

Iteration 51/82

→ management request
  PUT http://localhost:8081/models/resnet-18?number_gpu=10  
  202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       168µs   (cache)      (cache)         1ms              1ms        30µs      4ms   

  ✓  Successful  request

Iteration 52/82

→ management request
  PUT http://localhost:8081/models/resnet-18?number_gpu=-1  
  202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       191µs   (cache)      (cache)         1ms              1ms        30µs      4ms   

  ✓  Successful  request

Iteration 53/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true  
  200 OK ★ 3ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B
  │ {
  │   "status": "Workers scaled to 1 for model: resnet-18"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       173µs   (cache)      (cache)         1ms              1ms        32µs      3ms   

  ✓  Successful  request

Iteration 54/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false  
  202 Accepted ★ 3ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       162µs   (cache)      (cache)         1ms              981µs      26µs      3ms   

  ✓  Successful  request

Iteration 55/82

→ management request
  PUT http://localhost:8081/models/resnet-18?timeout=-1  
  202 Accepted ★ 3ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  866µs     141µs   (cache)      (cache)         1ms              836µs      22µs      2ms   

  ✓  Successful  request

Iteration 56/82

→ management request
  PUT http://localhost:8081/models/resnet-18?timeout=0  
  202 Accepted ★ 3ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  867µs     142µs   (cache)      (cache)         1ms              921µs      24µs      3ms   

  ✓  Successful  request

Iteration 57/82

→ management request
  POST http://localhost:8081/models?url=&model_name=resnet-18  
  404 Not Found ★ 3ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "empty url"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       168µs   (cache)      (cache)         1ms              1ms        28µs      3ms   

  ✓  Successful  request

Iteration 58/82

→ management request
  POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18  
  400 Bad Request ★ 958ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B
  │ {
  │   "code": 400,
  │   "type": "DownloadArchiveException",
  │   "message": "Failed to download archive from: https:/
  │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma
  │ r"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       541µs   181µs        213µs           954ms            1ms        40µs      958ms 

  ✓  Successful  request

Iteration 59/82

→ management request
  GET http://localhost:8081/models/invalid_squeezenet1_1  
  404 Not Found ★ 14ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: invalid_squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       487µs   156µs        210µs           11ms             1ms        28µs      14ms  

  ✓  Successful  request

Iteration 60/82

→ management request
  GET http://localhost:8081/models/squeezenet1_1/0.0  
  404 Not Found ★ 14ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       505µs   146µs        264µs           11ms             1ms        34µs      14ms  

  ✓  Successful  request

Iteration 61/82

→ management request
  GET http://localhost:8081/models?next_page_token=12  
  200 OK ★ 13ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B
  │ {
  │   "models": []
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       479µs   140µs        201µs           10ms             1ms        31µs      13ms  

  ✓  Successful  request

Iteration 62/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan  
  202 Accepted ★ 3ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       179µs   (cache)      (cache)         1ms              1ms        29µs      3ms   

  ✓  Successful  request

Iteration 63/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan  
  202 Accepted ★ 4ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       226µs   (cache)      (cache)         1ms              1ms        32µs      4ms   

  ✓  Successful  request

Iteration 64/82

→ management request
  PUT http://localhost:8081/models/resnet-18  
  202 Accepted ★ 4ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  986µs     168µs   (cache)      (cache)         1ms              1ms        31µs      4ms   

  ✓  Successful  request

Iteration 65/82

→ management request
  PUT http://localhost:8081/models/resnet181?min_worker=1  
  404 Not Found ★ 3ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: resnet181"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       165µs   (cache)      (cache)         1ms              1ms        29µs      3ms   

  ✓  Successful  request

Iteration 66/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1  
  400 Bad Request ★ 13ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B
  │ {
  │   "code": 400,
  │   "type": "BadRequestException",
  │   "message": "max_worker cannot be less than min_worke
  │ r."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       479µs   148µs        214µs           10ms             1ms        29µs      15ms  

  ✓  Successful  request

Iteration 67/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=1  
  202 Accepted ★ 13ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       465µs   599µs        233µs           9ms              1ms        28µs      13ms  

  ✓  Successful  request

Iteration 68/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=0  
  202 Accepted ★ 15ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  965µs     161µs   (cache)      (cache)         13ms             1ms        30µs      15ms  

  ✓  Successful  request

Iteration 69/82

→ management request
  PUT http://localhost:8081/models/resnet-18?min_worker=-1  
  500 Internal Server Error ★ 4ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B
  │ {
  │   "code": 500,
  │   "type": "IndexOutOfBoundsException",
  │   "message": "Index -1 out of bounds for length 0"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       191µs   (cache)      (cache)         2ms              1ms        30µs      4ms   

  ✓  Successful  request

Iteration 70/82

→ management request
  PUT http://localhost:8081/models/resnet-18?max_worker=-1  
  400 Bad Request ★ 13ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B
  │ {
  │   "code": 400,
  │   "type": "BadRequestException",
  │   "message": "max_worker cannot be less than min_worke
  │ r."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       480µs   150µs        217µs           10ms             1ms        31µs      13ms  

  ✓  Successful  request

Iteration 71/82

→ management request
  PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default  
  404 Not Found ★ 16ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: invalid_squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       515µs   159µs        252µs           11ms             3ms        30µs      16ms  

  ✓  Successful  request

Iteration 72/82

→ management request
  DELETE http://localhost:8081/models/resnet-18  
  200 OK ★ 16ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B
  │ {
  │   "status": "Model \"resnet-18\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       461µs   138µs        231µs           13ms             1ms        28µs      16ms  

  ✓  Successful  request

Iteration 73/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1/0.0  
  404 Not Found ★ 3ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       169µs   (cache)      (cache)         1ms              1ms        32µs      3ms   

  ✓  Successful  request

Iteration 74/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1  
  200 OK ★ 44ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       474µs   150µs        240µs           41ms             1ms        29µs      44ms  

  ✓  Successful  request

Iteration 75/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true  
  200 OK ★ 5ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       171µs   (cache)      (cache)         3ms              1ms        32µs      5ms   

  ✓  Successful  request

Iteration 76/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1  
  200 OK ★ 41ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       164µs   (cache)      (cache)         38ms             986µs      25µs      41ms  

  ✓  Successful  request

Iteration 77/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan  
  200 OK ★ 5ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       172µs   (cache)      (cache)         3ms              1ms        25µs      6ms   

  ✓  Successful  request

Iteration 78/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1  
  200 OK ★ 40ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       167µs   (cache)      (cache)         38ms             1ms        27µs      40ms  

  ✓  Successful  request

Iteration 79/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true  
  200 OK ★ 5ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       168µs   (cache)      (cache)         3ms              1ms        29µs      5ms   

  ✓  Successful  request

Iteration 80/82

→ management request
  POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1  
  200 OK ★ 44ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B
  │ {
  │   "status": "Model \"squeezenet1_1\" Version: 1.0 regi
  │ stered with 0 initial workers. Use scale workers API t
  │ o add workers for the model."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       218µs   (cache)      (cache)         41ms             1ms        32µs      44ms  

  ✓  Successful  request

Iteration 81/82

→ management request
  DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1  
  200 OK ★ 6ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Model \"squeezenet1_1\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       198µs   (cache)      (cache)         4ms              1ms        29µs      6ms   

  ✓  Successful  request

Iteration 82/82

→ management request
  DELETE http://localhost:8081/models/invalid_squeezenet1_1  
  404 Not Found ★ 3ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B
  │ {
  │   "code": 404,
  │   "type": "ModelNotFoundException",
  │   "message": "Model not found: invalid_squeezenet1_1"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       179µs   (cache)      (cache)         1ms              1ms        28µs      3ms   

  ✓  Successful  request

┌─────────────────────────┬─────────────────────┬────────────────────┐
│                         │            executed │             failed │
├─────────────────────────┼─────────────────────┼────────────────────┤
│              iterations │                  82 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│                requests │                  82 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│            test-scripts │                  82 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│      prerequest-scripts │                   0 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│              assertions │                  82 │                  0 │
├─────────────────────────┴─────────────────────┴────────────────────┤
│ total run duration: 20.2s                                          │
├────────────────────────────────────────────────────────────────────┤
│ total data received: 7.75kB (approx)                               │
├────────────────────────────────────────────────────────────────────┤
│ average response time: 228ms [min: 3ms, max: 5s, s.d.: 819ms]      │
├────────────────────────────────────────────────────────────────────┤
│ average DNS lookup time: 628µs [min: 121µs, max: 1ms, s.d.: 516µs] │
├────────────────────────────────────────────────────────────────────┤
│ average first byte time: 225ms [min: 1ms, max: 5s, s.d.: 819ms]    │
└────────────────────────────────────────────────────────────────────┘
newman: could not find "htmlextra" reporter
  ensure that the reporter is installed in the same directory as newman
  please install reporter using npm

newman

kf_api_test_collection

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true  
  200 OK ★ 3.4s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B
  │ {
  │   "status": "Model \"mnist\" Version: 1.0 registered w
  │ ith 1 initial workers"
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total 
  36ms      5ms    1ms          316µs           3.4s             5ms        465µs     3.4s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/v1/models/mnist:predict  
  200 OK ★ 27ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 409B
  │ 
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B
  │ {
  │   "predictions": [
  │     2
  │   ]
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  5ms       907µs   400µs        169µs           23ms             1ms        42µs      32ms  

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Explanations Model
  POST http://localhost:8080/v1/models/mnist:explain  
  200 OK ★ 72ms time ★ 680B↑ 25.4kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 409B
  │ 
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.17kB
  │ {
  │   "explanations": [
  │     [
  │       [
  │         [
  │           0.004570948738580575,
  │           0.006216969354192105,
  │           0.008197565570543713,
  │           0.00956357465709623,
  │           0.00899927455019611,
  │           0.009673474932247308,
  │           0.0075999051701512495,
  │           0.006361380884677352,
  │           0.005768828620731858,
  │           0.004394465852785626,
  │           0.004948218006895951,
  │           0.005273460552821625,
  │           0.005523799718367559,
  │           0.007789356415261442,
  │           0.008759363343907851,
  │           0.0043048051582686014,
  │           0.010970579130997414,
  │           0.0032484391603957454,
  │           0.005998033007716084,
  │           0.0037543659158808013,
  │           0.002765290843432279,
  │           0.004314086250885245,
  │           0.0014008573988940512,
  │           0.004841846505403689,
  │           0.0006374840586476711,
  │           0.001855853965693574,
  │           -0.0008280458167248678,
  │           -0.0
  │         ],
  │         [
  │           0.0016625159121928454,
  │           0.004443791781199298,
  │           0.012387840300503305,
  │           0.00945084395288644,
  │           0.016143820738507904,
  │           0.007797502048729162,
  │           0.0139422407842874,
  │           0.007557430473265285,
  │           0.0054790905833163525,
  │           0.009751320346115864,
  │           0.004764407372645556,
  │           0.007292147662234541,
  │           0.011797999372381933,
  │           0.006462684561652185,
  │           0.0033832142185796325,
  │           0.009225058876193865,
  │           0.0016750703115690345,
  │           0.007362304720604091,
  │           0.005270058056401144,
  │           0.005453598180888259,
  │           0.0043427257697093895,
  │           0.0058561332312329325,
  │           0.012168384678716265,
  │           0.009902719420388126,
  │           0.009813113255163137,
  │           0.0034427582108525158,
  │           0.0022924284763651136,
  │           -0.0
  │         ],
  │         [
  │           0.011528255254417058,
  │           0.009146989291478723,
  │           0.022265419864791573,
  │           0.017558217164003482,
  │           0.0247705545859691,
  │           0.021412693642065557,
  │  
  │ (showing 2.05kB/25.17kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       370µs   (cache)      (cache)         69ms             1ms        72µs      73ms  

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/mnist  
  200 OK ★ 25ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Model \"mnist\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       267µs   (cache)      (cache)         22ms             1ms        64µs      25ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 19ms time ★ 233B↑ 4.03kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.74kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 2.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 55.3
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com",} 8.439990997314453
  │ # HELP WorkerLoadTime Torchserve prometheus gauge metr
  │ ic with unit: Milliseconds
  │ # TYPE WorkerLoadTime gauge
  │ WorkerLoadTime{WorkerName="W-9000-mnist_1.0",Level="Ho
  │ st",Hostname="147dda19895c.ant.amazon.com",} 3272.0
  │ # HELP MemoryUsed Torchserve prometheus gauge metric w
  │ ith unit: Megabytes
  │ # TYPE MemoryUsed gauge
  │ MemoryUsed{Level="Host",Hostname="147dda19895c.ant.ama
  │ zon.com",} 8718.99609375
  │ # HELP PredictionTime Torchserve prometheus gauge metr
  │ ic with unit: ms
  │ # TYPE PredictionTime gauge
  │ PredictionTime{ModelName="mnist",Level="Model",Hostnam
  │ e="147dda19895c.ant.amazon.com",} 64.75
  │ # HELP MemoryAvailable Torchserve prometheus gauge met
  │ ric with unit: Megabytes
  │ # TYPE MemoryAvailable gauge
  │ MemoryAvailable{Level="Host",Hostname="147dda19895c.an
  │ t.amazon.com",} 7325.30859375
  │ # HELP Requests2XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests2XX counter
  │ Requests2XX{Level="Host",Hostname="147dda19895c.ant.am
  │ azon.com",} 4.0
  │ # HELP GPUMemoryUsed Torchserve prometheus gauge metri
  │ c with unit: Megabytes
  │ # TYPE GPUMemoryUsed gauge
  │ # HELP DiskAvailable Torchserve prometheus gauge metri
  │ c with unit: G
  │ (showing 2.05kB/3.74kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       635µs   166µs        235µs           16ms             1ms        65µs      19ms  

  ✓  Successful GET request

┌─────────────────────────┬─────────────────────┬─────────────────────┐
│                         │            executed │              failed │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│              iterations │                   1 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│                requests │                   5 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│            test-scripts │                   5 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│      prerequest-scripts │                   0 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│              assertions │                   7 │                   0 │
├─────────────────────────┴─────────────────────┴─────────────────────┤
│ total run duration: 3.8s                                            │
├─────────────────────────────────────────────────────────────────────┤
│ total data received: 29.06kB (approx)                               │
├─────────────────────────────────────────────────────────────────────┤
│ average response time: 716ms [min: 19ms, max: 3.4s, s.d.: 1361ms]   │
├─────────────────────────────────────────────────────────────────────┤
│ average DNS lookup time: 815µs [min: 166µs, max: 1ms, s.d.: 609µs]  │
├─────────────────────────────────────────────────────────────────────┤
│ average first byte time: 711ms [min: 16ms, max: 3.4s, s.d.: 1357ms] │
└─────────────────────────────────────────────────────────────────────┘
newman: could not find "htmlextra" reporter
  ensure that the reporter is installed in the same directory as newman
  please install reporter using npm

newman

kf_https_test_collection

→ HTTPS Inference API  Description
  OPTIONS https://localhost:8443  
  200 OK ★ 148ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB
  │ {
  │   "openapi": "3.0.1",
  │   "info": {
  │     "title": "TorchServe APIs",
  │     "description": "TorchServe is a flexible and easy 
  │ to use tool for serving deep learning models",
  │     "version": "0.8.1"
  │   },
  │   "paths": {
  │     "/": {
  │       "options": {
  │         "description": "Get openapi description.",
  │         "operationId": "apiDescription",
  │         "parameters": [],
  │         "responses": {
  │           "200": {
  │             "description": "A openapi 3.0.1 descriptor
  │ ",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "openapi",
  │                     "info",
  │                     "paths"
  │                   ],
  │                   "properties": {
  │                     "openapi": {
  │                       "type": "string"
  │                     },
  │                     "info": {
  │                       "type": "object"
  │                     },
  │                     "paths": {
  │                       "type": "object"
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           },
  │           "500": {
  │             "description": "Internal Server Error",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "code",
  │                     "type",
  │                     "message"
  │                   ],
  │                   "properties": {
  │                     "code": {
  │                       "type": "integer",
  │                       "description": "Error code."
  │                     },
  │                     "type": {
  │                       "type": "string",
  │                       "description": "Error type."
  │                     },
  │                     "message": {
  │                       "type": "string",
  │                       "description": "Error message."
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           }
  │         }
  │       }
  │     },
  │     "/ping": {
  │      
  │ (showing 2.05kB/23.41kB)
  └
  prepare   wait   dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  34ms      7ms    1ms          583µs           105ms           27ms             5ms        359µs     182ms 

  ✓  Status code is 200

→ HTTPS Management API Description
  OPTIONS https://localhost:8444  
  200 OK ★ 52ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB
  │ {
  │   "openapi": "3.0.1",
  │   "info": {
  │     "title": "TorchServe APIs",
  │     "description": "TorchServe is a flexible and easy 
  │ to use tool for serving deep learning models",
  │     "version": "0.8.1"
  │   },
  │   "paths": {
  │     "/": {
  │       "options": {
  │         "description": "Get openapi description.",
  │         "operationId": "apiDescription",
  │         "parameters": [],
  │         "responses": {
  │           "200": {
  │             "description": "A openapi 3.0.1 descriptor
  │ ",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "openapi",
  │                     "info",
  │                     "paths"
  │                   ],
  │                   "properties": {
  │                     "openapi": {
  │                       "type": "string"
  │                     },
  │                     "info": {
  │                       "type": "object"
  │                     },
  │                     "paths": {
  │                       "type": "object"
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           },
  │           "500": {
  │             "description": "Internal Server Error",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "code",
  │                     "type",
  │                     "message"
  │                   ],
  │                   "properties": {
  │                     "code": {
  │                       "type": "integer",
  │                       "description": "Error code."
  │                     },
  │                     "type": {
  │                       "type": "string",
  │                       "description": "Error type."
  │                     },
  │                     "message": {
  │                       "type": "string",
  │                       "description": "Error message."
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           }
  │         }
  │       }
  │     },
  │     "/models": {
  │    
  │ (showing 2.05kB/58.58kB)
  └
  prepare   wait   dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       1ms    151µs        325µs           32ms            15ms             1ms        67µs      52ms  

  ✓  Status code is 200

→ HTTPS Metrics API Description
  OPTIONS https://localhost:8445  
  200 OK ★ 39ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB
  │ {
  │   "openapi": "3.0.1",
  │   "info": {
  │     "title": "TorchServe APIs",
  │     "description": "TorchServe is a flexible and easy 
  │ to use tool for serving deep learning models",
  │     "version": "0.8.1"
  │   },
  │   "paths": {
  │     "/metrics": {
  │       "get": {
  │         "description": "Get TorchServe application met
  │ rics in prometheus format.",
  │         "operationId": "metrics",
  │         "parameters": [
  │           {
  │             "in": "query",
  │             "name": "name[]",
  │             "description": "Names of metrics to filter
  │ ",
  │             "required": false,
  │             "schema": {
  │               "type": "string"
  │             }
  │           }
  │         ],
  │         "responses": {
  │           "200": {
  │             "description": "TorchServe application met
  │ rics",
  │             "content": {
  │               "text/plain; version=0.0.4; charset=utf-
  │ 8": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "# HELP",
  │                     "# TYPE",
  │                     "metric"
  │                   ],
  │                   "properties": {
  │                     "# HELP": {
  │                       "type": "string",
  │                       "description": "Help text for To
  │ rchServe metric."
  │                     },
  │                     "# TYPE": {
  │                       "type": "string",
  │                       "description": "Type of TorchSer
  │ ve metric."
  │                     },
  │                     "metric": {
  │                       "type": "string",
  │                       "description": "TorchServe appli
  │ cation metric."
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           },
  │           "500": {
  │             "description": "Internal Server Error",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "code",
  │                     "type",
  │                     "message"
  │                   ],
  │                   "properties": {
  │                     "code": {
  │                
  │ (showing 2.05kB/2.55kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       933µs   153µs        368µs           32ms            3ms              1ms        70µs      40ms  

  ✓  Status code is 200

→ HTTPS Register Model - Mnist
  POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true  
  200 OK ★ 2.8s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B
  │ {
  │   "status": "Model \"mnist\" Version: 1.0 registered w
  │ ith 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       414µs   (cache)      (cache)         (cache)         2.8s             1ms        73µs      2.8s  

  ✓  Successful POST request

→ HTTPS Get Mnist Model Description
  GET https://localhost:8444/models/mnist  
  200 OK ★ 7ms time ★ 238B↑ 737B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 470B
  │ [
  │   {
  │     "modelName": "mnist",
  │     "modelVersion": "1.0",
  │     "modelUrl": "mnist.mar",
  │     "runtime": "python",
  │     "minWorkers": 1,
  │     "maxWorkers": 1,
  │     "batchSize": 1,
  │     "maxBatchDelay": 100,
  │     "loadedAtStartup": false,
  │     "workers": [
  │       {
  │         "id": "9000",
  │         "startTime": "2023-08-14T22:57:03.878Z",
  │         "status": "READY",
  │         "memoryUsage": 0,
  │         "pid": 18632,
  │         "gpu": false,
  │         "gpuUsage": "N/A"
  │       }
  │     ]
  │   }
  │ ]
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       312µs   (cache)      (cache)         (cache)         4ms              1ms        60µs      7ms   

  ✓  Successful GET request

→ HTTPS Scale up Workers -  Synchronous for Mnist
  PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true  
  200 OK ★ 3.3s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Workers scaled to 5 for model: mnist"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       278µs   (cache)      (cache)         (cache)         3.3s             1ms        40µs      3.3s  

  ✓  Successful PUT request

→ HTTPS Scale up Workers - Asynchronous for Mnist
  PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false  
  202 Accepted ★ 5ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       287µs   (cache)      (cache)         (cache)         2ms              1ms        52µs      6ms   

  ✓  Successful PUT request

→ HTTPS - Inference - Mnist_KF
  POST https://localhost:8443/v1/models/mnist:predict  
  200 OK ★ 18ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 409B
  │ 
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B
  │ {
  │   "predictions": [
  │     2
  │   ]
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  6ms       695µs   (cache)      (cache)         (cache)         15ms             1ms        47µs      23ms  

  ✓  Status code is 200

→ HTTPS - Explanations - Mnist_KF 
  POST https://localhost:8443/v1/models/mnist:explain  
  200 OK ★ 51ms time ★ 680B↑ 25.4kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 409B
  │ 
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.17kB
  │ {
  │   "explanations": [
  │     [
  │       [
  │         [
  │           0.004570948738580575,
  │           0.006216969354192105,
  │           0.008197565570543713,
  │           0.00956357465709623,
  │           0.00899927455019611,
  │           0.009673474932247308,
  │           0.0075999051701512495,
  │           0.006361380884677352,
  │           0.005768828620731858,
  │           0.004394465852785626,
  │           0.004948218006895951,
  │           0.005273460552821625,
  │           0.005523799718367559,
  │           0.007789356415261442,
  │           0.008759363343907851,
  │           0.0043048051582686014,
  │           0.010970579130997414,
  │           0.0032484391603957454,
  │           0.005998033007716084,
  │           0.0037543659158808013,
  │           0.002765290843432279,
  │           0.004314086250885245,
  │           0.0014008573988940512,
  │           0.004841846505403689,
  │           0.0006374840586476711,
  │           0.001855853965693574,
  │           -0.0008280458167248678,
  │           -0.0
  │         ],
  │         [
  │           0.0016625159121928454,
  │           0.004443791781199298,
  │           0.012387840300503305,
  │           0.00945084395288644,
  │           0.016143820738507904,
  │           0.007797502048729162,
  │           0.0139422407842874,
  │           0.007557430473265285,
  │           0.0054790905833163525,
  │           0.009751320346115864,
  │           0.004764407372645556,
  │           0.007292147662234541,
  │           0.011797999372381933,
  │           0.006462684561652185,
  │           0.0033832142185796325,
  │           0.009225058876193865,
  │           0.0016750703115690345,
  │           0.007362304720604091,
  │           0.005270058056401144,
  │           0.005453598180888259,
  │           0.0043427257697093895,
  │           0.0058561332312329325,
  │           0.012168384678716265,
  │           0.009902719420388126,
  │           0.009813113255163137,
  │           0.0034427582108525158,
  │           0.0022924284763651136,
  │           -0.0
  │         ],
  │         [
  │           0.011528255254417058,
  │           0.009146989291478723,
  │           0.022265419864791573,
  │           0.017558217164003482,
  │           0.0247705545859691,
  │           0.021412693642065557,
  │  
  │ (showing 2.05kB/25.17kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  2ms       342µs   (cache)      (cache)         (cache)         49ms             1ms        38µs      53ms  

  ✓  Status code is 200

→ HTTPS UnRegister Model Mnist
  DELETE https://localhost:8444/models/mnist  
  200 OK ★ 86ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Model \"mnist\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       240µs   (cache)      (cache)         (cache)         83ms             1ms        34µs      86ms  

  ✓  Successful DELETE request

┌─────────────────────────┬─────────────────────┬────────────────────┐
│                         │            executed │             failed │
├─────────────────────────┼─────────────────────┼────────────────────┤
│              iterations │                   1 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│                requests │                  10 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│            test-scripts │                  10 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│      prerequest-scripts │                   0 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│              assertions │                  10 │                  0 │
├─────────────────────────┴─────────────────────┴────────────────────┤
│ total run duration: 6.9s                                           │
├────────────────────────────────────────────────────────────────────┤
│ total data received: 110.43kB (approx)                             │
├────────────────────────────────────────────────────────────────────┤
│ average response time: 660ms [min: 5ms, max: 3.3s, s.d.: 1226ms]   │
├────────────────────────────────────────────────────────────────────┤
│ average DNS lookup time: 507µs [min: 151µs, max: 1ms, s.d.: 502µs] │
├────────────────────────────────────────────────────────────────────┤
│ average first byte time: 639ms [min: 2ms, max: 3.3s, s.d.: 1235ms] │
└────────────────────────────────────────────────────────────────────┘
newman: could not find "htmlextra" reporter
  ensure that the reporter is installed in the same directory as newman
  please install reporter using npm

newman

kfv2_api_test_collection

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true  
  200 OK ★ 2.7s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B
  │ {
  │   "status": "Model \"mnist\" Version: 1.0 registered w
  │ ith 1 initial workers"
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total 
  29ms      4ms    1ms          289µs           2.7s             6ms        346µs     2.7s  

  ✓  Successful POST request

→ Model Zoo - Inference Model
  POST http://localhost:8080/v2/models/mnist/infer  
  200 OK ★ 29ms time ★ 7kB↑ 473B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 6.73kB
  │ (showing 2.05kB/6.73kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 238B
  │ {
  │   "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298",
  │   "model_name": "mnist",
  │   "model_version": "1.0",
  │   "outputs": [
  │     {
  │       "name": "input-0",
  │       "shape": [],
  │       "datatype": "INT64",
  │       "data": [
  │         1
  │       ]
  │     }
  │   ]
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total 
  5ms       1ms    486µs        234µs           24ms             2ms        64µs      34ms  

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Explanations Model
  POST http://localhost:8080/v2/models/mnist/explain  
  200 OK ★ 46ms time ★ 7kB↑ 14.07kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 6.73kB
  │ (showing 2.05kB/6.73kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.83kB
  │ {
  │   "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298",
  │   "model_name": "mnist",
  │   "model_version": "1.0",
  │   "outputs": [
  │     {
  │       "name": "input-0",
  │       "shape": [
  │         1,
  │         28,
  │         28
  │       ],
  │       "datatype": "FP64",
  │       "data": [
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.004054752930745232,
  │         -0.00022612877885488927,
  │         -0.00012734136368717274,
  │         0.005648369486054885,
  │    
  │ (showing 2.05kB/13.83kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       507µs   (cache)      (cache)         43ms             1ms        83µs      47ms  

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/mnist  
  200 OK ★ 23ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Model \"mnist\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       306µs   (cache)      (cache)         21ms             1ms        52µs      24ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 21ms time ★ 233B↑ 4.02kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.73kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 2.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 57.4
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com",} 8.439990997314453
  │ # HELP WorkerLoadTime Torchserve prometheus gauge metr
  │ ic with unit: Milliseconds
  │ # TYPE WorkerLoadTime gauge
  │ WorkerLoadTime{WorkerName="W-9000-mnist_1.0",Level="Ho
  │ st",Hostname="147dda19895c.ant.amazon.com",} 2589.0
  │ # HELP MemoryUsed Torchserve prometheus gauge metric w
  │ ith unit: Megabytes
  │ # TYPE MemoryUsed gauge
  │ MemoryUsed{Level="Host",Hostname="147dda19895c.ant.ama
  │ zon.com",} 8744.7109375
  │ # HELP PredictionTime Torchserve prometheus gauge metr
  │ ic with unit: ms
  │ # TYPE PredictionTime gauge
  │ PredictionTime{ModelName="mnist",Level="Model",Hostnam
  │ e="147dda19895c.ant.amazon.com",} 38.18
  │ # HELP MemoryAvailable Torchserve prometheus gauge met
  │ ric with unit: Megabytes
  │ # TYPE MemoryAvailable gauge
  │ MemoryAvailable{Level="Host",Hostname="147dda19895c.an
  │ t.amazon.com",} 6975.07421875
  │ # HELP Requests2XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests2XX counter
  │ Requests2XX{Level="Host",Hostname="147dda19895c.ant.am
  │ azon.com",} 4.0
  │ # HELP GPUMemoryUsed Torchserve prometheus gauge metri
  │ c with unit: Megabytes
  │ # TYPE GPUMemoryUsed gauge
  │ # HELP DiskAvailable Torchserve prometheus gauge metri
  │ c with unit: Gi
  │ (showing 2.05kB/3.73kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       656µs   184µs        228µs           18ms             1ms        75µs      22ms  

  ✓  Successful GET request

┌─────────────────────────┬─────────────────────┬─────────────────────┐
│                         │            executed │              failed │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│              iterations │                   1 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│                requests │                   5 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│            test-scripts │                   5 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│      prerequest-scripts │                   0 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│              assertions │                   7 │                   0 │
├─────────────────────────┴─────────────────────┴─────────────────────┤
│ total run duration: 3s                                              │
├─────────────────────────────────────────────────────────────────────┤
│ total data received: 17.92kB (approx)                               │
├─────────────────────────────────────────────────────────────────────┤
│ average response time: 572ms [min: 21ms, max: 2.7s, s.d.: 1084ms]   │
├─────────────────────────────────────────────────────────────────────┤
│ average DNS lookup time: 747µs [min: 184µs, max: 1ms, s.d.: 456µs]  │
├─────────────────────────────────────────────────────────────────────┤
│ average first byte time: 567ms [min: 18ms, max: 2.7s, s.d.: 1080ms] │
└─────────────────────────────────────────────────────────────────────┘
Current date and time : 2023-08-14 22:06:06
## In directory /Volumes/workplace/python_env/serve | Executing command pip install --force-reinstall .
## In directory /Volumes/workplace/python_env/serve | Executing command pip install --force-reinstall model-archiver/.
## In directory /Volumes/workplace/python_env/serve | Executing command pip install --force-reinstall workflow-archiver/.
## Starting generate_mars, mar_config:/Volumes/workplace/python_env/serve/ts_scripts/../ts_scripts/mar_config.json, model_store_dir:/Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen

## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name fastrcnn --version 1.0 --model-file examples/object_detector/fast-rcnn/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --archive-format zip-store --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force

## fastrcnn.mar is generated.

## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name alexnet --version 1.0 --model-file examples/image_classifier/alexnet/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet-owt-7be5be79.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force

## alexnet.mar is generated.

## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name densenet161 --version 1.0 --model-file examples/image_classifier/densenet_161/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161-8d451a50.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force

## densenet161.mar is generated.

## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name mnist --version 1.0 --model-file examples/image_classifier/mnist/mnist.py --serialized-file examples/image_classifier/mnist/mnist_cnn.pt --handler examples/image_classifier/mnist/mnist_handler.py --archive-format zip-store --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force

## mnist.mar is generated.

## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force

## resnet-152-batch.mar is generated.

## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name resnet-18 --version 1.0 --model-file examples/image_classifier/resnet_18/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet18-f37072fd.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force

## resnet-18.mar is generated.

## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name squeezenet1_1 --version 1.0 --model-file examples/image_classifier/squeezenet/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1-b8a52dc0.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force

## squeezenet1_1.mar is generated.

## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name vgg16 --version 1.0 --model-file examples/image_classifier/vgg_16/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16-397923af.pth --handler examples/image_classifier/vgg_16/vgg_handler.py --extra-files examples/image_classifier/index_to_name.json --archive-format zip-store --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force

## vgg16.mar is generated.

## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name deeplabv3_resnet_101_eager --version 1.0 --model-file examples/image_segmenter/deeplabv3/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet101_coco-586e9e4e.pth --handler image_segmenter --extra-files examples/image_segmenter/deeplabv3/deeplabv3.py,examples/image_segmenter/deeplabv3/intermediate_layer_getter.py,examples/image_segmenter/deeplabv3/fcn.py --archive-format zip-store --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force

## deeplabv3_resnet_101_eager.mar is generated.

## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name fcn_resnet_101 --version 1.0 --model-file examples/image_segmenter/fcn/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet101_coco-7ecb50ca.pth --handler image_segmenter --extra-files examples/image_segmenter/fcn/fcn.py,examples/image_segmenter/fcn/intermediate_layer_getter.py --archive-format zip-store --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force

## fcn_resnet_101.mar is generated.

## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name maskrcnn --version 1.0 --model-file examples/object_detector/maskrcnn/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --archive-format zip-store --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force

## maskrcnn.mar is generated.

## Starting gen_mar: model_store

## Create symlink for mar files

## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully.
## Starting TorchServe
## Console logs redirected to file: ts_console.log
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs
## Successfully started TorchServe
## Stopping TorchServe
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: ['torchserve', '--stop']
## Successfully stopped TorchServe
## Starting gen_mar: model_store

## Create symlink for mar files

## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully.
## Starting TorchServe
## Console logs redirected to file: ts_console.log
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties
## Successfully started TorchServe
## Stopping TorchServe
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: ['torchserve', '--stop']
## Successfully stopped TorchServe
## Starting gen_mar: model_store

## Create symlink for mar files

## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully.
## Starting TorchServe
## Console logs redirected to file: ts_console.log
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties
## Successfully started TorchServe
## Stopping TorchServe
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: ['torchserve', '--stop']
## Successfully stopped TorchServe
## Starting gen_mar: model_store

## Create symlink for mar files

## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully.
## Starting TorchServe
## Console logs redirected to file: ts_console.log
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config.properties
## Successfully started TorchServe
## Stopping TorchServe
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: ['torchserve', '--stop']
## Successfully stopped TorchServe
## Starting gen_mar: model_store

## Create symlink for mar files

## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully.
## Starting TorchServe
## Console logs redirected to file: ts_console.log
newman: could not find "htmlextra" reporter
  ensure that the reporter is installed in the same directory as newman
  please install reporter using npm

newman

kfv2_https_test_collection

→ HTTPS Inference API  Description
  OPTIONS https://localhost:8443  
  200 OK ★ 156ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB
  │ {
  │   "openapi": "3.0.1",
  │   "info": {
  │     "title": "TorchServe APIs",
  │     "description": "TorchServe is a flexible and easy 
  │ to use tool for serving deep learning models",
  │     "version": "0.8.1"
  │   },
  │   "paths": {
  │     "/": {
  │       "options": {
  │         "description": "Get openapi description.",
  │         "operationId": "apiDescription",
  │         "parameters": [],
  │         "responses": {
  │           "200": {
  │             "description": "A openapi 3.0.1 descriptor
  │ ",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "openapi",
  │                     "info",
  │                     "paths"
  │                   ],
  │                   "properties": {
  │                     "openapi": {
  │                       "type": "string"
  │                     },
  │                     "info": {
  │                       "type": "object"
  │                     },
  │                     "paths": {
  │                       "type": "object"
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           },
  │           "500": {
  │             "description": "Internal Server Error",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "code",
  │                     "type",
  │                     "message"
  │                   ],
  │                   "properties": {
  │                     "code": {
  │                       "type": "integer",
  │                       "description": "Error code."
  │                     },
  │                     "type": {
  │                       "type": "string",
  │                       "description": "Error type."
  │                     },
  │                     "message": {
  │                       "type": "string",
  │                       "description": "Error message."
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           }
  │         }
  │       }
  │     },
  │     "/ping": {
  │      
  │ (showing 2.05kB/23.41kB)
  └
  prepare   wait   dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  32ms      7ms    1ms          709µs           112ms           27ms             5ms        354µs     188ms 

  ✓  Status code is 200

→ HTTPS Management API Description
  OPTIONS https://localhost:8444  
  200 OK ★ 52ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB
  │ {
  │   "openapi": "3.0.1",
  │   "info": {
  │     "title": "TorchServe APIs",
  │     "description": "TorchServe is a flexible and easy 
  │ to use tool for serving deep learning models",
  │     "version": "0.8.1"
  │   },
  │   "paths": {
  │     "/": {
  │       "options": {
  │         "description": "Get openapi description.",
  │         "operationId": "apiDescription",
  │         "parameters": [],
  │         "responses": {
  │           "200": {
  │             "description": "A openapi 3.0.1 descriptor
  │ ",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "openapi",
  │                     "info",
  │                     "paths"
  │                   ],
  │                   "properties": {
  │                     "openapi": {
  │                       "type": "string"
  │                     },
  │                     "info": {
  │                       "type": "object"
  │                     },
  │                     "paths": {
  │                       "type": "object"
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           },
  │           "500": {
  │             "description": "Internal Server Error",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "code",
  │                     "type",
  │                     "message"
  │                   ],
  │                   "properties": {
  │                     "code": {
  │                       "type": "integer",
  │                       "description": "Error code."
  │                     },
  │                     "type": {
  │                       "type": "string",
  │                       "description": "Error type."
  │                     },
  │                     "message": {
  │                       "type": "string",
  │                       "description": "Error message."
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           }
  │         }
  │       }
  │     },
  │     "/models": {
  │    
  │ (showing 2.05kB/58.58kB)
  └
  prepare   wait   dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       1ms    165µs        350µs           34ms            12ms             1ms        65µs      52ms  

  ✓  Status code is 200

→ HTTPS Metrics API Description
  OPTIONS https://localhost:8445  
  200 OK ★ 36ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB
  │ {
  │   "openapi": "3.0.1",
  │   "info": {
  │     "title": "TorchServe APIs",
  │     "description": "TorchServe is a flexible and easy 
  │ to use tool for serving deep learning models",
  │     "version": "0.8.1"
  │   },
  │   "paths": {
  │     "/metrics": {
  │       "get": {
  │         "description": "Get TorchServe application met
  │ rics in prometheus format.",
  │         "operationId": "metrics",
  │         "parameters": [
  │           {
  │             "in": "query",
  │             "name": "name[]",
  │             "description": "Names of metrics to filter
  │ ",
  │             "required": false,
  │             "schema": {
  │               "type": "string"
  │             }
  │           }
  │         ],
  │         "responses": {
  │           "200": {
  │             "description": "TorchServe application met
  │ rics",
  │             "content": {
  │               "text/plain; version=0.0.4; charset=utf-
  │ 8": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "# HELP",
  │                     "# TYPE",
  │                     "metric"
  │                   ],
  │                   "properties": {
  │                     "# HELP": {
  │                       "type": "string",
  │                       "description": "Help text for To
  │ rchServe metric."
  │                     },
  │                     "# TYPE": {
  │                       "type": "string",
  │                       "description": "Type of TorchSer
  │ ve metric."
  │                     },
  │                     "metric": {
  │                       "type": "string",
  │                       "description": "TorchServe appli
  │ cation metric."
  │                     }
  │                   }
  │                 }
  │               }
  │             }
  │           },
  │           "500": {
  │             "description": "Internal Server Error",
  │             "content": {
  │               "application/json": {
  │                 "schema": {
  │                   "type": "object",
  │                   "required": [
  │                     "code",
  │                     "type",
  │                     "message"
  │                   ],
  │                   "properties": {
  │                     "code": {
  │                
  │ (showing 2.05kB/2.55kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       902µs   148µs        349µs           29ms            3ms              1ms        64µs      36ms  

  ✓  Status code is 200

→ HTTPS Register Model - Mnist
  POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true  
  200 OK ★ 2.6s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B
  │ {
  │   "status": "Model \"mnist\" Version: 1.0 registered w
  │ ith 1 initial workers"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       447µs   (cache)      (cache)         (cache)         2.6s             1ms        97µs      2.6s  

  ✓  Successful POST request

→ HTTPS Get Mnist Model Description
  GET https://localhost:8444/models/mnist  
  200 OK ★ 7ms time ★ 238B↑ 737B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 470B
  │ [
  │   {
  │     "modelName": "mnist",
  │     "modelVersion": "1.0",
  │     "modelUrl": "mnist.mar",
  │     "runtime": "python",
  │     "minWorkers": 1,
  │     "maxWorkers": 1,
  │     "batchSize": 1,
  │     "maxBatchDelay": 100,
  │     "loadedAtStartup": false,
  │     "workers": [
  │       {
  │         "id": "9000",
  │         "startTime": "2023-08-14T22:57:17.944Z",
  │         "status": "READY",
  │         "memoryUsage": 0,
  │         "pid": 18705,
  │         "gpu": false,
  │         "gpuUsage": "N/A"
  │       }
  │     ]
  │   }
  │ ]
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       306µs   (cache)      (cache)         (cache)         4ms              1ms        62µs      7ms   

  ✓  Successful GET request

→ HTTPS Scale up Workers -  Synchronous for Mnist
  PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true  
  200 OK ★ 3s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B
  │ {
  │   "status": "Workers scaled to 5 for model: mnist"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       272µs   (cache)      (cache)         (cache)         3s               1ms        44µs      3s    

  ✓  Successful PUT request

→ HTTPS Scale up Workers - Asynchronous for Mnist
  PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false  
  202 Accepted ★ 5ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Processing worker updates..."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       329µs   (cache)      (cache)         (cache)         2ms              1ms        47µs      6ms   

  ✓  Successful PUT request

→ HTTPS - Inference - Mnist_KF
  POST https://localhost:8443/v2/models/mnist/infer  
  200 OK ★ 17ms time ★ 7kB↑ 473B↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 6.73kB
  │ (showing 2.05kB/6.73kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 238B
  │ {
  │   "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298",
  │   "model_name": "mnist",
  │   "model_version": "1.0",
  │   "outputs": [
  │     {
  │       "name": "input-0",
  │       "shape": [],
  │       "datatype": "INT64",
  │       "data": [
  │         1
  │       ]
  │     }
  │   ]
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  6ms       733µs   (cache)      (cache)         (cache)         14ms             1ms        42µs      23ms  

  ✓  Status code is 200

→ HTTPS - Explanations - Mnist_KF 
  POST https://localhost:8443/v2/models/mnist/explain  
  200 OK ★ 50ms time ★ 7kB↑ 14.07kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 6.73kB
  │ (showing 2.05kB/6.73kB)
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.83kB
  │ {
  │   "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298",
  │   "model_name": "mnist",
  │   "model_version": "1.0",
  │   "outputs": [
  │     {
  │       "name": "input-0",
  │       "shape": [
  │         1,
  │         28,
  │         28
  │       ],
  │       "datatype": "FP64",
  │       "data": [
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         -0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         0.0,
  │         -0.0,
  │         -0.004054752930745232,
  │         -0.00022612877885488927,
  │         -0.00012734136368717274,
  │         0.005648369486054885,
  │    
  │ (showing 2.05kB/13.83kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       283µs   (cache)      (cache)         (cache)         48ms             1ms        43µs      51ms  

  ✓  Status code is 200

→ HTTPS UnRegister Model Mnist
  DELETE https://localhost:8444/models/mnist  
  200 OK ★ 95ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Model \"mnist\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   ssl-handshake   transfer-start   download   process   total 
  1ms       286µs   (cache)      (cache)         (cache)         92ms             1ms        36µs      95ms  

  ✓  Successful DELETE request

┌─────────────────────────┬─────────────────────┬────────────────────┐
│                         │            executed │             failed │
├─────────────────────────┼─────────────────────┼────────────────────┤
│              iterations │                   1 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│                requests │                  10 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│            test-scripts │                  10 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│      prerequest-scripts │                   0 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│              assertions │                  10 │                  0 │
├─────────────────────────┴─────────────────────┴────────────────────┤
│ total run duration: 6.5s                                           │
├────────────────────────────────────────────────────────────────────┤
│ total data received: 99.3kB (approx)                               │
├────────────────────────────────────────────────────────────────────┤
│ average response time: 617ms [min: 5ms, max: 3s, s.d.: 1135ms]     │
├────────────────────────────────────────────────────────────────────┤
│ average DNS lookup time: 527µs [min: 148µs, max: 1ms, s.d.: 524µs] │
├────────────────────────────────────────────────────────────────────┤
│ average first byte time: 595ms [min: 2ms, max: 3s, s.d.: 1144ms]   │
└────────────────────────────────────────────────────────────────────┘
newman: could not find "htmlextra" reporter
  ensure that the reporter is installed in the same directory as newman
  please install reporter using npm

newman

explanation_api_test_collection

→ Model Zoo - Register Model
  POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true  
  200 OK ★ 3s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B
  │ {
  │   "status": "Model \"mnist\" Version: 1.0 registered w
  │ ith 1 initial workers"
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total 
  34ms      5ms    1ms          313µs           2.9s             6ms        402µs     3s    

  ✓  Successful POST request

→ Model Zoo - Explanations Model
  POST http://localhost:8080/explanations/mnist  
  200 OK ★ 71ms time ★ 538B↑ 23.23kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies
  ┌ ↑ file ★ 272B
  │ 
  └ 
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 22.99kB
  │ [
  │   [
  │     [
  │       -0.0003991944897845666,
  │       -0.00019002183247341165,
  │       -0.0008597191072726287,
  │       -0.0003293672080165372,
  │       -0.0009114924870760632,
  │       -0.00017816119210407766,
  │       -0.0005801029093699504,
  │       -5.752924167658201e-05,
  │       -0.0001303639263143984,
  │       -3.62233673383041e-05,
  │       1.2628184604974416e-05,
  │       -3.151631746970368e-05,
  │       -7.058358316301549e-05,
  │       -6.872771557622861e-05,
  │       -7.877429248010376e-05,
  │       -0.00013302446788670177,
  │       -3.925597174534168e-05,
  │       -0.0002292955422251281,
  │       1.7131089343971934e-05,
  │       -0.00038463530050298966,
  │       -0.0005095514626382591,
  │       -0.0003988010630332614,
  │       -0.0008459618762649931,
  │       -0.000503479098073749,
  │       -0.0006029825551944304,
  │       -0.0001768611590467448,
  │       -0.00015337617191751434,
  │       -0.0
  │     ],
  │     [
  │       -0.0004593670666297282,
  │       -0.0006972018195975884,
  │       -0.0006564253744532494,
  │       -0.0006832953638197262,
  │       -0.00043800711736966284,
  │       -0.0006188232666591454,
  │       8.717681738462818e-06,
  │       -0.00039305821681504525,
  │       -0.00010256508455661123,
  │       -0.00018236529665861955,
  │       -0.0005491941269239254,
  │       4.519850486390986e-05,
  │       -0.0004397756964768044,
  │       -0.0003280011955237874,
  │       -0.0003846252129601287,
  │       -0.0007280317190516349,
  │       -0.000380770012536674,
  │       -0.0006797484622773033,
  │       -0.0009410587129815517,
  │       -0.000532137644323727,
  │       -0.0010959785235309994,
  │       -0.0009239369563404389,
  │       -0.0007882726006158667,
  │       -0.0007635521758941316,
  │       -0.0003335173363825757,
  │       5.101228470974375e-05,
  │       7.699817174764277e-05,
  │       -0.0
  │     ],
  │     [
  │       -0.001316505642882128,
  │       -0.0009519237285693226,
  │       -0.0021000624054648156,
  │       -0.0007986434497030843,
  │       -0.0015434155160820046,
  │       -0.0008677095829840114,
  │       -0.0006676295170549929,
  │       -0.0010158373344518667,
  │       -0.0006892677787870458,
  │       -0.0010906412120094005,
  │       -0.0009184818727384202,
  │       -0.001018668423858874,
  │       -0.
  │ (showing 2.05kB/22.99kB)
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total 
  5ms       1ms    463µs        200µs           66ms             2ms        71µs      76ms  

  ✓  Successful POST request
  ✓  Test expected JSON response

→ Model Zoo - Unregister model
  DELETE http://localhost:8081/models/mnist  
  200 OK ★ 22ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B
  │ {
  │   "status": "Model \"mnist\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       525µs   (cache)      (cache)         19ms             1ms        81µs      23ms  

  ✓  Successful DELETE request

→ Model Zoo - Model Metrics
  GET http://localhost:8082/metrics  
  200 OK ★ 21ms time ★ 233B↑ 4kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.71kB
  │ # HELP GPUMemoryUtilization Torchserve prometheus gaug
  │ e metric with unit: Percent
  │ # TYPE GPUMemoryUtilization gauge
  │ # HELP ts_inference_requests_total Torchserve promethe
  │ us counter metric with unit: Count
  │ # TYPE ts_inference_requests_total counter
  │ ts_inference_requests_total{model_name="mnist",model_v
  │ ersion="default",hostname="147dda19895c.ant.amazon.com
  │ ",} 1.0
  │ # HELP MemoryUtilization Torchserve prometheus gauge m
  │ etric with unit: Percent
  │ # TYPE MemoryUtilization gauge
  │ MemoryUtilization{Level="Host",Hostname="147dda19895c.
  │ ant.amazon.com",} 57.4
  │ # HELP Requests4XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests4XX counter
  │ # HELP Requests5XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests5XX counter
  │ # HELP DiskUsage Torchserve prometheus gauge metric wi
  │ th unit: Gigabytes
  │ # TYPE DiskUsage gauge
  │ DiskUsage{Level="Host",Hostname="147dda19895c.ant.amaz
  │ on.com",} 8.439990997314453
  │ # HELP WorkerLoadTime Torchserve prometheus gauge metr
  │ ic with unit: Milliseconds
  │ # TYPE WorkerLoadTime gauge
  │ WorkerLoadTime{WorkerName="W-9000-mnist_1.0",Level="Ho
  │ st",Hostname="147dda19895c.ant.amazon.com",} 2840.0
  │ # HELP MemoryUsed Torchserve prometheus gauge metric w
  │ ith unit: Megabytes
  │ # TYPE MemoryUsed gauge
  │ MemoryUsed{Level="Host",Hostname="147dda19895c.ant.ama
  │ zon.com",} 8658.71875
  │ # HELP PredictionTime Torchserve prometheus gauge metr
  │ ic with unit: ms
  │ # TYPE PredictionTime gauge
  │ PredictionTime{ModelName="mnist",Level="Model",Hostnam
  │ e="147dda19895c.ant.amazon.com",} 46.16
  │ # HELP MemoryAvailable Torchserve prometheus gauge met
  │ ric with unit: Megabytes
  │ # TYPE MemoryAvailable gauge
  │ MemoryAvailable{Level="Host",Hostname="147dda19895c.an
  │ t.amazon.com",} 6975.15234375
  │ # HELP Requests2XX Torchserve prometheus counter metri
  │ c with unit: Count
  │ # TYPE Requests2XX counter
  │ Requests2XX{Level="Host",Hostname="147dda19895c.ant.am
  │ azon.com",} 3.0
  │ # HELP GPUMemoryUsed Torchserve prometheus gauge metri
  │ c with unit: Megabytes
  │ # TYPE GPUMemoryUsed gauge
  │ # HELP DiskAvailable Torchserve prometheus gauge metri
  │ c with unit: Giga
  │ (showing 2.05kB/3.71kB)
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       620µs   168µs        232µs           17ms             1ms        75µs      21ms  

  ✓  Successful GET request

┌─────────────────────────┬─────────────────────┬─────────────────────┐
│                         │            executed │              failed │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│              iterations │                   1 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│                requests │                   4 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│            test-scripts │                   4 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│      prerequest-scripts │                   0 │                   0 │
├─────────────────────────┼─────────────────────┼─────────────────────┤
│              assertions │                   5 │                   0 │
├─────────────────────────┴─────────────────────┴─────────────────────┤
│ total run duration: 3.3s                                            │
├─────────────────────────────────────────────────────────────────────┤
│ total data received: 26.83kB (approx)                               │
├─────────────────────────────────────────────────────────────────────┤
│ average response time: 779ms [min: 21ms, max: 3s, s.d.: 1284ms]     │
├─────────────────────────────────────────────────────────────────────┤
│ average DNS lookup time: 780µs [min: 168µs, max: 1ms, s.d.: 572µs]  │
├─────────────────────────────────────────────────────────────────────┤
│ average first byte time: 773ms [min: 17ms, max: 2.9s, s.d.: 1279ms] │
└─────────────────────────────────────────────────────────────────────┘
newman: could not find "htmlextra" reporter
  ensure that the reporter is installed in the same directory as newman
  please install reporter using npm

newman

management_api_collection

Iteration 1/11

→ workflow management request
  POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war  
  200 OK ★ 45.2s time ★ 321B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B
  │ {
  │   "status": "Workflow densenet has been registered and
  │  scaled successfully."
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total 
  33ms      5ms    1ms          313µs           45.2s            7ms        447µs     45.3s 

  ✓  Successful  request

Iteration 2/11

→ workflow management request
  GET http://localhost:8081/workflows  
  200 OK ★ 7ms time ★ 235B↑ 423B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 156B
  │ {
  │   "workflows": [
  │     {
  │       "workflowName": "densenet",
  │       "workflowUrl": "https://torchserve.s3.amazonaws.
  │ com/war_files/densenet_wf.war"
  │     }
  │   ]
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       738µs   (cache)      (cache)         4ms              1ms        82µs      9ms   

  ✓  Successful  request

Iteration 3/11

→ workflow management request
  GET http://localhost:8081/workflows/densenet  
  200 OK ★ 5ms time ★ 244B↑ 559B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 292B
  │ [
  │   {
  │     "workflowName": "densenet",
  │     "workflowUrl": "https://torchserve.s3.amazonaws.co
  │ m/war_files/densenet_wf.war",
  │     "minWorkers": 1,
  │     "maxWorkers": 1,
  │     "batchSize": 1,
  │     "maxBatchDelay": 50,
  │     "workflowDag": "{pre_processing=[densenet], densen
  │ et=[post_processing]}"
  │   }
  │ ]
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       260µs   (cache)      (cache)         2ms              1ms        71µs      6ms   

  ✓  Successful  request

Iteration 4/11

→ workflow management request
  POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war  
  500 Internal Server Error ★ 6ms time ★ 321B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B
  │ {
  │   "code": 500,
  │   "type": "FileAlreadyExistsException",
  │   "message": "densenet_wf.war"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       247µs   (cache)      (cache)         2ms              1ms        56µs      6ms   

  ✓  Successful  request

Iteration 5/11

→ workflow management request
  DELETE http://localhost:8081/workflows/densenet  
  200 OK ★ 46ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B
  │ {
  │   "status": "Workflow \"densenet\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       664µs   638µs        280µs           42ms             2ms        90µs      47ms  

  ✓  Successful  request

Iteration 6/11

→ workflow management request
  POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/does_not_exist.war  
  400 Bad Request ★ 782ms time ★ 324B↑ 441B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 170B
  │ {
  │   "code": 400,
  │   "type": "DownloadArchiveException",
  │   "message": "Failed to download archive from: https:/
  │ /torchserve.s3.amazonaws.com/war_files/does_not_exist.
  │ war"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       232µs   (cache)      (cache)         779ms            1ms        54µs      782ms 

  ✓  Successful  request

Iteration 7/11

→ workflow management request
  GET http://localhost:8081/workflows/does_not_exist  
  404 Not Found ★ 15ms time ★ 250B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B
  │ {
  │   "code": 404,
  │   "type": "WorkflowNotFoundException",
  │   "message": "Workflow not found: does_not_exist"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       651µs   205µs        272µs           11ms             1ms        47µs      15ms  

  ✓  Successful  request

Iteration 8/11

→ workflow management request
  DELETE http://localhost:8081/workflows/does_not_exist  
  404 Not Found ★ 14ms time ★ 253B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B
  │ {
  │   "code": 404,
  │   "type": "WorkflowNotFoundException",
  │   "message": "Workflow not found: does_not_exist"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       666µs   184µs        201µs           10ms             1ms        45µs      15ms  

  ✓  Successful  request

Iteration 9/11

→ workflow management request
  POST http://localhost:8081/workflows?url=malformed_url,?  
  404 Not Found ★ 15ms time ★ 275B↑ 396B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 127B
  │ {
  │   "code": 404,
  │   "type": "WorkflowNotFoundException",
  │   "message": "Workflow not found in workflow store: ma
  │ lformed_url,?"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       624µs   171µs        282µs           11ms             1ms        41µs      15ms  

  ✓  Successful  request

Iteration 10/11

→ workflow management request
  POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/custom_python_dep.war  
  500 Internal Server Error ★ 2.8s time ★ 327B↑ 531B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 250B
  │ {
  │   "code": 500,
  │   "type": "WorkflowException",
  │   "message": "Workflow custom_python_dep has failed to
  │  register. Failures: [Workflow Node custom_python_dep_
  │ _custom_python_dep failed to register. Details: Model 
  │ not found at: custom_python_dep.mar]"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       966µs   146µs        238µs           2.8s             1ms        36µs      2.8s  

  ✓  Successful  request

Iteration 11/11

→ workflow management request
  POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/loading-memory-error.war  
  500 Internal Server Error ★ 1762ms time ★ 330B↑ 543B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 262B
  │ {
  │   "code": 500,
  │   "type": "WorkflowException",
  │   "message": "Workflow loading-memory-error has failed
  │  to register. Failures: [Workflow Node loading-memory-
  │ error__loading-memory-error failed to register. Detail
  │ s: Model not found at: loading-memory-error.mar]"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total  
  1ms       619µs   164µs        275µs           1758ms           1ms        46µs      1762ms 

  ✓  Successful  request

┌─────────────────────────┬─────────────────────┬────────────────────┐
│                         │            executed │             failed │
├─────────────────────────┼─────────────────────┼────────────────────┤
│              iterations │                  11 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│                requests │                  11 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│            test-scripts │                  11 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│      prerequest-scripts │                   0 │                  0 │
├─────────────────────────┼─────────────────────┼────────────────────┤
│              assertions │                  11 │                  0 │
├─────────────────────────┴─────────────────────┴────────────────────┤
│ total run duration: 51.1s                                          │
├────────────────────────────────────────────────────────────────────┤
│ total data received: 1.7kB (approx)                                │
├────────────────────────────────────────────────────────────────────┤
│ average response time: 4.6s [min: 5ms, max: 45.2s, s.d.: 12.8s]    │
├────────────────────────────────────────────────────────────────────┤
│ average DNS lookup time: 792µs [min: 146µs, max: 1ms, s.d.: 615µs] │
├────────────────────────────────────────────────────────────────────┤
│ average first byte time: 4.6s [min: 2ms, max: 45.2s, s.d.: 12.8s]  │
└────────────────────────────────────────────────────────────────────┘
newman: could not find "htmlextra" reporter
  ensure that the reporter is installed in the same directory as newman
  please install reporter using npm

newman

workflow_inference

Iteration 1/3

→ Workflow - Register Workflow
  POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war  
  200 OK ★ 1m 6s time ★ 321B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B
  │ {
  │   "status": "Workflow densenet has been registered and
  │  scaled successfully."
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total   
  31ms      5ms    1ms          310µs           1m 6s            6ms        494µs     1m 6.1s 

  ✓  Successful POST request

→ Workflow - Inference
  POST http://localhost:8080/wfpredict/densenet  
  200 OK ★ 466ms time ★ 111.24kB↑ 450B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 183B
  │ {
  │   "tabby": 0.46661895513534546,
  │   "tiger_cat": 0.46449047327041626,
  │   "Egyptian_cat": 0.06614057719707489,
  │   "lynx": 0.0012924439506605268,
  │   "plastic_bag": 0.00022909804829396307
  │ }
  └
  prepare   wait   dns-lookup   tcp-handshake   transfer-start   download   process   total 
  6ms       1ms    272µs        340µs           461ms            2ms        56µs      471ms 

  ✓  Successful  request
  ✓  Test expected JSON response

→ Workflow - Unregister
  DELETE http://localhost:8081/workflows/densenet  
  200 OK ★ 42ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B
  │ {
  │   "status": "Workflow \"densenet\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       453µs   (cache)      (cache)         38ms             1ms        88µs      42ms  

  ✓  Successful DELETE request

Iteration 2/3

→ Workflow - Register Workflow
  POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/timeout-retry-test.war  
  200 OK ★ 1m 6.6s time ★ 328B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B
  │ {
  │   "status": "Workflow densenet has been registered and
  │  scaled successfully."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total   
  2ms       229µs   (cache)      (cache)         1m 6.6s          1ms        60µs      1m 6.6s 

  ✓  Successful POST request

→ Workflow - Inference
  POST http://localhost:8080/wfpredict/densenet  
  500 Internal Server Error ★ 12s time ★ 111.24kB↑ 383B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↑ file ★ 110.97kB
  │ (showing 2.05kB/110.97kB)
  └ 
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B
  │ {
  │   "code": 500,
  │   "type": "InternalServerException",
  │   "message": "Error executing pre_processing"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       312µs   (cache)      (cache)         12s              2ms        69µs      12s   

  ✓  Successful  request

→ Workflow - Unregister
  DELETE http://localhost:8081/workflows/densenet  
  200 OK ★ 39ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B
  │ {
  │   "status": "Workflow \"densenet\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  2ms       265µs   (cache)      (cache)         35ms             2ms        61µs      40ms  

  ✓  Successful DELETE request

Iteration 3/3

→ Workflow - Register Workflow
  POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/dog_breed_wf.war  
  200 OK ★ 1m 14.2s time ★ 322B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B
  │ {
  │   "status": "Workflow dog_breed_wf has been registered
  │  and scaled successfully."
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total    
  1ms       262µs   (cache)      (cache)         1m 14.2s         2ms        46µs      1m 14.2s 

  ✓  Successful POST request

→ Workflow - Inference
  POST http://localhost:8080/wfpredict/dog_breed_wf  
  200 OK ★ 500ms time ★ 77.75kB↑ 473B↓ size ★ 8↑ 7↓ headers ★ 0 cookies
  ┌ ↑ file ★ 77.47kB
  │ (showing 2.05kB/77.47kB)
  └ 
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 206B
  │ {
  │   "Cardigan_welsh_corgi": 0.6281960606575012,
  │   "Briard": 0.27462849020957947,
  │   "Icelandic_sheepdog": 0.07453601807355881,
  │   "Tibetan_mastiff": 0.015545750968158245,
  │   "Chow_chow": 0.002062038518488407
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  3ms       737µs   212µs        246µs           497ms            1ms        56µs      503ms 

  ✓  Successful  request
  ✓  Test expected JSON response

→ Workflow - Unregister
  DELETE http://localhost:8081/workflows/dog_breed_wf  
  200 OK ★ 37ms time ★ 251B↑ 323B↓ size ★ 7↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 57B
  │ {
  │   "status": "Workflow \"dog_breed_wf\" unregistered"
  │ }
  └
  prepare   wait    dns-lookup   tcp-handshake   transfer-start   download   process   total 
  1ms       236µs   (cache)      (cache)         33ms             2ms        42µs      37ms  

  ✓  Successful DELETE request

┌─────────────────────────┬───────────────────────┬──────────────────────┐
│                         │              executed │               failed │
├─────────────────────────┼───────────────────────┼──────────────────────┤
│              iterations │                     3 │                    0 │
├─────────────────────────┼───────────────────────┼──────────────────────┤
│                requests │                     9 │                    0 │
├─────────────────────────┼───────────────────────┼──────────────────────┤
│            test-scripts │                     9 │                    0 │
├─────────────────────────┼───────────────────────┼──────────────────────┤
│      prerequest-scripts │                     0 │                    0 │
├─────────────────────────┼───────────────────────┼──────────────────────┤
│              assertions │                    11 │                    0 │
├─────────────────────────┴───────────────────────┴──────────────────────┤
│ total run duration: 3m 40.4s                                           │
├────────────────────────────────────────────────────────────────────────┤
│ total data received: 901B (approx)                                     │
├────────────────────────────────────────────────────────────────────────┤
│ average response time: 24.4s [min: 37ms, max: 1m 14.2s, s.d.: 31.7s]   │
├────────────────────────────────────────────────────────────────────────┤
│ average DNS lookup time: 785µs [min: 212µs, max: 1ms, s.d.: 597µs]     │
├────────────────────────────────────────────────────────────────────────┤
│ average first byte time: 24.4s [min: 33ms, max: 1m 14.2s, s.d.: 31.7s] │
└────────────────────────────────────────────────────────────────────────┘
WARNING - Overwriting /private/var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T/workspace/model_store/densenet161_v1.mar ...
/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/grpc_tools/protoc.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
============================= test session starts ==============================
platform darwin -- Python 3.8.13, pytest-7.3.1, pluggy-1.0.0 -- /Users/lninga/opt/anaconda3/envs/py38/bin/python
cachedir: .pytest_cache
rootdir: /Volumes/workplace/python_env/serve
plugins: mock-3.10.0, cov-4.1.0
collecting ... collected 109 items

test_distributed_inference_handler.py::test_large_model_inference SKIPPED [  0%]
test_example_dcgan.py::test_model_archive_creation PASSED                [  1%]
test_example_dcgan.py::test_model_register_unregister PASSED             [  2%]
test_example_dcgan.py::test_image_generation_without_any_input_constraints PASSED [  3%]
test_example_dcgan.py::test_image_generation_with_input_constraints PASSED [  4%]
test_example_intel_extension_for_pytorch.py::test_single_worker_affinity SKIPPED [  5%]
test_example_intel_extension_for_pytorch.py::test_multi_worker_affinity SKIPPED [  6%]
test_example_intel_extension_for_pytorch.py::test_worker_scale_up_affinity SKIPPED [  7%]
test_example_intel_extension_for_pytorch.py::test_worker_scale_down_affinity SKIPPED [  8%]
test_example_micro_batching.py::test_single_example_inference[yaml_config] PASSED [  9%]
test_example_micro_batching.py::test_multi_example_inference[4-yaml_config] PASSED [ 10%]
test_example_micro_batching.py::test_multi_example_inference[4-no_config] PASSED [ 11%]
test_example_micro_batching.py::test_single_example_inference[no_config] 2023-08-14T23:23:34,820 [DEBUG] W-9001-image_classifier_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2023-08-14T23:23:34,820 [INFO ] W-9001-image_classifier_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 82
2023-08-14T23:23:34,821 [INFO ] W-9001-image_classifier_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:2.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692080614
PASSED [ 11%]
test_example_micro_batching.py::test_multi_example_inference[16-no_config] PASSED [ 12%]
test_example_micro_batching.py::test_multi_example_inference[16-yaml_config] PASSED [ 13%]
test_example_scriptable_tokenzier.py::test_handler PASSED                [ 14%]
test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_sample_text PASSED [ 15%]
test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_empty_string 2023-08-14T23:24:20,800 [DEBUG] W-9001-scriptable_tokenizer_untrained_1.0 org.pytorch.serve.job.Job - Waiting time ns: 186616, Backend time ns: 336224511
2023-08-14T23:24:20,801 [INFO ] W-9001-scriptable_tokenizer_untrained_1.0 TS_METRICS - QueueTime.Milliseconds:0.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692080660
2023-08-14T23:24:20,801 [DEBUG] W-9001-scriptable_tokenizer_untrained_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2023-08-14T23:24:20,801 [INFO ] W-9001-scriptable_tokenizer_untrained_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 335
2023-08-14T23:24:20,801 [INFO ] W-9001-scriptable_tokenizer_untrained_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:2.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692080660
PASSED [ 16%]
test_example_scriptable_tokenzier.py::test_inference_with_pretrained_model PASSED [ 17%]
test_gRPC_inference_api.py::test_inference_apis PASSED                   [ 18%]
test_gRPC_inference_api.py::test_inference_stream_apis PASSED            [ 19%]
test_gRPC_management_apis.py::test_management_apis PASSED                [ 20%]
test_handler.py::test_mnist_model_register_and_inference_on_valid_model PASSED [ 21%]
test_handler.py::test_mnist_model_register_using_non_existent_handler_with_nonzero_workers PASSED [ 22%]
test_handler.py::test_mnist_model_register_scale_inference_with_non_existent_handler 2023-08-15T00:01:14,810 [INFO ] W-9003-mnist_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9003 in 1 seconds.
2023-08-15T00:01:14,810 [INFO ] W-9002-mnist_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9002-mnist_1.0-stderr
2023-08-15T00:01:14,810 [WARN ] W-9002-mnist_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9002-mnist_1.0-stderr
2023-08-15T00:01:14,810 [WARN ] W-9002-mnist_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9002-mnist_1.0-stdout
2023-08-15T00:01:14,810 [INFO ] W-9002-mnist_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9002 in 1 seconds.
PASSED [ 22%]
test_handler.py::test_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 23%]
test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model PASSED [ 24%]
test_handler.py::test_kserve_mnist_model_register_scale_inference_with_non_existent_handler 2023-08-15T00:01:32,232 [INFO ] W-9002-mnist_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9002-mnist_1.0-stdout
2023-08-15T00:01:32,232 [INFO ] W-9002-mnist_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9002-mnist_1.0-stderr
2023-08-15T00:01:32,232 [WARN ] W-9002-mnist_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9002-mnist_1.0-stderr
2023-08-15T00:01:32,233 [WARN ] W-9002-mnist_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9002-mnist_1.0-stdout
2023-08-15T00:01:32,233 [INFO ] W-9002-mnist_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9002 in 1 seconds.
PASSED [ 25%]
test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 26%]
test_handler.py::test_huggingface_bert_batch_inference PASSED            [ 27%]
test_handler.py::test_MMF_activity_recognition_model_register_and_inference_on_valid_model SKIPPED [ 28%]
test_handler.py::test_huggingface_bert_model_parallel_inference PASSED   [ 29%]
test_handler.py::test_echo_stream_inference 2023-08-15T00:05:18,714 [DEBUG] W-9000-echo_stream_1.0 org.pytorch.serve.job.Job - Waiting time ns: 145662, Backend time ns: 2588202
2023-08-15T00:05:18,714 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - QueueTime.Milliseconds:0.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,714 [DEBUG] W-9000-echo_stream_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: false
2023-08-15T00:05:18,715 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - Requests2XX.Count:1.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,715 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - ts_inference_latency_microseconds.Microseconds:3234.577|#model_name:echo_stream,model_version:default|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,715 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - ts_queue_latency_microseconds.Microseconds:145.662|#model_name:echo_stream,model_version:default|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,715 [DEBUG] W-9000-echo_stream_1.0 org.pytorch.serve.job.Job - Waiting time ns: 145662, Backend time ns: 3767460
2023-08-15T00:05:18,715 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - QueueTime.Milliseconds:0.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,715 [DEBUG] W-9000-echo_stream_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: false
2023-08-15T00:05:18,716 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - Requests2XX.Count:1.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,716 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - ts_inference_latency_microseconds.Microseconds:4365.98|#model_name:echo_stream,model_version:default|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,716 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - ts_queue_latency_microseconds.Microseconds:145.662|#model_name:echo_stream,model_version:default|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,716 [DEBUG] W-9000-echo_stream_1.0 org.pytorch.serve.job.Job - Waiting time ns: 145662, Backend time ns: 4811969
2023-08-15T00:05:18,716 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - QueueTime.Milliseconds:0.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,717 [DEBUG] W-9000-echo_stream_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: false
2023-08-15T00:05:18,717 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - Requests2XX.Count:1.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,717 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - ts_inference_latency_microseconds.Microseconds:5429.19|#model_name:echo_stream,model_version:default|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,717 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - ts_queue_latency_microseconds.Microseconds:145.662|#model_name:echo_stream,model_version:default|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,717 [DEBUG] W-9000-echo_stream_1.0 org.pytorch.serve.job.Job - Waiting time ns: 145662, Backend time ns: 5809938
2023-08-15T00:05:18,717 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - QueueTime.Milliseconds:0.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,718 [DEBUG] W-9000-echo_stream_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2023-08-15T00:05:18,718 [INFO ] W-9000-echo_stream_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2
2023-08-15T00:05:18,718 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:5.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
FAILED                       [ 30%]
test_metrics.py::test_logs_created PASSED                                [ 31%]
test_metrics.py::test_logs_startup_cfg_created_snapshot_enabled PASSED   [ 32%]
test_metrics.py::test_logs_startup_cfg_created_snapshot_disabled PASSED  [ 33%]
test_metrics.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 33%]
test_metrics.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 34%]
test_metrics.py::test_log_location_var_snapshot_disabled PASSED          [ 35%]
test_metrics.py::test_log_location_var_snapshot_enabled PASSED           [ 36%]
test_metrics.py::test_async_logging PASSED                               [ 37%]
test_metrics.py::test_async_logging_non_boolean PASSED                   [ 38%]
test_metrics.py::test_metrics_location_var_snapshot_disabled PASSED      [ 39%]
test_metrics.py::test_metrics_location_var_snapshot_enabled PASSED       [ 40%]
test_metrics.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 41%]
test_metrics.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 42%]
test_metrics.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 43%]
test_metrics.py::test_metrics_log_mode PASSED                            [ 44%]
test_metrics.py::test_metrics_prometheus_mode PASSED                     [ 44%]
test_metrics.py::test_collect_system_metrics_when_not_disabled PASSED    [ 45%]
test_metrics.py::test_disable_system_metrics_using_config_properties PASSED [ 46%]
test_metrics.py::test_disable_system_metrics_using_environment_variable PASSED [ 47%]
test_metrics_kf.py::test_logs_created PASSED                             [ 48%]
test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_enabled PASSED [ 49%]
test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_disabled PASSED [ 50%]
test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 51%]
test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 52%]
test_metrics_kf.py::test_log_location_var_snapshot_disabled PASSED       [ 53%]
test_metrics_kf.py::test_log_location_var_snapshot_enabled PASSED        [ 54%]
test_metrics_kf.py::test_async_logging PASSED                            [ 55%]
test_metrics_kf.py::test_async_logging_non_boolean PASSED                [ 55%]
test_metrics_kf.py::test_metrics_location_var_snapshot_disabled PASSED   [ 56%]
test_metrics_kf.py::test_metrics_location_var_snapshot_enabled PASSED    [ 57%]
test_metrics_kf.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 58%]
test_metrics_kf.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 59%]
test_metrics_kf.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 60%]
test_model_archiver.py::test_multiple_model_versions_registration PASSED [ 61%]
test_model_archiver.py::test_duplicate_model_registration_using_local_url_followed_by_http_url PASSED [ 62%]
test_model_archiver.py::test_duplicate_model_registration_using_http_url_followed_by_local_url PASSED [ 63%]
test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_without_force PASSED [ 64%]
test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_with_force PASSED [ 65%]
test_model_archiver.py::test_model_archiver_without_handler_flag PASSED  [ 66%]
test_model_archiver.py::test_model_archiver_without_model_name_flag PASSED [ 66%]
test_model_archiver.py::test_model_archiver_without_model_file_flag PASSED [ 67%]
test_model_archiver.py::test_model_archiver_without_serialized_flag PASSED [ 68%]
test_onnx.py::test_convert_to_onnx PASSED                                [ 69%]
test_onnx.py::test_model_packaging_and_start PASSED                      [ 70%]
test_onnx.py::test_model_start PASSED                                    [ 71%]
test_onnx.py::test_inference PASSED                                      [ 72%]
test_onnx.py::test_stop PASSED                                           [ 73%]
test_pytorch_profiler.py::test_profiler_default_and_custom_handler[/Volumes/workplace/python_env/serve/test/pytest/profiler_utils/resnet_custom.py] PASSED [ 74%]
test_pytorch_profiler.py::test_profiler_default_and_custom_handler[image_classifier] PASSED [ 75%]
test_pytorch_profiler.py::test_profiler_arguments_override[/Volumes/workplace/python_env/serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 76%]
test_pytorch_profiler.py::test_batch_input[/Volumes/workplace/python_env/serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 77%]
test_sm_mme_requirements.py::test_no_model_loaded PASSED                 [ 77%]
test_sm_mme_requirements.py::test_oom_on_model_load SKIPPED (Test to...) [ 78%]
test_sm_mme_requirements.py::test_oom_on_invoke SKIPPED (Test to be ...) [ 79%]
test_snapshot.py::test_snapshot_created_on_start_and_stop PASSED         [ 80%]
test_snapshot.py::test_snapshot_created_on_management_api_invoke PASSED  [ 81%]
test_snapshot.py::test_start_from_snapshot PASSED                        [ 82%]
test_snapshot.py::test_start_from_latest PASSED                          [ 83%]
test_snapshot.py::test_start_from_read_only_snapshot PASSED              [ 84%]
test_snapshot.py::test_no_config_snapshots_cli_option PASSED             [ 85%]
test_snapshot.py::test_start_from_default PASSED                         [ 86%]
test_snapshot.py::test_start_from_non_existing_snapshot PASSED           [ 87%]
test_snapshot.py::test_torchserve_init_with_non_existent_model_store PASSED [ 88%]
test_snapshot.py::test_restart_torchserve_with_last_snapshot_with_model_mar_removed PASSED [ 88%]
test_snapshot.py::test_replace_mar_file_with_dummy PASSED                [ 89%]
test_snapshot.py::test_restart_torchserve_with_one_of_model_mar_removed PASSED [ 90%]
test_torch_compile.py::TestTorchCompile::test_archive_model_artifacts SKIPPED [ 91%]
test_torch_compile.py::TestTorchCompile::test_start_torchserve SKIPPED   [ 92%]
test_torch_compile.py::TestTorchCompile::test_server_status SKIPPED      [ 93%]
test_torch_compile.py::TestTorchCompile::test_registered_model SKIPPED   [ 94%]
test_torch_compile.py::TestTorchCompile::test_serve_inference SKIPPED    [ 95%]
test_torch_xla.py::TestTorchXLA::test_archive_model_artifacts SKIPPED    [ 96%]
test_torch_xla.py::TestTorchXLA::test_start_torchserve SKIPPED (PyTo...) [ 97%]
test_torch_xla.py::TestTorchXLA::test_server_status SKIPPED (PyTorch...) [ 98%]
test_torch_xla.py::TestTorchXLA::test_registered_model SKIPPED (PyTo...) [ 99%]
test_torch_xla.py::TestTorchXLA::test_serve_inference SKIPPED (PyTor...) [100%]

=================================== FAILURES ===================================
__________________________ test_echo_stream_inference __________________________

    def test_echo_stream_inference():
        test_utils.start_torchserve(no_config_snapshots=True, gen_mar=False)
        test_utils.register_model(
            "echo_stream", "https://torchserve.pytorch.org/mar_files/echo_stream.mar"
        )
    
        response = requests.post(
            TF_INFERENCE_API + "/predictions/echo_stream", data="foo", stream=True
        )
>       assert response.headers["Transfer-Encoding"] == "chunked"

test_handler.py:411: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = {'x-request-id': '12f801b4-00f7-4437-b278-4075ea77afe9', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache; no-store, must-revalidate, private', 'Expires': 'Thu, 01 Jan 1970 00:00:00 UTC', 'content-length': '5', 'connection': 'keep-alive'}
key = 'Transfer-Encoding'

    def __getitem__(self, key):
>       return self._store[key.lower()][1]
E       KeyError: 'transfer-encoding'

/Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/requests/structures.py:52: KeyError
----------------------------- Captured stdout call -----------------------------
2023-08-15T00:05:13,028 [INFO ] KQueueEventLoopGroup-3-5 ACCESS_LOG - /127.0.0.1:57822 "GET /models HTTP/1.1" 200 2
2023-08-15T00:05:13,028 [INFO ] KQueueEventLoopGroup-3-5 TS_METRICS - Requests2XX.Count:1.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083113
['torchserve', '--start', '--model-store', '/var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T/workspace/model_store/', '--no-config-snapshots']
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2023-08-15T00:05:16,179 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager...
2023-08-15T00:05:16,242 [INFO ] main org.pytorch.serve.metrics.configuration.MetricConfiguration - Successfully loaded metrics configuration from /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml
2023-08-15T00:05:16,431 [INFO ] main org.pytorch.serve.ModelServer -
Torchserve version: 0.8.1
TS Home: /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages
Current directory: /Volumes/workplace/python_env/serve/test/pytest
Temp directory: /var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T/
Metrics config path: /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml
Number of GPUs: 0
Number of CPUs: 12
Max heap size: 4096 M
Python executable: /Users/lninga/opt/anaconda3/envs/py38/bin/python3
Config file: N/A
Inference address: http://127.0.0.1:8080
Management address: http://127.0.0.1:8081
Metrics address: http://127.0.0.1:8082
Model Store: /private/var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T/workspace/model_store
Initial Models: N/A
Log dir: /Volumes/workplace/python_env/serve/test/pytest/logs
Metrics dir: /Volumes/workplace/python_env/serve/test/pytest/logs
Netty threads: 0
Netty client threads: 0
Default workers per model: 12
Blacklist Regex: N/A
Maximum Response Size: 6553500
Maximum Request Size: 6553500
Limit Maximum Image Pixels: true
Prefer direct buffer: false
Allowed Urls: [file://.*|http(s)?://.*]
Custom python dependency for model allowed: false
Enable metrics API: true
Metrics mode: log
Disable system metrics: false
Workflow Store: /private/var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T/workspace/model_store
Model config: N/A
2023-08-15T00:05:16,438 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager -  Loading snapshot serializer plugin...
2023-08-15T00:05:16,460 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: KQueueServerSocketChannel.
2023-08-15T00:05:16,511 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8080
2023-08-15T00:05:16,511 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: KQueueServerSocketChannel.
2023-08-15T00:05:16,512 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8081
2023-08-15T00:05:16,512 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: KQueueServerSocketChannel.
2023-08-15T00:05:16,513 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082
Model server started.
2023-08-15T00:05:16,784 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083116
2023-08-15T00:05:16,785 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:199.39812850952148|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083116
2023-08-15T00:05:16,785 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:8.439990997314453|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083116
2023-08-15T00:05:16,785 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:4.1|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083116
2023-08-15T00:05:16,785 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:7385.80859375|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083116
2023-08-15T00:05:16,786 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:8482.66796875|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083116
2023-08-15T00:05:16,786 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:54.9|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083116
2023-08-15T00:05:17,252 [DEBUG] KQueueEventLoopGroup-3-1 org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model echo_stream
2023-08-15T00:05:17,252 [DEBUG] KQueueEventLoopGroup-3-1 org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model echo_stream
2023-08-15T00:05:17,253 [INFO ] KQueueEventLoopGroup-3-1 org.pytorch.serve.wlm.ModelManager - Model echo_stream loaded.
2023-08-15T00:05:17,253 [DEBUG] KQueueEventLoopGroup-3-1 org.pytorch.serve.wlm.ModelManager - updateModel: echo_stream, count: 1
2023-08-15T00:05:17,260 [DEBUG] W-9000-echo_stream_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/Users/lninga/opt/anaconda3/envs/py38/bin/python3, /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T//.ts.sock.9000, --metrics-config, /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2023-08-15T00:05:18,649 [INFO ] W-9000-echo_stream_1.0-stdout MODEL_LOG - s_name_part0=/var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T//.ts.sock, s_name_part1=9000, pid=26603
2023-08-15T00:05:18,649 [INFO ] W-9000-echo_stream_1.0-stdout MODEL_LOG - Listening on port: /var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T//.ts.sock.9000
2023-08-15T00:05:18,658 [INFO ] W-9000-echo_stream_1.0-stdout MODEL_LOG - Successfully loaded /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2023-08-15T00:05:18,658 [INFO ] W-9000-echo_stream_1.0-stdout MODEL_LOG - [PID]26603
2023-08-15T00:05:18,659 [INFO ] W-9000-echo_stream_1.0-stdout MODEL_LOG - Torch worker started.
2023-08-15T00:05:18,659 [INFO ] W-9000-echo_stream_1.0-stdout MODEL_LOG - Python runtime: 3.8.13
2023-08-15T00:05:18,659 [DEBUG] W-9000-echo_stream_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-echo_stream_1.0 State change null -> WORKER_STARTED
2023-08-15T00:05:18,661 [INFO ] W-9000-echo_stream_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T//.ts.sock.9000
2023-08-15T00:05:18,665 [INFO ] W-9000-echo_stream_1.0-stdout MODEL_LOG - Connection accepted: /var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T//.ts.sock.9000.
2023-08-15T00:05:18,667 [INFO ] W-9000-echo_stream_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1692083118667
2023-08-15T00:05:18,685 [INFO ] W-9000-echo_stream_1.0-stdout MODEL_LOG - model_name: echo_stream, batchSize: 1
2023-08-15T00:05:18,687 [DEBUG] W-9000-echo_stream_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2023-08-15T00:05:18,687 [INFO ] W-9000-echo_stream_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 19
2023-08-15T00:05:18,687 [DEBUG] W-9000-echo_stream_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-echo_stream_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2023-08-15T00:05:18,687 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:1429.0|#WorkerName:W-9000-echo_stream_1.0,Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,688 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:2.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,691 [INFO ] KQueueEventLoopGroup-3-1 ACCESS_LOG - /127.0.0.1:57828 "POST /models?model_name=echo_stream&url=https%3A%2F%2Ftorchserve.pytorch.org%2Fmar_files%2Fecho_stream.mar&initial_workers=1&synchronous=true HTTP/1.1" 200 1945
2023-08-15T00:05:18,692 [INFO ] KQueueEventLoopGroup-3-1 TS_METRICS - Requests2XX.Count:1.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,711 [INFO ] KQueueEventLoopGroup-3-2 TS_METRICS - ts_inference_requests_total.Count:1.0|#model_name:echo_stream,model_version:default|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,711 [INFO ] W-9000-echo_stream_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd PREDICT to backend at: 1692083118711
2023-08-15T00:05:18,713 [INFO ] W-9000-echo_stream_1.0-stdout MODEL_LOG - Backend received inference at: 1692083118
2023-08-15T00:05:18,713 [INFO ] W-9000-echo_stream_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - result=[METRICS]PredictionTime.Milliseconds:0.06|#ModelName:echo_stream,Level:Model|#hostname:147dda19895c.ant.amazon.com,1692083118,12f801b4-00f7-4437-b278-4075ea77afe9, pattern=[METRICS]
2023-08-15T00:05:18,713 [INFO ] W-9000-echo_stream_1.0 ACCESS_LOG - /127.0.0.1:57830 "POST /predictions/echo_stream HTTP/1.1" 200 3
2023-08-15T00:05:18,713 [INFO ] W-9000-echo_stream_1.0-stdout MODEL_METRICS - PredictionTime.ms:0.06|#ModelName:echo_stream,Level:Model|#hostname:147dda19895c.ant.amazon.com,requestID:12f801b4-00f7-4437-b278-4075ea77afe9,timestamp:1692083118
2023-08-15T00:05:18,713 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - Requests2XX.Count:1.0|#Level:Host|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,714 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - ts_inference_latency_microseconds.Microseconds:1725.023|#model_name:echo_stream,model_version:default|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
2023-08-15T00:05:18,714 [INFO ] W-9000-echo_stream_1.0 TS_METRICS - ts_queue_latency_microseconds.Microseconds:145.662|#model_name:echo_stream,model_version:default|#hostname:147dda19895c.ant.amazon.com,timestamp:1692083118
----------------------------- Captured stderr call -----------------------------
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    19  100    19    0     0    990      0 --:--:-- --:--:-- --:--:--  1461
curl: (23) Failed writing body
=============================== warnings summary ===============================
../../../../../../Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/google/rpc/__init__.py:18
  /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/google/rpc/__init__.py:18: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    import pkg_resources

../../../../../../Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2871
  /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../../../../../../Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2871
  /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.logging')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../../../../../../Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2350
../../../../../../Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2350
  /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2350: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(parent)

../../../../../../Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2871
  /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../../../../../../Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2871
../../../../../../Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2871
  /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../../../../../../Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/google/rpc/__init__.py:20
  /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/google/rpc/__init__.py:20: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.rpc')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    pkg_resources.declare_namespace(__name__)

test/pytest/test_example_scriptable_tokenzier.py::test_handler
  /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/torch/jit/_recursive.py:266: UserWarning: 'batch_first' was found in ScriptModule constants, but was not actually set in __init__. Consider removing it.
    warnings.warn("'{}' was found in ScriptModule constants, "

test/pytest/test_example_scriptable_tokenzier.py::test_handler
  /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py:1501: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/NestedTensorImpl.cpp:179.)
    return forward_call(*args, **kwargs)

test/pytest/test_example_scriptable_tokenzier.py::test_handler
  /Volumes/workplace/python_env/serve/test/pytest/../../examples/text_classification_with_scriptable_tokenizer/handler.py:97: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
    data = F.softmax(data)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED test_handler.py::test_echo_stream_inference - KeyError: 'transfer-enco...
====== 1 failed, 90 passed, 18 skipped, 12 warnings in 4370.47s (1:12:50) ======
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties
## Successfully started TorchServe
## Stopping TorchServe
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: ['torchserve', '--stop']
## Successfully stopped TorchServe
## Starting gen_mar: model_store

## Create symlink for mar files

## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully.
## Starting TorchServe
## Console logs redirected to file: ts_console.log
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties
## Successfully started TorchServe
## Stopping TorchServe
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: ['torchserve', '--stop']
## Successfully stopped TorchServe
## Starting gen_mar: model_store

## Create symlink for mar files

## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully.
## Starting TorchServe
## Console logs redirected to file: ts_console.log
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kf.properties
## Successfully started TorchServe
## Stopping TorchServe
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: ['torchserve', '--stop']
## Successfully stopped TorchServe
## Starting gen_mar: model_store

## Create symlink for mar files

## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully.
## Starting TorchServe
## Console logs redirected to file: ts_console.log
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties
## Successfully started TorchServe
## Stopping TorchServe
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: ['torchserve', '--stop']
## Successfully stopped TorchServe
## Starting gen_mar: model_store

## Create symlink for mar files

## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully.
## Starting TorchServe
## Console logs redirected to file: ts_console.log
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kfv2.properties
## Successfully started TorchServe
## Stopping TorchServe
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: ['torchserve', '--stop']
## Successfully stopped TorchServe
## Starting gen_mar: model_store

## Create symlink for mar files

## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully.
## Starting TorchServe
## Console logs redirected to file: ts_console.log
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties
## Successfully started TorchServe
## Stopping TorchServe
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: ['torchserve', '--stop']
## Successfully stopped TorchServe
## Starting gen_mar: model_store

## Create symlink for mar files

## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully.
## Starting TorchServe
## Console logs redirected to file: ts_console.log
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --workflow-store=model_store --ncs
## Successfully started TorchServe
## Stopping TorchServe
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: ['torchserve', '--stop']
## Successfully stopped TorchServe
## Starting gen_mar: model_store

## Create symlink for mar files

## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully.
## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully.
## Starting TorchServe
## Console logs redirected to file: ts_console.log
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --workflow-store=model_store --ncs
## Successfully started TorchServe
## Stopping TorchServe
## In directory: /Volumes/workplace/python_env/serve/test | Executing command: ['torchserve', '--stop']
## Successfully stopped TorchServe
## Started regression tests
## Started densenet mar creation
## In directory: /private/var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T/workspace/model_store | Executing command: torch-model-archiver                 --model-name densenet161_v1                 --version 1.1                 --model-file /Volumes/workplace/python_env/serve/ts_scripts/../examples/image_classifier/densenet_161/model.py                 --serialized-file /var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T/workspace/model_store/densenet161-8d451a50.pth                 --extra-files /Volumes/workplace/python_env/serve/ts_scripts/../examples/image_classifier/index_to_name.json                 --handler image_classifier                 --force
## Started regression pytests
## In directory: /Volumes/workplace/python_env/serve/test/pytest | Executing command: python -m pytest -v ./
Removing file : inference_pb2_grpc.py
Removing file : inference_pb2.py
Removing file : management_pb2.py
Removing file : management_pb2_grpc.py
## TorchServe Regression Pytests Failed