Start model torchserve

[baopanda]

Hi all,
when I run my .mar model in the server I have encounter an error:

2020-07-10 10:41:04,568 [DEBUG] W-9005-segment_batch_ts_32_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432)
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:129)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

and the script that I am using to start the server

torchserve --start --model-store model-store --models segment_batch=segment_batch.mar

Can you help me to fix it, thank you in advance.

[harshbafna]

@baopanda : Could you please provide the complete logs from logs/ts_log.log. Also, share the model's mar file so that we can reproduce the issue at our end.

[baopanda]

Definitely, Mr @harshbafna , I am happy to provide it to you.
Logs from ts_log.log:

Torchserve version: 0.1.1
TS Home: /data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages
Current directory: /data/baobn/segment_api_torchserve
Temp directory: /tmp
Number of GPUs: 0
Number of CPUs: 32
Max heap size: 24144 M
Python executable: /data/baobn/miniconda3/envs/baobn1/bin/python
Config file: logs/config/20200710152925657-shutdown.cfg
Inference address: http://127.0.0.1:8080
Management address: http://127.0.0.1:8081
Model Store: /data/baobn/segment_api_torchserve/model-store
Initial Models: segment_batch=segment_batch.mar
Log dir: /data/baobn/segment_api_torchserve/logs
Metrics dir: /data/baobn/segment_api_torchserve/logs
Netty threads: 0
Netty client threads: 0
Default workers per model: 6
Blacklist Regex: N/A
Maximum Response Size: 6553500
Maximum Request Size: 6553500
Prefer direct buffer: false
2020-07-10 15:29:36,359 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Started restoring models from snapshot {
  "name": "20200710152925657-shutdown.cfg",
  "modelCount": 1,
  "created": 1594369765658,
  "models": {
    "segment_batch": {
      "1.0": {
        "defaultVersion": true,
        "marName": "segment_batch.mar",
        "minWorkers": 1,
        "maxWorkers": 1,
        "batchSize": 1,
        "maxBatchDelay": 50,
        "responseTimeout": 120
      }
    }
  }
}
2020-07-10 15:29:36,377 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Validating snapshot 20200710152925657-shutdown.cfg
2020-07-10 15:29:36,378 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Snapshot 20200710152925657-shutdown.cfg validated successfully
2020-07-10 15:29:36,807 [INFO ] main org.pytorch.serve.archive.ModelArchive - model folder already exists: 2233c6545081cc70f07c8df066e17622775b61da
2020-07-10 15:29:36,816 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model segment_batch
2020-07-10 15:29:36,817 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model segment_batch
2020-07-10 15:29:36,819 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model segment_batch
2020-07-10 15:29:36,819 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model segment_batch loaded.
2020-07-10 15:29:36,819 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: segment_batch, count: 1
2020-07-10 15:29:36,833 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2020-07-10 15:29:36,925 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8080
2020-07-10 15:29:36,925 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.
2020-07-10 15:29:36,927 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8081
2020-07-10 15:29:36,944 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Listening on port: /tmp/.ts.sock.9000
2020-07-10 15:29:36,945 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - [PID]49405
2020-07-10 15:29:36,945 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Torch worker started.
2020-07-10 15:29:36,946 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Python runtime: 3.8.2
2020-07-10 15:29:36,946 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-segment_batch_1.0 State change null -> WORKER_STARTED
2020-07-10 15:29:36,952 [INFO ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2020-07-10 15:29:36,966 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Connection accepted: /tmp/.ts.sock.9000.
2020-07-10 15:29:38,027 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Backend worker process died.
2020-07-10 15:29:38,027 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Traceback (most recent call last):
2020-07-10 15:29:38,027 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 175, in <module>
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     worker.run_server()
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 147, in run_server
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.handle_connection(cl_socket)
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 111, in handle_connection
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service, result, code = self.load_model(msg)
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 84, in load_model
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service = model_loader.load(model_name, model_dir, handler, gpu, batch_size)
2020-07-10 15:29:38,029 [INFO ] epollEventLoopGroup-4-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2020-07-10 15:29:38,029 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_loader.py", line 102, in load
2020-07-10 15:29:38,029 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2020-07-10 15:29:38,029 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
	at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432)
	at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:129)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
2020-07-10 15:29:38,029 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     entry_point(None, service.context)
2020-07-10 15:29:38,032 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/tmp/models/2233c6545081cc70f07c8df066e17622775b61da/MyHandle_ab_batch.py", line 109, in handle
2020-07-10 15:29:38,032 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: segment_batch, error: Worker died.
2020-07-10 15:29:38,032 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-segment_batch_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2020-07-10 15:29:38,032 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     _service.initialize(context)
2020-07-10 15:29:38,032 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-segment_batch_1.0-stderr
2020-07-10 15:29:38,032 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-segment_batch_1.0-stdout
2020-07-10 15:29:38,032 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/tmp/models/2233c6545081cc70f07c8df066e17622775b61da/MyHandle_ab_batch.py", line 29, in initialize
2020-07-10 15:29:38,033 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-segment_batch_1.0-stdout
2020-07-10 15:29:38,033 [INFO ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds.
2020-07-10 15:29:38,056 [INFO ] W-9000-segment_batch_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-segment_batch_1.0-stderr
2020-07-10 15:29:39,157 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Listening on port: /tmp/.ts.sock.9000
2020-07-10 15:29:39,157 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - [PID]49443
2020-07-10 15:29:39,157 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Torch worker started.
2020-07-10 15:29:39,157 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-segment_batch_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2020-07-10 15:29:39,157 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Python runtime: 3.8.2
2020-07-10 15:29:39,157 [INFO ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2020-07-10 15:29:39,160 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Connection accepted: /tmp/.ts.sock.9000.
2020-07-10 15:29:40,176 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Backend worker process died.
2020-07-10 15:29:40,176 [INFO ] epollEventLoopGroup-4-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2020-07-10 15:29:40,177 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Traceback (most recent call last):
2020-07-10 15:29:40,177 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2020-07-10 15:29:40,177 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 175, in <module>
2020-07-10 15:29:40,177 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
	at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432)
	at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:129)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
2020-07-10 15:29:40,177 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     worker.run_server()
2020-07-10 15:29:40,178 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: segment_batch, error: Worker died.
2020-07-10 15:29:40,178 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 147, in run_server
2020-07-10 15:29:40,179 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-segment_batch_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2020-07-10 15:29:40,179 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.handle_connection(cl_socket)
2020-07-10 15:29:40,179 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-segment_batch_1.0-stderr
2020-07-10 15:29:40,179 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 111, in handle_connection
2020-07-10 15:29:40,179 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-segment_batch_1.0-stdout
2020-07-10 15:29:40,179 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service, result, code = self.load_model(msg)
2020-07-10 15:29:40,180 [INFO ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds.
2020-07-10 15:29:40,180 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-segment_batch_1.0-stdout
2020-07-10 15:29:40,200 [INFO ] W-9000-segment_batch_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-segment_batch_1.0-stderr
2020-07-10 15:29:41,289 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Listening on port: /tmp/.ts.sock.9000
2020-07-10 15:29:41,289 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - [PID]49471
2020-07-10 15:29:41,289 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Torch worker started.
2020-07-10 15:29:41,289 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-segment_batch_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2020-07-10 15:29:41,289 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Python runtime: 3.8.2
2020-07-10 15:29:41,289 [INFO ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2020-07-10 15:29:41,292 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Connection accepted: /tmp/.ts.sock.9000.
2020-07-10 15:29:42,241 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Backend worker process died.
2020-07-10 15:29:42,242 [INFO ] epollEventLoopGroup-4-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2020-07-10 15:29:42,242 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Traceback (most recent call last):
2020-07-10 15:29:42,242 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2020-07-10 15:29:42,242 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 175, in <module>
2020-07-10 15:29:42,242 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
	at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432)
	at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:129)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
2020-07-10 15:29:42,242 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     worker.run_server()
2020-07-10 15:29:42,243 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: segment_batch, error: Worker died.
2020-07-10 15:29:42,243 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 147, in run_server
2020-07-10 15:29:42,243 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-segment_batch_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2020-07-10 15:29:42,243 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.handle_connection(cl_socket)
2020-07-10 15:29:42,243 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 111, in handle_connection
2020-07-10 15:29:42,243 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-segment_batch_1.0-stderr
2020-07-10 15:29:42,243 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service, result, code = self.load_model(msg)
2020-07-10 15:29:42,243 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-segment_batch_1.0-stdout
2020-07-10 15:29:42,244 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 84, in load_model
2020-07-10 15:29:42,244 [INFO ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 2 seconds.
2020-07-10 15:29:42,244 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-segment_batch_1.0-stdout
2020-07-10 15:29:42,263 [INFO ] W-9000-segment_batch_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-segment_batch_1.0-stderr
2020-07-10 15:29:44,359 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Listening on port: /tmp/.ts.sock.9000
2020-07-10 15:29:44,359 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - [PID]49492
2020-07-10 15:29:44,359 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Torch worker started.
2020-07-10 15:29:44,359 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-segment_batch_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2020-07-10 15:29:44,359 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Python runtime: 3.8.2
2020-07-10 15:29:44,360 [INFO ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2020-07-10 15:29:44,362 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Connection accepted: /tmp/.ts.sock.9000.
2020-07-10 15:29:45,356 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Backend worker process died.
2020-07-10 15:29:45,356 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Traceback (most recent call last):
2020-07-10 15:29:45,356 [INFO ] epollEventLoopGroup-4-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2020-07-10 15:29:45,356 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 175, in <module>
2020-07-10 15:29:45,357 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2020-07-10 15:29:45,357 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     worker.run_server()
2020-07-10 15:29:45,357 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
	at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432)
	at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:129)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
2020-07-10 15:29:45,357 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 147, in run_server
2020-07-10 15:29:45,357 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: segment_batch, error: Worker died.
2020-07-10 15:29:45,357 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.handle_connection(cl_socket)
2020-07-10 15:29:45,357 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-segment_batch_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2020-07-10 15:29:45,357 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 111, in handle_connection
2020-07-10 15:29:45,358 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service, result, code = self.load_model(msg)
2020-07-10 15:29:45,358 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-segment_batch_1.0-stderr
2020-07-10 15:29:45,358 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 84, in load_model
2020-07-10 15:29:45,358 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-segment_batch_1.0-stdout
2020-07-10 15:29:45,358 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service = model_loader.load(model_name, model_dir, handler, gpu, batch_size)
2020-07-10 15:29:45,358 [INFO ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 3 seconds.
2020-07-10 15:29:45,358 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-segment_batch_1.0-stdout
2020-07-10 15:29:45,378 [INFO ] W-9000-segment_batch_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-segment_batch_1.0-stderr
2020-07-10 15:29:48,474 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Listening on port: /tmp/.ts.sock.9000
2020-07-10 15:29:48,474 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - [PID]49514
2020-07-10 15:29:48,474 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Torch worker started.
2020-07-10 15:29:48,474 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Python runtime: 3.8.2
2020-07-10 15:29:48,474 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-segment_batch_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2020-07-10 15:29:48,474 [INFO ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2020-07-10 15:29:48,477 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Connection accepted: /tmp/.ts.sock.9000.
2020-07-10 15:29:49,453 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Backend worker process died.
2020-07-10 15:29:49,453 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Traceback (most recent call last):
2020-07-10 15:29:49,454 [INFO ] epollEventLoopGroup-4-5 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2020-07-10 15:29:49,454 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 175, in <module>
2020-07-10 15:29:49,454 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2020-07-10 15:29:49,454 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     worker.run_server()
2020-07-10 15:29:49,454 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
	at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432)
	at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:129)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
2020-07-10 15:29:49,454 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 147, in run_server
2020-07-10 15:29:49,455 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: segment_batch, error: Worker died.
2020-07-10 15:29:49,455 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.handle_connection(cl_socket)
2020-07-10 15:29:49,455 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-segment_batch_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2020-07-10 15:29:49,455 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 111, in handle_connection
2020-07-10 15:29:49,455 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service, result, code = self.load_model(msg)
2020-07-10 15:29:49,455 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-segment_batch_1.0-stderr
2020-07-10 15:29:49,455 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 84, in load_model
2020-07-10 15:29:49,455 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-segment_batch_1.0-stdout
2020-07-10 15:29:49,455 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service = model_loader.load(model_name, model_dir, handler, gpu, batch_size)
2020-07-10 15:29:49,455 [INFO ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 5 seconds.
2020-07-10 15:29:49,455 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-segment_batch_1.0-stdout
2020-07-10 15:29:49,480 [INFO ] W-9000-segment_batch_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-segment_batch_1.0-stderr
2020-07-10 15:29:53,600 [INFO ] epollEventLoopGroup-2-1 org.pytorch.serve.ModelServer - Inference model server stopped.
2020-07-10 15:29:53,600 [INFO ] epollEventLoopGroup-2-2 org.pytorch.serve.ModelServer - Management model server stopped.
2020-07-10 15:29:54,570 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Listening on port: /tmp/.ts.sock.9000
2020-07-10 15:29:54,570 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - [PID]49624
2020-07-10 15:29:54,570 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Torch worker started.
2020-07-10 15:29:54,570 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Python runtime: 3.8.2
2020-07-10 15:29:54,570 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-segment_batch_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2020-07-10 15:29:54,571 [INFO ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2020-07-10 15:29:54,572 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Connection accepted: /tmp/.ts.sock.9000.
2020-07-10 15:29:55,555 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Backend worker process died.
2020-07-10 15:29:55,555 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Traceback (most recent call last):
2020-07-10 15:29:55,555 [INFO ] epollEventLoopGroup-4-6 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2020-07-10 15:29:55,555 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 175, in <module>
2020-07-10 15:29:55,555 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2020-07-10 15:29:55,556 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
	at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432)
	at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:129)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
2020-07-10 15:29:55,556 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     worker.run_server()
2020-07-10 15:29:55,556 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: segment_batch, error: Worker died.
2020-07-10 15:29:55,556 [DEBUG] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-segment_batch_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2020-07-10 15:29:55,556 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 147, in run_server
2020-07-10 15:29:55,556 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.handle_connection(cl_socket)
2020-07-10 15:29:55,556 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-segment_batch_1.0-stderr
2020-07-10 15:29:55,557 [WARN ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-segment_batch_1.0-stdout
2020-07-10 15:29:55,557 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 111, in handle_connection
2020-07-10 15:29:55,557 [INFO ] W-9000-segment_batch_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 8 seconds.
2020-07-10 15:29:55,557 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-segment_batch_1.0-stdout
2020-07-10 15:29:55,586 [INFO ] W-9000-segment_batch_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-segment_batch_1.0-stderr
2020-07-10 15:29:55,853 [INFO ] main org.pytorch.serve.ModelServer - Torchserve stopped.

and file handle:

from ts.torch_handler.base_handler import BaseHandler
import transform_torch
import decode_torch
import json
import torch
import os
import requests
import numpy as np

proxies = {
    "http": None,
    "https": None,
}

import warnings

warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=UserWarning)


class MyHandle(BaseHandler):
    def __init__(self):
        super(MyHandle, self).__init__()
        self.initialized = False

    def initialize(self, ctx):
        properties = ctx.system_properties
        self.device = torch.device("cuda:" + str(properties.get("gpu_id")) if torch.cuda.is_available() else "cpu")
        model_dir = properties.get("model_dir")
        model_pt_path = os.path.join(model_dir, "model_cpu.pt")

        f_map_path = os.path.join(model_dir, "f_map.json")
        c_map_path = os.path.join(model_dir, "c_map.json")
        l_map_path = os.path.join(model_dir, "l_map.json")

        self.model = torch.jit.load(model_pt_path)

        with open(f_map_path, 'r') as f:
            self.f_map = json.load(f)
        with open(c_map_path, 'r') as f:
            self.c_map = json.load(f)
        with open(l_map_path, 'r') as f:
            self.l_map = json.load(f)

        self.model.to(self.device)
        self.model.eval()

    def preprocess_one_sentence(self,req):
        request_json = req["body"]
        sentence = request_json['sentence']

        word_tags, is_success = transform_torch.query_split_text_service(sentence)
        features, indexes = transform_torch.read_features_with_index(word_tags)
        f_f, f_p, b_f, b_p, w_f, mask_v = transform_torch.apply_model(features[0], self.f_map, self.c_map,
                                                                      self.f_map['<eof>'], self.c_map['\n'],
                                                                      torch.cuda.is_available())

        input = []
        input.extend((f_f, f_p, b_f, b_p, w_f, mask_v, features, indexes))

        return input

    def preprocess(self, requests):
        inputs = [self.preprocess_one_sentence(req) for req in requests]

        return inputs

    def inference(self, x):
        num = np.shape(x)[0]
        outputs = []

        for i in range(num):
            scores = self.model(x[i][0], x[i][1], x[i][2], x[i][3], x[i][4])

            output = []
            output.extend((scores, x[i][5], x[i][6], x[i][7]))
            outputs.append(output)

        return outputs

    def postprocess(self, outputs):
        tagset_size = len(self.l_map)
        start_tag = self.l_map['<start>']
        end_tag = self.l_map['<pad>']

        res = []

        for output in outputs:
            scores = output[0]
            mask = output[1]
            features = output[2]
            indexes = output[3]

            decoded = decode_torch.decode(scores, mask, start_tag, tagset_size, end_tag)

            result = decode_torch.decode_seg(features[0][0], decoded, indexes[0][0], self.l_map)

            res.append(result)

        return res


_service = MyHandle()


def handle(data, context):
    if not _service.initialized:
        _service.initialize(context)

    if data is None:
        return None

    data = _service.preprocess(data)
    data = _service.inference(data)
    data = _service.postprocess(data)

    return data

Thank you for helping me, wish you will have a good day

[harshbafna]

@baopanda: From the shared logs, I could only make out that your custom handler throws an exception from line:29 while loading the model.

2020-07-10 15:29:38,027 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 175, in <module>
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     worker.run_server()
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 147, in run_server
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.handle_connection(cl_socket)
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 111, in handle_connection
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service, result, code = self.load_model(msg)
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_service_worker.py", line 84, in load_model
2020-07-10 15:29:38,028 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     service = model_loader.load(model_name, model_dir, handler, gpu, batch_size)
2020-07-10 15:29:38,029 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/data/baobn/miniconda3/envs/baobn1/lib/python3.8/site-packages/ts/model_loader.py", line 102, in load
2020-07-10 15:29:38,029 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     entry_point(None, service.context)
2020-07-10 15:29:38,032 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/tmp/models/2233c6545081cc70f07c8df066e17622775b61da/MyHandle_ab_batch.py", line 109, in handle
2020-07-10 15:29:38,032 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     _service.initialize(context)
2020-07-10 15:29:38,032 [INFO ] W-9000-segment_batch_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/tmp/models/2233c6545081cc70f07c8df066e17622775b61da/MyHandle_ab_batch.py", line 29, in initialize

Unfortunately, it looks like the frontend input stream got closed before it could capture the complete stack trace.

Also,

• the handler file you have shared doesn't match the exact line sequence per the above stack trace. Could you please share your mar file (as it uses certain extra files as well).
• the handlers seem to be importing a couple of custom modules as well: transform_torch and decode_torch. Are these part of your model's archive file?

[baopanda]

@harshbafna
I have know the error and fixed it
I really appreciate your help with my project!
Once again, Thank you