Submitted job stops executing at client site with execution code: -9

[hitesh-ag1]

Hi, I have deployed NVFlare application on AWS, however, can't seem to get any job to completely execute.

Currently, I am trying to run the spleen CT segmentation job under MONAI bundle to run, however, after submitting the job all the client sites show the following message:

2023-01-19 07:50:57,007 - ClientEngine - INFO - Starting client app. rank: 0 2023-01-19 07:50:57,019 - ProcessExecutor - INFO - Worker child process ID: 20818 2023-01-19 07:51:00,362 - worker_process - INFO - Worker_process started. 2023-01-19 07:51:03,380 - MonaiAlgo - INFO - Initializing site-2 ... Loading dataset: 0%| | 0/32 [00:00<?, ?it/s] 2023-01-19 07:51:04,042 - ProcessExecutor - INFO - run (aeae4490-3b9b-41b7-b49f-387a6a1eedae): waiting for child worker process to finish. 2023-01-19 07:51:07,680 - ProcessExecutor - INFO - run (aeae4490-3b9b-41b7-b49f-387a6a1eedae): child worker process finished with execution code: -9

I am wondering what the last statement means - child worker process finished with execution code: -9 and how can I resolve it. The same job is working perfectly locally.

Any help regarding this issue would be greatly appreciated, thanks!

[hitesh-ag1]

Resolved this bug by upgrading my client EC2 instances to larger ones. Previously, it was a memory issue while loading the dataset as shown by:
Loading dataset: 0%| | 0/32 [00:00<?, ?it/s]

[YuanTingHsieh]

@hitesh-ag1 thanks for providing your solution!