[2023-07-19 15:13:59,136] [17948] [MainThread] [INFO] (__main__:37) - Initializing App from: C:\Users\berl\apps\radiology; studies: C:\Users\berl\datasets\Knee_CT\Intact; conf: {'models': 'segmentation_knee'}
[2023-07-19 15:14:00,969] [17948] [MainThread] [INFO] (numexpr.utils:148) - Note: NumExpr detected 32 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
[2023-07-19 15:14:00,969] [17948] [MainThread] [INFO] (numexpr.utils:160) - NumExpr defaulting to 8 threads.
[2023-07-19 15:14:03,064] [17948] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2023-07-19 15:14:03,070] [17948] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2023-07-19 15:14:03,071] [17948] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2023-07-19 15:14:03,071] [17948] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2023-07-19 15:14:03,071] [17948] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2023-07-19 15:14:03,072] [17948] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2023-07-19 15:14:03,072] [17948] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2023-07-19 15:14:03,073] [17948] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_knee.Segmentation'>
[2023-07-19 15:14:03,074] [17948] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_knee2.SegmentationSpleen'>
[2023-07-19 15:14:03,074] [17948] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2023-07-19 15:14:03,074] [17948] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2023-07-19 15:14:03,074] [17948] [MainThread] [INFO] (main:93) - +++ Adding Model: segmentation_knee => lib.configs.segmentation_knee.Segmentation
[2023-07-19 15:14:03,123] [17948] [MainThread] [INFO] (main:96) - +++ Using Models: ['segmentation_knee']
[2023-07-19 15:14:03,123] [17948] [MainThread] [INFO] (monailabel.interfaces.app:135) - Init Datastore for: C:\Users\berl\datasets\Knee_CT\Intact
[2023-07-19 15:14:03,123] [17948] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: False; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml']
[2023-07-19 15:14:03,132] [17948] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-07-19 15:14:03,133] [17948] [MainThread] [INFO] (main:126) - +++ Adding Inferer:: segmentation_knee => <lib.infers.segmentation.Segmentation object at 0x00000269BE282B90>
[2023-07-19 15:14:03,133] [17948] [MainThread] [INFO] (main:191) - {'segmentation_knee': <lib.infers.segmentation.Segmentation object at 0x00000269BE282B90>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x00000269C5C327A0>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x00000269C5C33190>}
[2023-07-19 15:14:03,133] [17948] [MainThread] [INFO] (main:206) - +++ Adding Trainer:: segmentation_knee => <lib.trainers.segmentation.Segmentation object at 0x00000269C5C33070>
[2023-07-19 15:14:03,133] [17948] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: C:\Users\berl\.cache\monailabel\sessions
[2023-07-19 15:14:03,133] [17948] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2023-07-19 15:14:03,133] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:432) - Train Request (input): {'model': 'segmentation_knee', 'name': 'train_01', 'pretrained': True, 'device': 'NVIDIA GeForce RTX 4090', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'client_id': 'user-xyz', 'local_rank': 0}
[2023-07-19 15:14:03,133] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:445) - CUDA_VISIBLE_DEVICES: None
[2023-07-19 15:14:03,134] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:450) - Distributed/Multi GPU is limited
[2023-07-19 15:14:03,134] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:465) - Distributed Training = FALSE
[2023-07-19 15:14:03,134] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:492) - 0 - Train Request (final): {'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': False, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'model': 'segmentation_knee', 'client_id': 'user-xyz', 'local_rank': 0, 'run_id': '20230719_151403'}
[2023-07-19 15:14:03,134] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:625) - 0 - Using Device: cuda; IDX: None
[2023-07-19 15:14:03,134] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:518) - Run/Output Path: C:\Users\berl\apps\radiology\model\segmentation_knee\train_01
[2023-07-19 15:14:03,134] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:534) - Tracking: mlflow
[2023-07-19 15:14:03,134] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:535) - Tracking URI: file:///C:/Users/berl/apps/radiology/model/segmentation_knee/train_01/mlruns;
[2023-07-19 15:14:03,134] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:536) - Tracking Experiment Name: segmentation_knee; Run Name: run_20230719_151403
[2023-07-19 15:14:03,134] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:410) - Total Records for Training: 8
[2023-07-19 15:14:03,134] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:411) - Total Records for Validation: 2
monai.handlers.stats_handler StatsHandler.__init__:name: Current default value of argument `name=None` has been deprecated since version 1.1. It will be changed to `name=StatsHandler` in version 1.3.
monai.transforms.io.dictionary LoadImaged.__init__:image_only: Current default value of argument `image_only=False` has been deprecated since version 1.1. It will be changed to `image_only=True` in version 1.3.
Loading dataset:   0%|          | 0/2 [00:00<?, ?it/s]
Loading dataset:  50%|#####     | 1/2 [00:28<00:28, 28.32s/it]
Loading dataset: 100%|##########| 2/2 [00:59<00:00, 29.92s/it]
Loading dataset: 100%|##########| 2/2 [00:59<00:00, 29.68s/it]
cache_num is greater or equal than dataset length, fall back to regular monai.data.CacheDataset.
[2023-07-19 15:15:02,588] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:328) - 0 - Records for Validation: 2
[2023-07-19 15:15:02,600] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:318) - 0 - Adding Validation to run every '1' interval
[2023-07-19 15:15:02,602] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:713) - 0 - Load Path C:\Users\berl\apps\radiology\model\segmentation_knee\train_01\model.pt
Loading dataset:   0%|          | 0/8 [00:00<?, ?it/s]
Loading dataset:  12%|#2        | 1/8 [00:25<02:56, 25.15s/it]
Loading dataset:  25%|##5       | 2/8 [00:50<02:32, 25.35s/it]
Loading dataset:  38%|###7      | 3/8 [01:18<02:12, 26.55s/it]
Loading dataset:  50%|#####     | 4/8 [01:49<01:52, 28.10s/it]
Loading dataset:  62%|######2   | 5/8 [02:17<01:25, 28.35s/it]
Loading dataset:  75%|#######5  | 6/8 [02:43<00:55, 27.50s/it]
Loading dataset:  88%|########7 | 7/8 [03:11<00:27, 27.74s/it]
Loading dataset: 100%|##########| 8/8 [03:41<00:00, 28.30s/it]
Loading dataset: 100%|##########| 8/8 [03:41<00:00, 27.68s/it]
[2023-07-19 15:18:44,074] [17948] [MainThread] [INFO] (monailabel.tasks.train.basic_train:264) - 0 - Records for Training: 8
[2023-07-19 15:18:44,075] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:876) - Engine run resuming from iteration 0, epoch 0 until 50 epochs
[2023-07-19 15:18:44,353] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:140) - Restored all variables from C:\Users\berl\apps\radiology\model\segmentation_knee\train_01\model.pt
[2023-07-19 15:18:44,973] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 1/50, Iter: 1/8 -- train_loss: 1.7931
[2023-07-19 15:18:45,863] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 1/50, Iter: 2/8 -- train_loss: 1.8407
[2023-07-19 15:18:46,745] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 1/50, Iter: 3/8 -- train_loss: 1.8611
[2023-07-19 15:18:47,622] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 1/50, Iter: 4/8 -- train_loss: 1.6930
[2023-07-19 15:18:48,500] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 1/50, Iter: 5/8 -- train_loss: 1.6988
[2023-07-19 15:18:49,385] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 1/50, Iter: 6/8 -- train_loss: 1.6823
[2023-07-19 15:18:50,262] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 1/50, Iter: 7/8 -- train_loss: 1.6590
[2023-07-19 15:18:51,148] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 1/50, Iter: 8/8 -- train_loss: 1.6306
[2023-07-19 15:18:51,154] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:259) - Got new best metric of train_mean_dice: 0.0
[2023-07-19 15:18:51,154] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:223) - Epoch[1] Metrics -- train_femur_L_mean_dice: 0.0000 train_femur_R_mean_dice: 0.0000 train_mean_dice: 0.0000 train_tibfib_L_mean_dice: 0.0000 train_tibfib_R_mean_dice: 0.0000
[2023-07-19 15:18:51,154] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:234) - Key metric: train_mean_dice best value: 0.0 at epoch: 1
[2023-07-19 15:18:51,168] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:876) - Engine run resuming from iteration 0, epoch 0 until 1 epochs
[2023-07-19 15:18:58,251] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:259) - Got new best metric of val_mean_dice: 0.0
[2023-07-19 15:18:58,251] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:223) - Epoch[1] Metrics -- val_femur_L_mean_dice: 0.0000 val_femur_R_mean_dice: 0.0000 val_mean_dice: 0.0000 val_tibfib_L_mean_dice: 0.0000 val_tibfib_R_mean_dice: 0.0000
[2023-07-19 15:18:58,251] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:234) - Key metric: val_mean_dice best value: 0.0 at epoch: 1
[2023-07-19 15:18:58,352] [17948] [MainThread] [INFO] (monailabel.tasks.train.handler:86) - New Model published: C:\Users\berl\apps\radiology\model\segmentation_knee\train_01\model.pt => C:\Users\berl\apps\radiology\model\segmentation_knee.pt
[2023-07-19 15:18:58,352] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:972) - Epoch[1] Complete. Time taken: 00:00:06.965
[2023-07-19 15:18:58,353] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:988) - Engine run complete. Time taken: 00:00:07.185
[2023-07-19 15:18:58,411] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:972) - Epoch[1] Complete. Time taken: 00:00:14.058
[2023-07-19 15:18:59,314] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 2/50, Iter: 1/8 -- train_loss: 1.7320
[2023-07-19 15:19:00,209] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 2/50, Iter: 2/8 -- train_loss: 1.6034
[2023-07-19 15:19:01,108] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 2/50, Iter: 3/8 -- train_loss: 1.6242
[2023-07-19 15:19:02,002] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 2/50, Iter: 4/8 -- train_loss: 1.5399
[2023-07-19 15:19:02,896] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 2/50, Iter: 5/8 -- train_loss: 1.4967
[2023-07-19 15:19:03,791] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 2/50, Iter: 6/8 -- train_loss: 1.4672
[2023-07-19 15:19:04,682] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 2/50, Iter: 7/8 -- train_loss: 1.4727
[2023-07-19 15:19:05,573] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 2/50, Iter: 8/8 -- train_loss: 1.4388
[2023-07-19 15:19:05,578] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:223) - Epoch[2] Metrics -- train_femur_L_mean_dice: 0.0000 train_femur_R_mean_dice: 0.0000 train_mean_dice: 0.0000 train_tibfib_L_mean_dice: 0.0000 train_tibfib_R_mean_dice: 0.0000
[2023-07-19 15:19:05,578] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:234) - Key metric: train_mean_dice best value: 0.0 at epoch: 1
[2023-07-19 15:19:05,583] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:876) - Engine run resuming from iteration 0, epoch 1 until 2 epochs
[2023-07-19 15:19:12,679] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:223) - Epoch[2] Metrics -- val_femur_L_mean_dice: 0.0000 val_femur_R_mean_dice: 0.0000 val_mean_dice: 0.0000 val_tibfib_L_mean_dice: 0.0000 val_tibfib_R_mean_dice: 0.0000
[2023-07-19 15:19:12,679] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:234) - Key metric: val_mean_dice best value: 0.0 at epoch: 1
[2023-07-19 15:19:12,730] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:972) - Epoch[2] Complete. Time taken: 00:00:06.922
[2023-07-19 15:19:12,732] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:988) - Engine run complete. Time taken: 00:00:07.149
[2023-07-19 15:19:12,732] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:972) - Epoch[2] Complete. Time taken: 00:00:14.320
[2023-07-19 15:19:13,633] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 3/50, Iter: 1/8 -- train_loss: 1.4314
[2023-07-19 15:19:14,533] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 3/50, Iter: 2/8 -- train_loss: 1.4297
[2023-07-19 15:19:15,437] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 3/50, Iter: 3/8 -- train_loss: 1.4218
[2023-07-19 15:19:16,334] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 3/50, Iter: 4/8 -- train_loss: 1.4421
[2023-07-19 15:19:17,231] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 3/50, Iter: 5/8 -- train_loss: 1.4099
[2023-07-19 15:19:18,136] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 3/50, Iter: 6/8 -- train_loss: 1.4059
[2023-07-19 15:19:19,028] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 3/50, Iter: 7/8 -- train_loss: 1.4034
[2023-07-19 15:19:19,921] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 3/50, Iter: 8/8 -- train_loss: 1.4005
[2023-07-19 15:19:19,927] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:223) - Epoch[3] Metrics -- train_femur_L_mean_dice: 0.0000 train_femur_R_mean_dice: 0.0000 train_mean_dice: 0.0000 train_tibfib_L_mean_dice: 0.0000 train_tibfib_R_mean_dice: 0.0000
[2023-07-19 15:19:19,927] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:234) - Key metric: train_mean_dice best value: 0.0 at epoch: 1
[2023-07-19 15:19:19,931] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:876) - Engine run resuming from iteration 0, epoch 2 until 3 epochs
[2023-07-19 15:19:27,048] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:223) - Epoch[3] Metrics -- val_femur_L_mean_dice: 0.0000 val_femur_R_mean_dice: 0.0000 val_mean_dice: 0.0000 val_tibfib_L_mean_dice: 0.0000 val_tibfib_R_mean_dice: 0.0000
[2023-07-19 15:19:27,048] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:234) - Key metric: val_mean_dice best value: 0.0 at epoch: 1
[2023-07-19 15:19:27,065] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:972) - Epoch[3] Complete. Time taken: 00:00:06.900
[2023-07-19 15:19:27,067] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:988) - Engine run complete. Time taken: 00:00:07.135
[2023-07-19 15:19:27,067] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:972) - Epoch[3] Complete. Time taken: 00:00:14.335
[2023-07-19 15:19:27,964] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 4/50, Iter: 1/8 -- train_loss: 1.3985
[2023-07-19 15:19:28,859] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 4/50, Iter: 2/8 -- train_loss: 1.5129
[2023-07-19 15:19:29,758] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 4/50, Iter: 3/8 -- train_loss: 1.4423
[2023-07-19 15:19:30,656] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 4/50, Iter: 4/8 -- train_loss: 1.3994
[2023-07-19 15:19:31,549] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 4/50, Iter: 5/8 -- train_loss: 1.3914
[2023-07-19 15:19:32,447] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 4/50, Iter: 6/8 -- train_loss: 1.3895
[2023-07-19 15:19:33,340] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 4/50, Iter: 7/8 -- train_loss: 1.3975
[2023-07-19 15:19:34,236] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 4/50, Iter: 8/8 -- train_loss: 1.3837
[2023-07-19 15:19:34,242] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:223) - Epoch[4] Metrics -- train_femur_L_mean_dice: 0.0000 train_femur_R_mean_dice: 0.0000 train_mean_dice: 0.0000 train_tibfib_L_mean_dice: 0.0000 train_tibfib_R_mean_dice: 0.0000
[2023-07-19 15:19:34,242] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:234) - Key metric: train_mean_dice best value: 0.0 at epoch: 1
[2023-07-19 15:19:34,247] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:876) - Engine run resuming from iteration 0, epoch 3 until 4 epochs
[2023-07-19 15:19:41,377] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:223) - Epoch[4] Metrics -- val_femur_L_mean_dice: 0.0000 val_femur_R_mean_dice: 0.0000 val_mean_dice: 0.0000 val_tibfib_L_mean_dice: 0.0000 val_tibfib_R_mean_dice: 0.0000
[2023-07-19 15:19:41,377] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:234) - Key metric: val_mean_dice best value: 0.0 at epoch: 1
[2023-07-19 15:19:41,385] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:972) - Epoch[4] Complete. Time taken: 00:00:06.910
[2023-07-19 15:19:41,386] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:988) - Engine run complete. Time taken: 00:00:07.138
[2023-07-19 15:19:41,386] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:972) - Epoch[4] Complete. Time taken: 00:00:14.319
[2023-07-19 15:19:42,288] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 5/50, Iter: 1/8 -- train_loss: 1.4142
[2023-07-19 15:19:43,187] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 5/50, Iter: 2/8 -- train_loss: 1.3794
[2023-07-19 15:19:44,083] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 5/50, Iter: 3/8 -- train_loss: 1.3774
[2023-07-19 15:19:45,004] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 5/50, Iter: 4/8 -- train_loss: 1.3752
[2023-07-19 15:19:45,910] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 5/50, Iter: 5/8 -- train_loss: 1.3795
[2023-07-19 15:19:46,807] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 5/50, Iter: 6/8 -- train_loss: 1.3868
[2023-07-19 15:19:47,699] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 5/50, Iter: 7/8 -- train_loss: 1.4227
[2023-07-19 15:19:48,596] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 5/50, Iter: 8/8 -- train_loss: 1.3695
[2023-07-19 15:19:48,601] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:223) - Epoch[5] Metrics -- train_femur_L_mean_dice: 0.0000 train_femur_R_mean_dice: 0.0000 train_mean_dice: 0.0000 train_tibfib_L_mean_dice: 0.0000 train_tibfib_R_mean_dice: 0.0000
[2023-07-19 15:19:48,601] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:234) - Key metric: train_mean_dice best value: 0.0 at epoch: 1
[2023-07-19 15:19:48,607] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:876) - Engine run resuming from iteration 0, epoch 4 until 5 epochs
[2023-07-19 15:19:55,962] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:223) - Epoch[5] Metrics -- val_femur_L_mean_dice: 0.0000 val_femur_R_mean_dice: 0.0000 val_mean_dice: 0.0000 val_tibfib_L_mean_dice: 0.0000 val_tibfib_R_mean_dice: 0.0000
[2023-07-19 15:19:55,962] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:234) - Key metric: val_mean_dice best value: 0.0 at epoch: 1
[2023-07-19 15:19:55,970] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:972) - Epoch[5] Complete. Time taken: 00:00:07.121
[2023-07-19 15:19:55,972] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:988) - Engine run complete. Time taken: 00:00:07.365
[2023-07-19 15:19:55,972] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:972) - Epoch[5] Complete. Time taken: 00:00:14.587
[2023-07-19 15:19:56,943] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 6/50, Iter: 1/8 -- train_loss: 1.3661
[2023-07-19 15:19:57,886] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 6/50, Iter: 2/8 -- train_loss: 1.3666
[2023-07-19 15:19:58,804] [17948] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:294) - Epoch: 6/50, Iter: 3/8 -- train_loss: 1.3634
[2023-07-19 15:19:58,883] [13412] [MainThread] [INFO] (monailabel.utils.async_tasks.utils:120) - Kill background task for train
[2023-07-19 15:19:58,891] [13412] [MainThread] [INFO] (monailabel.utils.async_tasks.utils:129) - Kill:: Process pid is 17948
[2023-07-19 15:19:58,892] [13412] [MainThread] [INFO] (monailabel.utils.async_tasks.utils:133) - Killed background process: 17948
[2023-07-19 15:20:00,523] [13412] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 1