[*INFO] process 52221, running on DEV-A15-HOST-GPU-32_76: starting (Wed Dec 13 11:46:26 2023) balanced sampler is not used Dataset has 1927774 samples Using Label Smoothing: 0.1 Using Following Mask: 0 Freq, 10 Time Using Mix-up with Rate 0.500000 Now Process as-full Number of Classes is 527 Now load features from whisper_large-v2/large-v2 Dataset has 18866 samples Using Label Smoothing: 0.0 Using Following Mask: 0 Freq, 0 Time Using Mix-up with Rate 0.000000 Now Process as-full Number of Classes is 527 Now load features from whisper_large-v2/large-v2 [*INFO] mode : lw_tr_1_8, model size:large-v2, num layer: 32, rep_dim: 1280 Creating experiment directory: ./exp/ Now starting training for 30 epochs running on cuda Total parameter number is : 40.030 million Total trainable parameter number is : 40.030 million The learning rate scheduler starts at 15 epoch with decay rate of 0.750 every 5 epoches now training with as-full, main metrics: mAP, loss function: BCEWithLogitsLoss(), learning rate scheduler: current #steps=0, #epochs=1 start training... --------------- 2023-12-13 11:46:35.786097 current #epochs=1, #steps=0 Epoch: [1][100/7530] Per Sample Total Time 0.07295 Per Sample Data Time 0.06710 Per Sample DNN Time 0.00585 | Train Loss 0.0263 Epoch: [1][200/7530] Per Sample Total Time 0.07290 Per Sample Data Time 0.06726 Per Sample DNN Time 0.00565 | Train Loss 0.0185 Epoch: [1][300/7530] Per Sample Total Time 0.07049 Per Sample Data Time 0.06491 Per Sample DNN Time 0.00558 | Train Loss 0.0152 Epoch: [1][400/7530] Per Sample Total Time 0.07100 Per Sample Data Time 0.06545 Per Sample DNN Time 0.00555 | Train Loss 0.0138 Epoch: [1][500/7530] Per Sample Total Time 0.06988 Per Sample Data Time 0.06435 Per Sample DNN Time 0.00553 | Train Loss 0.0149 Epoch: [1][600/7530] Per Sample Total Time 0.07181 Per Sample Data Time 0.06629 Per Sample DNN Time 0.00552 | Train Loss 0.0134 Epoch: [1][700/7530] Per Sample Total Time 0.07757 Per Sample Data Time 0.07206 Per Sample DNN Time 0.00551 | Train Loss 0.0130 start validation mAP: 0.017164 AUC: 0.635250 d_prime: 0.489021 train_loss: 0.029081 valid_loss: 0.025689 validation finished Epoch-1 lr: 5e-05 epoch 1 training time: 15749.759 --------------- 2023-12-13 16:09:05.544983 current #epochs=2, #steps=755 Epoch: [2][45/7530] Per Sample Total Time 0.07374 Per Sample Data Time 0.06824 Per Sample DNN Time 0.00550 | Train Loss 0.0132 Epoch: [2][145/7530] Per Sample Total Time 0.07316 Per Sample Data Time 0.06769 Per Sample DNN Time 0.00547 | Train Loss 0.0123 Epoch: [2][245/7530] Per Sample Total Time 0.07213 Per Sample Data Time 0.06666 Per Sample DNN Time 0.00547 | Train Loss 0.0122 Epoch: [2][345/7530] Per Sample Total Time 0.07155 Per Sample Data Time 0.06608 Per Sample DNN Time 0.00547 | Train Loss 0.0121 Epoch: [2][445/7530] Per Sample Total Time 0.07121 Per Sample Data Time 0.06574 Per Sample DNN Time 0.00547 | Train Loss 0.0127 Epoch: [2][545/7530] Per Sample Total Time 0.07120 Per Sample Data Time 0.06573 Per Sample DNN Time 0.00547 | Train Loss 0.0109 Epoch: [2][645/7530] Per Sample Total Time 0.07110 Per Sample Data Time 0.06563 Per Sample DNN Time 0.00547 | Train Loss 0.0112 Epoch: [2][745/7530] Per Sample Total Time 0.07114 Per Sample Data Time 0.06567 Per Sample DNN Time 0.00547 | Train Loss 0.0121 start validation mAP: 0.078722 AUC: 0.784905 d_prime: 1.115627 train_loss: 0.012295 valid_loss: 0.023188 validation finished Epoch-2 lr: 5e-05 epoch 2 training time: 14573.965 --------------- 2023-12-13 20:11:59.510533 current #epochs=3, #steps=1510 Epoch: [3][90/7530] Per Sample Total Time 0.07367 Per Sample Data Time 0.06823 Per Sample DNN Time 0.00544 | Train Loss 0.0114 Epoch: [3][190/7530] Per Sample Total Time 0.07055 Per Sample Data Time 0.06511 Per Sample DNN Time 0.00544 | Train Loss 0.0123 Epoch: [3][290/7530] Per Sample Total Time 0.07150 Per Sample Data Time 0.06606 Per Sample DNN Time 0.00545 | Train Loss 0.0108 Epoch: [3][390/7530] Per Sample Total Time 0.07063 Per Sample Data Time 0.06519 Per Sample DNN Time 0.00544 | Train Loss 0.0104 Epoch: [3][490/7530] Per Sample Total Time 0.07121 Per Sample Data Time 0.06577 Per Sample DNN Time 0.00544 | Train Loss 0.0112 Epoch: [3][590/7530] Per Sample Total Time 0.07027 Per Sample Data Time 0.06484 Per Sample DNN Time 0.00543 | Train Loss 0.0121 Epoch: [3][690/7530] Per Sample Total Time 0.07041 Per Sample Data Time 0.06497 Per Sample DNN Time 0.00544 | Train Loss 0.0111 start validation mAP: 0.138899 AUC: 0.850596 d_prime: 1.469359 train_loss: 0.011310 valid_loss: 0.021643 validation finished Epoch-3 lr: 5e-05 epoch 3 training time: 14396.779 --------------- 2023-12-14 00:11:56.289704 current #epochs=4, #steps=2265 Epoch: [4][35/7530] Per Sample Total Time 0.07707 Per Sample Data Time 0.07160 Per Sample DNN Time 0.00547 | Train Loss 0.0109 Epoch: [4][135/7530] Per Sample Total Time 0.06923 Per Sample Data Time 0.06377 Per Sample DNN Time 0.00545 | Train Loss 0.0104 Epoch: [4][235/7530] Per Sample Total Time 0.07061 Per Sample Data Time 0.06517 Per Sample DNN Time 0.00545 | Train Loss 0.0105 Epoch: [4][335/7530] Per Sample Total Time 0.06961 Per Sample Data Time 0.06416 Per Sample DNN Time 0.00545 | Train Loss 0.0110 Epoch: [4][435/7530] Per Sample Total Time 0.07036 Per Sample Data Time 0.06491 Per Sample DNN Time 0.00545 | Train Loss 0.0107 Epoch: [4][535/7530] Per Sample Total Time 0.06985 Per Sample Data Time 0.06440 Per Sample DNN Time 0.00546 | Train Loss 0.0108 Epoch: [4][635/7530] Per Sample Total Time 0.07021 Per Sample Data Time 0.06475 Per Sample DNN Time 0.00546 | Train Loss 0.0112 Epoch: [4][735/7530] Per Sample Total Time 0.06985 Per Sample Data Time 0.06438 Per Sample DNN Time 0.00547 | Train Loss 0.0106 start validation mAP: 0.174925 AUC: 0.884112 d_prime: 1.691113 train_loss: 0.010915 valid_loss: 0.020641 validation finished Epoch-4 lr: 5e-05 epoch 4 training time: 14455.500 --------------- 2023-12-14 04:12:51.789224 current #epochs=5, #steps=3020 Epoch: [5][80/7530] Per Sample Total Time 0.07544 Per Sample Data Time 0.06996 Per Sample DNN Time 0.00547 | Train Loss 0.0109 Epoch: [5][180/7530] Per Sample Total Time 0.07160 Per Sample Data Time 0.06613 Per Sample DNN Time 0.00547 | Train Loss 0.0105 Epoch: [5][280/7530] Per Sample Total Time 0.07100 Per Sample Data Time 0.06552 Per Sample DNN Time 0.00547 | Train Loss 0.0108 Epoch: [5][380/7530] Per Sample Total Time 0.07061 Per Sample Data Time 0.06513 Per Sample DNN Time 0.00547 | Train Loss 0.0109 Epoch: [5][480/7530] Per Sample Total Time 0.07016 Per Sample Data Time 0.06468 Per Sample DNN Time 0.00548 | Train Loss 0.0111 Epoch: [5][580/7530] Per Sample Total Time 0.07033 Per Sample Data Time 0.06486 Per Sample DNN Time 0.00548 | Train Loss 0.0108 Epoch: [5][680/7530] Per Sample Total Time 0.06986 Per Sample Data Time 0.06438 Per Sample DNN Time 0.00548 | Train Loss 0.0107 start validation mAP: 0.202702 AUC: 0.907072 d_prime: 1.870921 train_loss: 0.010643 valid_loss: 0.019731 validation finished Epoch-5 lr: 5e-05 epoch 5 training time: 14296.987 --------------- 2023-12-14 08:11:08.776470 current #epochs=6, #steps=3775 Epoch: [6][25/7530] Per Sample Total Time 0.08422 Per Sample Data Time 0.07872 Per Sample DNN Time 0.00549 | Train Loss 0.0104 Epoch: [6][125/7530] Per Sample Total Time 0.07035 Per Sample Data Time 0.06485 Per Sample DNN Time 0.00550 | Train Loss 0.0109 Epoch: [6][225/7530] Per Sample Total Time 0.07128 Per Sample Data Time 0.06579 Per Sample DNN Time 0.00549 | Train Loss 0.0105 Epoch: [6][325/7530] Per Sample Total Time 0.06977 Per Sample Data Time 0.06428 Per Sample DNN Time 0.00549 | Train Loss 0.0102 Epoch: [6][425/7530] Per Sample Total Time 0.07006 Per Sample Data Time 0.06458 Per Sample DNN Time 0.00548 | Train Loss 0.0104 Epoch: [6][525/7530] Per Sample Total Time 0.06921 Per Sample Data Time 0.06373 Per Sample DNN Time 0.00548 | Train Loss 0.0103 Epoch: [6][625/7530] Per Sample Total Time 0.06983 Per Sample Data Time 0.06435 Per Sample DNN Time 0.00548 | Train Loss 0.0102 Epoch: [6][725/7530] Per Sample Total Time 0.06944 Per Sample Data Time 0.06396 Per Sample DNN Time 0.00548 | Train Loss 0.0106 start validation mAP: 0.222847 AUC: 0.922920 d_prime: 2.015236 train_loss: 0.010523 valid_loss: 0.018996 validation finished Epoch-6 lr: 5e-05 epoch 6 training time: 14337.040 --------------- 2023-12-14 12:10:05.816440 current #epochs=7, #steps=4530 Epoch: [7][70/7530] Per Sample Total Time 0.07015 Per Sample Data Time 0.06465 Per Sample DNN Time 0.00550 | Train Loss 0.0107 Epoch: [7][170/7530] Per Sample Total Time 0.07180 Per Sample Data Time 0.06633 Per Sample DNN Time 0.00546 | Train Loss 0.0106 Epoch: [7][270/7530] Per Sample Total Time 0.06986 Per Sample Data Time 0.06440 Per Sample DNN Time 0.00545 | Train Loss 0.0101 Epoch: [7][370/7530] Per Sample Total Time 0.07044 Per Sample Data Time 0.06499 Per Sample DNN Time 0.00545 | Train Loss 0.0105 Epoch: [7][470/7530] Per Sample Total Time 0.06970 Per Sample Data Time 0.06426 Per Sample DNN Time 0.00545 | Train Loss 0.0103 Epoch: [7][570/7530] Per Sample Total Time 0.07018 Per Sample Data Time 0.06472 Per Sample DNN Time 0.00545 | Train Loss 0.0109 Epoch: [7][670/7530] Per Sample Total Time 0.06965 Per Sample Data Time 0.06420 Per Sample DNN Time 0.00546 | Train Loss 0.0107 start validation mAP: 0.233517 AUC: 0.931635 d_prime: 2.104466 train_loss: 0.010369 valid_loss: 0.018710 validation finished Epoch-7 lr: 5e-05 epoch 7 training time: 14289.395 --------------- 2023-12-14 16:08:15.210923 current #epochs=8, #steps=5285 Epoch: [8][15/7530] Per Sample Total Time 0.06978 Per Sample Data Time 0.06416 Per Sample DNN Time 0.00562 | Train Loss 0.0106 Epoch: [8][115/7530] Per Sample Total Time 0.07127 Per Sample Data Time 0.06576 Per Sample DNN Time 0.00551 | Train Loss 0.0107 Epoch: [8][215/7530] Per Sample Total Time 0.06944 Per Sample Data Time 0.06395 Per Sample DNN Time 0.00549 | Train Loss 0.0104 Epoch: [8][315/7530] Per Sample Total Time 0.07081 Per Sample Data Time 0.06532 Per Sample DNN Time 0.00549 | Train Loss 0.0103 Epoch: [8][415/7530] Per Sample Total Time 0.07045 Per Sample Data Time 0.06497 Per Sample DNN Time 0.00548 | Train Loss 0.0107 Epoch: [8][515/7530] Per Sample Total Time 0.07289 Per Sample Data Time 0.06742 Per Sample DNN Time 0.00548 | Train Loss 0.0104 Epoch: [8][615/7530] Per Sample Total Time 0.07398 Per Sample Data Time 0.06850 Per Sample DNN Time 0.00547 | Train Loss 0.0108 Epoch: [8][715/7530] Per Sample Total Time 0.07829 Per Sample Data Time 0.07282 Per Sample DNN Time 0.00547 | Train Loss 0.0106 start validation mAP: 0.247072 AUC: 0.938957 d_prime: 2.186482 train_loss: 0.010346 valid_loss: 0.018521 validation finished Epoch-8 lr: 5e-05 epoch 8 training time: 15922.349 --------------- 2023-12-14 20:33:37.560014 current #epochs=9, #steps=6040 Epoch: [9][60/7530] Per Sample Total Time 0.07493 Per Sample Data Time 0.06943 Per Sample DNN Time 0.00550 | Train Loss 0.0100 Epoch: [9][160/7530] Per Sample Total Time 0.07146 Per Sample Data Time 0.06597 Per Sample DNN Time 0.00549 | Train Loss 0.0109 Epoch: [9][260/7530] Per Sample Total Time 0.07148 Per Sample Data Time 0.06599 Per Sample DNN Time 0.00549 | Train Loss 0.0107 Epoch: [9][360/7530] Per Sample Total Time 0.07058 Per Sample Data Time 0.06509 Per Sample DNN Time 0.00549 | Train Loss 0.0108 Epoch: [9][460/7530] Per Sample Total Time 0.07079 Per Sample Data Time 0.06531 Per Sample DNN Time 0.00549 | Train Loss 0.0107 Epoch: [9][560/7530] Per Sample Total Time 0.07055 Per Sample Data Time 0.06506 Per Sample DNN Time 0.00549 | Train Loss 0.0108 Epoch: [9][660/7530] Per Sample Total Time 0.07053 Per Sample Data Time 0.06505 Per Sample DNN Time 0.00549 | Train Loss 0.0103 start validation mAP: 0.256824 AUC: 0.944409 d_prime: 2.252702 train_loss: 0.010280 valid_loss: 0.018149 validation finished Epoch-9 lr: 5e-05 epoch 9 training time: 14435.500 --------------- 2023-12-15 00:34:13.060410 current #epochs=10, #steps=6795 Epoch: [10][5/7530] Per Sample Total Time 0.09897 Per Sample Data Time 0.09333 Per Sample DNN Time 0.00564 | Train Loss 0.0110 Epoch: [10][105/7530] Per Sample Total Time 0.07390 Per Sample Data Time 0.06839 Per Sample DNN Time 0.00551 | Train Loss 0.0111 Epoch: [10][205/7530] Per Sample Total Time 0.07169 Per Sample Data Time 0.06619 Per Sample DNN Time 0.00550 | Train Loss 0.0109 Epoch: [10][305/7530] Per Sample Total Time 0.07228 Per Sample Data Time 0.06679 Per Sample DNN Time 0.00549 | Train Loss 0.0105 Epoch: [10][405/7530] Per Sample Total Time 0.07138 Per Sample Data Time 0.06589 Per Sample DNN Time 0.00549 | Train Loss 0.0104 Epoch: [10][505/7530] Per Sample Total Time 0.07173 Per Sample Data Time 0.06625 Per Sample DNN Time 0.00549 | Train Loss 0.0105 Epoch: [10][605/7530] Per Sample Total Time 0.07127 Per Sample Data Time 0.06578 Per Sample DNN Time 0.00548 | Train Loss 0.0106 Epoch: [10][705/7530] Per Sample Total Time 0.07185 Per Sample Data Time 0.06637 Per Sample DNN Time 0.00548 | Train Loss 0.0103 start validation mAP: 0.270451 AUC: 0.948035 d_prime: 2.299640 train_loss: 0.010249 valid_loss: 0.017759 validation finished Epoch-10 lr: 5e-05 epoch 10 training time: 14804.214 --------------- 2023-12-15 04:40:57.274259 current #epochs=11, #steps=7550 Epoch: [11][50/7530] Per Sample Total Time 0.07992 Per Sample Data Time 0.07443 Per Sample DNN Time 0.00549 | Train Loss 0.0103 Epoch: [11][150/7530] Per Sample Total Time 0.07326 Per Sample Data Time 0.06779 Per Sample DNN Time 0.00547 | Train Loss 0.0101 Epoch: [11][250/7530] Per Sample Total Time 0.07302 Per Sample Data Time 0.06755 Per Sample DNN Time 0.00547 | Train Loss 0.0105 Epoch: [11][350/7530] Per Sample Total Time 0.07168 Per Sample Data Time 0.06622 Per Sample DNN Time 0.00546 | Train Loss 0.0106 Epoch: [11][450/7530] Per Sample Total Time 0.07189 Per Sample Data Time 0.06643 Per Sample DNN Time 0.00546 | Train Loss 0.0096 Epoch: [11][550/7530] Per Sample Total Time 0.07137 Per Sample Data Time 0.06590 Per Sample DNN Time 0.00547 | Train Loss 0.0105 Epoch: [11][650/7530] Per Sample Total Time 0.07127 Per Sample Data Time 0.06580 Per Sample DNN Time 0.00547 | Train Loss 0.0099 Epoch: [11][750/7530] Per Sample Total Time 0.07073 Per Sample Data Time 0.06527 Per Sample DNN Time 0.00547 | Train Loss 0.0099 start validation mAP: 0.273605 AUC: 0.949193 d_prime: 2.315177 train_loss: 0.010195 valid_loss: 0.017649 validation finished Epoch-11 lr: 5e-05 epoch 11 training time: 14602.655 --------------- 2023-12-15 08:44:19.929306 current #epochs=12, #steps=8305 Epoch: [12][95/7530] Per Sample Total Time 0.07099 Per Sample Data Time 0.06550 Per Sample DNN Time 0.00550 | Train Loss 0.0111 Epoch: [12][195/7530] Per Sample Total Time 0.07485 Per Sample Data Time 0.06935 Per Sample DNN Time 0.00550 | Train Loss 0.0105 Epoch: [12][295/7530] Per Sample Total Time 0.07753 Per Sample Data Time 0.07204 Per Sample DNN Time 0.00550 | Train Loss 0.0099 Epoch: [12][395/7530] Per Sample Total Time 0.08071 Per Sample Data Time 0.07522 Per Sample DNN Time 0.00549 | Train Loss 0.0102 Epoch: [12][495/7530] Per Sample Total Time 0.08546 Per Sample Data Time 0.07997 Per Sample DNN Time 0.00549 | Train Loss 0.0107 Epoch: [12][595/7530] Per Sample Total Time 0.09682 Per Sample Data Time 0.09134 Per Sample DNN Time 0.00548 | Train Loss 0.0100 Epoch: [12][695/7530] Per Sample Total Time 0.11149 Per Sample Data Time 0.10601 Per Sample DNN Time 0.00548 | Train Loss 0.0098 start validation mAP: 0.278810 AUC: 0.951363 d_prime: 2.345069 train_loss: 0.010186 valid_loss: 0.017675 validation finished Epoch-12 lr: 5e-05 epoch 12 training time: 22009.965 --------------- 2023-12-15 14:51:09.894597 current #epochs=13, #steps=9060