opt.color_loss_items ['ray_masked_coarse_raycolor', 'ray_miss_coarse_raycolor', 'coarse_raycolor'] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Debug Mode ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ /usr/local/lib/python3.7/dist-packages/numpy/core/shape_base.py:420: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray. arrays = [asanyarray(arr) for arr in arrays] cam_points (336, 3) (336, 3) [1.81716196 2.44570022 1.96238958 2.16126027 2.09860581 1.31913184 2.3367693 1.96491161 2.67180893 2.26194129 2.60715615 1.74131191 2.8916889 2.83244113 2.56821701 2.62587521 2.34225579 2.17449685 2.05444007 2.01276715 2.67588859 2.66449982 1.96746667 2.88720236 2.36614622 2.66460748 1.92164603 1.90086848 1.73110557 2.27654528 1.84856773 2.26529174 1.86078119 2.5455486 2.70516926 2.87764254 2.36827083 2.62470783 2.72193956 2.88893599 2.75152799 2.28794224 2.49945202 1.37682749 1.79817419 2.77336373 2.47505956 2.15562537 2.59977839 1.93776372 1.65845967 2.6966041 2.47514613 2.51631626 1.36481633 2.30239115 2.33988582 1.34023171 2.14838412 2.28630439 2.88301412 2.4744344 1.76686721 2.2666648 2.20473681 2.27286507 2.50666569 2.09103486 1.89678514 2.74363018 2.68770224 2.94274601 2.87392307 2.30493472 1.87961396 1.42504995 1.52287357 2.1852683 2.8371983 2.82594996 1.48809217 2.2086535 2.64137929 1.761733 2.58479397 2.26771811 2.87231663 1.80708797 2.19661539 2.17406562 2.48021929 2.30167716 2.27985747 2.47982 1.51870918 2.25006141 1.92197214 2.70870916 2.28032155 1.97330576 2.56429468 2.75715415 1.80499589 2.71858427 2.747406 2.1252561 2.03128392 2.08701769 1.91598283 2.55093805 2.34233617 3.00990485 2.33926142 3.04156278 1.84102692 1.58988751 1.78108287 1.80726804 2.80246656 2.76454723 2.56199505 2.28856716 1.93882671 2.77692632 2.66771133 2.58324747 1.52715479 2.88723516 2.70747362 2.48964739 1.5025113 2.78180759 2.21769276 1.55662898 2.34112531 2.29066557 2.76145663 2.22434294 2.05607942 2.54442411 2.54112831 1.50728083 1.82085168 2.69961025 2.47589784 2.88416145 2.74964208 2.73576157 2.33282581 1.5985036 1.96986369 2.2781555 1.8889204 2.40331926 2.00948483 1.95359076 2.3563405 2.72794663 2.1901941 2.62406643 1.82345498 1.68841143 2.83907441 1.92386478 2.70183876 2.68563219 2.18717317 2.2906274 1.52039635 2.71592763 1.88740431 2.83390154 1.99631098 2.89543029 2.10291857 2.72202683 2.83590293 2.84714324 2.24419946 1.28916408 1.73075473 2.13111223 2.02142238 1.93053577 1.48310514 2.682173 1.3155245 2.55311495 2.29379679 2.69863647 1.63617749 2.77924394 2.51156645 2.52945505 1.51558375 2.21254253 3.02721855 2.45773082 2.42687143 2.66892472 2.49010462 2.45172775 2.19615062 2.23222378 2.46813432 2.79440424 2.5094242 2.71013564 2.98737759 2.57683712 2.02074765 2.91901682 1.92967957 2.47955637 1.97191932 2.49845286 2.25842758 2.54347242 2.04527998 1.8798868 2.77367789 1.94342771 2.49126209 2.76095895 1.47274357 2.2880897 2.0753395 2.33990874 2.23692891 2.349446 2.16889303 1.77029657 2.33664162 2.07220785 2.33029173 2.15095897 1.37353195 2.22786891 2.75748198 2.76007622 2.75175872 2.54991852 2.69575691 1.81417873 1.70900727 2.68822264 2.55557076 1.77598835 1.73689569 1.53088265 1.33587297 2.75485753 2.14352336 2.41895634 3.0438883 2.68446937 1.81403645 2.82557142 2.2098586 1.42253156 2.35162274 2.57726279 1.93370608 2.19941095 2.001088 2.71274333 2.41962569 1.76020152 2.81856426 2.27364855 2.08343274 1.35271422 2.45746395 2.24798433 2.15924567 2.23139916 2.44898242 1.32439266 2.4110307 3.00228587 2.43660996 2.8366013 2.69442943 2.32237519 2.81025667 2.12427491 1.90475138 2.71575907 2.23600809 2.70722477 2.25559416 2.65414862 2.30265541 2.01082681 2.72905277 2.50115848 1.86112915 2.77093417 1.32292202 1.85421765 2.02184304 2.67268925 2.66040904 2.16442052 2.30254759 2.29977021 2.28222669 2.34100944 2.61331155 2.55574742 2.79125685 2.81717489 2.33929794 2.77277598 2.09180486 2.83970692 2.28460986 2.42253021 1.85087111 1.99619342 1.51120762 2.41198928 2.26196053 2.8479847 1.92935362 1.52228841 2.17570879 2.63411908 2.83692657 2.24789731 2.42655606 2.89061672 2.23547722 2.2510608 2.21155326 1.96322447] dataset total: train 336 dataset [NerfSynthFtDataset] was created ../checkpoints/tanksntemples/barn/*_net_ray_marching.pth ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ test at 200000 iters Iter: 200000 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ opt.act_type!!!!!!!!! LeakyReLU self.points_embeding torch.Size([1, 1319235, 32]) querier device cuda:0 0 /content/pointnerf/run/../models/neural_points/query_point_indices_worldcoords.py:524: UserWarning: The CUDA compiler succeeded, but said the following: nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). kernel.cu(62): warning: function "cuda::atomicAdd(short *, short)" was declared but never referenced kernel.cu(58): warning: function "cuda::atomicAdd(int8_t *, int8_t)" was declared but never referenced kernel.cu(13): warning: function "cuda::atomicAdd(uint8_t *, uint8_t)" was declared but never referenced kernel.cu(92): warning: function "cuda::cas(float *, float, float)" was declared but never referenced kernel.cu(85): warning: function "cuda::cas(double *, double, double)" was declared but never referenced kernel.cu(101): warning: function "cuda::atomicCAS" was declared but never referenced """, no_extern_c=True) neural_params [('module.neural_points.xyz', torch.Size([1319235, 3]), False), ('module.neural_points.points_embeding', torch.Size([1, 1319235, 32]), True), ('module.neural_points.points_conf', torch.Size([1, 1319235, 1]), True), ('module.neural_points.points_dir', torch.Size([1, 1319235, 3]), True), ('module.neural_points.points_color', torch.Size([1, 1319235, 3]), True), ('module.neural_points.Rw2c', torch.Size([3, 3]), False)] model [MvsPointsVolumetricModel] was created opt.resume_iter!!!!!!!!! 200000 loading ray_marching from ../checkpoints/tanksntemples/barn/200000_net_ray_marching.pth ------------------- Networks ------------------- [Network ray_marching] Total number of parameters: 55.750M ------------------------------------------------ /usr/local/lib/python3.7/dist-packages/numpy/core/shape_base.py:420: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray. arrays = [asanyarray(arr) for arr in arrays] dataset total: test 48 dataset [NerfSynthFtDataset] was created -----------------------------------Testing----------------------------------- test set size 48, interval 1 data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.0 in 48 cases: time used: 33.245070457458496 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0006, device='cuda:0') tensor(31.9783, device='cuda:0') ray_masked_coarse_raycolor loss:0.00189653888810426, PSNR:27.220380783081055 dict_items([('coarse_raycolor', tensor(0.0006, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0019, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.1 in 48 cases: time used: 57.7176570892334 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0017, device='cuda:0') tensor(27.7940, device='cuda:0') ray_masked_coarse_raycolor loss:0.0022489121183753014, PSNR:26.480274200439453 dict_items([('coarse_raycolor', tensor(0.0017, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0022, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.2 in 48 cases: time used: 47.80196976661682 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0012, device='cuda:0') tensor(29.3484, device='cuda:0') ray_masked_coarse_raycolor loss:0.0016319644637405872, PSNR:27.872888565063477 dict_items([('coarse_raycolor', tensor(0.0012, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0016, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.3 in 48 cases: time used: 58.52335596084595 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0011, device='cuda:0') tensor(29.6383, device='cuda:0') ray_masked_coarse_raycolor loss:0.0012207103427499533, PSNR:29.13387107849121 dict_items([('coarse_raycolor', tensor(0.0011, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0012, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.4 in 48 cases: time used: 36.12207221984863 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0006, device='cuda:0') tensor(32.3557, device='cuda:0') ray_masked_coarse_raycolor loss:0.001546903746202588, PSNR:28.105363845825195 dict_items([('coarse_raycolor', tensor(0.0006, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0015, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.5 in 48 cases: time used: 37.15648102760315 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0007, device='cuda:0') tensor(31.7865, device='cuda:0') ray_masked_coarse_raycolor loss:0.001689524739049375, PSNR:27.722352981567383 dict_items([('coarse_raycolor', tensor(0.0007, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0017, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.6 in 48 cases: time used: 34.83007264137268 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0007, device='cuda:0') tensor(31.5959, device='cuda:0') ray_masked_coarse_raycolor loss:0.0017439342336729169, PSNR:27.584697723388672 dict_items([('coarse_raycolor', tensor(0.0007, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0017, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.7 in 48 cases: time used: 45.85961604118347 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0014, device='cuda:0') tensor(28.5135, device='cuda:0') ray_masked_coarse_raycolor loss:0.002066763350740075, PSNR:26.847089767456055 dict_items([('coarse_raycolor', tensor(0.0014, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0021, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.8 in 48 cases: time used: 45.80135726928711 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0015, device='cuda:0') tensor(28.1982, device='cuda:0') ray_masked_coarse_raycolor loss:0.0023651965893805027, PSNR:26.26132583618164 dict_items([('coarse_raycolor', tensor(0.0015, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0024, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.9 in 48 cases: time used: 43.13188600540161 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0015, device='cuda:0') tensor(28.2527, device='cuda:0') ray_masked_coarse_raycolor loss:0.0021701797377318144, PSNR:26.63504409790039 dict_items([('coarse_raycolor', tensor(0.0015, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0022, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.10 in 48 cases: time used: 33.215099811553955 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0008, device='cuda:0') tensor(30.8985, device='cuda:0') ray_masked_coarse_raycolor loss:0.0018614240689203143, PSNR:27.301546096801758 dict_items([('coarse_raycolor', tensor(0.0008, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0019, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.11 in 48 cases: time used: 51.48407292366028 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0017, device='cuda:0') tensor(27.7919, device='cuda:0') ray_masked_coarse_raycolor loss:0.0023374108131974936, PSNR:26.31264877319336 dict_items([('coarse_raycolor', tensor(0.0017, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0023, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.12 in 48 cases: time used: 28.61889910697937 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0004, device='cuda:0') tensor(34.4368, device='cuda:0') ray_masked_coarse_raycolor loss:0.0011697126319631934, PSNR:29.31920623779297 dict_items([('coarse_raycolor', tensor(0.0004, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0012, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.13 in 48 cases: time used: 56.73006200790405 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0016, device='cuda:0') tensor(27.8414, device='cuda:0') ray_masked_coarse_raycolor loss:0.0020620222203433514, PSNR:26.857067108154297 dict_items([('coarse_raycolor', tensor(0.0016, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0021, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.14 in 48 cases: time used: 45.80361199378967 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0021, device='cuda:0') tensor(26.8541, device='cuda:0') ray_masked_coarse_raycolor loss:0.0038017614278942347, PSNR:24.200149536132812 dict_items([('coarse_raycolor', tensor(0.0021, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0038, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.15 in 48 cases: time used: 53.408278465270996 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0044, device='cuda:0') tensor(23.5632, device='cuda:0') ray_masked_coarse_raycolor loss:0.006054313853383064, PSNR:22.17934799194336 dict_items([('coarse_raycolor', tensor(0.0044, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0061, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.16 in 48 cases: time used: 36.503607511520386 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0014, device='cuda:0') tensor(28.5779, device='cuda:0') ray_masked_coarse_raycolor loss:0.003340453375130892, PSNR:24.761943817138672 dict_items([('coarse_raycolor', tensor(0.0014, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0033, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.17 in 48 cases: time used: 62.85940504074097 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0012, device='cuda:0') tensor(29.0636, device='cuda:0') ray_masked_coarse_raycolor loss:0.0014236632268875837, PSNR:28.465925216674805 dict_items([('coarse_raycolor', tensor(0.0012, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0014, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.18 in 48 cases: time used: 67.02558135986328 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0012, device='cuda:0') tensor(29.3238, device='cuda:0') ray_masked_coarse_raycolor loss:0.001281438278965652, PSNR:28.923023223876953 dict_items([('coarse_raycolor', tensor(0.0012, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0013, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.19 in 48 cases: time used: 46.453757762908936 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0011, device='cuda:0') tensor(29.5593, device='cuda:0') ray_masked_coarse_raycolor loss:0.001818854478187859, PSNR:27.40201759338379 dict_items([('coarse_raycolor', tensor(0.0011, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0018, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.20 in 48 cases: time used: 44.56791353225708 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0013, device='cuda:0') tensor(29.0036, device='cuda:0') ray_masked_coarse_raycolor loss:0.0020137864630669355, PSNR:26.959863662719727 dict_items([('coarse_raycolor', tensor(0.0013, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0020, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.21 in 48 cases: time used: 52.38806867599487 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0021, device='cuda:0') tensor(26.8338, device='cuda:0') ray_masked_coarse_raycolor loss:0.002852906472980976, PSNR:25.44712257385254 dict_items([('coarse_raycolor', tensor(0.0021, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0029, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.22 in 48 cases: time used: 54.431315898895264 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0024, device='cuda:0') tensor(26.2692, device='cuda:0') ray_masked_coarse_raycolor loss:0.00301218475215137, PSNR:25.211181640625 dict_items([('coarse_raycolor', tensor(0.0024, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0030, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.23 in 48 cases: time used: 29.80834436416626 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0006, device='cuda:0') tensor(31.9887, device='cuda:0') ray_masked_coarse_raycolor loss:0.0023217389825731516, PSNR:26.34186553955078 dict_items([('coarse_raycolor', tensor(0.0006, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0023, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.24 in 48 cases: time used: 44.40573263168335 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0015, device='cuda:0') tensor(28.1164, device='cuda:0') ray_masked_coarse_raycolor loss:0.0023294761776924133, PSNR:26.327415466308594 dict_items([('coarse_raycolor', tensor(0.0015, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0023, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.25 in 48 cases: time used: 59.230486154556274 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0018, device='cuda:0') tensor(27.3827, device='cuda:0') ray_masked_coarse_raycolor loss:0.0022753991652280092, PSNR:26.429424285888672 dict_items([('coarse_raycolor', tensor(0.0018, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0023, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.26 in 48 cases: time used: 37.131062507629395 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0007, device='cuda:0') tensor(31.4633, device='cuda:0') ray_masked_coarse_raycolor loss:0.0015736191999167204, PSNR:28.031002044677734 dict_items([('coarse_raycolor', tensor(0.0007, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0016, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.27 in 48 cases: time used: 57.55204963684082 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0033, device='cuda:0') tensor(24.7646, device='cuda:0') ray_masked_coarse_raycolor loss:0.004254345316439867, PSNR:23.711671829223633 dict_items([('coarse_raycolor', tensor(0.0033, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0043, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.28 in 48 cases: time used: 31.15811252593994 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0006, device='cuda:0') tensor(31.9901, device='cuda:0') ray_masked_coarse_raycolor loss:0.002094773342832923, PSNR:26.78862953186035 dict_items([('coarse_raycolor', tensor(0.0006, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0021, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.29 in 48 cases: time used: 54.848225116729736 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0014, device='cuda:0') tensor(28.5069, device='cuda:0') ray_masked_coarse_raycolor loss:0.0018719237996265292, PSNR:27.277116775512695 dict_items([('coarse_raycolor', tensor(0.0014, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0019, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.30 in 48 cases: time used: 27.248998165130615 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0007, device='cuda:0') tensor(31.5604, device='cuda:0') ray_masked_coarse_raycolor loss:0.0026418722700327635, PSNR:25.780879974365234 dict_items([('coarse_raycolor', tensor(0.0007, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0026, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.31 in 48 cases: time used: 31.453717708587646 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0004, device='cuda:0') tensor(33.8698, device='cuda:0') ray_masked_coarse_raycolor loss:0.0011748801916837692, PSNR:29.30006217956543 dict_items([('coarse_raycolor', tensor(0.0004, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0012, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.32 in 48 cases: time used: 50.55198884010315 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0025, device='cuda:0') tensor(26.0323, device='cuda:0') ray_masked_coarse_raycolor loss:0.0041836840100586414, PSNR:23.784408569335938 dict_items([('coarse_raycolor', tensor(0.0025, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0042, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.33 in 48 cases: time used: 54.534178256988525 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0012, device='cuda:0') tensor(29.1858, device='cuda:0') ray_masked_coarse_raycolor loss:0.0015876052202656865, PSNR:27.992572784423828 dict_items([('coarse_raycolor', tensor(0.0012, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0016, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.34 in 48 cases: time used: 64.28956651687622 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0009, device='cuda:0') tensor(30.5289, device='cuda:0') ray_masked_coarse_raycolor loss:0.0010425829095765948, PSNR:29.818891525268555 dict_items([('coarse_raycolor', tensor(0.0009, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0010, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.35 in 48 cases: time used: 47.768404483795166 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0010, device='cuda:0') tensor(29.8683, device='cuda:0') ray_masked_coarse_raycolor loss:0.0014665352646261454, PSNR:28.337074279785156 dict_items([('coarse_raycolor', tensor(0.0010, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0015, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.36 in 48 cases: time used: 54.24845266342163 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0012, device='cuda:0') tensor(29.0868, device='cuda:0') ray_masked_coarse_raycolor loss:0.0017037768848240376, PSNR:27.68587303161621 dict_items([('coarse_raycolor', tensor(0.0012, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0017, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.37 in 48 cases: time used: 39.26139283180237 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0007, device='cuda:0') tensor(31.3886, device='cuda:0') ray_masked_coarse_raycolor loss:0.0017350130947306752, PSNR:27.606971740722656 dict_items([('coarse_raycolor', tensor(0.0007, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0017, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.38 in 48 cases: time used: 35.45406937599182 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0006, device='cuda:0') tensor(32.2148, device='cuda:0') ray_masked_coarse_raycolor loss:0.0014754002913832664, PSNR:28.310901641845703 dict_items([('coarse_raycolor', tensor(0.0006, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0015, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.39 in 48 cases: time used: 29.729392766952515 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0010, device='cuda:0') tensor(30.0422, device='cuda:0') ray_masked_coarse_raycolor loss:0.0035855129826813936, PSNR:24.454484939575195 dict_items([('coarse_raycolor', tensor(0.0010, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0036, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.40 in 48 cases: time used: 47.86436057090759 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0009, device='cuda:0') tensor(30.5783, device='cuda:0') ray_masked_coarse_raycolor loss:0.001228352659381926, PSNR:29.106767654418945 dict_items([('coarse_raycolor', tensor(0.0009, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0012, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.41 in 48 cases: time used: 30.053267240524292 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0007, device='cuda:0') tensor(31.3207, device='cuda:0') ray_masked_coarse_raycolor loss:0.002716704038903117, PSNR:25.659576416015625 dict_items([('coarse_raycolor', tensor(0.0007, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0027, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.42 in 48 cases: time used: 36.75830292701721 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0008, device='cuda:0') tensor(30.8160, device='cuda:0') ray_masked_coarse_raycolor loss:0.0017053002957254648, PSNR:27.681989669799805 dict_items([('coarse_raycolor', tensor(0.0008, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0017, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.43 in 48 cases: time used: 58.534098863601685 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0030, device='cuda:0') tensor(25.1864, device='cuda:0') ray_masked_coarse_raycolor loss:0.0038448795676231384, PSNR:24.15117073059082 dict_items([('coarse_raycolor', tensor(0.0030, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0038, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.44 in 48 cases: time used: 51.18449807167053 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0017, device='cuda:0') tensor(27.8225, device='cuda:0') ray_masked_coarse_raycolor loss:0.0022716692183166742, PSNR:26.436546325683594 dict_items([('coarse_raycolor', tensor(0.0017, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0023, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.45 in 48 cases: time used: 34.571436643600464 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0008, device='cuda:0') tensor(31.2277, device='cuda:0') ray_masked_coarse_raycolor loss:0.0017093643546104431, PSNR:27.67165184020996 dict_items([('coarse_raycolor', tensor(0.0008, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0017, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.46 in 48 cases: time used: 54.30385994911194 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0035, device='cuda:0') tensor(24.5089, device='cuda:0') ray_masked_coarse_raycolor loss:0.004597131162881851, PSNR:23.37512969970703 dict_items([('coarse_raycolor', tensor(0.0035, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0046, device='cuda:0'))]) data['gt_image'] coarse_raycolor:(640, 1088, 3) gt_image:torch.Size([696320, 3]) num.47 in 48 cases: time used: 30.88929843902588 s at ../checkpoints/tanksntemples/barn/test_200000/images coarse_raycolor tensor(0.0006, device='cuda:0') tensor(32.1757, device='cuda:0') ray_masked_coarse_raycolor loss:0.0020541020203381777, PSNR:26.873779296875 dict_items([('coarse_raycolor', tensor(0.0006, device='cuda:0')), ('ray_masked_coarse_raycolor', tensor(0.0021, device='cuda:0'))]) End of iteration 48 Number of batches 48 Time taken: 2199.86s [Average Loss] coarse_raycolor: 0.0013507337 coarse_raycolor_psnr: 29.3980293274 ray_masked_coarse_raycolor: 0.0022720045 ray_masked_coarse_raycolor_psnr: 26.7945842743 --------------------------------Finish Test Rendering-------------------------------- test id_list [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47] ../checkpoints/tanksntemples/barn/test_200000/images ../checkpoints/tanksntemples/barn/test_200000/images ../checkpoints/tanksntemples/barn/test_200000/images step-%04d-coarse_raycolor.png step-%04d-gt_image.png Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off] Downloading: "https://download.pytorch.org/models/alexnet-owt-7be5be79.pth" to /root/.cache/torch/hub/checkpoints/alexnet-owt-7be5be79.pth 100% 233M/233M [00:11<00:00, 21.7MB/s] Loading model from: /usr/local/lib/python3.7/dist-packages/lpips/weights/v0.1/alex.pth Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off] Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /root/.cache/torch/hub/checkpoints/vgg16-397923af.pth 100% 528M/528M [00:30<00:00, 18.0MB/s] Loading model from: /usr/local/lib/python3.7/dist-packages/lpips/weights/v0.1/vgg.pth 48 images computed psnr: 29.408589 ssim: 0.939953 lpips: 0.124517 vgglpips: 0.180550 rmse: 0.035239 --------------------------------Finish Evaluation-------------------------------- end loading end loading ------------------------------------------------------------------- PyCUDA ERROR: The context stack was not empty upon module cleanup. ------------------------------------------------------------------- A context was still active when the context stack was being cleaned up. At this point in our execution, CUDA may already have been deinitialized, so there is no way we can finish cleanly. The program will be aborted now. Use Context.pop() to avoid this problem. ------------------------------------------------------------------- dev_scripts/w_tt_ft/barn_test.sh: line 170: 914 Aborted (core dumped) python3 test_ft.py --name $name --scan $scan --data_root $data_root --dataset_name $dataset_name --model $model --which_render_func $which_render_func --which_blend_func $which_blend_func --out_channels $out_channels --num_pos_freqs $num_pos_freqs --num_viewdir_freqs $num_viewdir_freqs --random_sample $random_sample --random_sample_size $random_sample_size --batch_size $batch_size --gpu_ids $gpu_ids --checkpoints_dir $checkpoints_dir --pin_data_in_memory $pin_data_in_memory --test_num_step $test_num_step --test_color_loss_items $test_color_loss_items --bg_color $bg_color --split $split --which_ray_generation $which_ray_generation --near_plane $near_plane --far_plane $far_plane --dir_norm $dir_norm --which_tonemap_func $which_tonemap_func --resume_dir $resume_dir --resume_iter $resume_iter --agg_axis_weight $agg_axis_weight --agg_distance_kernel $agg_distance_kernel --radius_limit_scale $radius_limit_scale --depth_limit_scale $depth_limit_scale --vscale $vscale --kernel_size $kernel_size --SR $SR --K $K --P $P --NN $NN --agg_feat_xyz_mode $agg_feat_xyz_mode --agg_alpha_xyz_mode $agg_alpha_xyz_mode --agg_color_xyz_mode $agg_color_xyz_mode --raydist_mode_unit $raydist_mode_unit --agg_dist_pers $agg_dist_pers --agg_intrp_order $agg_intrp_order --shading_feature_mlp_layer0 $shading_feature_mlp_layer0 --shading_feature_mlp_layer1 $shading_feature_mlp_layer1 --shading_feature_mlp_layer2 $shading_feature_mlp_layer2 --shading_feature_mlp_layer3 $shading_feature_mlp_layer3 --shading_feature_num $shading_feature_num --dist_xyz_freq $dist_xyz_freq --shpnt_jitter $shpnt_jitter --shading_alpha_mlp_layer $shading_alpha_mlp_layer --shading_color_mlp_layer $shading_color_mlp_layer --which_agg_model $which_agg_model --color_loss_weights $color_loss_weights --num_feat_freqs $num_feat_freqs --dist_xyz_deno $dist_xyz_deno --apply_pnt_mask $apply_pnt_mask --point_features_dim $point_features_dim --color_loss_items $color_loss_items --visual_items $visual_items --act_type $act_type --point_conf_mode $point_conf_mode --point_dir_mode $point_dir_mode --point_color_mode $point_color_mode --normview $normview --alpha_range $alpha_range --ranges $ranges --mvs_img_wh $mvs_img_wh --img_wh $img_wh --vsize $vsize --wcoord_query $wcoord_query --max_o $max_o --debug