(yyvenvyy3) (base) yaser@vulcan07:/vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/nsff_exp$ python run_nerf.py --config configs/config_ams.txt factor None (288, 512, 3, 12) (288, 512, 12) Loaded /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/ams3_EXP2/ 25.367006217966598 102.02118748805 Loaded llff (12, 288, 512, 3) (60, 3, 5) [288. 512. 132.47626] /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/ams3_EXP2/ DEFINING BOUNDS NEAR FAR 0.0 1.0 YYXX ./logs ams3_F00-12 [] Begin TRAIN views are [ 0 1 2 3 4 5 6 7 8 9 10 11] TEST views are [] VAL views are [] expname ams3_F00-12 chain_bwd 1 lindisp False decay_iteration 25 Random FROM SINGLE IMAGE HARD MINING STAGE ! num_extra_sample 512 chain_5frames False /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/yyvenvyy3/lib/python3.7/site-packages/torch/nn/functional.py:1709: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead. warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.") /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/yyvenvyy3/lib/python3.7/site-packages/torch/nn/functional.py:1698: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead. warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.") w_depth 0.04 w_of 0.02 render_loss 0.5480020046234131 bidirection_loss 0.07860344648361206 sf_reg_loss 0.0003794229996856302 depth_loss 0.14743372797966003 flow_loss 0.10299813747406006 sf_sm_loss 0.01219163741916418 prob_reg_loss 0.09639997780323029 Step: 0, Loss: 0.9860084056854248, Time: 1.132138729095459 Saved checkpoints at ./logs/ams3_F00-12/000000.tar expname ams3_F00-12 chain_bwd 0 lindisp False decay_iteration 25 Random FROM SINGLE IMAGE HARD MINING STAGE ! num_extra_sample 512 chain_5frames False w_depth 0.04 w_of 0.02 render_loss 0.5382939577102661 bidirection_loss 0.07667422294616699 sf_reg_loss 0.00039361632661893964 depth_loss 0.14302153885364532 flow_loss 0.11433705687522888 sf_sm_loss 0.010468190535902977 prob_reg_loss 0.09648765623569489 Step: 1, Loss: 0.9796762466430664, Time: 0.96282958984375 expname ams3_F00-12 chain_bwd 1 lindisp False decay_iteration 25 Random FROM SINGLE IMAGE HARD MINING STAGE ! num_extra_sample 512 chain_5frames False w_depth 0.04 w_of 0.02 only bwd flow render_loss 0.5144094228744507 bidirection_loss 0.08542071282863617 sf_reg_loss 0.0004166632134001702 depth_loss 0.13744410872459412 flow_loss 0.06514854729175568 sf_sm_loss 0.02183498628437519 prob_reg_loss 0.09690804779529572 Step: 2, Loss: 0.9215824604034424, Time: 0.9498934745788574 expname ams3_F00-12 chain_bwd 0 lindisp False decay_iteration 25 Random FROM SINGLE IMAGE HARD MINING STAGE ! num_extra_sample 512 chain_5frames False w_depth 0.04 w_of 0.02 render_loss 0.4982748031616211 bidirection_loss 0.09155906736850739 sf_reg_loss 0.0004542608221527189 depth_loss 0.13313651084899902 flow_loss 0.15624740719795227 sf_sm_loss 0.015059935860335827 prob_reg_loss 0.09687482565641403 Step: 3, Loss: 0.9916068315505981, Time: 0.9602639675140381 expname ams3_F00-12 chain_bwd 1 lindisp False decay_iteration 25 Random FROM SINGLE IMAGE HARD MINING STAGE ! num_extra_sample 512 chain_5frames False w_depth 0.04 w_of 0.02 render_loss 0.5303755402565002 bidirection_loss 0.0894598588347435 sf_reg_loss 0.0004497336922213435 depth_loss 0.13513579964637756 flow_loss 0.14458894729614258 sf_sm_loss 0.013354028575122356 prob_reg_loss 0.09680234640836716 Step: 4, Loss: 1.0101662874221802, Time: 0.9628918170928955 expname ams3_F00-12 chain_bwd 0 lindisp False decay_iteration 25 Random FROM SINGLE IMAGE HARD MINING STAGE ! num_extra_sample 512 chain_5frames False w_depth 0.04 w_of 0.02 render_loss 0.5040347576141357 bidirection_loss 0.09653611481189728 sf_reg_loss 0.0004787869693245739 depth_loss 0.14950352907180786 flow_loss 0.16993746161460876 sf_sm_loss 0.014295339584350586 prob_reg_loss 0.0969347357749939 Step: 5, Loss: 1.031720757484436, Time: 0.961855411529541 expname ams3_F00-12 chain_bwd 1 lindisp False decay_iteration 25 Random FROM SINGLE IMAGE HARD MINING STAGE ! num_extra_sample 512 chain_5frames False w_depth 0.04 w_of 0.02 render_loss 0.5225862264633179 bidirection_loss 0.088686004281044 sf_reg_loss 0.0004327736678533256 depth_loss 0.1395830661058426 flow_loss 0.14392060041427612 sf_sm_loss 0.011364643462002277 prob_reg_loss 0.09695056080818176 Step: 6, Loss: 1.0035239458084106, Time: 0.9566366672515869 expname ams3_F00-12 chain_bwd 0 lindisp False decay_iteration 25 Random FROM SINGLE IMAGE HARD MINING STAGE ! num_extra_sample 512 chain_5frames False w_depth 0.04 w_of 0.02 render_loss 0.5132408142089844 bidirection_loss 0.09190072119235992 sf_reg_loss 0.00048530075582675636 depth_loss 0.14787255227565765 flow_loss 0.15651196241378784 sf_sm_loss 0.01362487394362688 prob_reg_loss 0.09705501049757004 *** Error in `python': corrupted size vs. prev_size: 0x00007f8e7002ced0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x7f474)[0x7f936f895474] /lib64/libc.so.6(+0x82c84)[0x7f936f898c84] /lib64/libc.so.6(__libc_malloc+0x4c)[0x7f936f89b78c] /opt/local/stow/gcc-7.5.0/lib64/libstdc++.so.6(_Znwm+0x18)[0x7f935e3d09e8] /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/yyvenvyy3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so(_ZNK5torch8autograd13SavedVariable6unpackESt10shared_ptrINS0_4NodeEE+0x924)[0x7f92eccacfb4] /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/yyvenvyy3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so(_ZN5torch8autograd9generated13AddmmBackward5applyEOSt6vectorIN2at6TensorESaIS5_EE+0x123)[0x7f92ec5b79f3] /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/yyvenvyy3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so(+0x320f361)[0x7f92ecc75361] /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/yyvenvyy3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so(_ZN5torch8autograd6Engine17evaluate_functionERSt10shared_ptrINS0_9GraphTaskEEPNS0_4NodeERNS0_11InputBufferERKS2_INS0_10ReadyQueueEE+0x153b)[0x7f92ecc7116b] /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/yyvenvyy3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so(_ZN5torch8autograd6Engine11thread_mainERKSt10shared_ptrINS0_9GraphTaskEE+0x49f)[0x7f92ecc71d8f] /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/yyvenvyy3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so(_ZN5torch8autograd6Engine11thread_initEiRKSt10shared_ptrINS0_10ReadyQueueEEb+0x89)[0x7f92ecc69569] /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/yyvenvyy3/lib/python3.7/site-packages/torch/lib/libtorch_python.so(_ZN5torch8autograd6python12PythonEngine11thread_initEiRKSt10shared_ptrINS0_10ReadyQueueEEb+0x53)[0x7f935d3f10c3] /opt/local/stow/gcc-7.5.0/lib64/libstdc++.so.6(+0xba2df)[0x7f935e3fb2df] /lib64/libpthread.so.0(+0x7ea5)[0x7f93702f4ea5] /lib64/libc.so.6(clone+0x6d)[0x7f936f9149fd] ======= Memory map: ======== 00400000-00401000 r-xp 00000000 00:28 3759616179 /opt/local/stow/Python3-3.7.6/bin/python3.7 00600000-00601000 r--p 00000000 00:28 3759616179 /opt/local/stow/Python3-3.7.6/bin/python3.7 00601000-00602000 rw-p 00001000 00:28 3759616179 /opt/local/stow/Python3-3.7.6/bin/python3.7 01cde000-73ff6000 rw-p 00000000 00:00 0 [heap] 200000000-200200000 ---p 00000000 00:00 0 200200000-200400000 rw-s 00000000 00:05 61790 /dev/nvidiactl 200400000-202400000 rw-s 00000000 00:05 61790 /dev/nvidiactl 202400000-205400000 rw-s 00000000 00:05 61790 /dev/nvidiactl 205400000-206000000 ---p 00000000 00:00 0 206000000-206200000 rw-s 00000000 00:05 61790 /dev/nvidiactl 206200000-206400000 rw-s 00000000 00:05 61790 /dev/nvidiactl 206400000-206600000 rw-s 206400000 00:05 70043 /dev/nvidia-uvm 206600000-206800000 rw-s 00000000 00:05 61790 /dev/nvidiactl 206800000-206a00000 ---p 00000000 00:00 0 206a00000-206c00000 rw-s 00000000 00:05 61790 /dev/nvidiactl 206c00000-300200000 ---p 00000000 00:00 0 10000000000-10104000000 ---p 00000000 00:00 0 7f8e3c000000-7f8e3c021000 rw-p 00000000 00:00 0 7f8e3c021000-7f8e40000000 ---p 00000000 00:00 0 7f8e42000000-7f8e70000000 ---p 00000000 00:00 0 7f8e70000000-7f8e7013f000 rw-p 00000000 00:00 0 7f8e7013f000-7f8e74000000 ---p 00000000 00:00 0 7f8e76000000-7f921c000000 ---p 00000000 00:00 0 7f921c000000-7f921c102000 rw-p 00000000 00:00 0 7f921c102000-7f9220000000 ---p 00000000 00:00 0 7f9222000000-7f9235800000 ---p 00000000 00:00 0 7f9235800000-7f9235a00000 rw-s 00000000 00:04 160844853 /dev/zero (deleted) 7f9235a00000-7f9240000000 ---p 00000000 00:00 0 7f9240000000-7f9240021000 rw-p 00000000 00:00 0 7f9240021000-7f9244000000 ---p 00000000 00:00 0 7f9246000000-7f9246400000 ---p 00000000 00:00 0 7f9246400000-7f9246600000 rw-s 00000000 00:04 160844848 /dev/zero (deleted) 7f9246600000-7f9246800000 rw-s 00000000 00:05 61790 /dev/nvidiactl 7f9246800000-7f9246a00000 rw-s 00000000 00:04 160844849 /dev/zero (deleted) 7f9246a00000-7f9246c00000 ---p 00000000 00:00 0 7f9246c00000-7f9246e00000 rw-s 00000000 00:05 61790 /dev/nvidiactl 7f9246e00000-7f9247000000 ---p 00000000 00:00 0 7f9247000000-7f9247200000 rw-s 00000000 00:04 160844851 /dev/zero (deleted) 7f9247200000-7f92474d6000 rw-s 00000000 00:05 61790 /dev/nvidiactl 7f92474d6000-7f9248000000 ---p 00000000 00:00 0 7f9249fa3000-7f924bfa3000 rw-p 00000000 00:00 0 7f924bfe0000-7f924c260000 rw-p 00000000 00:00 0 7f924c260000-7f924c261000 ---p 00000000 00:00 0 7f924c261000-7f924ce21000 rw-p 00000000 00:00 0 7f924ce21000-7f924ce22000 ---p 00000000 00:00 0 7f924ce22000-7f924d622000 rw-p 00000000 00:00 0 7f924d622000-7f924d822000 rw-s 00000000 00:04 160844850 /dev/zero (deleted) 7f924d822000-7f924d823000 ---p 00000000 00:00 0 7f924d823000-7f924e023000 rw-p 00000000 00:00 0 7f924e023000-7f924e024000 ---p 00000000 00:00 0 7f924e024000-7f924e864000 rw-p 00000000 00:00 0 7f924e864000-7f924e8b1000 r-xp 00000000 00:28 3759622457 /opt/local/stow/Python3-3.7.6/lib/python3.7/lib-dynload/_decimal.cpython-37m-x86_64-linux-gnu.so 7f924e8b1000-7f924eab0000 ---p 0004d000 00:28 3759622457 /opt/local/stow/Python3-3.7.6/lib/python3.7/lib-dynload/_decimal.cpython-37m-x86_64-linux-gnu.so 7f924eab0000-7f924eab1000 r--p 0004c000 00:28 3759622457 /opt/local/stow/Python3-3.7.6/lib/python3.7/lib-dynload/_decimal.cpython-37m-x86_64-linux-gnu.so 7f924eab1000-7f924eaba000 rw-p 0004d000 00:28 3759622457 /opt/local/stow/Python3-3.7.6/lib/python3.7/lib-dynload/_decimal.cpython-37m-x86_64-linux-gnu.so 7f924eaba000-7f924eb3a000 rw-p 00000000 00:00 0 7f924eb3a000-7f924eb3f000 r-xp 00000000 00:42 10329262 /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/yyvenvyy3/lib/python3.7/site-packages/Pillow.libs/libXdmcp-e15573e7.so.6.0.0 7f924eb3f000-7f924ed3e000 ---p 00005000 00:42 10329262 /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/yyvenvyy3/lib/python3.7/site-packages/Pillow.libs/libXdmcp-e15573e7.so.6.0.0 7f924ed3e000-7f924ed40000 rw-p 00004000 00:42 10329262 /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/yyvenvyy3/lib/python3.7/site-packages/Pillow.libs/libXdmcp-e15573e7.so.6.0.0 7f924ed40000-7f924ed42000 r-xp 00000000 00:42 10337451 /vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/yyvenvyy3/lib/python3.7/site-packages/Pillow.libs/libXau-312dbc56.so.6.0.0Aborted (core dumped) (yyvenvyy3) (base) yaser@vulcan07:/vulcanscratch/yaser/VIDEO/Neural-Scene-Flow-Fields/nsff_exp$