SplaTAM trying to allocate significant memory

[AlexanderVoldby]

I am trying to run SplaTAM on a dataset that i've collected offline using the method from #59. After doing the necessary processing I run $ python scripts/splatam.py configs/iphone/splatam.py
resulting in the folllowing error log

n-62-12-19(s214591) $ python scripts/splatam.py configs/iphone/splatam.py
System Paths:
/zhome/75/8/168142/Desktop/Bachelorprojekt/SplaTAM
/zhome/75/8/168142/Desktop/Bachelorprojekt/SplaTAM/scripts
/zhome/75/8/168142/miniconda3/envs/splatam/lib/python310.zip
/zhome/75/8/168142/miniconda3/envs/splatam/lib/python3.10
/zhome/75/8/168142/miniconda3/envs/splatam/lib/python3.10/lib-dynload
/zhome/75/8/168142/miniconda3/envs/splatam/lib/python3.10/site-packages
Seed set to: 0 (type: <class 'int'>)
Loaded Config:
{'workdir': '././experiments/iPhone_Captures/offline_demo', 'run_name': 'SplaTAM_iPhone', 'overwrite': False, 'depth_scale': 10.0, 'num_frames': 10, 'seed': 0, 'primary_device': 'cuda:0', 'map_every': 1, 'keyframe_every': 2, 'mapping_window_size': 32, 'report_global_progress_every': 100, 'eval_every': 1, 'scene_radius_depth_ratio': 3, 'mean_sq_dist_method': 'projective', 'report_iter_progress': False, 'load_checkpoint': False, 'checkpoint_time_idx': 130, 'save_checkpoints': False, 'checkpoint_interval': 5, 'use_wandb': False, 'data': {'dataset_name': 'nerfcapture', 'basedir': './experiments/iPhone_Captures', 'sequence': 'offline_demo', 'desired_image_height': 720, 'desired_image_width': 960, 'densification_image_height': 360, 'densification_image_width': 480, 'start': 0, 'end': -1, 'stride': 1, 'num_frames': 10}, 'tracking': {'use_gt_poses': False, 'forward_prop': True, 'visualize_tracking_loss': False, 'num_iters': 60, 'use_sil_for_loss': True, 'sil_thres': 0.99, 'use_l1': True, 'use_depth_loss_thres': True, 'depth_loss_thres': 20000, 'ignore_outlier_depth_loss': False, 'use_uncertainty_for_loss_mask': False, 'use_uncertainty_for_loss': False, 'use_chamfer': False, 'loss_weights': {'im': 0.5, 'depth': 1.0}, 'lrs': {'means3D': 0.0, 'rgb_colors': 0.0, 'unnorm_rotations': 0.0, 'logit_opacities': 0.0, 'log_scales': 0.0, 'cam_unnorm_rots': 0.001, 'cam_trans': 0.004}}, 'mapping': {'num_iters': 60, 'add_new_gaussians': True, 'sil_thres': 0.5, 'use_l1': True, 'ignore_outlier_depth_loss': False, 'use_sil_for_loss': False, 'use_uncertainty_for_loss_mask': False, 'use_uncertainty_for_loss': False, 'use_chamfer': False, 'loss_weights': {'im': 0.5, 'depth': 1.0}, 'lrs': {'means3D': 0.0001, 'rgb_colors': 0.0025, 'unnorm_rotations': 0.001, 'logit_opacities': 0.05, 'log_scales': 0.001, 'cam_unnorm_rots': 0.0, 'cam_trans': 0.0}, 'prune_gaussians': True, 'pruning_dict': {'start_after': 0, 'remove_big_after': 0, 'stop_after': 20, 'prune_every': 20, 'removal_opacity_threshold': 0.005, 'final_removal_opacity_threshold': 0.005, 'reset_opacities': False, 'reset_opacities_every': 500}, 'use_gaussian_splatting_densification': False, 'densify_dict': {'start_after': 500, 'remove_big_after': 3000, 'stop_after': 5000, 'densify_every': 100, 'grad_thresh': 0.0002, 'num_to_split_into': 2, 'removal_opacity_threshold': 0.005, 'final_removal_opacity_threshold': 0.005, 'reset_opacities_every': 3000}}, 'viz': {'render_mode': 'color', 'offset_first_viz_cam': True, 'show_sil': False, 'visualize_cams': True, 'viz_w': 600, 'viz_h': 340, 'viz_near': 0.01, 'viz_far': 100.0, 'view_scale': 2, 'viz_fps': 5, 'enter_interactive_post_online': False}}
Loading Dataset ...
0%| | 0/10 [00:00<?, ?it/sSaving parameters to: ././experiments/iPhone_Captures/offline_demo/SplaTAM_iPhonet/s]
Failed to evaluate trajectory.

Selected Keyframes at Frame 0: [0]
Tracking Result Time Step: 0: 0%| | 0/1 [00:00<?, ?it/s]
0%| | 0/10 [00:00<?, ?it/s]
Traceback (most recent call last): | 0/60 [00:00<?, ?it/s]
File "/zhome/75/8/168142/Desktop/Bachelorprojekt/SplaTAM/scripts/splatam.py", line 1007, in
rgbd_slam(experiment.config)
File "/zhome/75/8/168142/Desktop/Bachelorprojekt/SplaTAM/scripts/splatam.py", line 840, in rgbd_slam
loss, variables, losses = get_loss(params, iter_data, variables, iter_time_idx, config['mapping']['loss_weights'],
File "/zhome/75/8/168142/Desktop/Bachelorprojekt/SplaTAM/scripts/splatam.py", line 253, in get_loss
im, radius, _, = Renderer(raster_settings=curr_data['cam'])(**rendervar)
File "/zhome/75/8/168142/miniconda3/envs/splatam/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/zhome/75/8/168142/miniconda3/envs/splatam/lib/python3.10/site-packages/diff_gaussian_rasterization/init.py", line 186, in forward
return rasterize_gaussians(
File "/zhome/75/8/168142/miniconda3/envs/splatam/lib/python3.10/site-packages/diff_gaussian_rasterization/init.py", line 28, in rasterize_gaussians
return _RasterizeGaussians.apply(
File "/zhome/75/8/168142/miniconda3/envs/splatam/lib/python3.10/site-packages/diff_gaussian_rasterization/init.py", line 79, in forward
num_rendered, color, radii, geomBuffer, binningBuffer, imgBuffer, depth = _C.rasterize_gaussians(*args)
RuntimeError: CUDA out of memory. Tried to allocate 131065.38 GiB (GPU 0; 39.39 GiB total capacity; 79.66 MiB already allocated; 38.00 GiB free; 96.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Mapping Time Step: 0: 0%| | 0/60 [00:00<?, ?it/s]

As I understand from the runtime error, the program is trying to allocate 131065 gigabyte of memory on the GPU. This seems a bit crazy, since no GPU's have that kind of memory? Has anyone seen this error before or maybe someone knows how to reduce the amount of memory being allocated?

[Nik-V9]

Hi, this seems to be a blow-up of Gaussians due to tracking failure.

The offline version of NeRFCapture is broken and has incorrect depth. I would recommend using this new app: #59 (comment)