Online & Offline Demo Debugging

[zhaosiyuan1098]

Hi,the program is running well when I try to run the online demo and dataset collection.
However,when I try offine mode,I got some problems.

The detail steps I try is as follows:

1. open the nerfcapture app on my iphone which has lidar
2. switch to offine mode
3. touch the "start" button
4. keep touching "save frame"buttton for nearly 10 times
5. touch the "end"button
   6.run bash bash_scripts/nerfcapture.bash configs/iphone/nerfcapture.py

then I got the mistake:

save_path experiments/iPhone_Captures/offline_demo already exists
System Paths:
/home/syzhao15/Desktop/CODE/SplaTAM
/home/syzhao15/Desktop/CODE/SplaTAM/scripts
/opt/ros/humble/lib/python3.10/site-packages
/opt/ros/humble/local/lib/python3.10/dist-packages
/home/syzhao15/miniforge3/envs/splatam/lib/python310.zip
/home/syzhao15/miniforge3/envs/splatam/lib/python3.10
/home/syzhao15/miniforge3/envs/splatam/lib/python3.10/lib-dynload
/home/syzhao15/miniforge3/envs/splatam/lib/python3.10/site-packages
Seed set to: 0 (type: <class 'int'>)
Loaded Config:
{'workdir': '././experiments/iPhone_Captures/offline_demo', 'run_name': 'SplaTAM_iPhone', 'overwrite': False, 'depth_scale': 10.0, 'num_frames': 10, 'seed': 0, 'primary_device': 'cuda:0', 'map_every': 1, 'keyframe_every': 2, 'mapping_window_size': 32, 'report_global_progress_every': 100, 'eval_every': 1, 'scene_radius_depth_ratio': 3, 'mean_sq_dist_method': 'projective', 'report_iter_progress': False, 'load_checkpoint': False, 'checkpoint_time_idx': 130, 'save_checkpoints': False, 'checkpoint_interval': 5, 'use_wandb': False, 'data': {'dataset_name': 'nerfcapture', 'basedir': './experiments/iPhone_Captures', 'sequence': 'offline_demo', 'desired_image_height': 720, 'desired_image_width': 960, 'densification_image_height': 360, 'densification_image_width': 480, 'start': 0, 'end': -1, 'stride': 1, 'num_frames': 10}, 'tracking': {'use_gt_poses': False, 'forward_prop': True, 'visualize_tracking_loss': False, 'num_iters': 60, 'use_sil_for_loss': True, 'sil_thres': 0.99, 'use_l1': True, 'use_depth_loss_thres': True, 'depth_loss_thres': 20000, 'ignore_outlier_depth_loss': False, 'use_uncertainty_for_loss_mask': False, 'use_uncertainty_for_loss': False, 'use_chamfer': False, 'loss_weights': {'im': 0.5, 'depth': 1.0}, 'lrs': {'means3D': 0.0, 'rgb_colors': 0.0, 'unnorm_rotations': 0.0, 'logit_opacities': 0.0, 'log_scales': 0.0, 'cam_unnorm_rots': 0.001, 'cam_trans': 0.004}}, 'mapping': {'num_iters': 60, 'add_new_gaussians': True, 'sil_thres': 0.5, 'use_l1': True, 'ignore_outlier_depth_loss': False, 'use_sil_for_loss': False, 'use_uncertainty_for_loss_mask': False, 'use_uncertainty_for_loss': False, 'use_chamfer': False, 'loss_weights': {'im': 0.5, 'depth': 1.0}, 'lrs': {'means3D': 0.0001, 'rgb_colors': 0.0025, 'unnorm_rotations': 0.001, 'logit_opacities': 0.05, 'log_scales': 0.001, 'cam_unnorm_rots': 0.0, 'cam_trans': 0.0}, 'prune_gaussians': True, 'pruning_dict': {'start_after': 0, 'remove_big_after': 0, 'stop_after': 20, 'prune_every': 20, 'removal_opacity_threshold': 0.005, 'final_removal_opacity_threshold': 0.005, 'reset_opacities': False, 'reset_opacities_every': 500}, 'use_gaussian_splatting_densification': False, 'densify_dict': {'start_after': 500, 'remove_big_after': 3000, 'stop_after': 5000, 'densify_every': 100, 'grad_thresh': 0.0002, 'num_to_split_into': 2, 'removal_opacity_threshold': 0.005, 'final_removal_opacity_threshold': 0.005, 'reset_opacities_every': 3000}}, 'viz': {'render_mode': 'color', 'offset_first_viz_cam': True, 'show_sil': False, 'visualize_cams': True, 'viz_w': 600, 'viz_h': 340, 'viz_near': 0.01, 'viz_far': 100.0, 'view_scale': 2, 'viz_fps': 5, 'enter_interactive_post_online': False}}
Loading Dataset ...
Traceback (most recent call last):
  File "/home/syzhao15/Desktop/CODE/SplaTAM/scripts/splatam.py", line 1007, in <module>
    rgbd_slam(experiment.config)
  File "/home/syzhao15/Desktop/CODE/SplaTAM/scripts/splatam.py", line 514, in rgbd_slam
    dataset = get_dataset(
  File "/home/syzhao15/Desktop/CODE/SplaTAM/scripts/splatam.py", line 73, in get_dataset
    return NeRFCaptureDataset(basedir, sequence, **kwargs)
  File "/home/syzhao15/Desktop/CODE/SplaTAM/datasets/gradslam_datasets/nerfcapture.py", line 39, in __init__
    self.cams_metadata = self.load_cams_metadata()
  File "/home/syzhao15/Desktop/CODE/SplaTAM/datasets/gradslam_datasets/nerfcapture.py", line 72, in load_cams_metadata
    cams_metadata = json.load(open(cams_metadata_path, "r"))
FileNotFoundError: [Errno 2] No such file or directory: './experiments/iPhone_Captures/offline_demo/transforms.json'
Seed set to: 0 (type: <class 'int'>)
Traceback (most recent call last):
  File "/home/syzhao15/Desktop/CODE/SplaTAM/viz_scripts/final_recon.py", line 297, in <module>
    visualize(scene_path, viz_cfg)
  File "/home/syzhao15/Desktop/CODE/SplaTAM/viz_scripts/final_recon.py", line 170, in visualize
    w2c, k = load_camera(cfg, scene_path)
  File "/home/syzhao15/Desktop/CODE/SplaTAM/viz_scripts/final_recon.py", line 26, in load_camera
    all_params = dict(np.load(scene_path, allow_pickle=True))
  File "/home/syzhao15/miniforge3/envs/splatam/lib/python3.10/site-packages/numpy/lib/npyio.py", line 427, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '././experiments/iPhone_Captures/offline_demo/SplaTAM_iPhone/params.npz'

Can anyone tell me how can i fix it? Is there any mistake with my steps metioned above?
Thanks for help!!!

[zhaosiyuan1098]

I solved this problem and finally found out that the problem is caused by:

When I run bash bash_scripts/nerfcapture.bash configs/iphone/nerfcapture.pyfor the first time,the terminal had shown "waiting for frames...",howerver,the connection between my pc and iphone have some problems(maybe the nerfcapture app),so I used Ctrl+Cto simply kill the progress.

After that,an incomplete folder experiments/iPhone_Captures/offline_demo/keeps remaining on my pc，so when bash bash_scripts/nerfcapture.bash configs/iphone/nerfcapture.py is executed next time,the problem come out.

The simplest way to solve it is removing the experiments folder and run the offline mode again.hahah.

By the way,I noticed that the nerfcapture app connection is so fragile:

1. I have to restart it from time to time to keep connection available when running online mode
2. it is unworkable with my iphone's hotspot.

hope these will be fixed in the future～

[Buffyqsf]

Hello, I got a question. I found online or offline bash are "sudo" like. So these demos can't be run on the windows?

[nirmalsnair]

@Buffyqsf SplaTAM can be run on Windows, check this post: #9 (comment)

[xieodu]

I solved this problem and finally found out that the problem is caused by:

When I run bash bash_scripts/nerfcapture.bash configs/iphone/nerfcapture.pyfor the first time,the terminal had shown "waiting for frames...",howerver,the connection between my pc and iphone have some problems(maybe the nerfcapture app),so I used Ctrl+Cto simply kill the progress.

After that,an incomplete folder experiments/iPhone_Captures/offline_demo/keeps remaining on my pc，so when bash bash_scripts/nerfcapture.bash configs/iphone/nerfcapture.py is executed next time,the problem come out.

The simplest way to solve it is removing the experiments folder and run the offline mode again.hahah.

By the way,I noticed that the nerfcapture app connection is so fragile:

1. I have to restart it from time to time to keep connection available when running online mode

2. it is unworkable with my iphone's hotspot.

hope these will be fixed in the future～

Hello，the connection between my pc and iphone have some problems too, the terminal keeps showing "waiting for frames..."without any further progress. I wonder how did you ultimately connect the app to your pc?

[zhaosiyuan1098]

I have also been troubled by the same problem as you for a while.

• When I use my iphone's hotspot:
  no matter what I do,the progress keeps 'Waiting for frames'.issue
• When I use WIFI:
  1. online mode: if it shows 0 connection(s) at the upper right corner,I simply keep the progress run on my pc and kill the the nerfcapture app on my phone ,then restart it.The rate of reconnection successfully rate is nearly 90%.
  2. offline mode: I follow the steps mentioned when I create this issue,after waiting for several seconds,it starts to transfer data.However,the transfer speed is much lower than online mode.

[weiyunpei]

I have also been troubled by the same problem as you for a while.

• When I use my iphone's hotspot:
  no matter what I do,the progress keeps 'Waiting for frames'.issue

• When I use WIFI:

  1. online mode: if it shows 0 connection(s) at the upper right corner,I simply keep the progress run on my pc and kill the the nerfcapture app on my phone ,then restart it.The rate of reconnection successfully rate is nearly 90%.
  2. offline mode: I follow the steps mentioned when I create this issue,after waiting for several seconds,it starts to transfer data.However,the transfer speed is much lower than online mode.

Thx for your help. It did help me a lot!

[Buffyqsf]

@Buffyqsf SplaTAM can be run on Windows, check this post: #9 (comment)

Thx for your help. That helps me a lot.

[Nik-V9]

Hi @zhaosiyuan1098, Thanks for trying out the code and providing support in this thread!

Regarding your folder problem, you can set the overwrite config to True and the terminal will ask if you want to overwrite the existing folder:

SplaTAM/configs/iphone/nerfcapture.py

Line 11 in bbaf5cc

overwrite = False # Rewrite over dataset if it exists

Instead, if you don't want to overwrite the previous demo, you can change the name for the new scan in the config:

SplaTAM/configs/iphone/nerfcapture.py

Line 8 in bbaf5cc

scene_name = "offline_demo" # Scan Name

Linking the other things mentioned in this thread to this issue: #2
Please follow and comment on that issue thread for further updates.