Problem setting up data

[DeltaMarine101]

Hi, I was hoping you could help me understand what data goes where in the /data folder. After running sdf-gen, there are 6 output folders, --df_lowres_dir, --df_highres_dir, --df_if_dir, --chunk_lowres_dir, --chunk_highres_dir, and --chunk_if_dir. I think I've gathered that I should use --chunk_lowres_dir in the /sdf_008 folder, and --chunk_highres_dirin the /sdf_064 folder. I have a script for sampling a pointcloud from these using your function 'misc.sample_scene_point_clouds', and since the folder is called /pc_20K I assume I should sample 20k points?

After all this though trying to train for retrieval using trainer/train_retrieval.py gives error

No such file or directory: 'data\sdf_064\3DFront\00004f89-9aa5-43c2-ae3c-129586be8aaa__MasterBedroom-5863__64__000_000_000.npz'

Where do these .npz files come from, since sdf-gen returns .npy files? I did get .npz when sampling the pointcloud for /pc_20K, but I'm really not sure where they come from for the sdf's.

Any help in this matter would be greatly appreciated!

[nihalsid]

Hi, its very probably that at some point I switched from using npys to npzs because of space issues and didnt document / update the code (can't exactly remember since it was some time back). Can you try by replacing npy -> npz? If it helps I can try to dig up my filesystem and see if I still have processed data from 3DFront.

[DeltaMarine101]

Thank you for the fast reply! I'll try that out and reply here if it works.

[DeltaMarine101]

That fixed the issue, thank you. However when running the retrieval training, it uses over 100GB of memory and then runs out, how much RAM did you use to train it?

[nihalsid]

Hi, on the server where I was training this, I had access to large amounts of ram (>240G). There's a flag that prevents loading into the memory (preload_scenes and preload_retrievals). Can you try turning that off if its not too slow?

[DeltaMarine101]

That seems to have fixed the memory problem, however, now several more problems have appeared. Would it be possible for you to share trained checkpoints for 3DFront reconstruction? I've been struggling to get this training working for nearly 2 weeks already, but your work looks fantastic and I'd love to try it out!

[nihalsid]

Hi, I dont have a checkpoint anymore. Can you elaborate on the issues you're facing now?