problem about 'Save occupancy data'

[WeiChengTseng]

Hi all,
I ran python scripts/save_occ_data_parallel.py /path/to/raw/data 100000 2 --num-proc 40 to generate occupancy data with the pre-generated data provided by you. I found this line in python scripts/save_occ_data_parallel.py encounter *** AttributeError: 'str' object has no attribute 'copy'. The environmental setting is aligned with the description in README.md

Just wonder whether there is any solution for this issue.
Any help appreciated :)

[Steve-Tod]

Not very sure about that. Cause I don't see copy in that line😂 Can you try to debug by printing the mesh_pose_list_path before the line?

[WeiChengTseng]

I use pdb to print the value of mesh_pose_list_path, and the result is

mesh_pose_list_path='./data/data_packed_train_raw/mesh_pose_list/000000239c3e45008bde52d1a78d5c12.npz'

It seems that the code is correct since it correctly gets .npz file. My random guess is the format of .npz files is not shared by different computation infrastructures. I believe the correct is correct since I can use the scripts provided by the repo. to generate synthetic data and retrieve the occupancy of the generated synthetic data. However, using exactly the same dataset is still essential when comparing ML approaches 😂

Is it possible to obtain your raw data that is not saved in .npz?

[Steve-Tod]

I save the data directly to .npz with np.savez_compressed() as in here. Is it possible that the problem is with the version of numpy? I'm using 1.19.2.

[WeiChengTseng]

I also use numpy==1.19.2 😂.
After some testing, I think that the issue is not caused by the version of numpy. I guess the issue is caused by saving vgn to .npz fille.

I use your data generation script (generate_data_parallel.py) to create synthetic dataset, and here is a normal .npz

array([['data/urdfs/packed/train/Sauerkraut_800_tex_visual.obj',
        0.728894702126209,
        <vgn.utils.transform.Transform object at 0x7f6639ae03d0>],
       ['data/urdfs/packed/train/nutrigrain_apple_cinnamon_visual.obj',
        0.7023577187764145,
        <vgn.utils.transform.Transform object at 0x7f6639ae0430>],
       ['data/urdfs/packed/train/pringles_bbq_visual.obj',
        0.8754874818703685,
        <vgn.utils.transform.Transform object at 0x7f6639ae0460>]],
      dtype=object)

We can find that the .npz files contain vgn object, but the vgn package is installed locally which may also depend on the computation infrastructure. Therefore, the vgn objects saved with np.savez_compressed() are not valid across different devices. I also verify this hypothesis via creating a dataset with one computer and load the dataset with another one, and a similar issue happens.

[Steve-Tod]

I see your point. That's a possible cause. Maybe I should store the 4X4 transformation matrix rather than this Transform object.🤔

[WeiChengTseng]

I think that is a feasible solution. It will be wonderful if you can provide your dataset in that format.

[RolandZhu]

Hi, thanks for open-source the code! I encountered the same problem that .npz in the raw dataset cannot be loaded here.
Any idea how I can resolve this issue without generating the dataset again?

[Steve-Tod]

Sorry, I forgot to translate the data. I'll work on it in a few days.

[Steve-Tod]

Hi guys, I've updated the data and the code. Now we save and load the transformation matrix instead of the Transform object for the mesh poses. And I update the data. Note that only raw data need to be updated.

Please pull the new commit and try the new data and give me feedback, thank you!