Cuda OOM in plots.py during mesh extraction

[athena913]

Hi,

Thank you for sharing the code. I trained the model. When I ran eval.py to extract the mesh using the latest checkpoint I am getting Cuda OOM in the File "../code/utils/plots.py", line 197, in get_surface_high_res_mesh which is the line
grid_points = torch.cat(g, dim=0)
I tried to change the following code snippet in plots.py, but it did not help. My GPU has 11GB memory. I also tried using 2 gpus, but I think the eval code uses only a single gpu. I also tried torch.cuda.empty_cache() to clear the cuda cache but I still get the OOM error. Could you please provide some guidance on how to fix the OOM problem? Thanks.

g = []
for i, pnts in enumerate(torch.split(grid_points, 100000, dim=0)):
    g.append(torch.bmm(vecs.unsqueeze(0).repeat(pnts.shape[0], 1, 1).transpose(1, 2),
                       pnts.unsqueeze(-1)).squeeze() + s_mean)
grid_points = torch.cat(g, dim=0)

[iernstig]

I found a similar problem on another project (Neural Sparse Voxel Fields), they released unused cache. Maybe it could be useful here as well? facebookresearch/NSVF#34

[athena913]

Thanks. As I mentioned in my post above, I have tried to empty the cache, which is what the other post also has mentioned, I have also tried to reduce the 32-bit precision. But these changes did not help me.

However, this problem occurs only for some objects (e.g. SCAN ID 24), where the len(g)>2600 after the for loop. If len(g) < 2600 (e.g. SCAN ID 65), the mesh extraction works well. The paper uses a single Nvidia V100 which has 32GB gpu memory. I dont have such high gpu memory. So I have to look into using distributed memory across multiple gpus to see if that helps resolve the issue.

[lioryariv]

You can simply address this by giving a lower resolution parameter, e.g.: --resolution 256
where the default value is 512. It determines the grid resolution for the marching cube, so it will affect a bit the extracted mesh resolution but it does not suppose to be that transparent, 512 is quite big.

Best
Lior

[athena913]

Yes, that works. Thank you very much.