rgnn.py --in-memory option error

[drchangliu]

When running "rgnn.py" with the --in-memory option, the script cannot load in "full_feat.npy". The error message is: "OSError: Failed to interpret file "full_feat.npy" as a pickle. This is the original baseline script. Any suggestions? Thanks.

https://i.imgur.com/wcNdGXw.png

[rusty1s]

Mh, that works for me :( Which numpy version are you using? As a possible fix, you can try to set the allow_pickle=False option of np.load.

[weihua916]

Actually, torch.from_numpy(x) should be torch.from_numpy(self.x) This is fixed. Can you please try it again?

[skepsun]

torch.from_numpy(self.x) works, but torch.from_numpy(self.x).share_memory_() fails. Maybe it is caused by lower size (252GB) of shared memory（/dev/shm) on my instance, since the full feature tensor (>252GB) should be moved to shared memory.

[weihua916]

Got it. Without share_memory_(), the data would be duplicated for each worker of the dataloader. Isn't that an issue on your end?

[skepsun]

Increasing /dev/shm up to 400GB does not work. torch.from_numpy(self.x).share_memory_() seems to use more RAM memory. Maybe we can firstly allocate shared memory and assign values.

if self.in_memory:
    # import gc
    # self.x = np.empty((N, self.num_features), dtype=np.float16)
    self.x = torch.empty((N, self.num_features), dtype=torch.float16).share_memory_()
    node_chunk_size = 100000
    for i in tqdm(range(0, N, node_chunk_size)):
        j = min(N, i + node_chunk_size)
        x_inc = torch.from_numpy(x[i:j, :])
        self.x[i:j, :] = x_inc
        # del x_inc
        # gc.collect()

In my instance it works, though in memory training may not be as fast as expected.

Reading dataset...
  0%|                                                    | 0/2442 [00:00<?, ?it/s]rgnn.py:250: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:143.)
  x_inc = torch.from_numpy(x[i:j, :])
100%|█████████████████████████████████████████| 2442/2442 [21:36<00:00,  1.88it/s]
Reading adjacency matrix...
Done! [1912.03s]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]

  | Name  | Type       | Params
-------------------------------------
0 | convs | ModuleList | 9.2 M 
1 | norms | ModuleList | 4.1 K 
2 | skips | ModuleList | 1.8 M 
3 | mlp   | Sequential | 1.2 M 
4 | acc   | Accuracy   | 0     
-------------------------------------
12.3 M    Trainable params
0         Non-trainable params
12.3 M    Total params
49.022    Total estimated model params size (MB)
/home/scx/anaconda3/envs/gnn/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:52: UserWarning: The dataloader, val dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 64 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
Validation sanity check:   0%|                              | 0/2 [00:00<?, ?it/s]/home/scx/anaconda3/envs/gnn/lib/python3.8/site-packages/pytorch_lightning/core/step_result.py:148: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  value = torch.tensor(value, device=device, dtype=torch.float)
Epoch 0:  38%|█▌  | 462/1223 [05:15<08:39,  1.46it/s, loss=1.31, v_num=15, val_acc=0.00147]

[weihua916]

Glad that your solution works. 14 min per epoch is good