ModuleNotFoundError: No module named 'petrel_client'

[EliomEssaim]

File "/data/workspace/user/projects/OpenPilot/Openpilot-Deepdive/data.py", line 49, in _init_mc_
    from petrel_client.client import Client
ModuleNotFoundError: No module named 'petrel_client'

Hi,

I followed the steps in the readme. And my machine has executed "pip install -r requirements.txt " successfully.
When I start training, this error appears.
After searching for solutions, I installed petrel. But it seems to contribute nothing to this error.

I'm not familiar with this module which cannot be found in PyPI.

Could you give some advice to solve it? Thanks.

[ElectronicElephant]

Hi,

Sorry for the confusion. Petrel is an internal sdk to help speed up data loading. You can change use_memcache=True to use_memcache=False to avoid this problem.

https://github.com/OpenPerceptionX/Openpilot-Deepdive/blob/7ac6eedff5d8902fd4506fe9436836f3f14d37d6/main.py#L59

    train = Comma2k19SequenceDataset('data/comma2k19_train_non_overlap.txt', 's3://comma2k19/','train', use_memcache=True)
    val = Comma2k19SequenceDataset('data/comma2k19_val_non_overlap.txt', 's3://comma2k19/','demo', use_memcache=True)

[EliomEssaim]

@ElectronicElephant Thanks for your reply. I has set use_memcache=False.My programs no longer report this error.
But my programs meet with another problem:

File "/data/workspace/user/projects/OpenPilot/Openpilot-Deepdive/data.py", line 285, in __getitem__
    future_poses = torch.tensor(future_poses, dtype=torch.float32)
UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. 

It's a warning, but my programs do be slow. After the report message above, it stuck in here for about 30 minutes. Then I close my programs. Following UserWarning's advice:

# before
future_poses = torch.tensor(future_poses, dtype=torch.float32)
# after
future_poses = torch.tensor(np.array(future_poses), dtype=torch.float32)

After restarting my programs, it no longer reports UserWarning. But it is still very slow, stuck in "Loaded pre-trained weights for efficient net-b2" for about 30 minutes until now.

Is it ‘use_memcache=False’ result in programs executing with a very low efficiency? Or it's a normal situation, I just need to wait for another 30 minutes or longer?

All outputs in my programs:

Warning: DEMO mode is on.Warning: DEMO mode is on.
Warning: DEMO mode is on.
Warning: DEMO mode is on.

Loaded pretrained weights for efficientnet-b2
Loaded pretrained weights for efficientnet-b2
Loaded pretrained weights for efficientnet-b2
Loaded pretrained weights for efficientnet-b2

my configuration:

batch_size:48 world_size = 4
CUDA Version: 11.4

[ElectronicElephant]

Hi @EliomEssaim

Sorry for the confusion. We implemented and tested our code on the lab's slurm cluster, so it may be problematic when running on bare metal machines. We are looking into this issue and may fix the problems in a few days. Please take a coffee and have a break -:)

[ElectronicElephant]

Hi @EliomEssaim ,

Sorry for the delay. I have updated the code for bare-metal machines. Please pull the latest code and follow the instructions https://github.com/OpenPerceptionX/Openpilot-Deepdive#training-and-testing .

Also, please kindly note that the batch_size is actually the batch size per GPU, so you may want 6 instead of 48.

[EliomEssaim]

Oh, it works. Thanks. I will close this issue.