Cannot run the impala-based policy

[tunglm2203]

Hi, thanks for sharing your code.

I am trying to run with IMPALA-based policy in configuration-file mode, with configs/procgen/impala_procgen.json

The command that I used: python main.py --config-file configs/procgen/impala_procgen.json

However, I got all mean_episdoe_return equal to nan as follows (the outputs for setup are ignored):

Steps 0 @ 0.0 SPS. Mean return nan. Stats:
{'mean_episode_return': nan}
[INFO:4981 monobeast:782 2021-11-05 16:09:24,182] Steps 0 @ 0.0 SPS. Mean return nan. Stats:
{'mean_episode_return': nan}
Steps 0 @ 0.0 SPS. Mean return nan. Stats:
{'mean_episode_return': nan}
[INFO:4981 monobeast:782 2021-11-05 16:09:29,189] Steps 0 @ 0.0 SPS. Mean return nan. Stats:
{'mean_episode_return': nan}
Steps 0 @ 0.0 SPS. Mean return nan. Stats:
{'mean_episode_return': nan}
[INFO:4981 monobeast:782 2021-11-05 16:09:34,195] Steps 0 @ 0.0 SPS. Mean return nan. Stats:
{'mean_episode_return': nan}
Steps 0 @ 0.0 SPS. Mean return nan. Stats:
{'mean_episode_return': nan}
[INFO:4981 monobeast:782 2021-11-05 16:09:39,202] Steps 0 @ 0.0 SPS. Mean return nan. Stats:
{'mean_episode_return': nan}

I found that while interacting with the environment, the act() function is stopped at this point https://github.com/AGI-Labs/continual_rl/blob/bcf17d879e8a983340be233ff8f740c424d0f303/continual_rl/policies/impala/torchbeast/monobeast.py#L226. I have tried to check in forward() in the class ImpalaNet() and saw that it stopped at x = self._conv_net(x) https://github.com/AGI-Labs/continual_rl/blob/bcf17d879e8a983340be233ff8f740c424d0f303/continual_rl/policies/impala/nets.py#L55. I did some modifications by replacing it x = torch.rand(x.shape[0], 512), and the code can run successfully.

I guess the problem may be caused by the shared memory of the actor's model? Can you give some hints to fix that?

I have set up for 1 actor and 1 learner
Here is some OS information:

• OS: Ubuntu 20.04
• Python: 3.7.11 with conda
• GPU: RTX 3080
• RAM: 128 GB
• torch: 1.1.1+cu11.0
• Docker: No
• Branch used: develop

Update:
In ConvNet84x84 class, I manually set the intermediate_dim=1024 (for Procgen) and comment out this line https://github.com/AGI-Labs/continual_rl/blob/bcf17d879e8a983340be233ff8f740c424d0f303/continual_rl/utils/common_nets.py#L62, so code is runnable now. Maybe when forwarding the model with dummy input before calling init() causes problem?

[SamNPowers]

If you don't mind, could you try run:

OMP_NUM_THREADS=1 python main.py --config-file configs/procgen/impala_procgen.json

I think OMP_NUM_THREADS needs to be set before the first time one of our libraries (numpy, if memory serves) is used. We do try set it in IMPALA, but have found that setting it as the env variable in the command is more reliable. It could be that by doing the dummy_env we are using numpy too early, and OMP_NUM_THREADS isn't getting picked up properly.

By the way, thanks for debugging in and finding a fix that works.

[tunglm2203]

Oh, it worked now, I forgot to set OMP_NUM_THREADS=1. Thank you :)