-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Question] influence of buffer size when using vecenv and save customized replay buffer #1885
Comments
yes stable-baselines3/stable_baselines3/common/buffers.py Lines 196 to 197 in 5623d98
because the shape of the buffer is:
so at the end, the number of transitions stored is the same, you should not need to do anything. About the size of the replay buffer in general:
Not sure how custom you want it to be. |
Many thanks.
My task is about a robot from a random state to another random state, so that it might need as much recent data as possible, and I tested the buffer size here: print(replay_buffer.size())
replay_buffer.add(
self._last_original_obs, # type: ignore[arg-type]
next_obs, # type: ignore[arg-type]
buffer_action,
reward_,
dones,
infos,
) the outcome shows it will be full quickly. So for such a randomized task, should I increase the buffer size significantly? or maybe RL is not so capable for such task? |
❓ Question
My first question is should I resize the default
buffer_size
when using off-policy algorithm and vectorized env?I noticed that the default
buffer_size
of SAC is 1e6, for env_num=1. However, for vecenv, the buffer_size for each env isbuffer_size / num_env
, which means much less replay buffer for each env when I'm usingmake_vec_env
.For SAC, replay buffer contains$(s_t,a_t,s_{t+1},r_t)$ of different policies, with the help of max entropy it can avoid staying in a local minimum, which means searching advantages.
So will less buffer size cause bad behavior? And is there a limit of max buffer_size for each env?
My second question is near to issue 278, which tells me how to add a customized
replay_buffer
to buffer. But now I want to create a customized replay_buffer then saving it. Here is my code:is this way recommended or is there any other way to do this (create a customized replay buffer then saving it) ?
Checklist
The text was updated successfully, but these errors were encountered: