Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question] TD3_DEFAULT_CONFIG #55

Closed
sjYoondeltar opened this issue Jan 28, 2023 · 3 comments
Closed

[Question] TD3_DEFAULT_CONFIG #55

sjYoondeltar opened this issue Jan 28, 2023 · 3 comments

Comments

@sjYoondeltar
Copy link

sjYoondeltar commented Jan 28, 2023

Hi @Toni-SM

While utilizing TD3 agent on the pendulum-v1 env with the default config, I checked that "smooth_regularization_noise" is None in default config.

however, that cfg make this below error.

skrl/agents/torch/td3/td3.py in _update(self, timestep, timesteps)
393 # target policy smoothing
394 next_actions, _, _ = self.target_policy.act({"states": sampled_next_states, **rnn_policy}, role="target_policy")
--> 395 noises = torch.clamp(self._smooth_regularization_noise.sample(next_actions.shape),
396 min=-self._smooth_regularization_clip,
397 max=self._smooth_regularization_clip)

AttributeError: 'NoneType' object has no attribute 'sample'

I add the line cfg_td3["smooth_regularization_noise"] = GaussianNoise(mean=0, std=1) to avoid that error and it works.

Could you check this error and kindly let me know what would be a suitable default value for smooth_regularization_noise in TD3?

@sjYoondeltar sjYoondeltar changed the title TD3_DEFAULT_CONFIG [Question] TD3_DEFAULT_CONFIG Jan 28, 2023
@Toni-SM
Copy link
Owner

Toni-SM commented Jan 28, 2023

Hi @sjYoondeltar

I assume the regularization noise is always defined by the user for training.
I will fix the implementation to ignore its use when it's None (and show a warning message in this case)

A typical configuration can be found in the following example (td3_gym_pendulum.py):

cfg_td3["smooth_regularization_noise"] = GaussianNoise(0, 0.2, device=device)
cfg_td3["smooth_regularization_clip"] = 0.5

In this example, the parameters are the same as those used by the TD3 authors to evaluate the algorithm in the paper

Screenshot from 2023-01-28 17-21-59

@sjYoondeltar
Copy link
Author

Thanks for your answering @Toni-SM

@Toni-SM
Copy link
Owner

Toni-SM commented Feb 6, 2023

Hi @sjYoondeltar

Many thanks for reporting this issue.

A warning message has been added to the development version (develop branch) when the regularization noise is not specified and its availability is checked before applying it.

if self._smooth_regularization_noise is None:
logger.warning("agents:TD3: No smooth regularization noise specified to reduce variance during training")

if self._smooth_regularization_noise is not None:
noises = torch.clamp(self._smooth_regularization_noise.sample(next_actions.shape),
min=-self._smooth_regularization_clip,
max=self._smooth_regularization_clip)
next_actions.add_(noises)

@Toni-SM Toni-SM closed this as completed Feb 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants