Examples/minimal rl py #1205
Examples/minimal rl py #1205
Conversation
|
Hello @Scitator! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:
Comment last updated at 2021-05-10 06:14:18 UTC |
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit
pep8
examples/reinforcement_learning/dqn.py|45 col 1| D101 Missing docstring in public class
examples/reinforcement_learning/dqn.py|46 col 64| B008 Do not perform function calls in argument defaults. The call is performed only once at function definition time. All calls to your function will reuse the result of that definition-time function call. If this is intended, assign the function call to a module-level variable and use that variable as a default value.
examples/reinforcement_learning/dqn.py|50 col 1| D105 Missing docstring in magic method
examples/reinforcement_learning/dqn.py|55 col 1| D105 Missing docstring in magic method
examples/reinforcement_learning/dqn.py|68 col 1| D103 Missing docstring in public function
examples/reinforcement_learning/dqn.py|79 col 1| D103 Missing docstring in public function
examples/reinforcement_learning/dqn.py|89 col 9| B007 Loop control variable 't' not used within the loop body. If this is intended, start the name with an underscore.
examples/reinforcement_learning/dqn.py|105 col 1| D103 Missing docstring in public function
examples/reinforcement_learning/dqn.py|114 col 9| B007 Loop control variable 'i_episone' not used within the loop body. If this is intended, start the name with an underscore.
examples/reinforcement_learning/dqn.py|123 col 1| D103 Missing docstring in public function
examples/reinforcement_learning/dqn.py|144 col 1| D101 Missing docstring in public class
examples/reinforcement_learning/dqn.py|164 col 1| D102 Missing docstring in public method
examples/reinforcement_learning/dqn.py|185 col 1| D102 Missing docstring in public method
examples/reinforcement_learning/dqn.py|204 col 1| D102 Missing docstring in public method
examples/reinforcement_learning/dqn.py|222 col 1| D101 Missing docstring in public class
examples/reinforcement_learning/dqn.py|243 col 1| D102 Missing docstring in public method
examples/reinforcement_learning/dqn.py|251 col 1| D102 Missing docstring in public method
examples/reinforcement_learning/reinforce.py|16 col 76| E231 missing whitespace after ','
examples/reinforcement_learning/reinforce.py|16 col 77| C819 trailing comma prohibited
examples/reinforcement_learning/reinforce.py|19 col 1| D101 Missing docstring in public class
examples/reinforcement_learning/reinforce.py|24 col 1| D102 Missing docstring in public method
examples/reinforcement_learning/reinforce.py|27 col 1| D102 Missing docstring in public method
examples/reinforcement_learning/reinforce.py|36 col 1| D105 Missing docstring in magic method
examples/reinforcement_learning/reinforce.py|42 col 1| D101 Missing docstring in public class
examples/reinforcement_learning/reinforce.py|46 col 1| D105 Missing docstring in magic method
examples/reinforcement_learning/reinforce.py|52 col 1| D105 Missing docstring in magic method
examples/reinforcement_learning/reinforce.py|59 col 1| D103 Missing docstring in public function
examples/reinforcement_learning/reinforce.py|60 col 6| N806 variable 'G' in function should be lowercase
examples/reinforcement_learning/reinforce.py|67 col 1| D210 No whitespaces allowed surrounding docstring text
examples/reinforcement_learning/reinforce.py|75 col 1| D103 Missing docstring in public function
examples/reinforcement_learning/reinforce.py|83 col 1| D103 Missing docstring in public function
examples/reinforcement_learning/reinforce.py|94 col 9| B007 Loop control variable 't' not used within the loop body. If this is intended, start the name with an underscore.
examples/reinforcement_learning/reinforce.py|113 col 1| D103 Missing docstring in public function
examples/reinforcement_learning/reinforce.py|122 col 9| B007 Loop control variable 'i_episone' not used within the loop body. If this is intended, start the name with an underscore.
examples/reinforcement_learning/reinforce.py|131 col 1| D103 Missing docstring in public function
examples/reinforcement_learning/reinforce.py|152 col 1| D101 Missing docstring in public class
examples/reinforcement_learning/reinforce.py|158 col 1| D102 Missing docstring in public method
examples/reinforcement_learning/reinforce.py|167 col 1| D102 Missing docstring in public method
examples/reinforcement_learning/reinforce.py|180 col 1| D101 Missing docstring in public class
examples/reinforcement_learning/reinforce.py|188 col 1| D102 Missing docstring in public method
examples/reinforcement_learning/reinforce.py|204 col 10| N806 variable 'J_hat' in function should be lowercase
examples/reinforcement_learning/reinforce.py|232 col 88| C819 trailing comma prohibited
examples/reinforcement_learning/reinforce.py|247 col 72| C819 trailing comma prohibited
| ) | ||
|
|
||
|
|
||
| class ReplayBuffer: |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D101 Missing docstring in public class
| def __init__(self, capacity: int): | ||
| self.buffer = deque(maxlen=capacity) | ||
|
|
||
| def append(self, transition: Transition): |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D102 Missing docstring in public method
| def append(self, transition: Transition): | ||
| self.buffer.append(transition) | ||
|
|
||
| def sample(self, size: int) -> Sequence[np.array]: |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D102 Missing docstring in public method
| ) | ||
| return states, actions, rewards, dones, next_states | ||
|
|
||
| def __len__(self) -> int: |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D105 Missing docstring in magic method
|
|
||
| # as far as RL does not have some predefined dataset, | ||
| # we need to specify epoch length by ourselfs | ||
| class ReplayDataset(IterableDataset): |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D101 Missing docstring in public class
| ), | ||
| } | ||
|
|
||
| runner = CustomRunner(gamma=gamma, tau=tau, tau_period=tau_period,) |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
C819 trailing comma prohibited
| ) | ||
|
|
||
|
|
||
| class ReplayBuffer: |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D101 Missing docstring in public class
| def __init__(self, capacity: int): | ||
| self.buffer = deque(maxlen=capacity) | ||
|
|
||
| def append(self, transition: Transition): |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D102 Missing docstring in public method
| def append(self, transition: Transition): | ||
| self.buffer.append(transition) | ||
|
|
||
| def sample(self, size: int) -> Sequence[np.array]: |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D102 Missing docstring in public method
| ) | ||
| return states, actions, rewards, dones, next_states | ||
|
|
||
| def __len__(self) -> int: |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D105 Missing docstring in magic method
There was a problem hiding this comment.
Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit
pep8
examples/reinforcement_learning/reinforce.py|83 col 1| D103 Missing docstring in public function
examples/reinforcement_learning/reinforce.py|94 col 9| B007 Loop control variable 't' not used within the loop body. If this is intended, start the name with an underscore.
examples/reinforcement_learning/reinforce.py|113 col 1| D103 Missing docstring in public function
examples/reinforcement_learning/reinforce.py|122 col 9| B007 Loop control variable 'i_episone' not used within the loop body. If this is intended, start the name with an underscore.
examples/reinforcement_learning/reinforce.py|131 col 1| D103 Missing docstring in public function
examples/reinforcement_learning/reinforce.py|152 col 1| D101 Missing docstring in public class
examples/reinforcement_learning/reinforce.py|158 col 1| D102 Missing docstring in public method
examples/reinforcement_learning/reinforce.py|167 col 1| D102 Missing docstring in public method
examples/reinforcement_learning/reinforce.py|180 col 1| D101 Missing docstring in public class
examples/reinforcement_learning/reinforce.py|188 col 1| D102 Missing docstring in public method
examples/reinforcement_learning/reinforce.py|204 col 10| N806 variable 'J_hat' in function should be lowercase
examples/reinforcement_learning/reinforce.py|232 col 88| C819 trailing comma prohibited
examples/reinforcement_learning/reinforce.py|247 col 72| C819 trailing comma prohibited
|
|
||
| # as far as RL does not have some predefined dataset, | ||
| # we need to specify epoch length by ourselfs | ||
| class ReplayDataset(IterableDataset): |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D101 Missing docstring in public class
| # as far as RL does not have some predefined dataset, | ||
| # we need to specify epoch length by ourselfs | ||
| class ReplayDataset(IterableDataset): | ||
| def __init__(self, buffer: ReplayBuffer, epoch_size: int = int(1e3)): |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
B008 Do not perform function calls in argument defaults. The call is performed only once at function definition time. All calls to your function will reuse the result of that definition-time function call. If this is intended, assign the function call to a module-level variable and use that variable as a default value.
| self.buffer = buffer | ||
| self.epoch_size = epoch_size | ||
|
|
||
| def __iter__(self) -> Iterator[Sequence[np.array]]: |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D105 Missing docstring in magic method
| for i in range(len(dones)): | ||
| yield states[i], actions[i], rewards[i], dones[i], next_states[i] | ||
|
|
||
| def __len__(self) -> int: |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D105 Missing docstring in magic method
| # DQN | ||
|
|
||
|
|
||
| def get_action(env, network: nn.Module, state: np.array, epsilon: float = -1) -> int: |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D103 Missing docstring in public function
| yield states, actions, rewards | ||
| self.buffer.buffer.clear() | ||
|
|
||
| def __len__(self) -> int: |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D105 Missing docstring in magic method
| # REINFORCE | ||
|
|
||
|
|
||
| def get_cumulative_rewards(rewards, gamma=0.99): |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D103 Missing docstring in public function
|
|
||
|
|
||
| def get_cumulative_rewards(rewards, gamma=0.99): | ||
| G = [rewards[-1]] |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
N806 variable 'G' in function should be lowercase
|
|
||
|
|
||
| def to_one_hot(y, n_dims=None): | ||
| """ Takes an integer vector and converts it to 1-hot matrix. """ |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D210 No whitespaces allowed surrounding docstring text
| return y_one_hot | ||
|
|
||
|
|
||
| def get_action(env, network: nn.Module, state: np.array, epsilon: float = -1) -> int: |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D103 Missing docstring in public function
| return int(action) | ||
|
|
||
|
|
||
| def generate_session( |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D103 Missing docstring in public function
| states, actions, rewards = [], [], [] | ||
| state = env.reset() | ||
|
|
||
| for t in range(t_max): |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
B007 Loop control variable 't' not used within the loop body. If this is intended, start the name with an underscore.
| return total_reward, t | ||
|
|
||
|
|
||
| def generate_sessions( |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D103 Missing docstring in public function
| num_sessions: int = 100, | ||
| ) -> tp.Tuple[float, int]: | ||
| sessions_reward, sessions_steps = 0, 0 | ||
| for i_episone in range(num_sessions): |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
B007 Loop control variable 'i_episone' not used within the loop body. If this is intended, start the name with an underscore.
| return sessions_reward, sessions_steps | ||
|
|
||
|
|
||
| def get_network(env, num_hidden: int = 128): |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D103 Missing docstring in public function
| runner.epoch_metrics["_epoch_"]["v_reward"] = valid_rewards | ||
|
|
||
|
|
||
| class CustomRunner(dl.Runner): |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D101 Missing docstring in public class
| self.gamma: float = gamma | ||
| self._initialized = False | ||
|
|
||
| def handle_batch(self, batch: tp.Sequence[np.array]): |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
D102 Missing docstring in public method
| n_actions = probas.shape[1] | ||
| logprobas_for_actions = torch.sum(logprobas * to_one_hot(actions, n_dims=n_actions), dim=1) | ||
|
|
||
| J_hat = torch.mean(logprobas_for_actions * cumulative_returns) |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
N806 variable 'J_hat' in function should be lowercase
| model = get_network(env) | ||
| optimizer = torch.optim.Adam(model.parameters(), lr=lr) | ||
| loaders = { | ||
| "train_game": DataLoader(RolloutDataset(rollout_buffer), batch_size=batch_size,), |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
C819 trailing comma prohibited
| valid_metric="v_reward", | ||
| minimize_valid_metric=False, | ||
| load_best_on_end=True, | ||
| callbacks=[GameCallback(env=env, rollout_buffer=rollout_buffer,)], |
There was a problem hiding this comment.
[pep8] reported by reviewdog 🐶
C819 trailing comma prohibited
| def get_action( | ||
| env, network: nn.Module, state: np.array, sigma: Optional[float] = None | ||
| ) -> np.array: | ||
| state = torch.tensor(state, dtype=torch.float32).unsqueeze(0) |
There was a problem hiding this comment.
Could you please add info about the expected shapes of the inputs?
| class GameCallback(dl.Callback): | ||
| def __init__( | ||
| self, | ||
| *, |
There was a problem hiding this comment.
What are the main reasons to force users to use key-value?
Before submitting (checklist)
catalyst-make-codestyle && catalyst-check-codestyle(pip install -U catalyst-codestyle).make check-docs?pytest .?latestandminimalrequirements?Description
Related Issue
Type of Change
PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
FAQ
Please review the FAQ before submitting an issue: