Episode truncation & early stopping #581
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jinyu-W
reviewed
Feb 17, 2023
| @@ -267,6 +274,8 @@ def __init__( | |||
| self._transition_cache: List[CacheElement] = [] | |||
| self._agent_last_index: Dict[Any, int] = {} # Index of last occurrence of agent in self._transition_cache | |||
| self._reward_eval_delay = reward_eval_delay | |||
| self._max_episode_length = max_episode_length | |||
There was a problem hiding this comment.
feels like duplicated with the "max_tick", but the "truncated" info is different from "is_done". Strange
There was a problem hiding this comment.
max_tick is the concept of Env while max_episode_length is the concept of EnvSampler. A more readable name is more than welcome.
Jinyu-W
added a commit
that referenced
this pull request
Feb 17, 2023
* PPO, SAC, DDPG passed * Explore in SAC * Test GYM on server * Sync server changes * pre-commit * Ready to try on server * . * . * . * . * . * Performance OK * Move to tests * Remove old versions * PPO done * Start to test AC * Start to test SAC * SAC test passed * Multiple round in evaluation * Modify config.yml * Add Callbacks * [wip] SAC performance not good * [wip] still not good * update for some PR comments; Add a MARKDOWN file (#576) Co-authored-by: Jinyu Wang <wang.jinyu@microsoft.com> * Use FullyConnected to replace mlp * Update action bound * ??? * Change gym env wrapper metrics logci * Change gym env wrapper metrics logci * refine env_sampler.sample under step mode * Add DDPG. Performance not good... * Add DDPG. Performance not good... * wip * Sounds like sac works * Refactor file structure * Refactor file structure * Refactor file structure * Pre-commit * Pre commit * Minor refinement of CIM RL * Jinyu/rl workflow refine (#578) * remove useless files; add device mapping; update pdoc * add default checkpoint path; fix distributed worker log path issue; update example log path * update performance doc * remove tests/rl/algorithms folder * Resolve PR comments * Compare PPO with spinning up (#579) * [wip] compare PPO * PPO matching * Revert unnecessary changes * Minor * Minor * SAC Test parameters update (#580) * fix sac to_device issue; update sac gym test parameters * add rl test performance plot func * update sac eval interval config * update sac checkpoint interval config * fix callback issue * update plot func * update plot func * update plot func * update performance doc; upload performance images * Minor fix in callbacks; refine plot.py format. * Add n_interactions. Use n_interactions to plot curves. * pre-commit --------- Co-authored-by: Huoran Li <huo53926@126.com> Co-authored-by: Huoran Li <huoranli@microsoft.com> * Episode truncation & early stopping (#581) * Add truncated logic * (To be tested) early stop * Early stop test passed * Test passed * Random action. To be tested. * Warmup OK * Pre-commit * random seed * Revert pre-commit config --------- Co-authored-by: Jinyu-W <53509467+Jinyu-W@users.noreply.github.com> Co-authored-by: Jinyu Wang <wang.jinyu@microsoft.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
EnvSampler. This is not identical todurationinEnv.durationis an internal attribute ofEnv. WhenEnvhas executed all the steps in the duration, it "terminates".EnvSamplerand does not have to be perceived byEnv. When a single episode reaches maximum episode length, it is "truncated".Linked issue(s)/Pull request(s)
Type of Change
Related Component
Has Been Tested
Needs Follow Up Actions
Checklist