feat(off-policy): support off-policy lag #204

Gaiejj · 2023-04-05T10:39:10Z

Description

feat(off-policy): support off-policy lag

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds core functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (update in the documentation)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

I have read the CONTRIBUTION guide. (required)
My change requires a change to the documentation.
I have updated the tests accordingly. (required for a bug fix or a new feature)
I have updated the documentation accordingly.
I have reformatted the code using make format. (required)
I have checked the code using make lint. (required)
I have ensured make test pass. (required)

…ent#131)

Co-authored-by: borong <borongzh@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Co-authored-by: ruiyang sun <rockmagma02@gmail.com>

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: ruiyang sun <rockmagma02@gmail.com>

Co-authored-by: zmsn-2077 <73586554+zmsn-2077@users.noreply.github.com> Co-authored-by: borong <borongzh@gmail.com> Co-authored-by: friedmainfunction <73703265+friedmainfunction@users.noreply.github.com>

…nment#147) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…thms (PKU-Alignment#148) Co-authored-by: borong <borongzh@gmail.com> Co-authored-by: Gaiejj <524339208@qq.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: zmsn-2077 <jiamg.ji@gmail.com>

Co-authored-by: borong <borongzh@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: ruiyang sun <rockmagma02@gmail.com> Co-authored-by: zmsn-2077 <73586554+zmsn-2077@users.noreply.github.com> Co-authored-by: friedmainfunction <73703265+friedmainfunction@users.noreply.github.com> Co-authored-by: Gaiejj <524339208@qq.com> Co-authored-by: zmsn-2077 <jiamg.ji@gmail.com> Co-authored-by: Ruiyang Sun <rockmagma02@gmail.com> Co-authored-by: Jiayi Zhou <108712610+Gaiejj@users.noreply.github.com> Co-authored-by: 1Asan <99461435+1Asan@users.noreply.github.com> fix(algo): fix no return in algo_wrapper::learn (PKU-Alignment#122) fix(logger, wrapper): support csv file and velocity tasks (PKU-Alignment#131) fix typo. (PKU-Alignment#134) fix(ppo): fix entropy loss (PKU-Alignment#135) fix bugs (PKU-Alignment#136) fix: support new config for exp_grid (PKU-Alignment#142) fix(rollout, exp_grid): fix logdir path conflict (PKU-Alignment#145) fix(on-policy): fix the second order algorithms performance (PKU-Alignment#147)

…lignment#162) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: borong <borongzh@gmail.com>

Co-authored-by: borong <borongzh@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: ruiyang sun <rockmagma02@gmail.com> Co-authored-by: zmsn-2077 <73586554+zmsn-2077@users.noreply.github.com> Co-authored-by: friedmainfunction <73703265+friedmainfunction@users.noreply.github.com> Co-authored-by: Gaiejj <524339208@qq.com> Co-authored-by: zmsn-2077 <jiamg.ji@gmail.com> Co-authored-by: Ruiyang Sun <rockmagma02@gmail.com> Co-authored-by: Jiayi Zhou <108712610+Gaiejj@users.noreply.github.com> Co-authored-by: 1Asan <99461435+1Asan@users.noreply.github.com> fix(algo): fix no return in algo_wrapper::learn (PKU-Alignment#122) fix(logger, wrapper): support csv file and velocity tasks (PKU-Alignment#131) fix typo. (PKU-Alignment#134) fix(ppo): fix entropy loss (PKU-Alignment#135) fix bugs (PKU-Alignment#136) fix: support new config for exp_grid (PKU-Alignment#142) fix(rollout, exp_grid): fix logdir path conflict (PKU-Alignment#145) fix(on-policy): fix the second order algorithms performance (PKU-Alignment#147)

zmsn-2077 · 2023-04-06T09:08:56Z

omnisafe/algorithms/off_policy/ddpg_lag.py

@@ -0,0 +1,74 @@
+# Copyright 2022-2023 OmniSafe Team. All Rights Reserved.


zmsn-2077 · 2023-04-06T09:09:19Z

omnisafe/algorithms/off_policy/sac_lag.py

@@ -0,0 +1,75 @@
+# Copyright 2022-2023 OmniSafe Team. All Rights Reserved.


zmsn-2077 · 2023-04-06T09:09:33Z

omnisafe/algorithms/off_policy/td3_lag.py

@@ -0,0 +1,63 @@
+# Copyright 2022-2023 OmniSafe Team. All Rights Reserved.


zmsn-2077

LGTM.

muchvo · 2023-04-06T11:06:57Z

omnisafe/adapter/offpolicy_adapter.py

+        """Initialize the off-policy adapter.
+
+        Args:
+            env_id: The environment id.


how about considering type of parameters?

muchvo · 2023-04-06T11:10:56Z

omnisafe/adapter/online_adapter.py


+        Args:
+            env_id: The environment id.
+            num_envs: The number of environments.


how about considering type of parameters?

muchvo

LGTM.

rockmagma02 and others added 30 commits March 15, 2023 02:53

refactor: change architecture of omnisafe (PKU-Alignment#121)

4987e80

fix(algo): fix no return in algo_wrapper::learn (PKU-Alignment#122)

90f7549

chore: support num_thread setting (PKU-Alignment#124)

16be6a8

feat: update architecture of config.yaml (PKU-Alignment#126)

51e7f96

fix(logger, wrapper): support csv file and velocity tasks (PKU-Alignm…

2aa1417

…ent#131)

chore: fix typo. (PKU-Alignment#134)

04477eb

fix(ppo): fix entropy loss (PKU-Alignment#135)

fe686bb

test: add more test case, and fix bugs (PKU-Alignment#136)

63dc035

fix: support new config for exp_grid (PKU-Alignment#142)

8b32d73

Co-authored-by: borong <borongzh@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

feat: support policy evaluation (PKU-Alignment#137)

06dcbdb

Co-authored-by: ruiyang sun <rockmagma02@gmail.com>

feat(off-policy): add DDPG, TD3 SAC (PKU-Alignment#128)

c232fcf

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: ruiyang sun <rockmagma02@gmail.com>

fix(rollout, exp_grid): fix logdir path conflict (PKU-Alignment#145)

78ec6bd

Co-authored-by: zmsn-2077 <73586554+zmsn-2077@users.noreply.github.com> Co-authored-by: borong <borongzh@gmail.com> Co-authored-by: friedmainfunction <73703265+friedmainfunction@users.noreply.github.com>

fix(on-policy): fix the second order algorithms performance (PKU-Alig…

817edf9

…nment#147) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

docs: update README.md (PKU-Alignment#149)

cb633d6

docs: update changelog and release v0.1.0

ba48512

feat: support off-policy lagrangian algorithms

3e8de48

chore: update yaml (PKU-Alignment#155)

8463ca5

chore: update github workflow (PKU-Alignment#156)

5068262

feat: support cuda

6c1ed1d

feat: support cuda exp grid

3380d75

chore: merge dev branch

f43f32c

fix(onpolicy_adapter): fix the calculation of last state value (PKU-A…

973b363

…lignment#162) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: borong <borongzh@gmail.com>

Merge branch 'dev' into dev-action

cb1544c

fix: [pre-commit.ci] auto fixes [...]

2a294a4

fix: disable test cuda

065e22d

chore: update yaml (PKU-Alignment#155)

60c901e

chore: update github workflow (PKU-Alignment#156)

d955dd8

Gaiejj and others added 16 commits March 30, 2023 10:26

chore: merge dev

a420d39

fix: fix off-policy lag

07b4aac

feat: perfecting training workflow (PKU-Alignment#185)

14f5493

fix: fix off_policy algorithm

e121a8b

fix: fix cvpo

03103be

refactor: refactor off policy lag

c6b9e2e

chore: merge dev

3d6e7c5

chore: update configs

2b34c94

chore: rebase dev

26fa1c7

chore: clean the code

5bd5e21

chore: clean the code

d05f0ce

chore: update yaml

2a7d7a8

chore: clean the code

b800909

chore: update env version

8c5a8f2

chore: clean the code

c60aedf

chore: merge dev

f31a3e1

Gaiejj marked this pull request as ready for review April 6, 2023 08:43

Gaiejj requested review from rockmagma02 and muchvo April 6, 2023 08:43

fix: fix device error

6e71cee

zmsn-2077 reviewed Apr 6, 2023

View reviewed changes

omnisafe/algorithms/off_policy/ddpg_lag.py

@@ -0,0 +1,74 @@

# Copyright 2022-2023 OmniSafe Team. All Rights Reserved.

Copy link

Member

zmsn-2077 Apr 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2023

zmsn-2077 reviewed Apr 6, 2023

View reviewed changes

omnisafe/algorithms/off_policy/sac_lag.py

@@ -0,0 +1,75 @@

# Copyright 2022-2023 OmniSafe Team. All Rights Reserved.

Copy link

Member

zmsn-2077 Apr 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2022

zmsn-2077 reviewed Apr 6, 2023

View reviewed changes

omnisafe/algorithms/off_policy/td3_lag.py

@@ -0,0 +1,63 @@

# Copyright 2022-2023 OmniSafe Team. All Rights Reserved.

Copy link

Member

zmsn-2077 Apr 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2023

zmsn-2077 approved these changes Apr 6, 2023

View reviewed changes

muchvo reviewed Apr 6, 2023

View reviewed changes

docs(adapters): update comment

746371f

muchvo approved these changes Apr 6, 2023

View reviewed changes

Gaiejj merged commit 52aaf32 into PKU-Alignment:dev Apr 6, 2023
4 checks passed

Gaiejj deleted the dev-off branch April 6, 2023 14:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(off-policy): support off-policy lag #204

feat(off-policy): support off-policy lag #204

Gaiejj commented Apr 5, 2023

zmsn-2077 Apr 6, 2023

zmsn-2077 Apr 6, 2023

zmsn-2077 Apr 6, 2023

zmsn-2077 left a comment

muchvo Apr 6, 2023

muchvo Apr 6, 2023

muchvo left a comment

		@@ -0,0 +1,74 @@
		# Copyright 2022-2023 OmniSafe Team. All Rights Reserved.

		@@ -0,0 +1,75 @@
		# Copyright 2022-2023 OmniSafe Team. All Rights Reserved.

		@@ -0,0 +1,63 @@
		# Copyright 2022-2023 OmniSafe Team. All Rights Reserved.

feat(off-policy): support off-policy lag #204

feat(off-policy): support off-policy lag #204

Conversation

Gaiejj commented Apr 5, 2023

Description

Types of changes

Checklist

zmsn-2077 Apr 6, 2023

Choose a reason for hiding this comment

zmsn-2077 Apr 6, 2023

Choose a reason for hiding this comment

zmsn-2077 Apr 6, 2023

Choose a reason for hiding this comment

zmsn-2077 left a comment

Choose a reason for hiding this comment

muchvo Apr 6, 2023

Choose a reason for hiding this comment

muchvo Apr 6, 2023

Choose a reason for hiding this comment

muchvo left a comment

Choose a reason for hiding this comment