[rllib] example and docs on how to use parametric actions with DQN / PG algorithms #3384

ericl · 2018-11-22T07:24:07Z

What do these changes do?

Add examples of how to work with parametric action spaces (e.g., OpenAI 5 style).

Related issue number

ericl · 2018-11-22T07:24:59Z

python/ray/rllib/models/preprocessors.py

@@ -164,6 +165,8 @@ def _init_shape(self, obs_space, options):
        return (size, )

    def transform(self, observation):
+        if not isinstance(observation, OrderedDict):
+            observation = OrderedDict(sorted(list(observation.items())))


oops this is kind of an important bug fix

do you need to match the rest of the checks? https://github.com/openai/gym/blob/master/gym/spaces/dict_space.py#L34

No, note that that is a check against the space spec, this is sorting the observation dict.

AmplabJenkins · 2018-11-22T09:05:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9538/
Test FAILed.

AmplabJenkins · 2018-11-22T10:27:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9536/
Test FAILed.

AmplabJenkins · 2018-11-22T10:53:14Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9539/
Test FAILed.

AmplabJenkins · 2018-11-22T10:58:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9540/
Test FAILed.

AmplabJenkins · 2018-11-22T11:51:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9541/
Test FAILed.

AmplabJenkins · 2018-11-22T13:22:38Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9544/
Test FAILed.

AmplabJenkins · 2018-11-22T20:41:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9546/
Test FAILed.

AmplabJenkins · 2018-11-22T22:41:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9548/
Test FAILed.

AmplabJenkins · 2018-11-22T22:51:42Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9549/
Test FAILed.

AmplabJenkins · 2018-11-22T23:11:51Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9550/
Test FAILed.

AmplabJenkins · 2018-11-22T23:21:57Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9551/
Test FAILed.

AmplabJenkins · 2018-11-22T23:46:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9552/
Test FAILed.

AmplabJenkins · 2018-11-23T00:46:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9554/
Test FAILed.

AmplabJenkins · 2018-11-23T01:02:00Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9555/
Test FAILed.

AmplabJenkins · 2018-11-23T01:11:02Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9556/
Test FAILed.

zegerhoogeboom · 2018-11-23T13:43:56Z

Very sorry for my wrong previous comment, the issue was simply that I didn't install Ray from your branch. Both DQN and the PPO are working beautifully with the masking. I'm not using the action embeddings, but those are at least working in the example you created.

ericl · 2018-11-23T17:34:51Z

That should be fixed in the latest update -- try pulling.

AmplabJenkins · 2018-11-25T03:31:14Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9576/
Test FAILed.

ericl · 2018-11-27T09:12:24Z

Ping @richardliaw

richardliaw

tf.boolean_mask(x, tf.logical_not(tf.is_inf(x))) could be cleaner (for future ref)

richardliaw · 2018-11-24T08:04:28Z

python/ray/rllib/models/preprocessors.py

@@ -164,6 +165,8 @@ def _init_shape(self, obs_space, options):
        return (size, )

    def transform(self, observation):
+        if not isinstance(observation, OrderedDict):
+            observation = OrderedDict(sorted(list(observation.items())))


do you need to match the rest of the checks? https://github.com/openai/gym/blob/master/gym/spaces/dict_space.py#L34

ericl

The reason is_inf won't work is because inf is numerically unstable, so we use tf.float32.min instead

ericl added 3 commits November 21, 2018 22:27

pa model

95b2aa0

docs

2c3ca54

lint

9633107

ericl assigned richardliaw Nov 22, 2018

ericl commented Nov 22, 2018

View reviewed changes

ericl added 2 commits November 21, 2018 23:47

update

a95f0db

doc

caca39a

ericl force-pushed the pa-model branch from 4d86ce5 to caca39a Compare November 22, 2018 07:49

large

de5cd1b

ericl force-pushed the pa-model branch from e34e117 to de5cd1b Compare November 22, 2018 07:58

Update rllib-training.rst

26dcc8f

ericl added 5 commits November 22, 2018 09:40

Update parametric_action_cartpole.py

fd2058d

doc dqn

0a50f72

doc

3877d6a

Merge branch 'pa-model' of github.com:ericl/ray into pa-model

b1dd888

stop at 50

85d9fb8

ericl changed the title ~~[rllib] example and docs on how to use parametric actions with pg algorithms~~ [rllib] example and docs on how to use parametric actions with DQN / PG algorithms Nov 22, 2018

ericl added 4 commits November 22, 2018 11:49

0.2 gpu

6187de0

masked

b707bd7

improve docs

ee79f73

gym env

90b52e1

ericl added 2 commits November 22, 2018 12:43

custom lstm doc

084e8c7

token

115977c

ericl added 2 commits November 22, 2018 13:42

Merge remote-tracking branch 'upstream/master' into pa-model

1ca2992

support dueling

78c765c

ericl force-pushed the pa-model branch from e83ebde to 78c765c Compare November 22, 2018 22:00

Update rllib-models.rst

06148b4

ericl added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Nov 23, 2018

128

02b886d

richardliaw approved these changes Nov 28, 2018

View reviewed changes

ericl commented Nov 28, 2018

View reviewed changes

ericl merged commit f0df97d into ray-project:master Nov 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[rllib] example and docs on how to use parametric actions with DQN / PG algorithms #3384

[rllib] example and docs on how to use parametric actions with DQN / PG algorithms #3384

ericl commented Nov 22, 2018 •

edited

Loading

ericl Nov 22, 2018

richardliaw Nov 24, 2018

ericl Nov 28, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 23, 2018

AmplabJenkins commented Nov 23, 2018

AmplabJenkins commented Nov 23, 2018

zegerhoogeboom commented Nov 23, 2018 •

edited

Loading

ericl commented Nov 23, 2018

AmplabJenkins commented Nov 25, 2018

ericl commented Nov 27, 2018

richardliaw left a comment

richardliaw Nov 24, 2018

ericl left a comment

[rllib] example and docs on how to use parametric actions with DQN / PG algorithms #3384

[rllib] example and docs on how to use parametric actions with DQN / PG algorithms #3384

Conversation

ericl commented Nov 22, 2018 • edited Loading

What do these changes do?

Related issue number

ericl Nov 22, 2018

Choose a reason for hiding this comment

richardliaw Nov 24, 2018

Choose a reason for hiding this comment

ericl Nov 28, 2018

Choose a reason for hiding this comment

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 22, 2018

AmplabJenkins commented Nov 23, 2018

AmplabJenkins commented Nov 23, 2018

AmplabJenkins commented Nov 23, 2018

zegerhoogeboom commented Nov 23, 2018 • edited Loading

ericl commented Nov 23, 2018

AmplabJenkins commented Nov 25, 2018

ericl commented Nov 27, 2018

richardliaw left a comment

Choose a reason for hiding this comment

richardliaw Nov 24, 2018

Choose a reason for hiding this comment

ericl left a comment

Choose a reason for hiding this comment

ericl commented Nov 22, 2018 •

edited

Loading

zegerhoogeboom commented Nov 23, 2018 •

edited

Loading