Observation outside expected value range when executing /examples/rllib/traffic_light_grid.py

[c4cld]

#Bug Description

ValueError: ('Observation outside expected value range', Box(339,), array([0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0.05 ,
0.05 , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0.37301541, 0.42857143, 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 2. , 3. , 3. ,
0. , 0. , 3. , 3. , 0. ,
3. , 0. , 0. , 1. , 1. ,
0. , 1. , 1. , 1. , 0. ,
1. , 0. , 0. , 1. , 1. ,
0. , 0. , 1. , 0. ]))

Bug Reproduce

/examples/rllib/traffic_light_grid.py

The exact command that you did run that caused the bug:
python traffic_light_grid.py

Output Log

Traceback (most recent call last):
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 498, in _process_trial
result = self.trial_executor.fetch_result(trial)
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 342, in fetch_result
result = ray.get(trial_future[0])
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/worker.py", line 2247, in get
raise value
ray.exceptions.RayTaskError: �[36mray_PPO:train()�[39m (pid=10958, host=611dabai)
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 369, in train
raise e
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 358, in train
result = Trainable.train(self)
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/tune/trainable.py", line 171, in train
result = self._train()
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 126, in _train
fetches = self.optimizer.step()
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/rllib/optimizers/multi_gpu_optimizer.py", line 140, in step
self.num_envs_per_worker, self.train_batch_size)
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/rllib/optimizers/rollout.py", line 29, in collect_samples
next_sample = ray_get_and_free(fut_sample)
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/rllib/utils/memory.py", line 33, in ray_get_and_free
result = ray.get(object_ids)
ray.exceptions.RayTaskError: �[36mray_RolloutWorker:sample()�[39m (pid=11185, host=611dabai)
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/rllib/evaluation/rollout_worker.py", line 453, in sample
batches = [self.input_reader.next()]
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 56, in next
batches = [self.get_data()]
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 97, in get_data
item = next(self.rollout_provider)
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 313, in _env_runner
soft_horizon)
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/rllib/evaluation/sampler.py", line 401, in _process_observations
policy_id).transform(raw_obs)
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/rllib/models/preprocessors.py", line 166, in transform
self.check_shape(observation)
File "/home/dl_e/anaconda3/envs/flow_backup/lib/python3.6/site-packages/ray/rllib/models/preprocessors.py", line 61, in check_shape
self._obs_space, observation)
ValueError: ('Observation outside expected value range', Box(339,), array([0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0.05 ,
0.05 , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0.37301541, 0.42857143, 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 2. , 3. , 3. ,
0. , 0. , 3. , 3. , 0. ,
3. , 0. , 0. , 1. , 1. ,
0. , 1. , 1. , 1. , 0. ,
1. , 0. , 0. , 1. , 1. ,
0. , 0. , 1. , 0. ]))

Versions

• OS: ubuntu 16.04
• Flow version: 0.5.0.dev
• Flow commit number: fatal: there is no submission

[c4cld]

[eugenevinitsky]

Thanks for pointing this out! I will push up a PR tomorrow fixing this.

[c4cld]

@eugenevinitsky Thanks for your help. I'm waiting for your good news.^_^
And I have an idea to extend traffic_light_grid.py. I prepare to add some RL cars(like RL cars in stabilizing_highway.py) in traffic_light_grid, but I dont know whether this idea makes sense or not.
If so, I will try to make true this idea.

[sanakdag]

Having the same error when running /examples/rllib/traffic_light_grid.py
Would like to run a simple experiment with one RL-controlled traffic light
Wondering if there will be any update soon :)
Thanks

[eugenevinitsky]

PR is here #780, you can pull it now or it will be merged into master in the afternoon.

[jordan-schneider]

I'm seeing this same bug in the multi-agent traffic light grid, and PR #780 doesn't fix this.

[eugenevinitsky]

Good catch; this fix doesn't apply to the multi-agent version of this. I'll add a fix for that in a sec. The idea is the same though if you look at the PR; you need to increase the size of the bounds on the observation space.

[jordan-schneider]

Thanks!

[eugenevinitsky]

Just pushed; pull it and see if it fixes your problem?

[eugenevinitsky]

Btw, we have a Flow-users slack group that may be useful to you all. https://join.slack.com/t/flow-users/shared_invite/enQtNzk5NzE1OTkyNTE0LTU0YmE5ZTVkZmJmNDg1Nzg2YThjOThmY2ZlYWE2ZjQ0NTE1NjE3Y2YwMTljY2U5M2Y3MDhmMWUzZDM3YWViNmE

[jordan-schneider]

Yup, that looks like it fixed it. Thanks!

[c4cld]

@eugenevinitsky The problem has been solved!Thank you very much!

[c4cld]

@eugenevinitsky I have run /examples/rllib/traffic_light_grid.py successfully. And the output are saved in ~/ray_results. But when I use flow/visualize/visualizer_rllib.py to replay the trained policy, simulation ended at time:1.00. I dont know the reason. Because I am sure about the trained policy has been generated, and I successfully use tensorboard to check the curves of reward and loss.

[kjang96]

@ZJP-c4cld is there an accompanying traceback?

[c4cld]

@kjang96 No traceback.

[2855239858]

@ZJP-c4cld Btw, did you add Rl cars in traffic_light_grid successfully? I want to do it too. But I found it difficult to add Rl cars because it seems that I have to rewrite a lot of codes and handle basic API of sumo.

[2855239858]

I just want make sure that adding RL cars in traffic_light_grid is feasible.

[julienguegan]

Hey, I seems to have a problem quite similar after having trained the traffic_light_grid environment (and verify with tensorboard), I want to visualize the result with SUMO and the function visualizer_rllib.py but I get ValueError: ('Observation outside expected value range', Box(339,) at some point in the simulation ... Does someone already have that ? I did not touch anything from the python file of flow library