Dataset Collection Error

[omerXfaruq]

Hello There;

https://github.com/bradyz/2020_CARLA_challenge#data-collection
I am trying to follow your data collection, but it receives error, i think there is a problem with autopilot data-collection.

Also can I collect ground truth segmentation dataset with the autopilot, how can I do that?

Error:

> The sensor's configuration used is invalid:
> > Illegal sensor used. sensor.camera.semantic_segmentation are not allowed!
> 
> Traceback (most recent call last):
>   File "leaderboard/leaderboard/leaderboard_evaluator.py", line 273, in _load_and_run_scenario
>     AgentWrapper.validate_sensor_configuration(self.sensors, track, args.track)
>   File "/home/lenovo/carlaPath_0.9.10/CARLA_0.9.10.1/leaderboard/leaderboard/autoagents/agent_wrapper.py", line 205, in validate_sensor_configuration
>     raise SensorConfigurationInvalid("Illegal sensor used. {} are not allowed!".format(sensor['type']))
> leaderboard.envs.sensor_interface.SensorConfigurationInvalid: Illegal sensor used. sensor.camera.semantic_segmentation are not allowed!
> > Registering the route statistics
> Done. See sample_data/route_19.txt for detailed results.

[bradyz]

you should use the custom leaderboard repo that is included in this project - this custom leaderboard version disables these methods that allow for privileged information to be captured during training

additionally, i'm working on an updated version that should be public in the next month or so

[omerXfaruq]

Hello @bradyz , thanks for the response.

I used the original leaderboard repo in the $CARLA_ROOT directory (where I installed according to https://leaderboard.carla.org/get_started/#1-system-setup) and tried to run_agent.sh there.
It was missing
1-routes from 2020_carla_challenge/leaderboard/data , I copied the route directories.
2-team_agent autopilot, I copied leaderboard/team_code directory, from 2020_carla_challenge
3-carla_project files, carla_project from 2020_carla_challenge

However it still gave the privileged information capturing error error.

========= Preparing RouteScenario_19 (repetition 0) =========
> Setting up the agent
route_19_11_24_07_04_06

The sensor's configuration used is invalid:
> Illegal sensor used. sensor.camera.semantic_segmentation are not allowed!

Traceback (most recent call last):
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 270, in _load_and_run_scenario
    AgentWrapper.validate_sensor_configuration(self.sensors, track, args.track)
  File "/home/lenovo/carlaPath_0.9.10/CARLA_0.9.10.1/leaderboard/leaderboard/autoagents/agent_wrapper.py", line 205, in validate_sensor_configuration
    raise SensorConfigurationInvalid("Illegal sensor used. {} are not allowed!".format(sensor['type']))
leaderboard.envs.sensor_interface.SensorConfigurationInvalid: Illegal sensor used. sensor.camera.semantic_segmentation are not allowed!

My run_agent.sh:

#!/bin/bash
export PORT=2000                                                    # change to port that CARLA is running on
export ROUTES=leaderboard/data/routes_training/route_19.xml         # change to desired route
export TEAM_AGENT=auto_pilot.py                                     # no need to change
export TEAM_CONFIG=saveDataPath                                       # change path to checkpoint
export HAS_DISPLAY=1                                                # set to 0 if you don't want a debug window

export PYTHONPATH=$PYTHONPATH:$CARLA_ROOT/PythonAPI/carla
export PYTHONPATH=$PYTHONPATH:$CARLA_ROOT/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg
export PYTHONPATH=$PYTHONPATH:leaderboard
export PYTHONPATH=$PYTHONPATH:leaderboard/team_code
export PYTHONPATH=$PYTHONPATH:scenario_runner

if [ -d "$TEAM_CONFIG" ]; then
    CHECKPOINT_ENDPOINT="$TEAM_CONFIG/$(basename $ROUTES .xml).txt"
else
    CHECKPOINT_ENDPOINT="$(dirname $TEAM_CONFIG)/$(basename $ROUTES .xml).txt"
fi

python3 leaderboard/leaderboard/leaderboard_evaluator.py \
--track=SENSORS \
--scenarios=leaderboard/data/all_towns_traffic_scenarios_public.json  \
--agent=${TEAM_AGENT} \
--agent-config=${TEAM_CONFIG} \
--routes=${ROUTES} \
--checkpoint=${CHECKPOINT_ENDPOINT} \
--port=${PORT}

echo "Done. See $CHECKPOINT_ENDPOINT for detailed results."

[t27]

you should be using @bradyz 's version of the leaderboard repository.
This line should be present in your leaderboard_evaluator.py for it to work properly https://github.com/bradyz/leaderboard/blob/7104247204e0e591d43640d2c98490533bc5bbb8/leaderboard/leaderboard_evaluator.py#L42

[omerXfaruq]

@t27 thank you very much!

[omerXfaruq]

TL;DR
Finally I realize that the problem was path variables existing in ~/.bashrc like this: https://leaderboard.carla.org/get_started/#13-define-the-environment-variables

I deleted all path variables, and run the data_collection with leaderboard and scenario_runner from 2020_carla_challenge however i still got this error, it seems there is some small error in custom 2020_carla_challenge repo for data_collection
It stops just after running the run_agent and initializing vehicles on the world:

Current Error:

========= Preparing RouteScenario_19 (repetition 0) =========
> Setting up the agent
route_19_11_26_06_59_47
> Loading the world
> Running the route

Stopping the route, the agent has crashed:
> index 255 is out of bounds for axis 0 with size 26

Traceback (most recent call last):
  File "/home/lenovo/2020_CARLA_challenge/leaderboard/leaderboard/scenarios/scenario_manager.py", line 152, in _tick_scenario
    ego_action = self._agent()
  File "/home/lenovo/2020_CARLA_challenge/leaderboard/leaderboard/autoagents/agent_wrapper.py", line 77, in __call__
    return self._agent()
  File "/home/lenovo/2020_CARLA_challenge/leaderboard/leaderboard/autoagents/autonomous_agent.py", line 115, in __call__
    control = self.run_step(input_data, timestamp)
  File "/home/lenovo/2020_CARLA_challenge/leaderboard/team_code/auto_pilot.py", line 170, in run_step
    _topdown = Image.fromarray(COLOR[CONVERTER[topdown]])
IndexError: index 255 is out of bounds for axis 0 with size 26

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 345, in _load_and_run_scenario
    self.manager.run_scenario()
  File "/home/lenovo/2020_CARLA_challenge/leaderboard/leaderboard/scenarios/scenario_manager.py", line 136, in run_scenario
    self._tick_scenario(timestamp)
  File "/home/lenovo/2020_CARLA_challenge/leaderboard/leaderboard/scenarios/scenario_manager.py", line 159, in _tick_scenario
    raise AgentError(e)
leaderboard.autoagents.agent_wrapper.AgentError: index 255 is out of bounds for axis 0 with size 26
> Stopping the route

========= Results of RouteScenario_19 (repetition 0) ------ FAILURE =========

╒═════════════════════════════════╤═════════════════════╕
│ Start Time                      │ 2020-11-26 07:00:00 │
├─────────────────────────────────┼─────────────────────┤
│ End Time                        │ 2020-11-26 07:00:00 │
├─────────────────────────────────┼─────────────────────┤
│ Duration (System Time)          │ 0.19s               │
├─────────────────────────────────┼─────────────────────┤
│ Duration (Game Time)            │ 0.05s               │
├─────────────────────────────────┼─────────────────────┤
│ Ratio (System Time / Game Time) │ 0.27                │
╘═════════════════════════════════╧═════════════════════╛

you should use the custom leaderboard repo that is included in this project - this custom leaderboard version disables these methods that allow for privileged information to be captured during training

additionally, i'm working on an updated version that should be public in the next month or so

Hello @bradyz , I misunterstood you there, sorry! After looking through again, I understood you correctly now.
I initially run the example in your 2020_carla_challenge repo, and was using custom leaderboard. After your reply I thought I need to use leaderboard inside $CARLA_ROOT and tried the things in my second comment:

Hello @bradyz , thanks for the response.
...

My Final run_agent.sh recieving error "index 255 is out of bounds for axis 0 with size 26":

#!/bin/bash
export CARLA_ROOT=/home/lenovo/carlaPath_0.9.10/CARLA_0.9.10.1

export PORT=2000                                                    # change to port that CARLA is running on
export ROUTES=leaderboard/data/routes_training/route_19.xml         # change to desired route
export TEAM_AGENT=auto_pilot.py                                     # no need to change
export TEAM_CONFIG=sample_data                                      # change path to save data

export PYTHONPATH=$PYTHONPATH:$CARLA_ROOT/PythonAPI/carla
export PYTHONPATH=$PYTHONPATH:$CARLA_ROOT/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg
export PYTHONPATH=$PYTHONPATH:leaderboard
export PYTHONPATH=$PYTHONPATH:leaderboard/team_code
export PYTHONPATH=$PYTHONPATH:scenario_runner
	

if [ -d "$TEAM_CONFIG" ]; then
    CHECKPOINT_ENDPOINT="$TEAM_CONFIG/$(basename $ROUTES .xml).txt"
else
    CHECKPOINT_ENDPOINT="$(dirname $TEAM_CONFIG)/$(basename $ROUTES .xml).txt"
fi

python3 leaderboard/leaderboard/leaderboard_evaluator.py \
--track=SENSORS \
--scenarios=leaderboard/data/all_towns_traffic_scenarios_public.json  \
--agent=${TEAM_AGENT} \
--agent-config=${TEAM_CONFIG} \
--routes=${ROUTES} \
--checkpoint=${CHECKPOINT_ENDPOINT} \
--port=${PORT}

echo "Done. See $CHECKPOINT_ENDPOINT for detailed results."

[bradyz]

it looks like you'll have to pull from master for the carla_project submodule