Evaluation result

[lawrencewxj]

Dear Authors,

Thank you for your excellent work,

I trained the model with: CUDA_VISIBLE_DEVICES=0 python train.py --id transfuser --batch_size 56

And then I evaluate the model with: run_evaluation.sh

export ROUTES=leaderboard/data/evaluation_routes/routes_town05_long.xml
export TEAM_AGENT=leaderboard/team_code/transfuser_agent.py
export TEAM_CONFIG=transfuser/log/transfuser
export CHECKPOINT_ENDPOINT=results/transfuser_result.json
export SCENARIOS=leaderboard/data/scenarios/town05_all_scenarios.json

Finally, I got the results as
"labels": [
"Avg. driving score",
"Avg. route completion",
"Avg. infraction penalty",
"Collisions with pedestrians",
"Collisions with vehicles",
"Collisions with layout",
"Red lights infractions",
"Stop sign infractions",
"Off-road infractions",
"Route deviations",
"Route timeouts",
"Agent blocked"
],
"values": [
"19.388",
"51.060",
"0.539",
"0.000",
"0.472",
"0.000",
"0.221",
"0.008",
"0.008",
"0.000",
"0.000",
"0.999"
]

Is this Avg. driving score normal as I trained with the 14_weathers_minimal_data?

I noticed that the DS should be around 33% in the paper, would you please give me some suggestions about which part I made mistakes in.

Best

[ap229997]

For the results in Table 1(a) of the paper, models are trained on clear_weather_data.

For the transfuser model trained on 14_weathers_minimal_data, we observed variance due to training seeds as well as evaluation, and our scores ranged from 24 to 36.

Can you tell what hyperparameters you used for training? What changes did you make in the config file? Also, I'd suggest training with different seeds and evaluating multiple times to get an estimate of the variance.

[lawrencewxj]

Thank you for your reply, and for the config file (transfuser/transfuser/config.py), I only changed the root_dir and viz_root.

I will download the clear_weather_data to train again, with clear_weather_data, can I use the same training setting as in the transfuser/transfuser/config.py to get the results in Table 1(a)?

thank you so much.

[ap229997]

Yes, you should be able to reproduce the results in Table 1(a). Also, make sure that you train with multiple seeds and run multiple evaluations (for Table 1(a), we trained with 3 different seeds and evaluated each seed 3 times). You don't need to run 9 evaluations as we did, but at least 3 training seeds and 1 evaluation for each is desirable. Since, evaluation on Town05_long takes a lot of time, you can also try evaluating on Town05_short.

I'll also check again to make sure that there is no other source of variation.

[lawrencewxj]

Thank you so much, I will get back to you when I get the results.

[lawrencewxj]

Hi, I have downloaded the clear_weather_data, when I change the root_dir and viz_root, it still report an error,

Preloading 5301 sequences from /scratch/xijunwang/code/E2E/transfuser/data/clear_weather_data/Town10_tiny/rg_lidar_diag_pl_1_4.npy Preloading 3089 sequences from /scratch/xijunwang/code/E2E/transfuser/data/clear_weather_data/Town10_short/rg_lidar_diag_pl_1_4.npy 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:01<00:00, 9.47it/s] 0%| | 0/1 [00:00<?, ?it/s]Preloading 8284 sequences from /scratch/xijunwang/code/E2E/transfuser/data/clear_weather_data/Town05_short/rg_lidar_diag_pl_1_4.npy 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 20.58it/s] Total trainable parameters: 66293634 0%| | 0/2079 [00:00<?, ?it/s] Traceback (most recent call last): File "train.py", line 248, in <module> trainer.train() File "train.py", line 58, in train for data in tqdm(dataloader_train): File "/home/xijunwang/anaconda3/lib/python3.8/site-packages/tqdm/std.py", line 1178, in __iter__ for obj in iterable: File "/home/xijunwang/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__ data = self._next_data() File "/home/xijunwang/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data return self._process_data(data) File "/home/xijunwang/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data data.reraise() File "/home/xijunwang/anaconda3/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise raise self.exc_type(msg) FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0. Original Traceback (most recent call last): File "/home/xijunwang/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop data = fetcher.fetch(index) File "/home/xijunwang/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/xijunwang/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp> data = [self.dataset[idx] for idx in possibly_batched_index] File "/scratch/xijunwang/code/E2E/transfuser/transfuser/data.py", line 200, in __getitem__ scale_and_crop_image(Image.open(seq_fronts[i]), scale=self.scale, crop=self.input_resolution)))) File "/home/xijunwang/anaconda3/lib/python3.8/site-packages/PIL/Image.py", line 2912, in open fp = builtins.open(filename, "rb") FileNotFoundError: [Errno 2] No such file or directory: '/is/rg/avg/aprakash/carla9-10_data/opengl/all_towns_data/Town01_small/routes_town01_junctions_fixed_1_10_22_18_33_09/rgb_front/0004.png'

It seems that rg_lidar_diag_pl_1_4.npy has saved some old path of yours, should I delete all the rg_lidar_diag_pl_1_4.npy first, and then run the train.py, since I found that if it can't find the rg_lidar_diag_pl file, it will build a new one. Is that right?

Thanks.

[ap229997]

yes, you need to regenerate the preload files.