Question about result table on paper

[Kin-Zhang]

It's really a great job and clear codes as I said in the mail too. Thanks for your work!

But when I shared the paper with my labmates, who question about the result table on paper here:

1. As you can see here, we can know that tranfuser and other models may have variance but the number is too big, so did you try to explain the driving score belief.

2. As I know that, your expert is from

   The expert agent is based on the autopilot from this codebase.

   But the expert as we all know is rule-based, when our traffic random seed is always default in leaderboard_evaluator.py as 0, so all simulator's NPCs have same action as carla said here when you run Town05 Short or Long, why the expert will have 6.21 variance and 4.00 variance score?

Thanks again!

[ap229997]

1. The variance for transfuser and other models comes from two sources - training seed and evaluation variance. Training seed generally affects the initialization of the network and the order of the training data. Evaluation variance is due to the stochasticity in the behavior or other dynamic agents and traffic lights in the scene.

2. In our experiments, we found that the traffic manager in CARLA 0.9.10 still led to some evaluation variance. In the newer CARLA version, the traffic manager is supposed to be deterministic so that should reduce the variance but I have not experimented with the newer versions yet.

[Kin-Zhang]

• The variance for transfuser and other models comes from two sources - training seed and evaluation variance. Training seed generally affects the initialization of the network and the order of the training data. Evaluation variance is due to the stochasticity in the behavior or other dynamic agents and traffic lights in the scene.
• In our experiments, we found that the traffic manager in CARLA 0.9.10 still led to some evaluation variance. In the newer CARLA version, the traffic manager is supposed to be deterministic so that should reduce the variance but I have not experimented with the newer versions yet.

Thanks for replying, for first one yes the variance is from seed and initial But according to the result table and variance maybe shows that the Geometric method get higher scores since low variance here.

And for Carla, I check on 0.9.10 the documents didn’t have deterministic method but the leaderboard based on Carla 0.9.10.1 which use the getter method on code, I run several time on my computer find that with 0.9.10.1, it’s true for deterministic.

Finally Thanks for your reply, all the issue part helps me learn more about your method!