I'd like to know some detailed options for training and testing.

[jun-ja]

First of all, thank you for providing a good code and paper.
I have some questions about details of the code.

1. In the "Global config" code, only 'tiny' and 'short' are included in the training and 'long' are not included, is this intended? Can I add the rest of long town except Town 5? Did you take it off to prevent overfitting?

2. The results of my training and testing will not be completely consistent with your paper, but I want to make sure how similar they are. Can I know what the three seeds used for training are? If it's difficult to tell, wouldn't it matter if I use three seeds arbitrarily?

2-1. In order to change the random seed, which part of the training code should I modify?

Thank you.

[ap229997]

1. Long routes have a highly skewed distribution of navigational commands with ~90% being 'follow lane' which makes it difficult for the policy to learn turning and stopping behavior. With 'tiny' and 'short' routes, the distribution is much better (although still skewed). You can check our suppmat for more details. Overall, you can include the long routes as well but you should also check the distribution to make sure that the policy encounters substantial data to learn other behaviors except 'follow lane'.

2. You can use arbitrary seeds for training. We ran the training multiple times with arbitrary seeds as well (by default, it is randomized in PyTorch unless you set it explicitly) to make sure that the results are consistent. I agree that this doesn't make our results completely reproducible but we tried multiple seeds and the performance was within the variance reported in the paper.
   If you run the training code as is, PyTorch will use a random seed every time to initialize the model parameters (and in other sources of randomness) so you don't explicitly need to set the seed anywhere in the code. That being said, if you want a fully deterministic and reproducible protocol, check Hello, I want to know what ‘3 training seeds’ mean, can you talk about it in detail? #12.

[jun-ja]

Thank you for replying!