Question about how hyperparameters change as test domain and dataset changes

[wlaud1001]

Thank you for your great work.

I'm little curious about how hyperparameters change as test domain changes in your code.

I saw the instructions you provided in the readme for reproducing your results,
but I don't understand gow the HPs change as the test domain changes.

In your instructions (command lines for reproducing results), no other arguments for HPs is given and I think all the experiments will use default HPs regardless of test domain and dataset.

I'm just curious how HPs change as the test domain and dataset change in your instructions.

Thank you.

---

korean ver.
안녕하세요 좋은 연구 감사드립니다.

Domain generalization 관련 연구에 관심이 있어 공부를 하던 중 SWAD 논문을 보게 되었고,
이를 활용하고 싶어, 코드를 보던 중 질문이 있어 이슈를 남기게 되었습니다.

Readme에서 결과는 재현하기 위해 필요한 커맨드 라인을 제공해주셨는데,
해당 커멘드 라인으로 실험을 돌릴 시 dataset과 test domain이 바뀜에 따라 HPs가 어떻게 바뀌는지 이해가 가지 않아 질문을 드립니다.

해당 커멘드 라인에는 HP와 관련된 argument가 제공되지 않는 것으로 보이는데, 이럴 경우에는 모든 실험이 default HPs를 사용하여 진행될 것 같습니다.

dataset, test domain이 바뀜에 따라 사용되는 (찾아진) HP가 다를 것 같은데, 제공된 커멘드라인을 사용하여 코드를 돌렸을 경우 dataset, test domain이 바뀜에 따라 HP가 어떻게 바뀌는지 궁금하여 질문드립니다.

감사합니다.

[khanrc]

Thank you for the interest in our study.

The commands provide the HPs (See VLCS commands). For the other datasets, our searched results are same as default HPs ($N_s=3, N_e = 6, r = 0.3$). Note that we only tuned the SWAD HPs and left the remaining HPs (learning rate, weight decay, dropout rate) as default values. Further, we tuned the HPs in the dataset level, not in the test domain level (this is different to original DomainBed). So the same HPs are applied to the different test domains in the same dataset.

[wlaud1001]

Thank you for quick response.

I understand how SWAD HPs are tuned, but actually my question was about general HPs such as learning rate, weight decay, ... . (Sorry for the confusion.)

In the paper, it is mentioned that HPs are searched based on 396 (18 * 22(the number of test domains)) runs, which means HPs are searched in the test domain level (HPs such as lr, weight_decay, batch_size, ...). And with high probability, the best HPs will be different for each test domain.
Is there any reason why you applied the same HPs to the different test domains in the same dataset, even though HPs are searched in test domain level (396 runs)?

In addition, in the instruction you provided for reproducing results, it seems like the HPs such as learning rate, weight decay, are same across dataset, and i think this will not reproduce results correctly.

Thank you.

[khanrc]

Our HP search was done in the dataset level. Yes, each run for HP search was done in the test domain level. However, the results aggregation and the HP selection were done in the dataset level, for technical, historical, and environmental reasons. So, the provided commands can reproduce the reported results.

HP search for the algorithm-independent HPs (LR, weight decay, ...) were skipped an environmental reason; the computational resource was not sufficient at the first working period. After the submission, we checked the results with HP search, but the improvement was marginal (up to 0.1pp). So, we decided not to update the numbers to minimize the confusion of readers.

[wlaud1001]

Thanks for reply!

Sorry to keep asking.

Then, you mean, in your paper, the algorithm-independent HPs are not tuned based on Domainbed protocol, and just set to default value?

My curiousity is about supplementary B.2 table7 ,which describe hyperparameter search space.

According to your answer (algorithm-independent HP search were skipped), is the result of table.2 not the result of HP tuning using table.7? And, did the tuning using tabel.7 proceed after submission?

[khanrc]

Yes, it is written in the Section B.1.

Hence, we reduce the search space of SWAD for computational efficiency; batch size and learning rate are set to 32 for each domain and 5e-5, respectively. We set dropout probability and weight decay to zero. We only search Ns, Ne and r.

HP search space in Section B.2 (Table 7) is designed to compare SAM and MixStyle (+ERM) in DomainBed protocol, not to search HPs of SWAD.

[wlaud1001]

Thank you so much!!
Now i understand clearly.
Thanks for your kind response and thank you for great work!

Best regards,
Jimyeong