Issue in OOD data distribution when Grouper is set to "regions" for FMoW

[saraalemadi]

Hi,

I am trying to change the groupby from "year" to "region". I have followed the instructions in the README page and currently using the following command:
python3 wilds/examples/run_expt.py --dataset fmow --algorithm ERM --groupby_fields region --root_dir wilds_fmow/

However, the issue is that the training dataset is not being separated in terms of distinct regions for ID and OOD manner. That is, all regions are included in ID as well as OOD. Here is a screenshot of the output:

Therefore, I was wondering if that is a bug in the code or am I missing something?

Thanks
Sara A. Al-Emadi

[kohpangwei]

Hi Sara, this is expected: as described in our paper, the training, val, and test data come from (different time ranges in) the same regions.

[saraalemadi]

Hi @kohpangwei,

Thanks for your reply. However, I would like to clarify that when running the experiment without changing the grouper (mentioned in https://github.com/p-lambda/wilds#domain-information using CombinatorialGrouper), I get the following output, which shows a clear separation between ID and OOD defined by year. Here is an example:

But when I do specify the CombinatorialGrouper in terms of "region" rather than "year", I get a mixed output as shown in my previous comment. Therefore, shouldn't the CombinatorialGrouper split the data into distinct groups (ID and OOD) based on the specified group?

Would appreciate your clarification.

Thanks
Sara A. Al-Emadi

[kohpangwei]

Hi Sara,

This is expected behavior. The data splits are unaffected by the grouper. The grouper just groups the (fixed) data splits according to the specified group-by feature.