Add MapInWild dataset

[burakekim]

Closes #1096

Here is the MapInWild dataset instance. There are things I solved with ugly fixes and quick workarounds but I hope this pull request still is a good-enough starting point.

TODO and issues

• Checksum: So far no checksum. I can not seem to find the MD5s from Hugging Face. There are only SH256s.
• Now it is not really straightforward to distinguish the modalities in the __getitem__ making it harder to apply specific normalization steps (i.e., s2/10000).
• Documentation and tests

Nice to do

• Picking modalities band-wise to enable the use of various combinations of bands. This could even supersede the modality-wise picking.
• Improving the plot function. A normalization function (i.e., percentile_normalization) is needed for at least the Sentinel-2 data. For that, an argument and modality-aware logic inside this function could be helpful.
• Remove the pandas and move to a lower-level library.

Any help is appreciated. Thanks in advance for your time in reviewing this pull request.

P.S. I am by no means a GitHub pro, so any heads-up is appreciated.

The signature:
mapinwild = MapInWild(root="data", download=True, modality=["mask","esa_wc","s2_temporal_subset","viirs","s1"], split="train", transforms=None, checksum=False)

Some sample images:

VIIRS Nighttime light band

Sentinel-2 single temporal subset (the Sentinel-2 season with the highest score)

ESA WorldCover map

Sentinel-1

[adamjstewart]

This looks like a great start! Let me know if you have any specific questions.

Checksum: So far no checksum. I can not seem to find the MD5s from Hugging Face. There are only SH256s.

You can calculate them yourself using md5 <filename> on the command line. We'll likely move from MD5 to SHA256 someday.

Remove the pandas and move to a lower-level library.

You should probably just keep pandas. I've been thinking about making it a required dep of TorchGeo because so many datasets use it.

[burakekim]

@microsoft-github-policy-service agree company="unibwmunich"

[burakekim]

Thanks a bunch for your revisions!

I applied some quick changes and tests. I will make sure there is a full test coverage after applying the necessary additions and changes, which are for now:

• Applying MD5 checks.
• Finding a generic way to handle modalities one by one in the __getitem__ so that we can apply modality-specific normalizations. As a last resort, I will create some mess by simply using self._load_raster(id, modals) to load the modalities separately.
• Making plot function modality aware to apply proper visualization scalings.

Issues

• Do you happen to know why the line pyupgrade --py37-plus $(find . -name "*.py") outputs the error below? It works without the query argument $(find . -name "*.py"). Is this argument needed?

usage: pyupgrade [-h] [--exit-zero-even-if-changed] [--keep-percent-format] [--keep-mock] [--keep-runtime-typing]
                 [--py3-plus] [--py36-plus] [--py37-plus] [--py38-plus] [--py39-plus] [--py310-plus] [--py311-plus]
                 [filenames ...]
pyupgrade: error: unrecognized arguments: -name *.py)

• I shortened the plain texts that are going over 89 characters and applied # noqa: E501 for the links, is this OK?

[adamjstewart]

Do you happen to know why the line pyupgrade --py37-plus $(find . -name "*.py") outputs the error below?

Nope, that syntax should be correct. Note that our tests use the more complicated:

$ pyupgrade --py38-plus $(find . -path ./docs/src -prune -o -name "*.py" -print)

I shortened the plain texts that are going over 89 characters and applied # noqa: E501 for the links, is this OK?

Yes this is perfect.

[adamjstewart]

Note that we just dropped Python 3.8 support, so --py38-plus is now --py39-plus

[burakekim]

@adamjstewart I have improved the dataset class (mapinwild.py) and added the code to create test data (mapinwild\data.py) with the last commit.

Now, I assume the next step is to populate the test_mapinwild.py.

The progress is quite slow as I have very little bandwidth for this side hustle -sorry for that! In fact, feel free to contribute to the test_mapinwild.py to wrap up things faster, otherwise, it could take up quite some time for me.

[adamjstewart]

No rush. Most of us are currently working towards a paper deadline, so we prob won't be able to help much either. But might be able to help after June.

[burakekim]

Could you please help me understand the check "license/cla Expected — Waiting for status to be reported"? I can not seem to complete that.

Also, please let me know if there is anything else to be done from my side before the release. Thank you for your time in helping me out here, it was a great learning curve for me. I could have never done it without your great reviews :) I plan to contribute more to the library as much as I can!

[adamjstewart]

@calebrob6 any idea what's wrong with the CLA bot? It was accepted here: #1131 (comment)

Glad my harsh reviews didn't scare you away 😆. I also see that you're in Munich? I just started a postdoc at TUM, maybe I'll see you around!

I'll review everything in detail again tomorrow, but I think we're probably pretty close now.

[calebrob6]

No idea, let's see if the ol' close and reopen trick works :)

[adamjstewart]

I hate how reliable ol' close and reopen is...

[calebrob6]

Yep, I think that did it! Thanks for all the hard work on this @burakekim!

[burakekim]

I also see that you're in Munich? I just started a postdoc at TUM, maybe I'll see you around!

@adamjstewart Yes, I am doing my PhD here in Munich. Definitely, it would be great to meet up! I will shoot you an e-mail :)

[burakekim]

Thanks for all the hard work on this @burakekim!

Thanks a bunch, @calebrob6! Contributing to the library has been an enjoyable experience :)

[calebrob6]

What are the requested changes here?

[adamjstewart]

Need to re-review tonight

[adamjstewart]

Two remaining easy changes and I think this will be ready to merge!

[adamjstewart]

Great work on this! Glad to have your dataset in TorchGeo!

[adamjstewart]

Bumping tests...