Add L8 Biome Dataset #1200

shradhasehgal · 2023-03-28T01:17:59Z

Overview

Adds L8 Biome dataset as a RasterDataset to TorchGeo.

Main changes:

Added functions to initialize the L8 Biome dataset, sample from it, and plot images.
Added data generation script and artificial data corresponding to the same.
Wrote tests to check the functionality of the L8 Biome class.
Integrated L8Biome Data Module and included different splits.
Included dataset details into the docs.

Here is a sample plot of an image and its mask from the dataset:

adamjstewart · 2023-03-28T02:23:38Z

Used getitem and plotting functions from CDL dataset. How should these be changed for L8 Biome?

CDL is useless for __getitem__ and plot since this dataset is drastically different. CDL is only useful for copying the downloading code. For __getitem__, LandCoverAIGeo is a better reference. For plotting, every dataset is different, there's isn't a great example, but any land cover mapping dataset should have a similar image + mask plotting setup.

What will the cmap be for this dataset?

First of all, cmap isn't required for any dataset. But you do need a way to plot the image, and imshow + colormap is the best way to do this. Oftentimes, like in the case of CDL, this cmap is hardcoded into the GeoTIFF file, and TorchGeo will automatically read and use it without listing it in the dataset, but I don't think that's the case with your masks. In the case of this dataset, the authors purposefully chose 0, 64, 128, 192, 255 for the class values instead of 0, 1, 2, 3, 4. I'm guessing that's because these are the cmap values they recommend. Basically, we plot the image in grayscale with each class having these color values. Note that in the __getitem__ you'll need to undo this and ensure that these classes are 0, 1, 2, 3, 4.

torchgeo/datasets/l8biome.py

adamjstewart · 2023-03-28T02:31:30Z

For the failing tests, see https://torchgeo.readthedocs.io/en/stable/user/contributing.html#linters

torchgeo/datasets/l8biome.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

…orchgeo into datasets/l8biome

…claration

shradhasehgal · 2023-03-30T06:32:16Z

Latest modifications:

Fixed L8Biome dataset verification and download
Resolved comments from previous review
Changed auxiliary files for dataset addition
Created L8Biome datamodule

Doubts:

Used getitem function from LandCoverAI as per this comment. Wondering if anything needs to change for L8Biome in particular?
Altered cmap values to 0-4 as per this discussion. What do the values in this dictionary represent and how to obtain them for L8 Biome?
Some tests are still failing after running linters. How to resolve them?

TODOs:

Write tests
Plotting function

Tagging @adamjstewart for review.

adamjstewart

This is very close to being done (aside from plotting and testing)!

docs/api/datasets.rst

docs/api/geo_datasets.csv

torchgeo/datamodules/__init__.py

torchgeo/datamodules/l8biome.py

torchgeo/datasets/l8biome.py

…l files

tests/data/l8biome/data.py

tests/datasets/test_l8biome.py

tests/data/l8biome/data.py

tests/datasets/test_l8biome.py

adamjstewart

Hopefully my last review!

Docs look good: https://torchgeo--1200.org.readthedocs.build/en/1200/api/datasets.html#l8-biome

Can you plot an image and add the plot to the PR description so we can make sure everything looks visually correct? Colors might be better now that it isn't only B2 data.

torchgeo/datasets/l8biome.py

Downloading raw data directly Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

torchgeo/datasets/l8biome.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

adamjstewart

Looks great, congrats on your first PR! Hopefully the first of many if the extensive review process didn't scare you away 😅

* Added varification of data * Added getitem and plotting functions - partial * Update torchgeo/datasets/l8biome.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Changes corresponding to draft PR * Changed auxilliary files for dataset addition; added L8Biome datamodule * Fixed dataset verification and download; minor changes in variable declaration * Running linters to properly format code * Fixing failed linter tests from previous commit * Fixing comments from review microsoft#2 * Added script to generate artificial data in similar format as original files * Removed dependency on existing dataset * Fixes to the getitem function; tested with samples * Making suggested changes; fixing mask; writing tests * Changes to plot RGB image * Changes to plot RGB image - running into errors related to band variables declaration * Fixed plotting of dataset * Added masks to artificial data generation; tested with pytest - all tests pass * Resolving file and code formatting with linters * Added artificial data * Fixing lists in docstrings * Fixing comments from review * Added new tests * Regenerated artificial data * Added test for l8datamodule; linters formatting * Fixed image generation with more rigorous check on filepath * docstring formatting change * Resolving CI test errors * Added conf/yaml for datamodules test * Fixing CI test failure bug; Changes from review; Samplers in datamodule WIP * Added more artificial data * Fixed yaml and datamodule changes * Running linters to properly format code * Reducing batch_size and length for datamodules tests * Reduced patch size in l8biome yaml to fit within artificial data * Declaring all samplers each time * Reverting to stages checks for samplers * Changed data generation file and generated data again * Fix getitem * Included more encompassing tests * Adding test for RGB bands being absent; running linters * Added check for mask in getitem test * Changes corresponding to review * Update torchgeo/datasets/l8biome.py Downloading raw data directly Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Update l8biome.py * Update torchgeo/datasets/l8biome.py: Changing RGB bands order Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Change B2 band to B1 --------- Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

shradhasehgal added 2 commits March 27, 2023 19:52

Added varification of data

6606195

Added getitem and plotting functions - partial

db96c6c

github-actions bot added datasets Geospatial or benchmark datasets documentation Improvements or additions to documentation labels Mar 28, 2023

adamjstewart added this to In progress in SSL4EO-L via automation Mar 28, 2023

adamjstewart reviewed Mar 28, 2023

View reviewed changes

adamjstewart marked this pull request as draft March 28, 2023 02:33

adamjstewart changed the title ~~Adding L8Biome Dataset - WIP~~ Add L8 Biome Dataset Mar 28, 2023

adamjstewart added this to the 0.5.0 milestone Mar 28, 2023

adamjstewart reviewed Mar 28, 2023

View reviewed changes

torchgeo/datasets/l8biome.py Outdated Show resolved Hide resolved

shradhasehgal and others added 6 commits March 28, 2023 15:59

Update torchgeo/datasets/l8biome.py

0a32908

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Changes corresponding to draft PR

779ca31

Merge branch 'datasets/l8biome' of https://github.com/shradhasehgal/t…

8579f15

…orchgeo into datasets/l8biome

Changed auxilliary files for dataset addition; added L8Biome datamodule

dd98acb

Fixed dataset verification and download; minor changes in variable de…

f375d84

…claration

Running linters to properly format code

63c8484

github-actions bot added the datamodules PyTorch Lightning datamodules label Mar 30, 2023

Fixing failed linter tests from previous commit

69145b7

adamjstewart reviewed Mar 30, 2023

View reviewed changes

torchgeo/datasets/l8biome.py Outdated Show resolved Hide resolved

Fixing comments from review #2

467d9ab

adamjstewart reviewed Apr 1, 2023

View reviewed changes

torchgeo/datasets/l8biome.py Outdated Show resolved Hide resolved

Added script to generate artificial data in similar format as origina…

38216b0

…l files

github-actions bot added the testing Continuous integration testing label Apr 1, 2023

adamjstewart requested changes Apr 1, 2023

View reviewed changes

tests/data/l8biome/data.py Outdated Show resolved Hide resolved

tests/datasets/test_l8biome.py Outdated Show resolved Hide resolved

tests/datasets/test_l8biome.py Outdated Show resolved Hide resolved

tests/datasets/test_l8biome.py Outdated Show resolved Hide resolved

shradhasehgal added 2 commits April 1, 2023 13:28

Removed dependency on existing dataset

21f1c88

Fixes to the getitem function; tested with samples

62d7e1b

shradhasehgal added 4 commits April 3, 2023 22:57

Added more artificial data

654149a

Fixed yaml and datamodule changes

2a1c030

Running linters to properly format code

bb0dec5

Reducing batch_size and length for datamodules tests

3fa2a09

adamjstewart reviewed Apr 4, 2023

View reviewed changes

tests/data/l8biome/data.py Outdated Show resolved Hide resolved

shradhasehgal and others added 7 commits April 4, 2023 08:50

Reduced patch size in l8biome yaml to fit within artificial data

4634458

Declaring all samplers each time

e0f0eaf

Reverting to stages checks for samplers

e5e33ce

Changed data generation file and generated data again

369cbdc

Fix getitem

2e718c6

Included more encompassing tests

9a30a38

Adding test for RGB bands being absent; running linters

1d088a0

adamjstewart reviewed Apr 5, 2023

View reviewed changes

tests/datasets/test_l8biome.py Show resolved Hide resolved

Added check for mask in getitem test

918f440

shradhasehgal requested a review from adamjstewart April 5, 2023 03:06

adamjstewart requested changes Apr 5, 2023

View reviewed changes

Changes corresponding to review

e2ada6b

shradhasehgal requested a review from adamjstewart April 5, 2023 04:22

adamjstewart reviewed Apr 5, 2023

View reviewed changes

torchgeo/datasets/l8biome.py Outdated Show resolved Hide resolved

Update torchgeo/datasets/l8biome.py

415a2ce

Downloading raw data directly Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

adamjstewart reviewed Apr 5, 2023

View reviewed changes

torchgeo/datasets/l8biome.py Outdated Show resolved Hide resolved

Update l8biome.py

f4712db

adamjstewart reviewed Apr 5, 2023

View reviewed changes

torchgeo/datasets/l8biome.py Outdated Show resolved Hide resolved

adamjstewart reviewed Apr 5, 2023

View reviewed changes

torchgeo/datasets/l8biome.py Outdated Show resolved Hide resolved

shradhasehgal and others added 2 commits April 5, 2023 13:02

Update torchgeo/datasets/l8biome.py: Changing RGB bands order

9eaa5a9

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Change B2 band to B1

25e5e62

adamjstewart approved these changes Apr 5, 2023

View reviewed changes

adamjstewart merged commit 24afe5a into microsoft:main Apr 5, 2023
20 checks passed

SSL4EO-L automation moved this from In progress to Done Apr 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add L8 Biome Dataset #1200

Add L8 Biome Dataset #1200

shradhasehgal commented Mar 28, 2023 •

edited

adamjstewart commented Mar 28, 2023

adamjstewart commented Mar 28, 2023

shradhasehgal commented Mar 30, 2023

adamjstewart left a comment

adamjstewart left a comment

adamjstewart left a comment

Add L8 Biome Dataset #1200

Add L8 Biome Dataset #1200

Conversation

shradhasehgal commented Mar 28, 2023 • edited

Overview

Main changes:

adamjstewart commented Mar 28, 2023

adamjstewart commented Mar 28, 2023

shradhasehgal commented Mar 30, 2023

adamjstewart left a comment

Choose a reason for hiding this comment

adamjstewart left a comment

Choose a reason for hiding this comment

adamjstewart left a comment

Choose a reason for hiding this comment

shradhasehgal commented Mar 28, 2023 •

edited