L7 Irish: add new dataset/datamodule #1197

yichiac · 2023-03-26T14:25:02Z

Overview

Adding Landsat 7 Cloud Cover Assessment Validation Data as a RasterDataset to TorchGeo.

Main changes:

Added functions to initialize the L7 Irish dataset, sample from it, and plot images.
Added data generation script and artificial data corresponding to the same.
Wrote tests to check the functionality of the L7 Irish class.
Integrated L7Irish Data Module and included different splits.
Included dataset details into the docs.

Here is a sample plot of an image and its mask from the dataset:

adding [Landsat 7 Cloud Cover Assessment Validation Data](https://landsat.usgs.gov/landsat-7-cloud-cover-assessment-validation-data) from USGS. The files are .tif format. The naming pattern is slightly different from the Landsat class. Thus, the customized RasterDataset class is used.

adamjstewart

Obviously still needs documentation and tests

.DS_Store

torchgeo/datasets/l7irish.py

adamjstewart · 2023-03-26T16:56:25Z

See https://torchgeo.readthedocs.io/en/stable/user/contributing.html#linters for how to fix the failing tests.

torchgeo/datasets/l7irish.py

# deleted: .DS_Store # deleted: torchgeo/.DS_Store # # Changes not staged for commit: # modified: .gitignore # modified: torchgeo/datasets/l7irish.py

# modified: .gitignore # modified: torchgeo/datasets/l7irish.py

adamjstewart

Still a lot of failing tests. See Contributing docs for how to fix style issues.

.gitignore

torchgeo/datasets/l7irish.py

modify: .gitignore and l7irish.py

# modified: .gitignore # new file: tests/data/l7irish/data.py # modified: torchgeo/datasets/l7irish.py

.gitignore

tests/data/l7irish/data.py

torchgeo/datasets/l7irish.py

torchgeo/datasets/__init__.py, torchgeo/datasets/l7irish.py

modified l7irish.py

Added l7irish to __init__ Added L7Irish details to geo_datasets.csv Will work on datamodules/l7irish.py

adamjstewart

Almost done. Just a few last changes. The 2015 version also has cloud shadow, so we should add that back.

docs/api/geo_datasets.csv

tests/conf/l7irish.yaml

tests/data/l7irish/data.py

torchgeo/datasets/l7irish.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

docs/api/geo_datasets.csv

torchgeo/datasets/l7irish.py

tests/data/l7irish/data.py

torchgeo/datasets/l7irish.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

torchgeo/datasets/l7irish.py

* Landsat 7 Irisih: add new dataset/datamodule adding [Landsat 7 Cloud Cover Assessment Validation Data](https://landsat.usgs.gov/landsat-7-cloud-cover-assessment-validation-data) from USGS. The files are .tif format. The naming pattern is slightly different from the Landsat class. Thus, the customized RasterDataset class is used. * # Changes to be committed: # deleted: .DS_Store # deleted: torchgeo/.DS_Store # # Changes not staged for commit: # modified: .gitignore # modified: torchgeo/datasets/l7irish.py * # Changes to be committed: # modified: .gitignore # modified: torchgeo/datasets/l7irish.py * add: test_l7irish.py and data.py modify: .gitignore and l7irish.py * # Changes to be committed: # modified: .gitignore # new file: tests/data/l7irish/data.py # modified: torchgeo/datasets/l7irish.py * modified: tests/datasets/test_l7irish.py, torchgeo/datasets/__init__.py, torchgeo/datasets/l7irish.py * added data.py, austral.tar.gz, test_l7irish.py modified l7irish.py * remove comments in test_l7irish.py * resolve black and flake8 issues * Fixed _getitem Added l7irish to __init__ Added L7Irish details to geo_datasets.csv Will work on datamodules/l7irish.py * Added L7 Irish datamodule * fix flake8 space error * fix black test error * chmod +x for data.py * Update docs/api/datamodules.rst Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Update docs/api/datasets.rst Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Update docs/api/geo_datasets.csv Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Update torchgeo/datasets/l7irish.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Update torchgeo/datasets/l7irish.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Resolved minor issues in l7irish.py Added mask assert in test_l7irish.py * Improved _getitem and plot functions * Added new artificial data with 5 scenes Updated test_l7irish.py * remove comments in l7irish.py * resolve black, flake8, and isort errors * add l7irish.yaml and refine test_segmentation.py * modified l7irish.yaml * revert a change in .gitignore * add function test_rgb_bands_absent_plot() * resolve black test issue * Update torchgeo/datasets/l7irish.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Update torchgeo/datasets/l7irish.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Updaye l7irish.py and create new test data * update l7irish.py for style tests * remove old test data * Update tests/data/l7irish/data.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Update torchgeo/datasets/l7irish.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * update data.py and l7irish.py * update md5s, citations, masks, and thermal bands * update mask mapping * update formatting * update mask path * Update torchgeo/datasets/l7irish.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Update torchgeo/datasets/l7irish.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Update tests/data/l7irish/data.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Update docs/api/geo_datasets.csv Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Update tests/conf/l7irish.yaml Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * resolve issues from comments * Update L7 Irish link Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * update mask data generation and review changes * Fix checksums --------- Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

github-actions bot added the datasets Geospatial or benchmark datasets label Mar 26, 2023

adamjstewart requested changes Mar 26, 2023

View reviewed changes

adamjstewart changed the title ~~Landsat 7 Irisih: add new dataset/datamodule~~ L7 Irish: add new dataset/datamodule Mar 26, 2023

adamjstewart added this to In progress in SSL4EO-L via automation Mar 26, 2023

adamjstewart reviewed Mar 27, 2023

View reviewed changes

torchgeo/datasets/l7irish.py Outdated Show resolved Hide resolved

adamjstewart reviewed Mar 27, 2023

View reviewed changes

torchgeo/datasets/l7irish.py Outdated Show resolved Hide resolved

adamjstewart added this to the 0.5.0 milestone Mar 28, 2023

adamjstewart reviewed Mar 28, 2023

View reviewed changes

torchgeo/datasets/l7irish.py Show resolved Hide resolved

yichiac marked this pull request as ready for review March 28, 2023 03:05

# Changes to be committed:

1211d33

# deleted: .DS_Store # deleted: torchgeo/.DS_Store # # Changes not staged for commit: # modified: .gitignore # modified: torchgeo/datasets/l7irish.py

yichiac dismissed a stale review via 1211d33 March 28, 2023 03:24

# Changes to be committed:

9963dba

# modified: .gitignore # modified: torchgeo/datasets/l7irish.py

adamjstewart requested changes Mar 28, 2023

View reviewed changes

add: test_l7irish.py and data.py

9e22c85

modify: .gitignore and l7irish.py

github-actions bot added the testing Continuous integration testing label Apr 1, 2023

# Changes to be committed:

ad8bfa8

# modified: .gitignore # new file: tests/data/l7irish/data.py # modified: torchgeo/datasets/l7irish.py

adamjstewart requested changes Apr 1, 2023

View reviewed changes

yichiac and others added 6 commits April 1, 2023 13:01

modified: tests/datasets/test_l7irish.py,

ba3c8c6

torchgeo/datasets/__init__.py, torchgeo/datasets/l7irish.py

added data.py, austral.tar.gz, test_l7irish.py

63d29a3

modified l7irish.py

remove comments in test_l7irish.py

de91df1

Merge branch 'main' into main

6d65ec7

resolve black and flake8 issues

56dce86

Fixed _getitem

00e91f9

Added l7irish to __init__ Added L7Irish details to geo_datasets.csv Will work on datamodules/l7irish.py

github-actions bot added datamodules PyTorch Lightning datamodules documentation Improvements or additions to documentation labels Apr 4, 2023

yichiac added 3 commits April 4, 2023 21:12

Added L7 Irish datamodule

0b54298

fix flake8 space error

6f0c2b6

fix black test error

fc9c788

update data.py and l7irish.py

9b6c868

yichiac requested a review from adamjstewart April 7, 2023 17:29

yichiac and others added 7 commits April 11, 2023 10:03

update md5s, citations, masks, and thermal bands

340d8b6

update mask mapping

8cb6480

update formatting

4bfa372

update mask path

b4e5d17

Merge branch 'main' into datasets/l7irish

eb78c37

Merge branch 'main' into datasets/l7irish

e23ae78

Merge branch 'main' into datasets/l7irish

d4fa226

adamjstewart requested changes Apr 12, 2023

View reviewed changes

yichiac and others added 7 commits April 11, 2023 21:56

Merge branch 'main' into datasets/l7irish

65ea08c

Update torchgeo/datasets/l7irish.py

6b38751

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Update torchgeo/datasets/l7irish.py

d7806ba

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Update tests/data/l7irish/data.py

64353ce

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Update docs/api/geo_datasets.csv

50db19d

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Update tests/conf/l7irish.yaml

9dba933

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

resolve issues from comments

cfa20d3

adamjstewart requested changes Apr 12, 2023

View reviewed changes

yichiac and others added 5 commits April 12, 2023 10:32

Merge branch 'main' into datasets/l7irish

79cc501

Update L7 Irish link

fe42927

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Merge branch 'main' into datasets/l7irish

01e1e80

update mask data generation and review changes

11149db

Merge branch 'main' into datasets/l7irish

796a607

adamjstewart requested changes Apr 12, 2023

View reviewed changes

torchgeo/datasets/l7irish.py Outdated Show resolved Hide resolved

Fix checksums

1107eaf

adamjstewart approved these changes Apr 12, 2023

View reviewed changes

adamjstewart merged commit a0131ac into microsoft:main Apr 12, 2023
20 checks passed

SSL4EO-L automation moved this from In progress to Done Apr 12, 2023

yichiac deleted the datasets/l7irish branch February 24, 2024 14:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

L7 Irish: add new dataset/datamodule #1197

L7 Irish: add new dataset/datamodule #1197

yichiac commented Mar 26, 2023 •

edited

adamjstewart left a comment

adamjstewart commented Mar 26, 2023

adamjstewart left a comment

adamjstewart left a comment

L7 Irish: add new dataset/datamodule #1197

L7 Irish: add new dataset/datamodule #1197

Conversation

yichiac commented Mar 26, 2023 • edited

adamjstewart left a comment

Choose a reason for hiding this comment

adamjstewart commented Mar 26, 2023

adamjstewart left a comment

Choose a reason for hiding this comment

adamjstewart left a comment

Choose a reason for hiding this comment

yichiac commented Mar 26, 2023 •

edited