Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSL4EO-L: add new dataset #1332

Merged
merged 5 commits into from
May 14, 2023
Merged

SSL4EO-L: add new dataset #1332

merged 5 commits into from
May 14, 2023

Conversation

adamjstewart
Copy link
Collaborator

Remaining TODOs:

  • Add data module
  • Add download/extraction support

etm_toa
tm_sr
oli_tirs_toa
oli_sr

@adamjstewart adamjstewart added this to the 0.5.0 milestone May 13, 2023
@adamjstewart adamjstewart added this to In progress in SSL4EO-L via automation May 13, 2023
@github-actions github-actions bot added datasets Geospatial or benchmark datasets documentation Improvements or additions to documentation testing Continuous integration testing labels May 13, 2023
def dataset(self, monkeypatch: MonkeyPatch, request: SubRequest) -> SSL4EOS12:
monkeypatch.setitem(
monkeypatch.setitem( # type: ignore[arg-type]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self.seasons = seasons
self.transforms = transforms

self.scenes = sorted(os.listdir(root))
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will replace this with integer indexing after we remap the folder names.

Comment on lines 228 to 230
_Metadata = TypedDict(
"_Metadata", {"filename": str, "md5": str, "bands": list[str]}
)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TypedDict allows us to remove all uses of Any and cast! We should try to use this in more places.

@@ -121,7 +298,7 @@ def __init__(
RuntimeError: if dataset is missing or checksum fails
"""
assert split in self.metadata
assert seasons in range(4)
assert seasons in range(1, 5)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a bug, oops

@github-actions github-actions bot added the datamodules PyTorch Lightning datamodules label May 13, 2023
@adamjstewart adamjstewart merged commit d7773a6 into main May 14, 2023
18 checks passed
SSL4EO-L automation moved this from In progress to Done May 14, 2023
@adamjstewart adamjstewart deleted the datasets/ssl4eo-l branch May 14, 2023 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datamodules PyTorch Lightning datamodules datasets Geospatial or benchmark datasets documentation Improvements or additions to documentation testing Continuous integration testing
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

None yet

2 participants