Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated xcube server's dataset configuration extraction methodology #1048

Merged

Conversation

konstntokas
Copy link
Contributor

@konstntokas konstntokas commented Jul 18, 2024

The xcube server's dataset configuration extraction methodology has been updated. When the data resource ID is provided in the Path field, xcube will attempt to access the dataset using the given ID. If wildcard patterns are used, the server will crawl through the data store to find matching data IDs. This process may result in a long setup time if the data store contains numerous data IDs. A UserWarning will be issued for the "stac" data store.

Checklist:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/source/*
  • Changes documented in CHANGES.md
  • GitHub CI passes
  • AppVeyor CI passes
  • Test coverage remains or increases (target 100%)

@konstntokas konstntokas changed the title Konstntokas xxx update xcube server dataset context for stac Updated xcube server's dataset configuration extraction methodology Jul 18, 2024
@konstntokas konstntokas marked this pull request as ready for review July 18, 2024 13:16
@konstntokas konstntokas requested a review from forman July 18, 2024 13:16
@konstntokas
Copy link
Contributor Author

I tested it with the stac configuration example (snippet shown below), I showed last week to @gunbra32 and @forman, and the setup of the server runs without any noticeable delay.

DataStores:
  - Identifier: stac
    StoreId: stac
    StoreParams:
      url: https://s3.eu-central-1.wasabisys.com/stac/odse/catalog.json
    Datasets:
      - Identifier: Example
        Path: lcv_blue_landsat.glad.ard/lcv_blue_landsat.glad.ard_1999.12.02..2000.03.20/lcv_blue_landsat.glad.ard_1999.12.02..2000.03.20.json
        Style: S2L2A

Copy link
Member

@forman forman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good! I approve, however, please remove that remaining print().

data_store.get_data_ids(data_type=MULTI_LEVEL_DATASET_TYPE),
)
for store_dataset_id in store_dataset_ids:
print(store_dataset_id)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove.

@konstntokas konstntokas force-pushed the konstntokas-xxx-update_xcube_server_dataset_context_for_stac branch from a34f4bd to 3db4e50 Compare July 22, 2024 10:24
@konstntokas konstntokas merged commit 799fe53 into main Jul 22, 2024
2 of 3 checks passed
@konstntokas konstntokas deleted the konstntokas-xxx-update_xcube_server_dataset_context_for_stac branch July 22, 2024 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants