Skip to content

Improvements for dataset loading#395

Merged
ChrisJohnNOAA merged 7 commits intoERDDAP:mainfrom
ChrisJohnNOAA:dataset_loading
Nov 20, 2025
Merged

Improvements for dataset loading#395
ChrisJohnNOAA merged 7 commits intoERDDAP:mainfrom
ChrisJohnNOAA:dataset_loading

Conversation

@ChrisJohnNOAA
Copy link
Contributor

Description

Some improvements to speed up dataset loading.

  • config option to allow just reading data headers for some datasets (nc file based datasets) when the actual_range is included in the file attributes
  • config option for EDDTable datasets to generate their subset table as a task rather than a blocking part of the constructor
  • jsonlCSV optimizations
  • config option to only run the touch thread when there are items to touch
  • improved file rename and delete

Fixes #368

These improvements should help with the server load during dataset loading. Note that to get the full advantage of the benefits each variable of each data file needs to have the attribute actual_range defined (and accurate).

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Checklist before requesting a review

  • I have performed a self-review of my code
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [Coming later] I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@ChrisJohnNOAA ChrisJohnNOAA merged commit 2612a15 into ERDDAP:main Nov 20, 2025
3 checks passed
@ChrisJohnNOAA ChrisJohnNOAA deleted the dataset_loading branch February 26, 2026 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant