Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CDA direct file access #134

Merged
merged 14 commits into from
May 30, 2024
Merged

CDA direct file access #134

merged 14 commits into from
May 30, 2024

Conversation

jeandet
Copy link
Member

@jeandet jeandet commented May 14, 2024

This PR adds to Speasy the ability to automatically find CDF files on the CDA archive based on information given in this file. This allows to reduce the latency (no WS involved) but could imply to download more data than needed (each CDF has many variables).

The CDAWeb module can now work in 3 different modes:

  • BEST -> let Speasy choose between using the REST API or directly download files from archive based on some future heuristics
  • API -> always use the REST API
  • FILE -> directly download cdf files when possible. It automatically fallback to API when files are not CDF or when the variable is virtual (generated on the fly by the REST API)

The current limitations are:

  • while CDF files are cached, Speasy still needs to know if they changed on the server, this takes ~100ms. As a workaround Speasy will only check each 24h.
  • not all CDF are ISTP compliant so we overlay the master cdf given by the CDA which are compliant on top of each individual CDF to merge them.

Impact on Speasy dependencies:

  • we depend on PyISTP>=0.5.0 to support this overlay mechanism.
  • I found at least one CDF written with the 2.4 CDF distribution so I had to implement support for CDF<2.5 into CDF++ and release the 0.7.0 version.
  • at some point before the 1.3.0 release this has to work with the Speasy proxy

Unrelated fixes:

  • SSCWeb module was silently broken since last update, the trajectory file loader didn't convert values from string to float...

@jeandet jeandet requested a review from brenard-irap May 14, 2024 21:07
@jeandet jeandet added the enhancement New feature or request label May 14, 2024
@jeandet jeandet added this to the 1.3.0 milestone May 14, 2024
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
… a simple scalar

When the result is a scalar we don't wrap it back into a SpeasyVariable

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
…e and there is space

Only caching the variable for 24H was not a good idea since CDF files on
remote archives don't change much, CDF++ is fast and we download several
times the same CDF for each variables if we request several vars from
the same dataset.

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
…verter

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
…t CDF files

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
…he data

as much as possible

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
…trings...

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>
@jeandet jeandet marked this pull request as ready for review May 23, 2024 10:33
@jeandet jeandet merged commit c3bc411 into SciQLop:main May 30, 2024
17 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants