Cdaweb direct file access #89

jeandet · 2023-05-04T15:01:19Z

Big PR, mostly adding direct archive support plus some refactoring around http module.
Direct Archive support means, Speasy can access servers with no webservices or any local folder as long as:

remote server enables mod_dir or generates some html files for each folder where Speasy can extract the file list
files are split regularly or at least start date can be parsed from file names
file are ISTP compliant CDFs

Most interesting parts for review are:

archive module itself here and here for downloaders strategies
former HTTP module has been split into:
- New AnyFiles module, it abstracts access to a file either locally or remotely for speasy with opt-in caching for remote files.
- Reworked HTTP module that focuses on http transactions with retry and timeouts.
- New URL utils module that focuses on URL manipulation and parsing
AMDA and CDAWeb have been refactored to use new HTTP AnyFiles an URL modules
Test are here:

The module is documented here and human readable here.

This PR also adds PyYAML as new dependency.

speasy/webservices/generic_archive/__init__.py

speasy/core/file_access.py

speasy/core/any_files.py

speasy/core/http.py


 RETRY_AFTER_LIST = [429, 503]  # Note: Specific treatment for 429 & 503 error codes (see below)

+_HREF_REGEX = re.compile(' href="([A-Za-z0-9.-_]+)">')


speasy/webservices/amda/_impl.py

@@ -10,9 +10,9 @@
 from .utils import load_catalog, load_csv, load_timetable
 # General modules
 from ...config import amda as amda_cfg
+from ...core.any_files import any_loc_open


speasy/core/http.py


 RETRY_AFTER_LIST = [429, 503]  # Note: Specific treatment for 429 & 503 error codes (see below)

+_HREF_REGEX = re.compile(' href="([A-Za-z0-9.-_]+)">')


speasy/webservices/csa/__init__.py

speasy/core/cdf/__init__.py

speasy/core/any_files.py

speasy/webservices/csa/__init__.py

speasy/core/any_files.py

speasy/webservices/csa/__init__.py

@@ -104,13 +105,13 @@
    return root


-def _read_cdf(response: requests.Response, variable: str) -> SpeasyVariable:
-    with tarfile.open(fileobj=BytesIO(response.content)) as tar:
+def _load_variable(archive: io.BytesIO, variable: str) -> SpeasyVariable:


speasy/core/any_files.py

speasy/core/direct_archive_downloader/direct_archive_downloader.py

tests/test_url_utils.py

+# -*- coding: utf-8 -*-
+
+"""Tests for `speasy.common` package."""
+import re


speasy/core/url_utils.py

tests/test_url_utils.py

+
+from speasy.core.url_utils import ensure_url_scheme, is_local_file
+
+_HERE_ = os.path.dirname(os.path.abspath(__file__))


This reduces a lot functions signature size... Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

…tring Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

…y extractor Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

… and remote files + regex filter and file list for archive module Since cda archive datasets have different cdf versions across the same year, this PR adds the capability to list files from remote location parsing html pages such as apache mod_dir generated ones. This allows to give some predictable url base with a small regex part to filter matching files from either remote or local location. Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

codecov · 2023-06-28T09:15:53Z

Codecov Report

Patch coverage: 88.32% and project coverage change: +2.62 🎉

Comparison is base (3187bdb) 83.86% compared to head (1df08de) 86.49%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #89      +/-   ##
==========================================
+ Coverage   83.86%   86.49%   +2.62%     
==========================================
  Files          45       51       +6     
  Lines        2771     3043     +272     
  Branches      437      473      +36     
==========================================
+ Hits         2324     2632     +308     
+ Misses        316      276      -40     
- Partials      131      135       +4

Flag	Coverage Δ
unittests	`85.86% <88.32%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
speasy/webservices/amda/inventory.py	`80.18% <0.00%> (ø)`
speasy/webservices/amda/_impl.py	`77.67% <44.44%> (+9.53%)`	⬆️
speasy/core/cdf/__init__.py	`75.47% <57.14%> (+2.97%)`	⬆️
speasy/core/dataprovider.py	`89.61% <66.66%> (+0.13%)`	⬆️
speasy/core/inventory/indexes.py	`90.90% <75.00%> (ø)`
speasy/webservices/cda/__init__.py	`81.91% <83.33%> (-0.56%)`	⬇️
speasy/core/cdf/inventory_extractor.py	`85.71% <85.71%> (ø)`
speasy/webservices/csa/__init__.py	`87.39% <88.88%> (+2.59%)`	⬆️
speasy/core/any_files.py	`89.47% <89.47%> (ø)`
speasy/webservices/generic_archive/__init__.py	`89.55% <89.55%> (ø)`
... and 16 more

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

speasy/core/direct_archive_downloader/direct_archive_downloader.py

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

headers usage Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

jeandet · 2023-09-20T09:17:14Z

@brenard-irap can we merge this now?

jeandet added enhancement New feature or request WIP Work In Progress (don't merge) labels May 4, 2023

jeandet force-pushed the cdaweb_direct_file_access branch 2 times, most recently from 9195a9c to 385964e Compare May 23, 2023 16:19

github-advanced-security bot found potential problems May 23, 2023

View reviewed changes

speasy/webservices/generic_archive/__init__.py Fixed Show fixed Hide fixed

jeandet force-pushed the cdaweb_direct_file_access branch from 9bb8937 to 842fe48 Compare May 24, 2023 13:35

jeandet marked this pull request as ready for review May 24, 2023 13:56

github-advanced-security bot found potential problems May 24, 2023

View reviewed changes

speasy/core/file_access.py Fixed Show fixed Hide fixed

jeandet force-pushed the cdaweb_direct_file_access branch from f5cf14e to 2a535b9 Compare May 25, 2023 08:54

github-advanced-security bot found potential problems May 25, 2023

View reviewed changes

speasy/core/file_access.py Fixed Show fixed Hide fixed

speasy/core/file_access.py Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems May 25, 2023

View reviewed changes

speasy/core/file_access.py Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems May 26, 2023

View reviewed changes

jeandet force-pushed the cdaweb_direct_file_access branch 2 times, most recently from 9b35369 to 7ad431e Compare May 26, 2023 15:06

github-advanced-security bot found potential problems May 26, 2023

View reviewed changes

speasy/core/direct_archive_downloader/direct_archive_downloader.py Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems May 26, 2023

View reviewed changes

jeandet added 13 commits June 19, 2023 09:15

Type alias for functions accepting almost anything as a datetime

ed1359f

This reduces a lot functions signature size... Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Basic POC of direct file downloader

162c13b

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Forgot that SpeasyVariable slicing don't support (yet) datetimes as s…

f3efeda

…tring Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Move direct archive downloader into core since it's not exclusive to CDA

adc2203

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Adds basic bits to build ParameterIndexes from any ISTP "compliant" file

d3557a7

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Make urlopen_with_retry support local files without scheme set

a2c8ec0

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Refac CDA inventory builder to use shared code from core.cdf inventor…

77239aa

…y extractor Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

New "GenericArchive" provider WIP

ad7704e

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

[GenericArchive] Basic "uncached" get_data implementation

1d6ff3e

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

[GenericArchive] fixes typo and adds simple get_data test

62977ec

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

importlib.resources is usable with Python 3.9+ :/

318e3c8

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

fixes few misses on generic_archive integration

061dce3

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Adds caching option for remote file access

48d5af9

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

jeandet force-pushed the cdaweb_direct_file_access branch from 7a0c591 to 48d5af9 Compare June 27, 2023 15:39

jeandet added 2 commits June 28, 2023 09:48

CSA doc example get FGM data to make CI less flaky

67e45a7

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Fix broken proxy after http module refactoring 😅

8662193

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

jeandet added 4 commits June 28, 2023 11:46

Yeah on CI network is almost as fast as disks...

eacfd7f

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Allow to subclass inventory indexes

7fff9cb

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Remove dead code

392b3de

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Fixes remote files list regex

de5e088

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

github-advanced-security bot found potential problems Jun 29, 2023

View reviewed changes

speasy/core/direct_archive_downloader/direct_archive_downloader.py Fixed Show fixed Hide fixed

Archive burst data preliminary support implemented

bb1ce68

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

jeandet force-pushed the cdaweb_direct_file_access branch from c87626a to bb1ce68 Compare June 30, 2023 13:26

jeandet added this to the 1.2.0 milestone Jun 30, 2023

jeandet added 2 commits June 30, 2023 17:22

Adds split_rules test

e034f6e

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Adds Monthly split burst data tests and fixes

b1a1e8b

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

jeandet force-pushed the cdaweb_direct_file_access branch from c100877 to b1a1e8b Compare July 4, 2023 13:44

jeandet added 2 commits July 5, 2023 18:57

A bit of doc and inventory generator notebook

af19a0e

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Caches archive inventory cdf files and fixes case insensitive HTTP

256f49a

headers usage Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

jeandet force-pushed the cdaweb_direct_file_access branch from 2acbe77 to 256f49a Compare July 5, 2023 16:59

jeandet added 5 commits July 6, 2023 18:46

A bit of documentation

83ce744

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Cacches inventory indexes from CDF extractor

3e84e08

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

Documentation WIP

40f4ffc

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

fix typo in previous refac

9763a4e

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

improve direct archive module doc

1df08de

Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

jeandet requested a review from brenard-irap July 10, 2023 16:45

jeandet added the New WS label Jul 10, 2023

jeandet removed the WIP Work In Progress (don't merge) label Jul 19, 2023

brenard-irap approved these changes Sep 20, 2023

View reviewed changes

jeandet merged commit 5372fdf into SciQLop:main Sep 20, 2023
19 of 20 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cdaweb direct file access #89

Cdaweb direct file access #89

jeandet commented May 4, 2023 •

edited

codecov bot commented Jun 28, 2023 •

edited

jeandet commented Sep 20, 2023


		RETRY_AFTER_LIST = [429, 503] # Note: Specific treatment for 429 & 503 error codes (see below)

		_HREF_REGEX = re.compile(' href="([A-Za-z0-9.-_]+)">')


		from speasy.core.url_utils import ensure_url_scheme, is_local_file

		_HERE_ = os.path.dirname(os.path.abspath(__file__))

Cdaweb direct file access #89

Cdaweb direct file access #89

Conversation

jeandet commented May 4, 2023 • edited

codecov bot commented Jun 28, 2023 • edited

Codecov Report

jeandet commented Sep 20, 2023

jeandet commented May 4, 2023 •

edited

codecov bot commented Jun 28, 2023 •

edited