Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix cache issue with ArgoIndex #346

Merged
merged 12 commits into from
Apr 18, 2024
Merged

Fix cache issue with ArgoIndex #346

merged 12 commits into from
Apr 18, 2024

Conversation

gmaze
Copy link
Member

@gmaze gmaze commented Apr 16, 2024

closes #345

@gmaze gmaze self-assigned this Apr 16, 2024
@gmaze gmaze linked an issue Apr 16, 2024 that may be closed by this pull request
Cache is not working with the ArgoIndex #345
Closed
@gmaze
Copy link
Member Author

gmaze commented Apr 16, 2024

Managed to reproduce the error using only fsspec methods and raised the issue:
fsspec/filesystem_spec#1575

gmaze added 10 commits April 16, 2024 16:45
@gmaze
blabla
cd7b91f
@gmaze
all that was missing was this f* save_cache !
42f935b
@gmaze
cleaner load methods
2bfa016 
- fix bug where by nrows in the index was overwriten by that of a search
@gmaze
More fix
3c842b5
@gmaze
Update mocked_http.py
f3ff09d 
- add server home page with content listing (otherwise "exists" with a cached http fsspec store won't work)
@gmaze
other minor improvement to mocked servers
ad2f8b8
@gmaze
Update test_fetchers_data_erddap_bgc.py
b418f3d 
- fix bug whereby the ArgoIndex was always cached and not following tests arguments
@gmaze
Update caching.py
f6b67f8 
- new option to lscache to handl errors
@gmaze
pinned upstream to xarray 2024.2.0
564eab6 
because of pydata/xarray#8909 (comment)
@gmaze
Improve ArgoIndex.load design
73427f6
@gmaze
Copy link
Member Author

gmaze commented Apr 18, 2024

I think I also found another issue leading to basically the same error:

When using a cached http fsspec store (a WholeFileCacheFileSystem on top of a HTTPFileSystem), then checking for the existence of a binary file fails if the parent web page does not link back to the existing binary file

This seems to be due to the fact that the HTTPFileSystem raises a UnicodeDecodeError when doing ls on a binary file, the list of links is empty and _ls_real return []

@gmaze gmaze merged commit cdfe8d4 into master Apr 18, 2024
8 of 9 checks passed
@gmaze gmaze deleted the fix-cache branch April 18, 2024 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@gmaze gmaze

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Cache is not working with the ArgoIndex
1 participant
@gmaze