-
Notifications
You must be signed in to change notification settings - Fork 16
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adding
reader_options
kwargs to open_virtual_dataset. (#67)
* adding reader_options kwargs to open_virtual_dataset * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix typing * modifies _automatically_determine_filetype to open file with fsspec to allows for reading of cloud storage * using UPath to get file protocol and open with fsspec * tests passing locally. Reading over s3/local w+w/o indexes & guessing filetypes * add s3fs to test * typing school 101 * anon * tying * test_anon update * anon failing * double down on storage_options * fsspec nit * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * seting s3 defaults as empty to try to appease the cruel boto3 gods * added fpath to SingleHDF5ToZarr * hardcode in empty storage opts for s3 * hardcode default + unpack test * changed reader_options defaults * updated docs install * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * changed docstring type in utils to numpy style * added TYPE_CHECKING for fsspec and s3fs mypy type hints * fixed TYPE_CHECKING import * pinned xarray to latest commit on github * re-add upath * merged w/ main * ådds section to usage * Minor formatting nit of code example in docs --------- Co-authored-by: Tom Nicholas <tom@cworthy.org> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
- Loading branch information
1 parent
b15c07b
commit 8923b8c
Showing
8 changed files
with
145 additions
and
15 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -13,4 +13,4 @@ dependencies: | |
- "sphinx_design" | ||
- "sphinx_togglebutton" | ||
- "sphinx-autodoc-typehints" | ||
- -e .. | ||
- -e "..[test]" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
from __future__ import annotations | ||
|
||
from typing import TYPE_CHECKING, Optional | ||
|
||
if TYPE_CHECKING: | ||
from fsspec.implementations.local import LocalFileOpener | ||
from s3fs.core import S3File | ||
|
||
|
||
def _fsspec_openfile_from_filepath( | ||
*, | ||
filepath: str, | ||
reader_options: Optional[dict] = { | ||
"storage_options": {"key": "", "secret": "", "anon": True} | ||
}, | ||
) -> S3File | LocalFileOpener: | ||
"""Converts input filepath to fsspec openfile object. | ||
Parameters | ||
---------- | ||
filepath : str | ||
Input filepath | ||
reader_options : _type_, optional | ||
Dict containing kwargs to pass to file opener, by default {'storage_options':{'key':'', 'secret':'', 'anon':True}} | ||
Returns | ||
------- | ||
S3File | LocalFileOpener | ||
Either S3File or LocalFileOpener, depending on which protocol was supplied. | ||
Raises | ||
------ | ||
NotImplementedError | ||
Raises a Not Implemented Error if filepath protocol is not supported. | ||
""" | ||
|
||
import fsspec | ||
from upath import UPath | ||
|
||
universal_filepath = UPath(filepath) | ||
protocol = universal_filepath.protocol | ||
|
||
if protocol == "": | ||
fpath = fsspec.open(filepath, "rb").open() | ||
|
||
elif protocol in ["s3"]: | ||
s3_anon_defaults = {"key": "", "secret": "", "anon": True} | ||
if not bool(reader_options): | ||
storage_options = s3_anon_defaults | ||
|
||
else: | ||
storage_options = reader_options.get("storage_options") # type: ignore | ||
|
||
# using dict merge operator to add in defaults if keys are not specified | ||
storage_options = s3_anon_defaults | storage_options | ||
|
||
fpath = fsspec.filesystem(protocol, **storage_options).open(filepath) | ||
|
||
else: | ||
raise NotImplementedError( | ||
"Only local and s3 file protocols are currently supported" | ||
) | ||
|
||
return fpath |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters