Enhancements in Filesystem Configuration #869
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
…nation factory Updated FilesystemConfiguration's __init__ method to accept two additional optional parameters: kwargs and client_kwargs. This will provide greater flexibility in configuration, facilitating custom arguments that can be passed from the Filesystem configuration. Tests are adapted to reflect these changes, and further modifications were made to implement these new parameters within fsspec_filesystem.py.
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
) Refactored common storage configuration and revised imports in line with PEP rules. Also updated the string formatting in filesystem related tests and moved `test_s3_wrong_certificate` method to `test_filesystem_client.py`. There’s an addition of `kwargs` and `client_kwargs` parameters to `fsspec_filesystem` function allowing for a more configurable function. Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
…ith linting (#814) Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Updated the test case for S3 SSL certificate verification to use a self-signed certificate generated during the test. This change ensures the test is more reliable, as it doesn't depend on external, possibly changing, data. Improved the test setup by creating and using a temporary certificate file, which gets cleaned up after use. Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Updated the filesystem.md documentation to improve clarity, fix grammar mistakes, and ensure better readability. The changes include breaking up longer sentences, correcting spellings and punctuations. (#814) Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
…rguments (#814) Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
|
Documentation on usage added to filesystem destination |
sh-rp
requested changes
Jan 2, 2024
Pipboyguy
added
documentation
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
This commit updates the handling of filesystem and credential configuration for different cloud storage protocols. Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>
sh-rp
approved these changes
Jan 5, 2024
This was referenced Jan 10, 2024
deanja
added a commit
to deanja/dlt-verified-sources
that referenced
this pull request
Jan 16, 2024
kwargs or client_kwargs can be passed when constructing fsspec filesystem instances. See dlt-hub/dlt#869
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Re-submission of Enhancements in Filesystem Configuration after changing default branch (see #856 (review))