Add azure blob storage filesystem/staging destination #592

steinitzu · 2023-08-26T22:01:33Z

Resolves #560

netlify · 2023-08-26T22:02:07Z

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	`24e4c2e`
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/64f0db279ebce400087cf4d6

rudolfix

just a few comments. please check

dlt/common/configuration/specs/azure_credentials.py

rudolfix · 2023-08-27T16:39:03Z

dlt/common/configuration/specs/azure_credentials.py

+
+    def to_adlfs_credentials(self) -> Dict[str, Any]:
+        """Return a dict that can be passed as kwargs to adlfs"""
+        return dict(


adlfs does not need this token. (maybe it does that itself?)

fs = fsspec.filesystem("adl", account_name=adls_account_name, account_key=adls_account_key) print(fs.ls("dlt-ci-test-bucket/"))

works for me

So the SAS is needed for snowflake when not using named stage https://github.com/dlt-hub/dlt/pull/592/files#diff-577441894b0e65b848310ea282c52bab00b6f649eb2c1b9402a5423d70e4d3e4R84
Otherwise we don't need it

dlt/common/configuration/specs/azure_credentials.py

dlt/destinations/filesystem/configuration.py

dlt/destinations/filesystem/filesystem_client.py

pyproject.toml

rudolfix · 2023-08-27T16:43:56Z

@sh-rp please take a look

if tests setup is OK
please add required secrets to CI (check 1password)
please setup required Snowflake stage
update documentation if needed

thanks!

rudolfix · 2023-08-27T16:45:53Z

@mjducutoae so blob storage will be available in alpha most probably on Tuesday. you can take a look at the implementation here. I think it is fairly standard way of using adlfs

tests/load/filesystem/test_azure_credentials.py

sh-rp · 2023-08-30T09:28:57Z

@steinitzu: when providing the AZ credentials I am getting

dlt.destinations.exceptions.DatabaseTerminalException: 091003 (22000): 01aea694-3201-ee94-0002-0a0a026d6e8e: Failure using stage area. Cause: [The specifed resource name contains invalid characters. (Status Code: 400; Error Code: InvalidResourceName)]

when running "pytest tests/load/pipeline/test_stage_loading.py::test_all_data_types" with the "az-authorization" destination config. Can you see this locally too? I can't quite figure out which resource name is invalid.

sh-rp · 2023-08-30T12:18:10Z

I have set up the az integration on snowflake and this seems to work, we only have to figure out this error for direct authentication.

steinitzu · 2023-08-30T13:38:12Z

@sh-rp my bad. I was stripping out the bucket name when generating the snowflake compatible URL. Fixed in last commit, runs for me locally now.

sh-rp · 2023-08-30T16:21:30Z

@steinitzu perfect, this works now. There is one failing test remaining which is:

pytest tests/load/pipeline/test_filesystem_pipeline.py::test_pipeline_merge_write_disposition

I am getting

('Failed to remove %s for %s', ['dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/_dlt_pipeline_state', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/_dlt_pipeline_state/', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/_dlt_pipeline_state/1693412282.432455.178b58ce42.jsonl', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/_dlt_pipeline_state/1693412284.13063.fdec11df2b.jsonl', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/other_data', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/other_data/', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/other_data/1693412284.13063.a5a78fdcf1.jsonl', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/some_data', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/some_data/', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/some_data/1693412284.13063.f72071cafb.jsonl', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/some_source._dlt_loads.1693412282.432455', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/some_source._dlt_loads.1693412283.199035', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/some_source._dlt_loads.1693412284.13063'], ResourceExistsError('This operation is not permitted on a non-empty directory.\nRequestId:d7e03825-801e-002d-305d-db4ce4000000\nTime:2023-08-30T16:18:06.3619677Z\nErrorCode:DirectoryIsNotEmpty'))

On the console for that, so maybe you're deleting a non empty dir which fails or something like that.

sh-rp · 2023-08-30T16:22:02Z

One additional thing, azure is super verbose on the console even if the tests do not fail, maybe there is a way to limit that a bit?

steinitzu · 2023-08-30T20:09:26Z

@steinitzu perfect, this works now. There is one failing test remaining which is:

pytest tests/load/pipeline/test_filesystem_pipeline.py::test_pipeline_merge_write_disposition

I am getting

('Failed to remove %s for %s', ['dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/_dlt_pipeline_state', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/_dlt_pipeline_state/', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/_dlt_pipeline_state/1693412282.432455.178b58ce42.jsonl', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/_dlt_pipeline_state/1693412284.13063.fdec11df2b.jsonl', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/other_data', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/other_data/', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/other_data/1693412284.13063.a5a78fdcf1.jsonl', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/some_data', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/some_data/', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/some_data/1693412284.13063.f72071cafb.jsonl', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/some_source._dlt_loads.1693412282.432455', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/some_source._dlt_loads.1693412283.199035', 'dlt-ci-test-bucket/test_23db9bc8501f4dc572e0c89497831423/some_source._dlt_loads.1693412284.13063'], ResourceExistsError('This operation is not permitted on a non-empty directory.\nRequestId:d7e03825-801e-002d-305d-db4ce4000000\nTime:2023-08-30T16:18:06.3619677Z\nErrorCode:DirectoryIsNotEmpty'))

On the console for that, so maybe you're deleting a non empty dir which fails or something like that.

I couldn't replicate this specific error. But I was getting fail in this test because of the listings cache, so I disabled this for all fs clients f47473d
Hopefully everything works now 🙏

steinitzu · 2023-08-30T20:17:18Z

One additional thing, azure is super verbose on the console even if the tests do not fail, maybe there is a way to limit that a bit?

The root logger is always set to INFO. I can't figure out why, happens for me regardless of what's set in dlt config.
Not sure if it's dlt or some other package setting it?

This particular log can be disabled with:

log = logging.getLogger('azure.core.pipeline.policies.http_logging_policy')
log.setLevel(logging.WARNING)

But imo this is best solved by not overriding the root log level, then this verbosity is opt-in.

rudolfix · 2023-08-30T21:00:38Z

@steinitzu hmmmm we do not touch the root logger. dlt was logging to root but that was removed several months ago.
my take is that pytest is doing something. we have following setting

[pytest]
python_paths= dlt
norecursedirs= .direnv .eggs build dist
addopts= -v --showlocals --durations 10
xfail_strict= true
log_cli= 1
log_cli_level= INFO
python_files = test_*.py *_test.py

log_cli_level is probably messing up with root logger

during the tests several logs are throttled in tests/conftest.py ie

    # disable sqlfluff logging
    for log in ["sqlfluff.parser", "sqlfluff.linter", "sqlfluff.templater", "sqlfluff.lexer"]:
        logging.getLogger(log).setLevel("ERROR")

    # disable snowflake logging
    for log in ["snowflake.connector.cursor", "snowflake.connector.connection"]:
        logging.getLogger(log).setLevel("ERROR")

steinitzu · 2023-08-30T21:11:06Z

Hey @rudolfix
I'm getting this outside pytest too. Something happens when calling dlt.pipeline(), after the log level is changed:

import dlt
import logging

log = logging.getLogger()

print(log)

dlt.pipeline('my_pipe')

print(log)

output ->

<RootLogger root (WARNING)>
<RootLogger root (INFO)>

steinitzu · 2023-08-30T21:21:07Z

It's coming from airflow, just from being installed. export AIRFLOW__LOGGING__LOGGING_LEVEL=WARNING overrides it

steinitzu marked this pull request as ready for review August 27, 2023 15:49

rudolfix requested changes Aug 27, 2023

View reviewed changes

rudolfix requested a review from sh-rp August 27, 2023 16:42

rudolfix assigned sh-rp and rudolfix Aug 27, 2023

sh-rp reviewed Aug 29, 2023

View reviewed changes

tests/load/filesystem/test_azure_credentials.py Outdated Show resolved Hide resolved

steinitzu force-pushed the sthor/azure-blob branch from 1a3e42d to 284dc5f Compare August 29, 2023 12:42

steinitzu added 12 commits August 30, 2023 18:31

Add azure fsspec dependency

81c2f1b

Add azure blob storage filesystem/staging destination

dfa3c58

Add az in test drivers

4a5a76f

Add docs for azure blob storage

c15e683

Test az snowflake stage with storage integration and explicit creds

389d90b

Test azure credentials

787ae16

Add abfs alias

f65b910

Add sas token permission config

fede7bc

Match adlfs dependency version

f7e729a

Use module level pytest skip

881fbe0

Update lockfile

e389aad

Fix az bucket url for snowflake

d8dbc1f

sh-rp force-pushed the sthor/azure-blob branch from 437131e to d8dbc1f Compare August 30, 2023 16:31

Disable filesystem listings cache

f47473d

rudolfix added 2 commits August 31, 2023 17:01

Merge branch 'devel' into sthor/azure-blob

864f696

runs airflow tests on 2.7.0

410b1ed

rudolfix force-pushed the sthor/azure-blob branch from 6797065 to 410b1ed Compare August 31, 2023 18:08

stops weaviate cluster tests

24e4c2e

rudolfix approved these changes Sep 1, 2023

View reviewed changes

rudolfix merged commit c945650 into devel Sep 1, 2023
36 checks passed

rudolfix deleted the sthor/azure-blob branch September 1, 2023 06:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add azure blob storage filesystem/staging destination #592

Add azure blob storage filesystem/staging destination #592

steinitzu commented Aug 26, 2023

netlify bot commented Aug 26, 2023 •

edited

Loading

rudolfix left a comment

rudolfix Aug 27, 2023

steinitzu Aug 27, 2023

rudolfix commented Aug 27, 2023

rudolfix commented Aug 27, 2023

sh-rp commented Aug 30, 2023

sh-rp commented Aug 30, 2023

steinitzu commented Aug 30, 2023

sh-rp commented Aug 30, 2023

sh-rp commented Aug 30, 2023

steinitzu commented Aug 30, 2023

steinitzu commented Aug 30, 2023

rudolfix commented Aug 30, 2023

steinitzu commented Aug 30, 2023

steinitzu commented Aug 30, 2023

Add azure blob storage filesystem/staging destination #592

Add azure blob storage filesystem/staging destination #592

Conversation

steinitzu commented Aug 26, 2023

netlify bot commented Aug 26, 2023 • edited Loading

✅ Deploy Preview for dlt-hub-docs canceled.

rudolfix left a comment

Choose a reason for hiding this comment

rudolfix Aug 27, 2023

Choose a reason for hiding this comment

steinitzu Aug 27, 2023

Choose a reason for hiding this comment

rudolfix commented Aug 27, 2023

rudolfix commented Aug 27, 2023

sh-rp commented Aug 30, 2023

sh-rp commented Aug 30, 2023

steinitzu commented Aug 30, 2023

sh-rp commented Aug 30, 2023

sh-rp commented Aug 30, 2023

steinitzu commented Aug 30, 2023

steinitzu commented Aug 30, 2023

rudolfix commented Aug 30, 2023

steinitzu commented Aug 30, 2023

steinitzu commented Aug 30, 2023

netlify bot commented Aug 26, 2023 •

edited

Loading