[dagster-deltalake] Fix GcsConfig() ImportError and TypeError for partitioned assets #22054
Conversation
|
Hi @ion-elgreco - I see that you last worked on this section of code, and wanted to get your input. Does this look good to you? |
|
@cmpadden the config one is fine, but changing the expression param in the filter seems to not solve anything. Pyarrow11+ supports the filter method, and they all use expression as a parameter. But I would say these are the least of the issues, there are many issues and missing implementations in the current io manager. I tried upstreaming this a while ago but there was no traction, so I now internally maintain the improved versions |
|
Thanks for the insight, @ion-elgreco ; sorry to hear that getting it upstream has been difficult. @thmswt the build kite tests pass, however it looks like |
|
@cmpadden imports are sorted now as suggested by @ion-elgreco thanks for your input. Tested it with Pyarrow12+. Version 16.1.0 works with expression param though. |
…titioned assets (dagster-io#22054) ## Summary & Motivation Closes [[dagster-deltalake] GcsConfig ImportError and TypeError for partitioned assets dagster-io#22053](dagster-io#22053) ## How I Tested These Changes `python -m pytest python_modules/libraries/dagster-deltalake/dagster_deltalake_tests`
Summary & Motivation
Closes [dagster-deltalake] GcsConfig ImportError and TypeError for partitioned assets #22053
How I Tested These Changes
python -m pytest python_modules/libraries/dagster-deltalake/dagster_deltalake_tests