openlineage: add (S)FTP extractors#30888
Conversation
mobuchowski
force-pushed
the
openlineage-s-ftp-extractors
branch
from
e3f09aa to
8fd5975
Compare
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
mobuchowski
force-pushed
the
openlineage-s-ftp-extractors
branch
2 times, most recently
from
93893d4 to
97e3b59
Compare
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
mobuchowski
force-pushed
the
openlineage-s-ftp-extractors
branch
from
97e3b59 to
671d5f3
Compare
| disable_source_code: | ||
| description: | | ||
| If disabled, OpenLineage events do not contain source code of particular | ||
| operators, like PythonOperator. | ||
| default: "True" |
There was a problem hiding this comment.
Since this is disabled by default, I wonder if we should flip this into add_source_code.
| """ | ||
| :meta private: | ||
| """ |
There was a problem hiding this comment.
Why is the __init__ file marked as private? Do we not want people to import from this file (but submodules instead?) and in that case, why are the classes re-exported from submodules into this file?
There was a problem hiding this comment.
Because this is just temporary. The idea was that first extractors will be added "as is" and then they will be merged into operators. This is what @mobuchowski plan is. We will not release openlineage provider (it is currently excluded from release) until this refactoring is done.
There was a problem hiding this comment.
The question then becomes why the imports are added to this file.
| """ | ||
| :meta private: | ||
| """ |
There was a problem hiding this comment.
I believe the general advice is to put the module-level docstring above the imports instead of below.
3 participants
This PR adds (S)FTP extractors to OpenLineage provider.
As rest of currently open extractor PRs, this depends on #30713, and will be rebased as soon as it's merged.