-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor AppendOperator
#439
Conversation
3fdae0f
to
1a9caab
Compare
columns=["sell", "living"], | ||
main_table=load_main, | ||
append_table=load_append, | ||
source_to_target_columns_map={"sell": "sell", "living": "living"}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Although using a dictionary gives us more flexibility, it does look odd in the cases when the columns in the source and target tables are exactly the same. It would be great to support just a tuple with the column values, as opposed to always enforcing a dictionary. What are your thoughts, @kaxil ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree, it feels tedious and probably most of the times the col names would be the same. Thoughts on the name of the parameter? source_to_target_columns_map
doesn't seem appropriate , how about just columns
and we allow passing either a) list or tuples containing col names b) dict containing source to target mapping.
If you agree, should we do the same for merge too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both suggestions look great, @kaxil (interface + adjust merge)!
BTW: I'm fine with us doing this change in a separate PR if you preferer
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes I will tackle them separately, thanks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kaxil I just noticed coverage doesn't seem to be running for this PR - this issue was not introduced in this PR, but we probably want to restore that before merging further PRs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall LGTM. But certain tests seem to be failing.
source_table: Table, | ||
target_table: Table, | ||
source_to_target_columns_map: Optional[Dict[str, str]] = None, | ||
task_id: str = "", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is task_id
required here as this operator is inherited from BaseOperator
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mainly because of the following line:
task_id = task_id or get_unique_task_id("append_table") |
This PR refactors the Append operator to use the new interfaces. Breaking Changes: - Removed casting functionality - changed the interface from which uses `source_to_target_columns_map` instead of `columns`: ``` def append( self, main_table: Table, columns: List[str], casted_columns: dict, append_table: Table, ): ``` to ``` def append( source_table: Table, target_table: Table, source_to_target_columns_map: Optional[Dict[str, str]] = None, **kwargs, ): ``` This PR intends to keep the same logic as now and just use new interfaces so we can delete all the old files from old interfaces. The scope to whether or not we should have a single function or not is kept for #383 closes #343 closes #335 Co-Authored-By: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com> Co-Authored-By: Utkarsh Sharma <utkarsharma2@gmail.com>
addresses #439 (comment) Tt feels tedious and probably most of the times the col names would be the same. This commit renames `source_to_target_columns_map` param in Append and Merge operator to `columns` and allows passing either a) list or tuples containing col names b) dict containing source to target mapping. Before: ```python append( source_table=filtered_data, target_table=Table(name="homes_reporting", conn_id=SNOWFLAKE_CONN_ID), source_to_target_columns_map={ "sell": "sell", "list": "list", "variable": "variable", "value": "value", }, ) ``` After: ```python append( source_table=filtered_data, target_table=Table(name="homes_reporting", conn_id=SNOWFLAKE_CONN_ID), columns=["sell", "list", "variable", "value"] ) ```
addresses #439 (comment) Tt feels tedious and probably most of the times the col names would be the same. This commit renames `source_to_target_columns_map` param in Append and Merge operator to `columns` and allows passing either a) list or tuples containing col names b) dict containing source to target mapping. Before: ```python append( source_table=filtered_data, target_table=Table(name="homes_reporting", conn_id=SNOWFLAKE_CONN_ID), source_to_target_columns_map={ "sell": "sell", "list": "list", "variable": "variable", "value": "value", }, ) ``` After: ```python append( source_table=filtered_data, target_table=Table(name="homes_reporting", conn_id=SNOWFLAKE_CONN_ID), columns=["sell", "list", "variable", "value"] ) ```
addresses #439 (comment) It feels tedious and probably most of the times the col names would be the same. This commit renames `source_to_target_columns_map` param in Append and Merge operator to `columns` and allows passing either a) list or tuples containing col names b) dict containing source to target mapping. Before: ```python append( source_table=filtered_data, target_table=Table(name="homes_reporting", conn_id=SNOWFLAKE_CONN_ID), source_to_target_columns_map={ "sell": "sell", "list": "list", "variable": "variable", "value": "value", }, ) ``` After: ```python append( source_table=filtered_data, target_table=Table(name="homes_reporting", conn_id=SNOWFLAKE_CONN_ID), columns=["sell", "list", "variable", "value"] ) ```
This PR refactors the Append operator to use the new interfaces.
Breaking Changes that I would love to get some feedback on:
source_to_target_columns_map
instead ofcolumns
:to
This PR intends to keep the same logic as now and just use new interfaces so we can delete all the old files from old interfaces. The scope to whether or not we should have a single function or not is kept for #383
closes #343
closes #335
Co-Authored-By: Utkarsh Sharma utkarsharma2@gmail.com