-
Notifications
You must be signed in to change notification settings - Fork 722
Major - Make path argument in wr.s3.to functions optional #598
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…ption if existing table path is different to input
jaidisido
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Clarified some code sections
| df = pd.DataFrame({"c1": [None, 1, None]}, dtype="Int16") | ||
| wr.s3.to_parquet( | ||
| df=df, | ||
| path=path, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since path is now optional, it was removed from some tests to check that they still pass even without referencing it
| def test_missing_or_wrong_path(path, glue_database, glue_table): | ||
| # Missing path | ||
| df = pd.DataFrame({"FooBoo": [1, 2, 3]}) | ||
| with pytest.raises(wr.exceptions.InvalidArgumentValue): | ||
| wr.s3.to_parquet(df=df) | ||
| with pytest.raises(wr.exceptions.InvalidArgumentCombination): | ||
| wr.s3.to_parquet(df=df, dataset=True) | ||
| with pytest.raises(wr.exceptions.InvalidArgumentValue): | ||
| wr.s3.to_parquet(df=df, dataset=True, database=glue_database, table=glue_table) | ||
|
|
||
| # Wrong path | ||
| wr.s3.to_parquet(df=df, path=path, dataset=True, database=glue_database, table=glue_table) | ||
| wrong_path = "s3://bucket/prefix" | ||
| with pytest.raises(wr.exceptions.InvalidArgumentValue): | ||
| wr.s3.to_parquet(df=df, path=wrong_path, dataset=True, database=glue_database, table=glue_table) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In certain cases, Wrangler must throw an error if the path is missing or wrong
igorborgest
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Issue #, if available:
#586
Description of changes:
pathargument is now optional. The code changes implementing this feature are based on the below logic tree:Added and modified
s3tests to evaluate impactBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.