str_cat with Pandas .str.cat() interface
#1496
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 30 commits
April 11, 2016 10:38
Ad-hoc testing with postgres. Still need automated tests.
- input error checking in str_cat - clean up compute_up() functions
GH 1476 _child attribute had special meaning in blaze. For this function, we only need a 'lhs' and 'rhs' argument so replace '_child' and 'col' with 'lhs' and 'rhs' where the data in 'rhs' concatenates to the data in 'lhs' elementwise.
GH 1476 also raise an exception if 'sep' kwarg is not a string
GH 1476 Table needs at least two string columns for good test so added a new table 'accounts2' for this test.
Reformat the example in docstring to see if that fixes the error thrown by doctest.
- use existing DataFrame (dfbig) for previous tests - add a fixture to append row will null values and test str_cat()
GH 1476 If a row in either column has a NULL, str_cat() returns None which is consistent with pandas for na_rep kwarg set to None; current default behavior. todo: add na_rep as a kwarg.
GH 1476 - if lhs or rhs arguments contain Nulls, then output of str_cat() will also contain Null values.
GH 1476 Since str_cat() is a binary operation, we should be able to chain it to concat more than one String columns - similar to pandas.
The table/DataFrame needs three string columns so updated test variables as well.
- Handles the cases where str_cat operates on expression with WHERE
clause: eg:
s = symbol('s', ds2)
t = s[s.amount <= 200]
c1 = t.comment.str_cat(t.sex, sep=' -- ')
- pulled out manipuating data arrays into separate function
- added a test
- chaining str_cat() with WHERE clause is not consistent with pandas yet
- also unable to add (Select, Select) in existing compute_up decorator - it fails
The exception raised when trying to concat string columns from different tables is not yet implemented fully for all use cases in the refactored code. For now, marked a test as xfail.
| @@ -1185,7 +1188,7 @@ def compute_up(t, s, **kwargs): | |||
| string_func_names = { | |||
| # <blaze function name>: <SQL function name> | |||
| 'str_upper': 'upper', | |||
| 'str_lower': 'lower', | |||
| 'str_lower': 'lower' | |||
There was a problem hiding this comment.
If we are making a style edit, should this just be:
string_func_names = {'str_upper': upper,
'str_lower': lower}Uses `reconstruct_select()` properly.
|
Thanks for the pointers @llllllllll -- |
| @pytest.mark.parametrize("expr", | ||
| [t.name.str_cat(t_str_cat.comment), | ||
| t.name.str_cat(t_str_cat.comment.str_cat(t_str_cat.name))]) | ||
| def test_str_cat_runtime_exception(expr): |
There was a problem hiding this comment.
Are we checking that the str_cat operates on the same table? If not, then we should take this test out - I saw a comment that we need to do this check more consistently throughout other blaze operations so perhaps remove the test?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
.str_cat().str_cat()rolls up concatenation ops in SQL query..str.cat()when.straccessor namespace is implemented.ping @sandhujasmine