coalesce and cast #1409

llllllllll · 2016-02-09T15:59:31Z

cast is needed when blaze is incorrect about the types of an expression. For example, the case I opened earlier about reductions of var * A being typed A instead of ?A when the collection could be empty. To make it easy to work around this we should be able to explicitly overwrite the type.

coalesce is useful for a lot of queries.

llllllllll · 2016-02-18T00:47:06Z

ping @kwmsmith I would like someone to look this one over because I am adding some new expressions.

llllllllll · 2016-02-22T20:38:19Z

freshly rebased against master

llllllllll · 2016-03-10T19:09:52Z

@kwmsmith I would like to merge this soon

llllllllll · 2016-03-11T02:20:26Z

If you don't have any comments I plan on merging by the end of tomorrow

kwmsmith · 2016-03-11T16:43:36Z

blaze/expr/expressions.py

+class Cast(Expr):
+    """Cast an expression to a different type.
+
+    This is only an expression time operation.


Can you provide a brief Examples section?

kwmsmith · 2016-03-11T17:19:23Z

@llllllllll Apologies for taking so long to get to this -- I've been pinned down on a number of internal things lately.

This looks good to me; my only question is on the names; new users may be confused on the distinction between cast and coerce, and coalesce isn't obvious at first what it does unless you're coming from SQL :)

Because of the possibly confusing or non-obvious names, it's important that we have good docstrings with good examples.

llllllllll · 2016-03-11T17:41:49Z

No worries, I just wanted to make sure didn't get forgotten. I completely agree about the naming and documentation issue. I will go through and try to add some good examples and maybe some prose docs about the difference between cast and coerce. If you have an alternative name for cast please let me know. This operation is not at all the same as sql cast which can change values, this is really more a C cast, but idk if our users will have the same intuition for what a C cast is vs a sql cast.

kwmsmith · 2016-03-11T18:02:49Z

If you have an alternative name for cast please let me know.

Taking a page from Numpy / pandas, which have astype(): what about as_dshape instead of cast? I'm trying to capture the idea that we're just changing the dshape of the expression, not the underlying data.

llllllllll · 2016-03-11T18:03:48Z

numpy astype actually does conversions though, the equivalent in numpy is view

kwmsmith · 2016-03-11T18:09:37Z

For cast, what happens if the user casts a dataframe with integer column to a dataframe with float column?

bz.compute(df.int_column.cast('float32'))

Since cast is a no-op at compute time, the result here would be an integer pandas series, correct? So in that sense, it isn't analogous to Numpy's view, since the cast is completely ignored at compute time. I don't think I can come up with any Numpy / Pandas analogue for cast here, can you?

llllllllll · 2016-03-11T18:11:52Z

I guess it is closer to view which is a C style cast. Really this is just a type level function for allowing us to use different expressions when blaze gets the type wrong. The use case I have for this is when there are bugs in the blaze type system I need to be able to fix them in my app code before I have time to fix them in blaze. For example, reductions over var returning A instead of ?A

kwmsmith · 2016-03-11T18:16:56Z

Yes, I totally understand the usecase for correcting blaze / datashape's option / non option types. That makes sense. It's for the other cases when the user tries to cast from integer to floats, for example, and it has no effect in the computed results that gets more confusing.

Is dealing with option types the only usecase here, or are there others? Because my preference would be to keep this limited to handling casts between option / non-option types of the same underlying type for now, and only expand it to more general casts if there's a clear usecase for that.

llllllllll · 2016-03-11T18:19:08Z

The use case is for any time that the blaze type system is incorrect. This was happening before with reductions on timedeltas giving back floats. This was corrected but but I would like to have the freedom to work around this class of issue in the future.

kwmsmith · 2016-03-11T18:42:02Z

I'm OK with this for the option / non-option case, of course, and I might envision other cases as well (string & string[10], for example). But if it turns out users misuse cast frequently and it leads to confusion--the int to float cast, for example, and the user expecting some change in the result--then we may need to modify the types cast is allowed to cast between.

My preference would be to only allow cast for the clear pain points, and expand it as other cases become clear in the future.

I'm OK with merging this as long as we're able to restrict the cases for cast in the future if it becomes a place where users get frequently tripped up.

Cast is a type level function that reinterpretes a node as a different type. This is like c cast, or c++ reinterpret_cast. This allows users to correct mistakes in the blaze type system. Coalesce is a binary operator much like sql coalesce. It picks the first non null element.

updates tests for coalesce

llllllllll · 2016-03-15T17:01:16Z

Added more tests, including some xfail tests around promotion of mixed shaped records which I think should work in the future.

@kwmsmith tests are passing

kwmsmith · 2016-03-15T18:01:43Z

blaze/compute/tests/test_postgresql_compute.py

+def test_coalesce(sqla):
+    t = symbol('t', discover(sqla))
+    assert (
+        odo(compute(coalesce(t.B, -1), {t: sqla}), list) ==


nit: for consistency, we've been using compute(expr, sql_data, return_type=list) for simple cases like this rather than an explicit odo call.

kwmsmith · 2016-03-15T18:11:26Z

@llllllllll just some consistency nits, otherwise +1.

As I mentioned above, in the future, we may need to restrict cast's applicability if we find that non power-users misuse it frequently. And if we find a better name for cast, then I'm fine with a rename in the future.

Also, please make a separate issue to add sections for coerce and cast in the blaze docs.

llllllllll · 2016-03-15T18:16:14Z

I like the return_type use here, I think it reads nicely. I added these exprs to the API section, but I will add an issue to write actual docs for this

kwmsmith · 2016-03-15T18:22:56Z

LGTM.

llllllllll · 2016-03-15T18:24:20Z

Thanks for looking at this

llllllllll · 2016-03-15T18:26:36Z

merged on cli

llllllllll added expression core new expression user experience labels Feb 9, 2016

llllllllll force-pushed the c-funcs branch 3 times, most recently from 3e008de to d290fc6 Compare February 12, 2016 05:25

llllllllll mentioned this pull request Feb 12, 2016

condense the blaze query quantopian/zipline#994

Merged

llllllllll force-pushed the c-funcs branch 4 times, most recently from 6e0ca18 to 0b6e761 Compare February 18, 2016 00:25

llllllllll force-pushed the c-funcs branch from 0b6e761 to 7da76ef Compare February 22, 2016 20:37

llllllllll force-pushed the c-funcs branch from 7da76ef to b42cab5 Compare March 1, 2016 22:01

richafrank force-pushed the c-funcs branch from b42cab5 to 3142218 Compare March 4, 2016 15:38

kwmsmith reviewed Mar 11, 2016
View reviewed changes

kwmsmith added this to the 0.10 milestone Mar 11, 2016

Joe Jevnik added 3 commits March 14, 2016 19:56

TST: fix uri to use :memory:

572e3f0

DOC: Adds whatsnew and api entries

a092233

llllllllll force-pushed the c-funcs branch from cb51585 to f78e0e5 Compare March 15, 2016 00:08

Joe Jevnik added 3 commits March 14, 2016 20:08

DOC: update docs

f78e0e5

ENH: fix fields optional records

cbaa70b

updates tests for coalesce

DOC: fix doctest

1e90a0c

kwmsmith reviewed Mar 15, 2016
View reviewed changes

STY: use return_type=list instead of explicit odo

7e4b5ce

llllllllll closed this Mar 15, 2016

llllllllll deleted the c-funcs branch March 15, 2016 18:26

llllllllll mentioned this pull request Mar 15, 2016

write docs for coalesce and cast #1449

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

coalesce and cast #1409

coalesce and cast #1409

llllllllll commented Feb 9, 2016

llllllllll commented Feb 18, 2016

llllllllll commented Feb 22, 2016

llllllllll commented Mar 10, 2016

llllllllll commented Mar 11, 2016

kwmsmith Mar 11, 2016

llllllllll Mar 15, 2016

kwmsmith commented Mar 11, 2016

llllllllll commented Mar 11, 2016

kwmsmith commented Mar 11, 2016

llllllllll commented Mar 11, 2016

kwmsmith commented Mar 11, 2016

llllllllll commented Mar 11, 2016

kwmsmith commented Mar 11, 2016

llllllllll commented Mar 11, 2016

kwmsmith commented Mar 11, 2016

llllllllll commented Mar 15, 2016

kwmsmith Mar 15, 2016

kwmsmith commented Mar 15, 2016

llllllllll commented Mar 15, 2016

kwmsmith commented Mar 15, 2016

llllllllll commented Mar 15, 2016

llllllllll commented Mar 15, 2016

coalesce and cast #1409

coalesce and cast #1409

Conversation

llllllllll commented Feb 9, 2016

llllllllll commented Feb 18, 2016

llllllllll commented Feb 22, 2016

llllllllll commented Mar 10, 2016

llllllllll commented Mar 11, 2016

kwmsmith Mar 11, 2016

Choose a reason for hiding this comment

llllllllll Mar 15, 2016

Choose a reason for hiding this comment

kwmsmith commented Mar 11, 2016

llllllllll commented Mar 11, 2016

kwmsmith commented Mar 11, 2016

llllllllll commented Mar 11, 2016

kwmsmith commented Mar 11, 2016

llllllllll commented Mar 11, 2016

kwmsmith commented Mar 11, 2016

llllllllll commented Mar 11, 2016

kwmsmith commented Mar 11, 2016

llllllllll commented Mar 15, 2016

kwmsmith Mar 15, 2016

Choose a reason for hiding this comment

kwmsmith commented Mar 15, 2016

llllllllll commented Mar 15, 2016

kwmsmith commented Mar 15, 2016

llllllllll commented Mar 15, 2016

llllllllll commented Mar 15, 2016