Use containertype to determine array type for array broadcast #19745

andreasnoack · 2016-12-28T21:09:01Z

When trying to get broadcast working for DArrays the line deleted in this PR caused some issues because it forced all broadcasting of AbstractArrays into Arrays. With the line deleted, this line is called instead and it allows for overloading containertype for DArrays.

cc: @pabloferz

pabloferz · 2016-12-29T02:51:55Z

LGTM. This shouldn't affect performance, but I'd check anyway (you cannot be too cautious here).

tkelman · 2016-12-29T02:53:54Z

should add a test so it doesn't regress again

andreasnoack · 2016-12-29T02:59:26Z

@tkelman Do you mean testing that broadcast can be extended to a new AbstractArray?

andreasnoack · 2016-12-29T03:08:22Z

@nanosoldier runbenchmarks("broadcast", vs = ":master")

nanosoldier · 2016-12-29T04:56:44Z

Your benchmark job has completed - no performance regressions were detected. A full report can be found here. cc @jrevels

tkelman · 2016-12-29T04:59:31Z

testing that broadcast can be extended to a new AbstractArray?

sure, if that's what having this line broke

tkelman · 2016-12-29T13:49:14Z

test/broadcast.jl

+    @test a .* a' == @inferred(aa .* aa')
+    @test isa(aa .+ 1, Array19745)
+    @test isa(aa .* aa', Array19745)
+


blank lines right after begin and right before end don't look very good and waste space imo, we don't format most testset blocks like this

andreasnoack · 2016-12-29T19:21:06Z

base/broadcast.jl

@@ -313,7 +301,7 @@ ziptype{T}(::Type{T}, A) = typestuple(T, A)
 ziptype{T}(::Type{T}, A, B) = (Base.@_pure_meta; Iterators.Zip2{typestuple(T, A), typestuple(T, B)})
 @inline ziptype{T}(::Type{T}, A, B, C, D...) = Iterators.Zip{typestuple(T, A), ziptype(T, B, C, D...)}

-_broadcast_type{S}(::Type{S}, f, T::Type, As...) = Base._return_type(S, typestuple(S, T, As...))
+_broadcast_type{S}(::Type{S}, f, T::Type, As...) = Base._return_type(f, typestuple(S, T, As...))


@TotalVerb I'm not sure what the purpose of the first type argument is but it seemed wrong to pass it as the first argument to _return_type and it also caused a test failure in the test at line 366 below. Does this change seem right to you?

The first argument is used to differentiate when Nullable should be treated as scalar (when S === Any) or as container (when S === Array), but should be passed as an argument only for typestuple. This was certainly a mistake.

Yes, this was a mistake.

andreasnoack · 2016-12-29T19:23:29Z

base/broadcast.jl

@@ -10,21 +10,9 @@ import Base: broadcast, broadcast!
 export bitbroadcast, dotview
 export broadcast_getindex, broadcast_setindex!

-## Broadcasting utilities ##
-
-broadcast_array_type() = Array


I think we can avoid defining these if we delete the method in line 26 and just use this method which was the origin purpose of this PR.

Yes, you can remove this, it should do no harm. But I believe the ## Broadcasting utilities ## is still useful for organizing the code.

Yep, this was an annoying special case I had to add. Thanks for addressing it!

andreasnoack · 2016-12-29T19:24:20Z

@TotalVerb This PR had conflicts with #16961 so please take a look and see if I've messed something up.

stevengj · 2016-12-29T19:58:18Z

@andreasnoack, when you link to lines in source code on github, be sure to type y so that github gives you a link tied to a particular commit. Your source-code link at the top of this PR is broken by a recent update to broadcast.jl.

andreasnoack · 2016-12-29T20:01:00Z

@stevengj I've updated the link

TotalVerb

I am a little concerned about the difficulty in getting custom array types working with nullables, but that's only tangentially related to this PR. After the 0.6 release, we will probably have to move the Nullable handling out of promote_containertype, else risk an exponential explosion of special cases for that method.

This PR looks good. I was wondering about the purpose of the AbstractArray special case. Glad to see it's not needed.

Sacha0

+1, had this change in mind as well. (The line at issue forced (1) an ad hoc reimplementation of the broadcast container-type dispatch mechanism in #16961, (2) a similar workaround in code extending sparse broadcast to structured matrices that I haven't submitted yet, and (3) specific structuring of some of the sparse broadcast methods.) Best!

tkelman · 2016-12-30T11:21:08Z

@nanosoldier runbenchmarks(ALL, vs = ":master")

KristofferC · 2016-12-30T11:22:30Z

@nanosoldier sleeping? @jrevels

andreasnoack · 2016-12-30T11:55:19Z

Maybe a conflict is a problem for the soldier? I'll try again @nanosoldier runbenchmarks(ALL, vs = ":master")

tkelman · 2016-12-30T12:03:30Z

was there one? last night's daily build didn't finish either, probably network error again?

andreasnoack · 2016-12-30T12:05:06Z

A tiny one from #19722 because it also added a test at the bottom of test/broadcast.jl.

tkelman · 2016-12-30T12:08:38Z

ah right. that wasn't there when I asked for nanosoldier an hour ago.

jrevels · 2016-12-30T13:55:51Z

It was the getaddrinfo error again, merge conflicts shouldn't be a problem for Nanosoldier.

@nanosoldier runbenchmarks(ALL, vs = ":master")

OT: I can't wait until v0.6 is released and the web stack is updated, #19699 will make it so much easier to debug network issues...

nanosoldier · 2016-12-30T17:35:45Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @jrevels

tkelman added the needs tests Unit tests are required for this change label Dec 28, 2016

andreasnoack force-pushed the anj/bc branch from 40c98c1 to 7767d8b Compare December 29, 2016 13:34

andreasnoack removed the needs tests Unit tests are required for this change label Dec 29, 2016

tkelman reviewed Dec 29, 2016

View reviewed changes

andreasnoack force-pushed the anj/bc branch 2 times, most recently from db674c3 to 30c29b0 Compare December 29, 2016 19:18

andreasnoack commented Dec 29, 2016

View reviewed changes

andreasnoack force-pushed the anj/bc branch from 30c29b0 to 79388a3 Compare December 29, 2016 20:26

stevengj approved these changes Dec 29, 2016

View reviewed changes

TotalVerb approved these changes Dec 29, 2016

View reviewed changes

Sacha0 approved these changes Dec 29, 2016

View reviewed changes

TotalVerb mentioned this pull request Dec 30, 2016

RFC: Nullables as collections #16961

Merged

tkelman mentioned this pull request Dec 30, 2016

StackOverflowError in summarysize during misc test #19707

Closed

Use containertype to determine array type for array broadcast

4d62ffe

andreasnoack force-pushed the anj/bc branch from 79388a3 to 4d62ffe Compare December 30, 2016 11:53

This was referenced Dec 30, 2016

faster String allocation #19449

Merged

RFC: Speedier, simpler and more systematic index conversions #19730

Merged

tkelman merged commit ab984a5 into master Dec 30, 2016

tkelman deleted the anj/bc branch December 30, 2016 17:37

tkelman mentioned this pull request Dec 31, 2016

LLVM 3.9 miss-compiles inlined broadcast for Array & SubArray #19792

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use containertype to determine array type for array broadcast #19745

Use containertype to determine array type for array broadcast #19745

andreasnoack commented Dec 28, 2016 •

edited

Loading

pabloferz commented Dec 29, 2016

tkelman commented Dec 29, 2016

andreasnoack commented Dec 29, 2016

andreasnoack commented Dec 29, 2016

nanosoldier commented Dec 29, 2016

tkelman commented Dec 29, 2016

tkelman Dec 29, 2016

andreasnoack Dec 29, 2016

pabloferz Dec 29, 2016

TotalVerb Dec 29, 2016

andreasnoack Dec 29, 2016

pabloferz Dec 29, 2016 •

edited

Loading

TotalVerb Dec 29, 2016

andreasnoack commented Dec 29, 2016

stevengj commented Dec 29, 2016

andreasnoack commented Dec 29, 2016

TotalVerb left a comment •

edited

Loading

Sacha0 left a comment

tkelman commented Dec 30, 2016

KristofferC commented Dec 30, 2016

andreasnoack commented Dec 30, 2016

tkelman commented Dec 30, 2016

andreasnoack commented Dec 30, 2016

tkelman commented Dec 30, 2016

jrevels commented Dec 30, 2016

nanosoldier commented Dec 30, 2016

Use containertype to determine array type for array broadcast #19745

Use containertype to determine array type for array broadcast #19745

Conversation

andreasnoack commented Dec 28, 2016 • edited Loading

pabloferz commented Dec 29, 2016

tkelman commented Dec 29, 2016

andreasnoack commented Dec 29, 2016

andreasnoack commented Dec 29, 2016

nanosoldier commented Dec 29, 2016

tkelman commented Dec 29, 2016

tkelman Dec 29, 2016

Choose a reason for hiding this comment

andreasnoack Dec 29, 2016

Choose a reason for hiding this comment

pabloferz Dec 29, 2016

Choose a reason for hiding this comment

TotalVerb Dec 29, 2016

Choose a reason for hiding this comment

andreasnoack Dec 29, 2016

Choose a reason for hiding this comment

pabloferz Dec 29, 2016 • edited Loading

Choose a reason for hiding this comment

TotalVerb Dec 29, 2016

Choose a reason for hiding this comment

andreasnoack commented Dec 29, 2016

stevengj commented Dec 29, 2016

andreasnoack commented Dec 29, 2016

TotalVerb left a comment • edited Loading

Choose a reason for hiding this comment

Sacha0 left a comment

Choose a reason for hiding this comment

tkelman commented Dec 30, 2016

KristofferC commented Dec 30, 2016

andreasnoack commented Dec 30, 2016

tkelman commented Dec 30, 2016

andreasnoack commented Dec 30, 2016

tkelman commented Dec 30, 2016

jrevels commented Dec 30, 2016

nanosoldier commented Dec 30, 2016

andreasnoack commented Dec 28, 2016 •

edited

Loading

pabloferz Dec 29, 2016 •

edited

Loading

TotalVerb left a comment •

edited

Loading