Pair construction can bork convert #8631

Closed
timholy opened this Issue Oct 8, 2014 · 36 comments

Projects

None yet
@timholy
Member
timholy commented Oct 8, 2014

A minimal testcase for the issue reported here:

julia> immutable RGB
           r::Float32
           g::Float32
           b::Float32
       end

julia> 
WARNING: deprecated syntax "{a=>b, ...}".
Use "Dict{Any,Any}(a=>b, ...)" instead.
julia> const colormaps_sequential = {
           #single hue
           #name        hue   w     d     c     s     b     wcolor      dcolor
           "blues"   => (255, 0.3,  0.25, 0.88, 0.6,  0.75, RGB(1,1,0), RGB(0,0,1)),
           "greens"  => (120, 0.15, 0.18, 0.88, 0.55, 0.9,  RGB(1,1,0), RGB(0,0,1)),
           "grays"   => (0,   0.0,  0.0,  1.0,  0.0,  0.75, RGB(1,1,0), RGB(0,0,1)),
           "oranges" => (20,  0.5,  0.4,  0.83, 0.95, 0.85, RGB(1,1,0), RGB(1,0,0)),
           "purples" => (265, 0.15, 0.2,  0.88, 0.5,  0.7,  RGB(1,0,1), RGB(1,0,0)),
           "reds"    => (12,  0.15, 0.25, 0.8,  0.85, 0.6,  RGB(1,1,0), RGB(0.3,0.1,0.1))
       }

WARNING: deprecated syntax "{a=>b, ...}".
Use "Dict{Any,Any}(a=>b, ...)" instead.
Dict{Any,Any} with 6 entries:
  "reds"    => (12,0.15,0.25,0.8,0.85,0.6,RGB(1.0f0,1.0f0,0.0f0),RGB(0.3f0,0.1f0,0.1f0))
  "purples" => (265,0.15,0.2,0.88,0.5,0.7,RGB(1.0f0,0.0f0,1.0f0),RGB(1.0f0,0.0f0,0.0f0))
  "oranges" => (20,0.5,0.4,0.83,0.95,0.85,RGB(1.0f0,1.0f0,0.0f0),RGB(1.0f0,0.0f0,0.0f0))
  "greens"  => (120,0.15,0.18,0.88,0.55,0.9,RGB(1.0f0,1.0f0,0.0f0),RGB(0.0f0,0.0f0,1.0f0))
  "blues"   => (255,0.3,0.25,0.88,0.6,0.75,RGB(1.0f0,1.0f0,0.0f0),RGB(0.0f0,0.0f0,1.0f0))
  "grays"   => (0,0.0,0.0,1.0,0.0,0.75,RGB(1.0f0,1.0f0,0.0f0),RGB(0.0f0,0.0f0,1.0f0))

julia> 
WARNING: deprecated syntax "{a=>b, ...}".
Use "Dict{Any,Any}(a=>b, ...)" instead.
julia> const colormaps_diverging = {
           #name         h1   h2    w     d1    d2    c      s     b      wcolor      dcolor      dcolor2
           "rdbu"    => (12,  255,  0.2,  0.6,  0.0,  0.85,  0.6,  0.65,  RGB(1,1,0), RGB(1,0,0), RGB(0,0,1))
       }

WARNING: deprecated syntax "{a=>b, ...}".
Use "Dict{Any,Any}(a=>b, ...)" instead.
Dict{Any,Any} with 1 entry:
  "rdbu" => (12,255,0.2,0.6,0.0,0.85,0.6,0.65,RGB(1.0f0,1.0f0,0.0f0),RGB(1.0f0,0.0f0,0.0f0),RGB(0.0f0,0.0f0,1.0f0))

And now:

julia> convert((Int...), (uint(1),))
ERROR: `convert` has no method matching convert(::Type{Int64...}, ::Uint64)
 in convert at base.jl:9
 in convert at base.jl:17
@timholy
Member
timholy commented Oct 8, 2014

That's just a copy-paste of the maps_data.jl file from Color.

@timholy timholy changed the title from Deprecated Dict syntax can bork convert to Dict construction can bork convert Oct 8, 2014
@timholy
Member
timholy commented Oct 8, 2014

Actually, even if you use the new syntax, the error remains.

@timholy
Member
timholy commented Oct 8, 2014

An even more minimal case:

julia> immutable RGB
           r::Float32
           g::Float32
           b::Float32
       end

julia> const colormaps_sequential = Dict(
           "blues"   => (255, 0.3,  0.25, 0.88, 0.6,  0.75, RGB(1,1,0), RGB(0,0,1)))
Dict{ASCIIString,(Int64,Float64,Float64,Float64,Float64,Float64,RGB,RGB)} with 1 entry:
  "blues" => (255,0.3,0.25,0.88,0.6,0.75,RGB(1.0f0,1.0f0,0.0f0),RGB(0.0f0,0.0f0,1.0f0))

julia> convert((Int...), (uint(1),))
ERROR: `convert` has no method matching convert(::Type{Int64...}, ::Uint64)
 in convert at base.jl:9
 in convert at base.jl:17
@simonster
Member

Even more minimal:

julia> 1 => (1,1,1,1,1,1,1,1,1.,false);

julia> convert((Int...), (uint(1),))
ERROR: `convert` has no method matching convert(::Type{Int64...}, ::Uint64)
 in convert at base.jl:9
 in convert at base.jl:17
@simonster simonster changed the title from Dict construction can bork convert to Pair construction can bork convert Oct 8, 2014
@JeffBezanson
Member

Good, this is the level of breakage I was hoping to get to in 0.4-devel :) On the other hand, oh f*** there goes my week.

@timholy
Member
timholy commented Oct 8, 2014

I sympathize. Meanwhile, I'll be curious to see whether this might be the same bug that's also behind #8618, #8128 (which seems to be working now, even though #8618 is not). Those others aren't obviously pair-related, of course, so the pessimistic view is that there is more than one 😦.

@jakebolewski
Member

Playing around with this

julia> 1=>(1,1,1,1,false)
1=>(1,1,1,1,false)

julia> convert((Int...), (uint(1),))
(1,)
julia> 1=>(1,1,1,1,1,false)
1=>(1,1,1,1,1,false)

julia> convert((Int...), (uint(1),))
ERROR: `convert` has no method matching convert(::Type{Int64...}, ::Uint64)
julia> 1=>(1,1,1,1,2.0)
1=>(1,1,1,1,2.0)

julia> convert((Int...), (uint(1),))
(1,)
julia> 1=>(1,1,1,1,1,2.0)
1=>(1,1,1,1,1,2.0)

julia> convert((Int...), (uint(1),))
ERROR: `convert` has no method matching convert(::Type{Int64...}, ::Uint64)
julia> 1=>(1,1,1,1,1,2.0,1)
1=>(1,1,1,1,1,2.0,1)

julia> convert((Int...), (uint(1),))
(1,)
julia> 1=>(1,1,1,1,1,1,2.0,1)
1=>(1,1,1,1,1,1,2.0,1)

julia> convert((Int...), (uint(1),))
ERROR: `convert` has no method matching convert(::Type{Int64...}, ::Uint64)
julia> 1=>(1.0,2,3,4, false)
1=>(1.0,2,3,4,false)

julia> convert((Int...), (uint(1),))
(1,)
julia> 1=>(1.0,2,3,4,5, false)
1=>(1.0,2,3,4,5,false)

julia> convert((Int...), (uint(1),))
ERROR: `convert` has no method matching convert(::Type{Int64...}, ::Uint64)
julia> 1=>(false,1.0,2,3,4)
1=>(false,1.0,2,3,4)

julia> convert((Int...), (uint(1),))
(1,)
julia> 1=>(false,1.0,2,3,4,5)
1=>(false,1.0,2,3,4,5)

julia> convert((Int...), (uint(1),))
ERROR: `convert` has no method matching convert(::Type{Int64...}, ::Uint64)
julia> 1 => (false,1,2,3,4,5)
1=>(false,1,2,3,4,5)

julia> convert((Int...), (uint(1),))
(1,)

So it seems like it is potentially related to these magic numbers although I don't offhand know why it starts at N+1.

@jakebolewski
Member

Actually changing those numbers does not seem to impact behavior.

@garborg
Member
garborg commented Oct 9, 2014

In case it's not old news, convert is breaking for me on 0.3:

julia> aa = convert(Array{Ufixed8}, a)
ERROR: `convert` has no method matching convert(::Type{Array{UfixedBase{Uint8,8},N}}, ::Array{Float64,2})
 in convert at base.jl:13

julia> Pkg.status()
ERROR: `convert` has no method matching convert(::Type{UTF8String}, ::ASCIIString)
 in convert at base.jl:13
 in dir at pkg/git.jl:10
 in git at pkg/git.jl:16
 in success at pkg/git.jl:27
 in iscommit at pkg/git.jl:44
 in installed_version at ./pkg/read.jl:65
 in installed at ./pkg/read.jl:122
 in status at pkg/entry.jl:107
 in anonymous at pkg/dir.jl:28
 in cd at ./file.jl:20
 in cd at pkg/dir.jl:28
 in status at pkg.jl:28 (repeats 2 times)

julia> versioninfo()
Julia Version 0.3.2-pre+48
Commit bf35ce6 (2014-10-08 14:44 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Core(TM) i5-4258U CPU @ 2.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3
@bjarthur bjarthur referenced this issue in JuliaArchive/Color.jl Oct 15, 2014
Closed

convert error #68

@timholy
Member
timholy commented Oct 16, 2014

@bjarthur noted that this can be triggered on 0.3, too, although the minimal testcase above is not sufficient to do it.
JuliaArchive/Color.jl#68

@davidssmith
Contributor

I'm getting this on my 3.3 install when I try to return a tuple of Float64 matrices from fetch. The code is hundreds of lines long, and I can't really reduce it to a minimal test case. I could try to write one if that would help, but I suspect it is the same issue with convert.

Julia Version 0.3.3-pre+8
Commit 142798d (2014-10-23 10:59 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin14.0.0)
  CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3
@timholy
Member
timholy commented Nov 2, 2014

I needed to take a short break from more tedious stuff, so I got started here; only a tiny step forward, but I thought I'd throw it out there since I still have deadlines that will occupy most of my attention.

I began by inserting a print statement:

diff --git a/src/gf.c b/src/gf.c
index ee226d9..0b21779 100644
--- a/src/gf.c
+++ b/src/gf.c
@@ -1841,6 +1843,8 @@ static jl_value_t *ml_matches(jl_methlist_t *ml, jl_value_t *type,
         ti = lookup_match(type, (jl_value_t*)ml->sig, &env, ml->tvars);
         if (ti != (jl_value_t*)jl_bottom_type) {
             assert(ml->func->linfo);  // no builtin methods
+           if (strcmp(ml->func->linfo->name->name, "convert") == 0)
+               jl_printf(JL_STDOUT, "match function %s, file %s, line %d\n", ml->func->linfo->name->name, ml->func->linfo->file->name, ml->func->linfo->line);
             assert(jl_is_tuple(env));

             int skip = 0;

and then used the following debug file:

1 => (1,2)  # A non-breaking warmup
# 1 => (1,1,1,1,1,1,1,1,1.,false);
println("Starting")
convert((Int...), (uint(1),))

If I leave the comment in place (so convert is not broken), then the log after Starting looks like this:

match function convert, file bool.jl, line 4
match function convert, file int.jl, line 154
match function convert, file int.jl, line 156
match function convert, file int.jl, line 154
match function convert, file int.jl, line 156
match function convert, file bool.jl, line 4
match function convert, file int.jl, line 154
match function convert, file int.jl, line 156
match function convert, file int.jl, line 154
match function convert, file int.jl, line 156
match function convert, file bool.jl, line 3
match function convert, file bool.jl, line 4
match function convert, file int.jl, line 163
match function convert, file int.jl, line 163
match function convert, file int.jl, line 163
match function convert, file int.jl, line 163
match function convert, file base.jl, line 36
match function convert, file base.jl, line 36
match function convert, file base.jl, line 36
match function convert, file base.jl, line 36

but if I uncomment that line, then it looks like this:

match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file bool.jl, line 4
match function convert, file int.jl, line 154
match function convert, file int.jl, line 156
match function convert, file int.jl, line 154
match function convert, file int.jl, line 156
match function convert, file base.jl, line 39
match function convert, file bool.jl, line 4
match function convert, file int.jl, line 154
match function convert, file int.jl, line 156
match function convert, file int.jl, line 154
match function convert, file int.jl, line 156
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file bool.jl, line 3
match function convert, file bool.jl, line 4
match function convert, file int.jl, line 163
match function convert, file int.jl, line 163
match function convert, file int.jl, line 163
match function convert, file int.jl, line 163
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 36
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 36
match function convert, file base.jl, line 39
match function convert, file base.jl, line 36
match function convert, file base.jl, line 36
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file pointer.jl, line 10
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file base.jl, line 39
match function convert, file pointer.jl, line 10

Basically, the two start out identically except for the presence of a bunch of base.jl line 39 matches. This is my line 39. Kinda weird.

@timholy
Member
timholy commented Nov 3, 2014

Surprisingly, "convertalypse" (or at least, this specific manifestation) turns out to be a previously undetected consequence of @vtjnash's inlining patches. The problem lies somewhere in this range of commits:
cd4666d
e9e5606
31969bf
ec2fa85
4f71f59
5b91981
2bc684b
350c8e7
19d20ac
51c9a5b
baa1cb4
f4357ac
f8bfa2f

I can't narrow it down any better than this, because julia won't build within this range.

Here was my test script:

type Pair{A,B}
    first::A
    second::B
end
Pair(1, (1,1,1,1,1,1,1,1,1.,false));
convert((Int...), (uint(1),))
@vtjnash
Member
vtjnash commented Nov 3, 2014

In the inlining work, I assume that the subtype relationship must hold between the input arguments and the function type signature, however, for Tuples of typevars, Julia computes the wrong answer for subtype. Perhaps that is at issue here? I already opened an issue for it.

@timholy
Member
timholy commented Nov 3, 2014

Not sure I follow entirely. Would that explain why it works in a fresh julia session, but that once you construct a Pair it's broken henceforth?

The Tuple may be a red herring, if it turns out that a single bug underlies #8818, #8836. I think it's too early to tell, but we should at least be aware of the possibility.

@timholy
Member
timholy commented Nov 4, 2014

Unfortunately, there seems to be more than one problem underlying "convertalypse." Disabling all inlining by building julia with

diff --git a/base/inference.jl b/base/inference.jl
index 0ecb096..ed6765e 100644
--- a/base/inference.jl
+++ b/base/inference.jl
@@ -2032,6 +2032,7 @@ end
 # static parameters are ok if all the static parameter values are leaf types,
 # meaning they are fully known.
 function inlineable(f, e::Expr, atypes, sv, enclosing_ast)
+    return NF
     if !(isa(f,Function) || isstructtype(f) || isa(f,IntrinsicFunction))
         return NF
     end

"fixes" the problem here but does not fix #8818. So your suggestion that this may be limited to tuple-types is probably well-founded.

@GunnarFarneback GunnarFarneback added a commit to GunnarFarneback/Color.jl that referenced this issue Dec 6, 2014
@GunnarFarneback GunnarFarneback Workaround for Julia bug
Switching the colormaps_sequential and colormaps_diverging values from tuples to vectors works around the Julia bug JuliaLang/julia#8631, solving #68 and JuliaIO/HDF5.jl#160.
ddfe8d5
@GunnarFarneback GunnarFarneback referenced this issue in JuliaArchive/Color.jl Dec 6, 2014
Merged

Workaround for Julia bug #73

@aviks
Member
aviks commented Dec 9, 2014

Any thoughts? Seems like a very nasty bug to be lurking around in 0.3.

@timholy
Member
timholy commented Dec 9, 2014

The most recent version of Color works around this, but I certainly don't have a fix for the underlying problem.

@vtjnash
Member
vtjnash commented Dec 17, 2014

the naive fix for this appears to be:

diff --git a/src/gf.c b/src/gf.c
index fdf295e..e3a7f02 100644
--- a/src/gf.c
+++ b/src/gf.c
@@ -777,7 +777,7 @@ static jl_function_t *cache_method(jl_methtable_t *mt, jl_tuple_t *type,
         need_guard_entries = 1;
     }

-    if (need_guard_entries) {
+    if (1) {
         temp = ml_matches(mt->defs, (jl_value_t*)type, lambda_sym, -1);
         for(i=0; i < jl_array_len(temp); i++) {
             jl_value_t *m = jl_cellref(temp, i);

so we just need to be better about setting that need_guard_entries flag when there are vargs tuples anywhere in the method sig

note, the most direct what to trap this is:

lldb> break set -n jl_breakpoint

lldb> r -q

julia> f()=(ccall(:jl_breakpoint,Void,());convert((Int...), x))
f (generic function with 1 method)

julia> x=(uint(1),)
(0x0000000000000001,)

julia> f()

lldb> break set -n jl_apply_generic

lldb> c
@timholy
Member
timholy commented Dec 17, 2014

Thanks for the trap suggestion, that's hugely helpful. I'll work through this shortly as a useful learning exercise.

And of course, thanks for diving into this and diagnosing the problem!

@vtjnash
Member
vtjnash commented Dec 28, 2014

the issue may be that in the call to ml_matches in need_guard_entries (
https://github.com/JuliaLang/julia/blob/fb730f33ecc922e33e676c32ba50dd9a434d024e/src/gf.c#L781) there is a method missing from the returned list.

in particular, the following call:

julia> methods(convert, ((Type, Type...), (Any, Any...)))
2-element Array{Any,1}:
 convert(::Type{Tuple},x::Tuple) at base.jl:39                   
 convert(T::(Type,Type...),x::(Any,Any...)) at base.jl:44

is missing the following function definition:

convert{T}(::Type{(T...)}, x::Tuple) = cnvt_all(T, x...)

therefore, when we specialize convert(::(Type{Int64}, Type{Int64}, Type{Int64}, Type{Int64}, Type{Int64}, Type{Int64}, Type{Int64}, Type{Float64}, Type{Bool}), ::(Int64, Int64, Int64, Int64, Int64, Int64, Int64, Float64, Bool)) to match convert(::(Type, Type...), ::(Any, Any...)) the guard entry for Type... matching the empty list is not generated.

@timholy
Member
timholy commented Dec 30, 2014

Nice detective work. I assume you don't yet have an idea why that method isn't showing up?

@vtjnash
Member
vtjnash commented Dec 30, 2014

no, usually i just hope jeff will chime in at some point.

although, looking back at it, i'm not sure if my statement was entirely correct either. It's not exactly missing the match for the empty list so much as missing the match for when (Type, Type...) can be reduced to (T...)

but it does let us make a more reduced test case:

julia> broken(::(Type, Type...), ::(Any, Any...)) = 1
broken (generic function with 1 method)

julia> broken{T}(::Type{(T...)}, x::Tuple) = 2
broken (generic function with 2 methods)

julia> methods(broken, ((Type, Type...), (Any, Any...)))
1-element Array{Any,1}:
 broken(::(Type{T<:Top},Type{T<:Top}...),::(Any,Any...)) at none:1

and sticking a breakpoint in jl_matching_methods, i can see that lookup_match returns Union() when attempting to compare the requested signature to that first method signature.

tracing through the code execution, we see that (Type, Type...) turns into Type{(Top, Top...)}. Note the presence of Top there. The type intersection of Type{(T<:Any...)} and Type{(T<:Top...)} is Union()

which means that we can "fix" it (and the original bug) with one liner (kids, don't try this at home):

julia> broken(::(Type, Type...), ::(Any, Any...)) = 1
broken (generic function with 1 method)

julia> broken{T}(::Type{(T...)}, x::Tuple) = 2
broken (generic function with 2 methods)

julia> methods(broken, ((Type, Type...), (Any, Any...)))
1-element Array{Any,1}:
 broken(::(Type{T<:Top},Type{T<:Top}...),::(Any,Any...)) at none:1

julia> Type.parameters = (TypeVar(:T),)
(T,)

julia> methods(broken, ((Type, Type...), (Any, Any...)))
2-element Array{Any,1}:
 broken{T}(::Type{(T...,)},x::(Any...,)) at none:1      
 broken(::(Type{T},Type{T}...),::(Any,Any...)) at none:1

so, now that you got me started again, it seems I do now have some idea why this isn't showing up

@vtjnash
Member
vtjnash commented Dec 30, 2014

@JeffBezanson can you verify the validity of the following patch (since you can't make a tuple containing #undef)

fwiw, the following change doesn't seem to break things (and it fixes this bug):

diff --git a/src/jltypes.c b/src/jltypes.c
index 7d4d53c..50a838e 100644
--- a/src/jltypes.c
+++ b/src/jltypes.c
@@ -751,10 +751,12 @@ static int tuple_to_Type(jl_tuple_t *a, jl_tuple_t **ptemp)
     for(i=0; i < alen; i++) {
         jl_value_t *el = jl_tupleref(a, i);
         if (jl_is_type_type(el)) {
-            jl_tupleset(*ptemp, i, jl_tparam0(el));
+            jl_value_t *T = jl_type_intersection(jl_tparam0(el), (jl_value_t*)jl_any_type); // removes Undef
+            jl_tupleset(*ptemp, i, T);
         }
         else if (i==alen-1 && jl_is_vararg_type(el) && jl_is_type_type(jl_tparam0(el))) {
-            jl_tupleset(*ptemp, i, jl_wrap_vararg(jl_tparam0(jl_tparam0(el))));
+            jl_value_t *T = jl_type_intersection(jl_tparam0(jl_tparam0(el)), (jl_value_t*)jl_any_type); // removes Undef
+            jl_tupleset(*ptemp, i, jl_wrap_vararg(T));
         }
         else {
             *ptemp = NULL;

with the following test case:

f8631(::(Type, Type...), ::(Any, Any...)) = 1
f8631{T}(::Type{(T...)}, x::Tuple) = 2
@test length(methods(f8631, ((Type, Type...), (Any, Any...)))) == 2

i'm not sure if this is entirely complete, however, or if the following answer is also wrong:

julia> Base.typeintersect(Type{(TypeVar(:T,Core.Top), TypeVar(:T,Core.Top)...)}, Type{(TypeVar(:T)...,)})
Union()
@JeffBezanson
Member

That patch looks safe. In the second part T might need to be rooted before passing it to jl_wrap_vararg.

@vtjnash
Member
vtjnash commented Dec 31, 2014

ok. however, i don't feel like that patch is complete. i can't quite decide if the following are correct:

julia> Base.typeintersect((Type{TypeVar(:A,Integer)}, Type{TypeVar(:B)}...), Type{(TypeVar(:C,Integer)...,)})
Union()

julia> Base.typeintersect((Type{TypeVar(:A)}, Type{TypeVar(:B)}...), Type{(TypeVar(:C,Integer)...,)})
Type{(C<:Integer...,)}

it seems to be a covariance/invariance complication arising from the Type{(T,)}, (Type{T},) duality. in the former case, we expect that Type{(T,)} must be matched invariantly (including the TypeVar, because it is in a type parameter), but in the second case, we expect that the TypeVar can be matched covariantly. however, it seems that these two type specifications are assumed to be equivalent (by the existence of tuple_to_Type).

so that leads to this alternative patch, which also fixes the convert bug, but in a very different way:

diff --git a/src/jltypes.c b/src/jltypes.c
index 7d4d53c..69982b1 100644
--- a/src/jltypes.c
+++ b/src/jltypes.c
@@ -539,6 +539,14 @@ static jl_value_t *intersect_tag(jl_datatype_t *a, jl_datatype_t *b,
                     continue;
                 }
             }
+            else if (jl_is_tuple(ap)) {
+                if (!jl_is_tuple(bp)) {
+                    JL_GC_POP();
+                    return jl_bottom_type;
+                }
+                ti = intersect_tuple((jl_tuple_t*)ap, (jl_tuple_t*)bp, penv,eqc,invariant);
+                jl_tupleset(p, i, ti);
+            }
             else {
                 int tva = jl_has_typevars_(ap,0);
                 int tvb = jl_has_typevars_(bp,0);

and now we get:

julia> Base.typeintersect((Type{TypeVar(:A,Integer)}, Type{TypeVar(:B)}...), Type{(TypeVar(:C,Integer)...,)})
Type{(C<:Integer,C<:Integer...)}

julia> Base.typeintersect((Type{TypeVar(:A)}, Type{TypeVar(:B)}...), Type{(TypeVar(:C,Integer)...,)})
Type{(C<:Integer...,)}

(although now this fails other tests)

additionally, 98a1165 seems closely related to this issue.

@vtjnash
Member
vtjnash commented Dec 31, 2014

or instead:

diff --git a/src/jltypes.c b/src/jltypes.c
index 7d4d53c..8ccb59b 100644
--- a/src/jltypes.c
+++ b/src/jltypes.c
@@ -539,6 +539,9 @@ static jl_value_t *intersect_tag(jl_datatype_t *a, jl_datatype_t *b,
                     continue;
                 }
             }
+            else if (jl_is_tuple(ap) && jl_is_tuple(bp)) {
+                ti = intersect_tuple((jl_tuple_t*)ap, (jl_tuple_t*)bp, penv,eqc,invariant);
+            }
             else {
                 int tva = jl_has_typevars_(ap,0);
                 int tvb = jl_has_typevars_(bp,0);

although that can mucks things up pretty badly, since it turns
convert(::Type{Tuple}, x::Tuple) = x
into the equivalent:
convert(::Type{(Any...)}, x::Tuple) = x
and therefore starts to consider it to be a possible (and preferable) method match for pretty much any tuple:

julia> methods(convert, typeof(((Int...), (l,))))
2-element Array{Any,1}:
 convert(::Type{(Any...,)},x::(Any...,)) at base.jl:39                   
 convert(T::(Type{T<:Top},Type{T<:Top}...),x::(Any,Any...)) at base.jl:44
@garrison
Member
garrison commented Feb 4, 2015

No comments here in over a month, so I'll ask: What is the current status of this issue with convert? From the above breakthroughs I thought it was nearly fixed and was hoping it would go in 0.3.5. Now there is talk of 0.3.6 (planning issue: #10058) and I'm curious how likely it is that this will be fixed soon.

@JeffBezanson JeffBezanson added a commit that closed this issue Feb 14, 2015
@JeffBezanson JeffBezanson eliminate Undef and Top types. fixes #8631
thanks to @vtjnash for the test case.

It's hard to tell whether this is the real fix for the problem, but
it works, and having a type that's not a subtype of Any around was
just a huge pain. Things are simpler without it.

Inside the compiler, undefinedness is no longer part of a variable's
type. Instead there is a bit per variable telling whether it is ever
used when undefined. This has the disadvantage of being coarser, since
it doesn't give the undefinedness of each variable use. However it has
the advantage of giving better type info: if a variable is either a
Float64 or undefined, its type is just "Float64" and we can optimize
accordingly.

This combines well with the future optimization of storing
possibly-undefined variables unboxed (#6914).
For that we will add a run time 1-bit flag to track definedness, and
then LLVM can hopefully eliminate checks along paths where the
flag is known to be set, gaining back the previous granularity.
84e80f4
@JeffBezanson JeffBezanson added a commit that referenced this issue Feb 14, 2015
@JeffBezanson JeffBezanson Revert "eliminate Undef and Top types. fixes #8631"
This reverts commit 84e80f4.

Major performance regressions. This commit will be back shortly when
fixed.
c631a08
@ivarne
Contributor
ivarne commented Feb 14, 2015

🍰 🎆

@ivarne ivarne reopened this Feb 14, 2015
@aviks
Member
aviks commented Feb 14, 2015

👏

@ivarne did you intend to reopen this?

[edit: saw the revert, sorry for the noise]

@tkelman
Member
tkelman commented Feb 15, 2015

Nice work @JeffBezanson! How much different would this look like on release-0.3? We're probably tagging 0.3.6 soon, might be good to do this as one of the first commits for 0.3.7 (assuming it doesn't introduce any new issues on master).

@JeffBezanson
Member

Thanks! This is a tricky one. Since this removes the Top and Undef types it's technically a breaking change, though user code should never have used those types. Tough call. Other than that it should be possible to backport.

@tkelman
Member
tkelman commented Feb 15, 2015

Right, don't want to rush this, but wanted to get the question out there since this has been bothering people for a while. Let's see who can break it first :)

@timholy
Member
timholy commented Feb 15, 2015

Great to have this closed! Many thanks.

@tlnagy
Contributor
tlnagy commented Mar 10, 2016

Is this supposed to be working in 0.4.3?

julia> 1 => (1,1,1,1,1,1,1,1,1.,false)
1=>(1,1,1,1,1,1,1,1,1.0,false)

julia> convert((Int...), (uint(1),))
ERROR: MethodError: `start` has no method matching start(::Type{Int64})
 in append_any at essentials.jl:127
@JeffBezanson
Member

convert(Tuple{Vararg{Int}}, (UInt(1),)) is what you want.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment