Modules and multiple Dispatch

[ingmarschuster]

Either there is a parametric knot in my head by now or there are problems with julias multiple dispatch and how it interacts with modules.
the following code snippet

module As
    export aT, aTfunc
    abstract aT{T1, T2}

    function aTfunc{T1 <: Int64, T2}(bI::aT{T1, T2})
             println("called aTfunc")
             bfunc(bI)
    end

end

module Bs
    export bT, bfunc
    using As
    type bT <: aT{Int64, Float64}
             c::Int64
    end

    function bfunc(bI::bT)
             println("called bfunc")
    end

end

using As, Bs

aTfunc(bT(5))

does not work in Commit 5f6c86c* (0.2.0-rc1+7).

while the same thing without modules

abstract aT{T1, T2}

type bT <: aT{Int64, Float64}
         c::Int64
end

function aTfunc{T1 <: Int64, T2}(bI::aT{T1, T2})
         println("called aTfunc")
         bfunc(bI)
end

function bfunc(bI::bT)
         println("called bfunc")
end

aTfunc(bT(5))

works as expected. I see what the technical reasons might be in the case with modules, still as far as I see it, multiple dispatch should work across module boundaries.

[JeffBezanson]

The identifier bfunc is not visible inside module As.

[ingmarschuster]

Sorry for not answering right away, I'm in very a different time zone. This issue is not "Why does this not work" (which is what you are answering) but rather "Shouldn't this work?".
Take for example the equivalent code in Python (2.7). If you put the following code into As.py and Bs.py:

### file/module As.py ###
class aT:
    def __init__(self):
        pass

    def aTfunc(self):
        print("called aTfunc")
        self.bfunc()

and

### file/module Bs.py ###
from As import aT
class bT(aT):
    _c = 0
    def __init__(self, c):
        self._c = c

    def bfunc(self):
        print("called bfunc")

bT(5).aTfunc()

That works as expected. I think if this doesn't work, it becomes very difficult to do good software design. Do you agree or disagree? If you disagree, what would you suggest as a design pattern here?

[StefanKarpinski]

The original code is a bit hard to follow, so to help people following along that haven't run your example code, the resulting error is this:

julia> aTfunc(bT(5))
called aTfunc
ERROR: bfunc not defined
 in aTfunc at none:7

I'm not sure how this could be otherwise. The bfunc binding is not visible in the As module, which is the entire point of modules: you can't just randomly see things in other modules. It seems rather strange to me that the As module calls a function that it knows nothing about – it neither imports nor defines bfunc. This is strange in the Python example too. What happens if you call the aTfunc method on an aT object rather than a bT object? The fact that bT(5).aTfunc() happens to work there seems like it's actually hiding a deeper design error – namely that you are using bfunc from aT although an aT object might not have any bfunc method. You'll note that what you're doing would not work in a statically type checked single o.o. language either. The obvious solution here is to define the bfunc function in the As module and then extend it for the bT type in the Bs module. The same solution would fix be necessary to make this work in a static language.

[StefanKarpinski]

To be concrete, the proposed solution is this:

module As
    export aT, aTfunc
    abstract aT{T1,T2}
    function aTfunc{T1<:Int64,T2}(bI::aT{T1,T2})
         println("called aTfunc")
         bfunc(bI)
    end
    function bfunc(aI::aT)
         println("bfunc called on aT")
    end
end
module Bs
    export bT, bfunc
    using As
    type bT <: aT{Int64, Float64}
         c::Int64
    end
    function As.bfunc(bI::bT)
         println("called bfunc on bT")
    end
end

using As, Bs

julia> aTfunc(bT(5))
called aTfunc
called bfunc on bT

[ingmarschuster]

Thanks a lot Stefan. You are absolutely correct that this would not work in a statically typed language. But as Julia is a Dynamic language, I assumed dynamic dispatch (i.e. method dispatch at runtime). This is also what happens in Python: at runtime, Python asks bT for a method bfunc and calls it. If it doesn't exists, it throws a runtime error. As far as I know, the same process is used in dispatching Objective C messages and other languages with dynamic dispatch.

This makes sense because you would often want to extend a type from another module. Take the case of writing code for a very special distribution for which it makes no sense to put it in the Distributions.jl package. Still, you'd want to write

using Distributions

type SpecialDist <: Distribution
    ...
end

function rand(d::SpecialDist)
    ...
end

Using an approach like this facilitates abstract types as general interfaces.

[ingmarschuster]

Oh, and thanks for your suggestion of an immediate solution! (even if not entirely satisfying to me in the long run)

[ingmarschuster]

Just to prove the point I tried it in another language with dynamic dispatch, Ruby. If you create two files

#file As.rb

module As
    class AT
        def aTfunc
            print("called aTfunc\n")
            self.bfunc()
        end
    end
end

and

#file Bs.rb
require "As"

module Bs
    class BT < As::AT

        def bfunc
            print("called bfunc\n")
        end
    end
end

Bs::BT.new.aTfunc()

now calling ruby Bs.rb produces

called aTfunc
called bfunc

[JeffBezanson]

This has nothing to do with dynamic dispatch. It has to do with the visibility of the identifier bfunc. In python and ruby you have effectively a single global namespace of method names. There is no way to say "x.bfunc(), but in the sense of bfunc defined by module Z". But this is how it works in julia, since functions are not inside types.

In bfunc(bI) one could imagine bfunc being looked up inside bI somehow, but bfunc itself is also a first-class object. If module As contained the code f(bfunc), how would it work? What would the identifier bfunc refer to?

[ingmarschuster]

There is no way to say "x.bfunc() in the sense of module Z" because the x-reference already implies the containing module (if I'm not mistaken; I think this is correct at least for Python).
But you are right in that Julias behavior is a general consequence of methods not being members of types. I still think that overriding methods across module boundaries makes sense for the reasons stated above, so maybe Stefans suggestion is the way to go in Julia. Possibly with a slight change of implementation of As.bfunc to make its intent clear:

module As
    export aT, aTfunc, bfunc
    abstract aT{T1,T2}

    ...

    function bfunc(aI::aT)
         error("As.bfunc(x::",typeof(aI),") not implemented.")
    end
end

[StefanKarpinski]

Yes, I think this is the right way to do this.