Support defining methods of a function in multiple cells

[karlwessel]

The other solution is much harder to implement consistently - i.e. being able to detect multiple methods with the same argument types (instead of just detecting multiple functions with the same name) and deleting only the methods from one cell, while leaving the others.

All of this is possible within the scope of Pluto, but not something for the next couple of months :)

Originally posted by @fonsp in #109 (comment)

[karlwessel]

I would like to track this feature in an actual issue, since at least I am really really missing this.

Multiple dispatch and therefore multiple methods of the same function is a key feature of Julia. Often those methods are implemented in completely different contexts (for example Base.show). Having to put them all in the same cell makes exploratory implementation of custom types really hard.

[fonsp]

(split into two issues: #178)

[fonsp]

It would be helpful if someone with an eye for edge cases (@karlwessel) would write the test cases for this before thinking up a solution.

(Writing test cases is fairly easy, and they don't need to be perfect yet.)

[fonsp]

Strictly speaking, two cells f(x::Int) = 1 & f(x::Int) = 2 should not be allowed, and we can find those cases and throw a MultipleDefinitionsError. However, there are some very difficult situations like:

f(x::Int) = 1 & f(x::Int64) = 2 (built in type synonyms)

A = Number & f(x::A) = 1 & f(x::Number) = 2 (dynamic types)

A = rand(Bool) ? Number : String & f(x::A) = 1 & f(x::Number) = 2 (aaah)

f = 1 & f(x) = 1 (variables vs methods)

f(x::Int) = 1; f(x::String) = 2 && f(x::String) = 3; f(x::Vector) = 4 (multiple methods per cell)

f(x::Int) = 1; f(x::Int) = 2 && f(x::Vector) = 3 (number of methods in code might not equal number of defined methods)

f() = 1 && f(; s) = 2 (kwargs)

f(x::T) where T = 1 && f(x) = 2

f(x::(rand(Bool) ? Int64 : Float64)) = 1 &
f(x::(rand(Bool) ? Int64 : Float64)) = 2

If all of these situations are allowed without a MultipleDefinitionsError (which is the easy solution), then a notebook could be ambiguous (ignoring the SECRET fact that the order in which cells are placed is used as a tie-breaker when two cells are mutually independent.) They need to be handled correctly.

[karlwessel]

Is it possible to detect those cases during execution of a notebook by comparing the methodswith count for each function for which there are cells defining methods for? And throw a runtime exception if one cell that should increase the count doesn't because it replaced another method?

[fonsp]

That's going in the right direction! (Although my 5th example would not get detected with that method)

[karlwessel]

Well, maybe we don't have to catch all corner cases...

[fonsp]

Building on your counting methods idea:

It's very nice if we can keep all of the syntax analysis in one place, before running the code, and then run cells in a precomputed order. Checking the number of defined methods after running cells (in the middle of a reactive run), and then taking that information back to do something special makes that tricky. But there are three reasons to do this kind of "dynamic" code analysis instead of static anaylsis:

• Support defining methods of a function in multiple cells #177 (this issue)
• Reactive response to removing using Module #24
• Macros and DSLs #196

The common hurdles in these three are:

• the dependency graph can only be determined after runtime
• the dependency graph depends on the global state
• we'd like to know things about external packages without
  • analysing their source code, nor
  • importing them into the server process

Solution?

We need to do some dynamic analysis, but we can try to keep it simple and isolated. The runner process will have three new APIs:

do_signatures_overlap(signatures::Array{NTuple{Type}})::Bool

symbols_exported_by_module(name::Symbol)::Set{Symbol}

macroexpand(expr::Expr)::Expr

to solve these three problems, respectively.

do_signatures_overlap?

Create a new test module, import all globals into the test module, and then define a dummy functions with the given signatures. If the number of methods is less than the number of signatures, then they overlap. (Thanks @karlwessel!)

symbols_exported_by_module

Create a new test module, call the using statement there, and then return Core.eval(testmodule, :(names($(name_of_usinged_module))) ).

macroexpand

This is just Base.macroexpand :)

Separate using+import from all other code

using and import already get some of special treatment - they always execute first in a "tie" scenario of a reactive run. This is a heuristic: we know that our syntax analysis fails for using so this is a hacky way to fix most scenarios. All using X sub-expressions are also extracted from cell code and stored separately.

To support the solutions above, we need to go from:

1. syntax analysis (in server process)
2. reactive run (in worker process)

to:

1. mini syntax analysis to find using (in server process)
2. call the using statements (in worker process)
3. syntax analysis (in server process), with symbols_exported_by_module to fill in the gaps
4. reactive run (in worker process), but use do_signatures_overlap during the run to somehow figure out if cells should err.

[fonsp]

We can simplify the do_signatures_overlap quite a bit by dropping support for:
A = B
f(x::A) = 1
f(x::B) = 2

Although it is common to have user-defined types, I hope that it's uncommon for user defined aliases to cause this kind of trouble. So:

do_signatures_overlap?

Create a new test module, import all globals into the test module, create dummy types for all unknown types, and then define dummy functions with the given signatures. If the number of methods is less than the number of signatures, then they overlap.

So:

function f(x::Int, y::MyType, z::random_type())
    asdf
end

will become

struct var"MyType" end
struct var"random_type()" end

function f(x::Int, y::var"MyType", z::var"random_type()")
end

(var"..." is how you define variables with possibly illegal names in Julia).

1. mini syntax analysis to find using (in server process)
2. call the using statements (in worker process)
3. syntax analysis (in server process), with symbols_exported_by_module and do_signatures_overlap to fill in the gaps
4. reactive run (in worker process)

[karlwessel]

f(x::Int) = 1 & f(x::Int64) = 2 (built in type synonyms)

A = Number & f(x::A) = 1 & f(x::Number) = 2 (dynamic types)

A = rand(Bool) ? Number : String & f(x::A) = 1 & f(x::Number) = 2 (aaah)

f = 1 & f(x) = 1 (variables vs methods)

f(x::Int) = 1; f(x::String) = 2 && f(x::String) = 3; f(x::Vector) = 4 (multiple methods per cell)

f(x::Int) = 1; f(x::Int) = 2 && f(x::Vector) = 3 (number of methods in code might not equal number of defined methods)

f() = 1 && f(; s) = 2 (kwargs)

f(x::T) where T = 1 && f(x) = 2

f(x::(rand(Bool) ? Int64 : Float64)) = 1 &
f(x::(rand(Bool) ? Int64 : Float64)) = 2

I guess the next step is me implementing those tests :-)