Multiple definitions; good design for limiting allocation?

[timholy]

Revise can get desynchronized from the file state by the following sequence of user actions:

1. cut the definition of foo(x::T) from fileA.jl, but do not yet save the file
2. paste definition into fileB.jl (another file in the same package) and save the file
3. trigger a revision
4. save fileA.jl and trigger a revision

After item 3, there are technically duplicated definitions of the method (and if the user had quit and rebuilt the package at this point s/he would get a precompile error). After item 4, Revise will helpfully delete the method, not realizing it had been duplicated. So attempts to use the method will henceforth fail.

I used to think of this as "user error" (the user should have saved fileA.jl before triggering the first revision) but lately I've come to think it should be treated as a Revise bug. The question is how best to implement a fix. It seems the most thorough fix would involve making method_info a sig=>list-of-definitions dict rather than a sig=>definition dict. If list-of-definitions is a Vector, though, there are going to be an awful lot of vectors with just one element. Any thoughts on a better way to do this?

[timholy]

Any thoughts on the memory issue? Maybe just allow either a sig=>definition or sig=>list-of-definitions? My biggest concern there is that I think get(method_info, sig, nothing) will then have 4 possible return types: Nothing, Missing, (LineNumberNode,Expr), and Vector{(LineNumberNode,Expr)}, and I think the limit for automatic union-splitting is 3.

EDIT: thinking about it a bit more, perhaps consumers will need to be designed to do manual union-splitting anyway. So maybe this isn't a big problem.

[KristofferC]

How many methods are there in practice? Perhaps n_methods * size_of_array is negligible. The Julia GC is pretty good with many small allocations.

[timholy]

Good question. This simulates what would happen if someone touched every file in Base, which is hopefully a proxy for something like the worst-case scenario:

julia> using Revise, CodeTracking

julia> Revise.track(Base)

julia> pkgdata = Revise.pkgdatas[Base.PkgId(Base)];

julia> for file in pkgfiles(Base).files
           Revise.maybe_parse_from_cache!(pkgdata, file)
       end

julia> length(CodeTracking.method_info)
10362

So about 10k vectors. If the overhead is 80 bytes (I'm actually not sure what it is these days), I guess that's only 0.8Mb. Geez. My sensibilities are still stuck in 1998, this is not even worth worrying about. Sorry for the noise!

I'll leave this open until the bigger issue is solved.