Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
New attempt at fixing MPR#7726 #1676
The new approach works by checking explicitly for such types after applying non-trivial substitutions, namely functor applications and
The error reporting is a bit weak, but this PR is already open for discussion.
Sorry, for my slow reply on this PR. The reason is that what I actually wanted to do is to implement the alternative suggested by Jacques:
because I strongly dislike the approach taken in this current PR. It is completely different from how we deal with recursive type checks everywhere else in the compiler, and type-checking of abstract types more generally.
If a functor definition is accepted then any module which has the module type of the parameter should be allowed. This PR instead adds an extra condition that has nothing to do with the parameter type of the functor. We should not be checking that a substitution is valid after the fact -- this is the approach to substitution of C++ templates.
However, I haven't had time to implement the alternative, and this change is clearly forwards compatible with doing the more principled fix later, so I guess we can put it in for now.
I'll review the code itself for correctness this afternoon.
lpw25 left a comment •
The code looks fine in that it does what it is intended to. However, it's not sufficient to prevent all instances of MPR#7726. For example:
OCaml version 4.07.0+dev0-2017-09-18 # module type T = sig type t end module Fix(F:(T -> T)) = struct module rec Fixed : T with type t = F(Fixed).t = F(Fixed) end;; module type T = sig type t end module Fix : functor (F : T -> T) -> sig module rec Fixed : sig type t = F(Fixed).t end end # module Id (X:sig type t end) = struct type t = X.t end;; module Id : functor (X : sig type t end) -> sig type t = X.t end # let f (x : Fix(Id).Fixed.t) = x;; val f : Fix(Id).Fixed.t -> Fix(Id).Fixed.t = <fun> # f 5;; Fatal error: exception Stack overflow
Well spotted. Actually, this is related to the fact module applications inside paths are not properly typed.
Concerning your doubts about checking after substitution, I think this is perfectly fine from a theoretical point of view. Usually, we just cannot do that because of abstraction and separate compilation. But in the case of recursive modules, what you see is what you get. All recursive modules must be declared simultaneously and typed together. Actually, one could say that what I did with Keiko is similar: assume that we know the types or aliases of all the recursive modules, and from that check well-foundedness. The difference is that, in presence of abstraction, one has to redo it every time something is made concrete.
A question is whether some other checks could benefit from this approach. Something that I have delayed for a long time is recomputing variances after functor applications. There could be some others, but all this strongly depends on working only at the level of types: we cannot break abstraction, so this is only about cases where making thing concrete changes some behavior.
Oops, the error message shows that actually the problem with generative functors was already fixed :)
This is still clearly not sufficient. You'll need to re-check every type recursively through every functor application. For example:
# module type T = sig type t end;; module type T = sig type t end # module Fix(F:(T -> T)) = struct module rec Fixed : T with type t = F(Fixed).t = F(Fixed) end;; module Fix : functor (F : T -> T) -> sig module rec Fixed : sig type t = F(Fixed).t end end # module Id (X:sig type t end) = struct type t = X.t end;; module Id : functor (X : sig type t end) -> sig type t = X.t end # module Foo (F : T -> T) = struct let f (x : Fix(F).Fixed.t) = x end;; module Foo : functor (F : T -> T) -> sig val f : Fix(F).Fixed.t -> Fix(F).Fixed.t end # module M = Foo(Id);; module M : sig val f : Fix(Id).Fixed.t -> Fix(Id).Fixed.t end # M.f 5;; Fatal error: exception Stack overflow
I really don't think this is a good approach to solving this issue. We should just be conservatively assuming that the the results of functor applications might be equal to any of the types passed in to them as arguments.
Thank you for this new example. Indeed, by commuting one more abstraction, one sees that we would have to recheck all types (or rather all modules in type constructor paths). This is doable, but might be costly.
I'd still very much want to see an example of unsoundness before introducing a drastic restriction such as assuming the non-contractiveness of all abstract functors, as this would prevent the definition of useful functors building fixpoints. I do not exclude that such unsoundness exists, maybe doing something like MPR#5343.
Actually, there is an easy enough solution: call
Do you still see cases that wouldn't be caught by that approach?
Another approach, conform with what is done for the module system, is to accept that since type checking is undecidable, we may sometimes have non-termination...
Since this PR has changed quite a bit since its first version, here is an explanation of the goal, the choices, and the limitations.
As stated before, the fundamental problem is that when a recursive module is defined inside a functor, the well-foundedness check for types sees abstract types coming from functor arguments as external types, and assumes that they cannot cause loops. However, when one of the functor arguments is a functor, and it is given one of the modules in the recursion as argument, it may actually return an internal type from this module, and by instantiation the functor one could create a loop.
My understanding is that, contrary to undetected non-contractiveness, which can be used to equate two arbitrary types, and break soundness, this problem only can cause non-termination of the type-checker (usually stack overflow). But of course, even if we know that this is not satisfactory.
A fix would be to be conservative, and just say that indeed any type returned by the argument functor can be any type from the modules it was given, rejecting all programs where this could result in ill-foundedness.
module type T = sig type t end module Fix(F:(T -> T)) = struct module rec Fixed : T with type t = F(Fixed).t = F(Fixed) end;; module F1(X : T) = struct type t = X.t end;; module M1 = Fix(F1);;
Here trying to expand
However, this restriction is actually pretty drastic. It prohibits safe uses of the
module F2(X:T) = struct type t = Z | S of X.t end;; module M2 = Fix(F2);; let x : M2.Fixed.t = S Z;;
The problem here is that, due to the contractiveness condition, it is already impossible to have a functor construct the fixpoint of a parameterized type given as argument, so that the above
An alternative approach is to delay the check until the application of the outside functor. However, as @lpw25 pointed, since functor applications can appear inside type constructors, this requires checking all the types exported by (almost) any functor for all functor applications included in their expansion.
What this PR implements is a lazy version of this alternative approach. We still re-check recursive modules in the result of functor applications, in module expressions and type annotations, but we delay the check of functor applications inside type constructors to their lookup in the environment (which is cached, and already applies a substitution to the result type of the functor). The cost of this check is further reduced by constructing the internal environment lazily during this check, i.e. the cost only appears when the functor return type contains recursive modules.
My opinion is that, since recursive modules are still an experimental feature, we can live with these limitations.
Note that the last 3 points could be alleviated by checking all constructor paths in principal mode.
Of course, it might still be worth exploring the first approach, to see whether we really need to keep all this expressiveness.
Right, I've read the code and I think that it does what it is intended to and that it will prevent the stack overflows from the MPR.
Against my better judgement I'm going to approve this PR, based on this reasoning:
I really hope that we can remove this code at some point and replace it with something more principled, but for an experimental feature like recursive modules -- which already have plenty of ugly corner cases -- I think it is fine like this for now.
My only remaining concern is that it might adversely effect the compile-time performance of code bases with a lot of functors (e.g. Jane Street's). I think it is probably fine, and I'll do a general check on the performance of the compiler on our code base before 4.08 is released so we'll see then if there are any issues.