Stackoverflow on DAML-LF with too much nesting

[cocreature]

The following DAML Script will produce a stackoverflow when loaded into sandbox (using the 1.1.0 snapshot but it’s not specific to that) https://gist.github.com/cocreature/d6b3d222dec9937d0590e0862eaf5591

This is not actually specific to DAML Script. The only somewhat surprising fact here is that a large do block in DAML Script produces nested DAML-LF but that is somewhat unavoidable. Even if you inline the typeclass methods you end up with a free monad which is nested.

While we could solve the issue in the decoder which is the part that is exploding here this only slightly bumps the limit until we run into the protobuf recursion limit.

@remyhaemmerle-da proposed the following steps which I support:

1. Define a nesting level for DAML-LF that matches the protobuf level. Anything with more nesting will be a hard error.
2. Ensure that the DAML-LF decoder, typechecker and so on do not stackoverflow on that level. This could be by trampolining or simply by making sure that we do not use that much stack up to this recursion level (note that this is somewhat tricky to control since the decoder is also a library so we don’t know how deep we are on the stack when we start)
3. Ensure that the compiler produces at least a sensible error and ideally automatically outlines expressions that run over the limit so you do not run into this error.

Workaround in the meantime

Manually outline things, e.g., turn

a = do
  x 
  y
  z

into

a = do
  x
  b
b = do
  y
  z

[bame-da]

Have you considered flattening things in DAML-LF? We do this for transaction trees (at least on the Ledger API) already so why not do it for LF as well?

[cocreature]

Outlining is flatting things to some degree but without changing DAML-LF. Flattening everything would be an option and was something that we considered at some point but it’s a fairly invasive DAML-LF change.

[remyhaemmerle-da]

This is similar of https://discuss.daml.com/t/maximum-number-of-record-fields/2073/2