fix: Float RecApp out of applications

[nomeata]

This didn't work before

def f (n : Nat) : Nat :=
  match n with
  | 0 => 0
  | n + 1 => (f) n

because the RecApp metadata marker gets in the way. More practically
relevant, such code is to be produced when using rw or simp in
recursive theorems (see included test case).

We can fix this by preprocessing the definitions and floating the
.mdata marker out of applications.

For structural recursion, there already exists a preprocess function;
this now also floats out .mdata markers.

For well-founded recursion, this introduces an analogous preprocess
function.

Fixes #2810.

One test case output changes: With the .mdata out of the way, we get a
different error message. Seems fine.

Alternative approaches are:

• Leaving the .mdata marker where it is, and looking around it.
  Tried in fix: Structural recursion in the presence of parentheses #2813, but not nice (many many places where withApp etc.
  need to be adjusted).
• Moving the .mdata inside the application, so that withApp still
  works. Tried in fix: Move RecApp marker further inside the expression #2814. Also not nice, the invariant that the .mdata
  is around the .const is tedious to maintain.

[nomeata]

Open question: Should there be one preprocess function invoked by addPreDefinitions for both strategies? Would the well-founded recursion code benefit or be hurt by the beta-reductions done for structural recursion?

[leodemoura]

Open question: Should there be one preprocess function invoked by addPreDefinitions for both strategies? Would the well-founded recursion code benefit or be hurt by the beta-reductions done for structural recursion?

Great question. I don't see any reason why it would hurt well-founded recursion.
Let's try a single preprocess function for both of them.
Note that the current preprocess assumes there is only one function on the recursive block. This is not sufficient for well-founded recursion since it already supports mutual recursion. We need support for this too.

[nomeata]

Let's try a single preprocess function for both of them.

Will do (tomorrow)

[semorrison]

(Apologies for force pushing to your branch, I am still diagnosing some CI weirdnesses!)

[leanprover-community-mathlib4-bot]

• ❗ Mathlib CI will not be attempted unless you rebase your PR onto the 'nightly' branch. (2023-11-06 03:23:43)
• ❗ Mathlib CI can not be attempted yet, as the 'nightly-testing-2023-11-06' branch does not exist there yet. We will retry when you push more commits. It may be necessary to rebase onto 'nightly' tomorrow. (2023-11-06 10:52:58)
• ✅ Mathlib branch lean-pr-testing-2818 has successfully built against this PR. (2023-11-06 12:09:47) View Log

[nomeata]

Great question. I don't see any reason why it would hurt well-founded recursion.
Let's try a single preprocess function for both of them.

Tried in #2823. Beta-reduction (if done as it is defined now) can remove let_fun redexes that we want to keep because of the termination argument they bring into scope; example for a test case now failing:

def f (p : Pos) : Pos :=
  match h : p.view with -- It would also be nice to have a feature to force Lean to applies "views" automatically for us.
  | 1 => 1
  | .succ x =>
    have : sizeOf x < sizeOf p := sizeof_lt_of_view_eq h -- See comment at `sizeof_lt_of_view_eq`
    f x + x + 1

Maybe one could use a more careful form of beta-reduction that works for both, but I’d say it’s not worth for the gain, and who knows, maybe the next derecursifier that comes along may need some other set of preprocessors again.

I left a note in the code that the difference between the two functions is deliberate.

[digama0]

Note: the fact that the WF preprocessor does not remove beta redexes during preprocessing is a bug, which causes unusedHavesSuffices linter failures on the generated equation lemmas. Of course it is understandable due to the way WF receives hypotheses, but it should remove betas after doing the termination proofs.

It may or may not be possible to remove betas without breaking the WF termination prover with the current preprocessor design, I have not looked in detail. Perhaps we need a postprocessor too.

[nomeata]

Thanks! That sounds indeed like you want to post-process them. Separate issue, I’d say?

[nomeata]

!bench

[leanprover-bot]

Here are the benchmark results for commit 47d1e07.
There were no significant changes against commit 8b86bee.

[nomeata]

Will merge this on Wednesday.