Add ExprSplitter

[timholy]

This implements an "expression-splitting iterator" that returns individual expressions ready for evaluation to the caller. This allows the caller to decide what to do with them (e.g., evaluation, signature analysis, whatever). It's basically a proposed replacement for prepare_thunk and split_expressions; by placing the consumer in charge, we get rid of any of the weird "should I eval if I can't lower?" questions that have dogged the previous framework.

For the moment this is parallel to our older framework; I've not yet actually rebased JuliaInterpreter on this, although the Frame(::Module, ::CodeInfo) constructor is a step in this direction. My local copy of Revise#revise3 is now based on this and it seems to work well.

@pfitzseb and @aviatesk, are there callers of these functions outside JuliaInterpreter, Debugger, and Revise? If so we'd want to think about a careful deprecation path. Indeed we might want to do that anyway since split_expressions, even though not exported, is pretty essential in some contexts (so there may be unknown users).

Closes #406
Closes #419

[pfitzseb]

I'm pretty sure we're using split_expressions in VSCode and Juno, yes.

[KristofferC]

Instead of working hard on deprecation and then just have everyone immediately update to the new one (effectively making the work on the deprecation pointless) we could also release a new breaking version and have people just update to that. Of course, this depends on how hard it is to do the deprecation and how hard it is to do the upgrade.

[timholy]

Since these are officially internal (though well-used) functions, I decided to go with the breaking change. The transition is pretty straightforward as can be seen by the tests. The main thing I'm unsure of is whether we want to export a utility for detecting and stripping out @doc exprs, or whether that was relevant only for Revise.

[timholy]

I wasn't quite sure what to do about these.

[timholy]

Ugh, this is ugly getting it over the finish line. The various bugs that different Julia versions have wrt LineNumberNodes are really annoying.

Are the remaining errors anything to worry about? Travis had a failure on Windows 1.4 (a strange string-indexing error), appveyor on 1.2 (a problem with stat). The second seems more worrisome, since it seems to have happened on both 32- and 64-bit. A few older PRs were not triggering this.

[aviatesk]

Since these are officially internal (though well-used) functions, I decided to go with the breaking change. The transition is pretty straightforward as can be seen by the tests. The main thing I'm unsure of is whether we want to export a utility for detecting and stripping out @doc exprs, or whether that was relevant only for Revise.

as far as I know, the only consumer who uses the utility for at-doc exprs is only Revise, so I would like to say it's your call :)

[timholy]

Thanks! If Revise is the only one, I've already got that covered from within Revise. There's isn't much to it, really, just grabbing the 4th argument of the expression so it wouldn't be much of a utiliy anyway.

[KristofferC]

Are the remaining errors anything to worry about? Travis had a failure on Windows 1.4 (a strange string-indexing error), appveyor on 1.2 (a problem with stat)

I would not really worry about unmaintained julia versions (1.1 - 1.4). Might be time to drop compat from them completely so they will keep working at the state they are in but not get updates.

[timholy]

Sounds good. Looks like JuliaLang/julia#26685. I just pushed a debug commit; let's see if that reveals an easy fix, and if not I'll just make 1.3 the lowest Julia version supported set the version compat to "~1.0, 1.5".

[timholy]

That is one weird filename 😄.https://ci.appveyor.com/project/KristofferC/juliainterpreter-jl/builds/35055810/job/ubmbpu36otugvd9e

[codecov]

Codecov Report

Merging #425 into master will increase coverage by 0.22%.
The diff coverage is 92.55%.

@@            Coverage Diff             @@
##           master     #425      +/-   ##
==========================================
+ Coverage   88.73%   88.96%   +0.22%     
==========================================
  Files          12       12              
  Lines        2006     2256     +250     
==========================================
+ Hits         1780     2007     +227     
- Misses        226      249      +23     

Impacted Files	Coverage Δ
src/packagedef.jl	88.57% <ø> (-8.10%)	⬇️
src/interpret.jl	85.88% <57.14%> (-3.85%)	⬇️
src/types.jl	75.71% <66.66%> (-0.90%)	⬇️
src/construct.jl	90.69% <98.55%> (-1.91%)	⬇️
src/precompile.jl	100.00% <100.00%> (ø)
src/utils.jl	87.46% <100.00%> (+3.82%)	⬆️
src/localmethtable.jl	96.49% <0.00%> (-3.51%)	⬇️
... and 7 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 64d719f...06ff99c. Read the comment docs.

[timholy]

A JuliaHub search indicates that the only known users of prepare_thunk and split_expressions are the ones we know about. So this crew (I should have also CCed @davidanthoff) can decide.

[KristofferC]

That is one weird filename

Wow, where did that come from?

[timholy]

I am not sure. Seems likely to be a memory bug or something.

[timholy]

Revise@3.0.0 is officially ready to be announced, pending the resolution of this PR. 👍 or 👎 votes accepted, preferably with reasons for the latter.

[aviatesk]

lgtm

[timholy]

Seems like folks are willing to do the work of updating for the breaking change. Many thanks, all!

[timholy]

Before merging I'm going to back out the commit disabling the ambiguity test. That means the test will fail, but we've at least seen the tests pass.