MRG: First draft of jupyter_compat mode

[matthew-brett]

This is a first attempt to replicate the default Jupyter behavior with
Reticulate.

That is:

• Always display plots
• Do not display results of code expressions unless at last code line.
• Display result of final expression unless suppressed with ;

See: gh-1391

[t-kalinowski]

This LGTM! Thank you!
Can you please add a NEWS entry?

[matthew-brett]

I've added a NEWS entry - look ok?

I had two worries:

• There are currently two ways of preventing code output - the suppress option, that changes the compile_mode, used now for the standard mode, and my post-hoc deletion of the return value for jupyter_compat. Do we need both?
• I guess one problem will be, that we can only detect Matplotlib plotting when the code uses standalone expressions, rather than statements. Is that right? If so, it would be easy to miss with code like arrs, bins = plt.hist(some_array).

[t-kalinowski]

Thanks

• In the news entry, can you please include a link to this pr, the original issue, and if you'd like, your github handle?
• I agree, consolidating the code paths makes sense. I'd be OK always parsing in "single" mode, and then handling or discarding the output afterwards, in eng_python_autoprint() or just after it. Additionally, with this change, it might make sense to completely supplant the 'inspect-the-last-value-and-call-plt.show()-if-its-a-matplotlib-object' logic with the new logic based around matplotlib_pending_show. Seems like that's a cleaner solution over-all.
• We might not even need a jupyter_compat option, and my preference is not to expand the API if it's not needed. If I'm understanding correctly:
  • The repr of a matplotlib plot is by default always suppressed.
  • With this new approach, a plt.show() is always called at the end of a chunk if a matplotlib plot object was returned by any of the individual expressions.
  • The presence or absence of a trailing ; on the last expression of a chunk returning a matplotlib object is then, effectively, a no-op---it makes no difference since the plt.show() will be called regardless, and the repr() is suppressed regardless.

[matthew-brett]

I've added the links to the NEWS entry.

For the logic as currently implemented with jupyter_compat

• Only the final expression gets output displayed.
• Semi-colon will suppress that final output, if present, but otherwise, yes, it's a no-op.
• I defered to eng_python_autoprint for the repr for the Matplotlib-returning expressions, so yes, these are currently suppressed regardless. I guess for full Jupyter compatibility, we should not suppress reprs for the final expression (without a semi-colon).

Did you look at the Matplotlib figure manager? I have a vague memory that is where you can look for any new but not-yet-displayed figures.

[t-kalinowski]

Thank you!

Did you look at the Matplotlib figure manager? I have a vague memory that is where you can look for any new but not-yet-displayed figures.

I have not. I remember having a brief, unsuccessful, look for something like a "pending undisplayed matplotlib plots queue". I'm open to other suggestions for how to to better detect if there is a pending plot, though I think this new approach embodied in this PR seems good.

[t-kalinowski]

I guess for full Jupyter compatibility, we should not suppress reprs for the final expression (without a semi-colon).

Is this necessary? The repr seems to me output that nobody will want in their polished and rendered report, perhaps excepting someone writing a matplotlib tutorial.

I'm inclined to keep the current behavior.

[matthew-brett]

Is this necessary? The repr seems to me output that nobody will want in their polished and rendered report, perhaps excepting someone writing a matplotlib tutorial.

Fair enough - it's certainly niche - although I can imagine a student being surprised they get the output in the Jupyter notebook, and don't see it in the report.

[matthew-brett]

I asked for advice on detecting new plots over on the Matplotlib discourse ....

[matthew-brett]

I've updated the PR with what I think you were suggesting - is it OK?

As you've implied, this is a behavior change - as you can now longer suppress plot output with terminating semicolons - but my guess is that this was very rarely used, intentionally.

The heuristic for detecting plots will miss this:

res = plt.plot(range(10))

but that would need a completely different approach - maybe soneone in the Matplotlib community has an idea. Jupyter has to do this, so it must be possible.

[matthew-brett]

Also tried asking about plot detection over at Jupyter Discourse.

[t-kalinowski]

LGTM! Thank you!