New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prototype async REPL using IPython, take III #11265

Merged
merged 32 commits into from Sep 7, 2018

Conversation

Projects
None yet
6 participants
@Carreau
Member

Carreau commented Aug 14, 2018

This is a squash and a rebase of a large number of commits from Min and
I. For simplicity of managing it, history has been reduced to a single
commit, but more historical versions can be found, in particular in
PR #11155, or commit aedb5d6 to be more
exact.

--

I'm expecting some errors in the squash and resolve of conflicts, so will iterate on it.

Carreau added some commits Mar 10, 2017

Prototype async REPL using IPython, take III
This is a squash and a rebase of a large number of commits from Min and
I. For simplicity of managing it, history has been reduced to a single
commit, but more historical versions can be found, in particular in
PR 11155, or commit aedb5d6 to be more
exact.

Carreau added some commits Aug 14, 2018

@Carreau

This comment has been minimized.

Member

Carreau commented Aug 14, 2018

I'm thinking of dropping 3.4, as maintaining compat with it will be painful:

There are no specific plans for the next release of Python 3.4. However, the expectation is that Python 3.4.10 will be released in March of 2019, and this will be the final release of Python 3.4.

We'll also be supporting 3.5, 3.6 and 3.7.

@minrk and @takluyver thoughts ?

@minrk

This comment has been minimized.

Member

minrk commented Aug 16, 2018

asyncio support is pretty valuable and long overdue, so if 3.4 makes it too hard, I think the trade-off is worth it.

setup.py Outdated
@@ -201,6 +201,7 @@
extras_require.update({
':python_version == "3.4"': ['typing'],
':python_version >= "3.5"': ['trio', 'curio'],

This comment has been minimized.

@minrk

minrk Aug 16, 2018

Member

Neither of these are dependencies, right? We shouldn't require these packages.

This comment has been minimized.

@Carreau

Carreau Aug 16, 2018

Member

One of them is necessary if we want to have nested embed(). I can make that optional, I probably added them here to have the test passing.

@Carreau Carreau added this to the 7.0 milestone Aug 16, 2018

Carreau added some commits Aug 16, 2018

dalejung added a commit to dalejung/ipython that referenced this pull request Aug 21, 2018

Only trigger async loop runner when needed. Allows running files that
call async loops synchronously on their own.
#11265

Handle early return conditions from coro
@dalejung

This comment has been minimized.

Contributor

dalejung commented Aug 21, 2018

@Carreau I ran into an issue running python files that have their own loop calls. So if I have a file that calls run_until_complete, it'll break when %run since it nests within the loop_runners own call.

Can we change the run_cell_async coro to first yield interactivity so it can call the pseudo sync when async is not required?

dalejung@8d38b3c

is a working example. await statements in the shell work while running or calling funcs/files that invoke their own loops work as well.

@Carreau

This comment has been minimized.

Member

Carreau commented Aug 21, 2018

I have to better understand the implication. It is likely possible, though it will likely work only on terminal IPython.

It might also not stay forever, as in the end I want IPython to always run on an event loop. Though, I guess we could run only on the pseudo-sync, or a non-asyncio one.

Once that hapend we can figure out how to patch %run in order to patch run_until_complete

@njsmith

This comment has been minimized.

Contributor

njsmith commented Aug 22, 2018

The way I would imagine it working in the long run is:

  • The async annotations spread out from run_cell to infect the rest of IPython, so that the whole program runs under a single call to the event loop
  • By default IPython starts out in 'sync' mode, which means:
    • Using the pseudo-sync runner
    • Raise SyntaxError if someone tries to use a top-level await in a cell
  • You can switch from 'sync' mode to 'asyncio' mode, 'trio' mode etc., which switches IPython to using these loops
    • Internally, this loop switcheroo can be accomplished by yielding a special value to the pseudo-sync runner, which then resumes the same coroutine object under the new loop. This is serious black magic but should work...
    • Also it sets a flag internally to start allowing top-level await inside cells
  • You can use configuration, pass a command-line flag, etc., to preemptively switch into one of the async modes at startup

Of course there's a ways to go before we use async annotations everywhere like that. But we need to eventually reach a point of keeping the same loop alive over multiple cells, if we want this to be really useful. In any case, I think the basic idea of tracking whether we're in sync or async mode, and turning the magic parser on-or-off as appropriate, is a good one to expose to the user. And that's something that can be done now and would solve @dalejung's problem.

@njsmith

This comment has been minimized.

Contributor

njsmith commented Aug 22, 2018

...of course when writing the above I forgot about prompt-toolkit. Besides it needing some work to integrate with all these different loops, I'm not sure that it'll be happy switching between loops mid-stream...

Well, the architecture I described above ought to work for the kernel, or for jupyter console, at least :-). (With no GUI integration enabled.)

Refactor and tweak async-in-function tests
This improves Python3.5 compatibility, since "yield" in an async
function was a SyntaxError in 3.5, but not in later versions

Also adds a test for async methods.
@Carreau

This comment has been minimized.

Member

Carreau commented Aug 27, 2018

Thanks ! Much appreciated.

@Carreau

This comment has been minimized.

Member

Carreau commented Aug 28, 2018

That would close #9166 as well.

@Carreau

This comment has been minimized.

Member

Carreau commented Aug 28, 2018

Todo, catch exception when the runner crash, EG, when using autoawait trio, and then await asyncio.sleep(0).

[EDIT]: Now done.

@Carreau Carreau referenced this pull request Aug 28, 2018

Merged

async support #323

Carreau added some commits Aug 28, 2018

Test that using wrong runner/coroutine pair does not crash.
Make sure that if the coroutine runner encounter an exception (likely
when encountering the wrong coroutine from another aio library), it is
set on the result and does not crash the interpreter.
@Carreau

This comment has been minimized.

Member

Carreau commented Aug 31, 2018

Timeit, time, and prun magics will need updates to work with async code.
They still work for sync code though, so I don't think it is a blocker.

@@ -1,3 +1,6 @@
Depreations

This comment has been minimized.

@minrk

minrk Sep 4, 2018

Member

deprecations

This comment has been minimized.

@Carreau

Carreau Sep 4, 2018

Member

Thanks, Fixed.

interactivity = 'async'
# yield interactivity so let run_cell decide whether to use
# an async loop_runner
yield interactivity

This comment has been minimized.

@minrk

minrk Sep 4, 2018

Member

Adding yield interactivity here makes this a not-entirely-valid coroutine, doesn't it? When I try to call await run_cell_async from ipykernel, I get a bad yield: 'last_expr'. How can we communicate this required info without faking that this isn't actually a coroutine? Does it need to be an actual async generator?

@Carreau

This comment has been minimized.

Member

Carreau commented Sep 4, 2018

@Carreau Carreau force-pushed the Carreau:async-repl-aug-2018 branch from 3156d6d to 85db703 Sep 6, 2018

@Carreau

This comment has been minimized.

Member

Carreau commented Sep 7, 2018

Ok, seem like we've reached quasi static state and most of the latest updates were documentation. Plus ipykernel works with that. So I'm going to merge (thanks all involved !), and we can refine if necessary.

@Carreau Carreau merged commit 0acd580 into ipython:master Sep 7, 2018

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@Carreau Carreau deleted the Carreau:async-repl-aug-2018 branch Sep 7, 2018

@minrk

This comment has been minimized.

Member

minrk commented Sep 7, 2018

Fair enough. I'm just wondering if there's a better way to check if it should be async explicitly first, rather than relying on the fact that coroutines are actually just a special kind of generators, and having to call it in a special way that no other coro can be called.

@minrk

This comment has been minimized.

Member

minrk commented Sep 7, 2018

#11289 makes run_cell_async a regular coroutine by moving the 'should it be async' check to a separate method.

Since in general, running it with a coroutine runner ought to work (I know there are edge cases for now), skipping this check should be safe.

clrpackages pushed a commit to clearlinux-pkgs/ipython that referenced this pull request Oct 3, 2018

[update to 7.0.0] Don't let parentheses level go below 0
Alyssa Whitwell (3):
      Change signature to be imported from inspect library
      Add .python-version to gitignore
      Deprecate use of imp library, condense module_paths module to only one function, update tests

Anatol Ulrich (1):
      fix #11082 - use dpaste.com, also remove py2 compatibility imports

Benjamin Ragan-Kelley (19):
      upgrade pip on appveyor
      Merge pull request #11073 from Carreau/remove-io-meth
      Merge pull request #11072 from Carreau/remove-profile
      pretty-rendering for os.environ by default
      Merge pull request #11177 from takluyver/prompt-toolkit-2
      Merge pull request #11185 from gpotter2/spec-missing
      Merge pull request #11172 from cedric05/master
      Merge pull request #11174 from Carreau/close-bg
      Merge pull request #11178 from toddrme2178/patch-1
      Merge pull request #11250 from Carreau/invalid-escape-sequence
      Merge pull request #11278 from Carreau/cleanup-doctest
      make `run_cell_async` a regular coroutine
      exercise calling run_cell_async as a coroutine
      catch BaseException in coroutine runner
      Merge pull request #11287 from Carreau/bash-raise
      Merge pull request #11307 from Carreau/safe-input-transformer
      Merge pull request #11305 from Carreau/more-autoawait-docs
      Merge pull request #11311 from Carreau/rst-formatting
      Merge pull request #11301 from Carreau/cleanup-manifest

Chaz Reid (1):
      Document -o flag for logstart magic

Christoph (5):
      Disable qt inputhook backend when DISPLAY is not set on linux, because the qt lib does not handle this correct
      apply review comments
      type
      check also WAYLAND_DISPLAY
      apply comments

Dale Jung (1):
      Only trigger async loop runner when needed. Allows running files that call async loops synchronously on their own. ipython/ipython#11265

Dave Hirschfeld (1):
      Allow specifying the name of the root element of a JSON display object

Fernando Perez (1):
      Merge pull request #11107 from zichongkao/repr_doc

Fred Mitchell (1):
      Create LICENSE

Gabriel Potter (1):
      Fix autogen_shortcuts.py for prompt_toolkit 2.0

Grant Nestor (2):
      Drop jquery dependency in IPython.display.Javascript
      Take jquery's getScript approach

Hugo (1):
      Drop support for Python 3.3

Jonathan Slenders (1):
      Upgrade to prompt_toolkit 2.0

Jörg Dietrich (2):
      Add ipython_tokens for syntax highlighting following cell magic
      add three tests for cell magic syntax highlighting

Kyle Kelley (1):
      Merge pull request #11267 from gnestor/display-javascript

M Pacer (1):
      create new --raise-error argument to raise an error if nonzero exitcode

Matthew Seal (2):
      Added fix for display hook call output format
      Added missing imports for test

Matthias Bussonnier (145):
      Remove Deprecation Warning, add since when things were deprecated.
      Merge pull request #10833 from takluyver/req-py34
      remove some py3compat usage
      Fix readme: Quote to Codeblock
      Update LICENCE and Copying to match.
      ignore .editorconfig
      Remove deprecated profile line magic.
      Deprecate some methods from IPython.utils.io
      Merge pull request #10648 from srinivasreddy/srinivasreddy-patch-1
      Merge pull request #11110 from takluyver/skip-jedi-omission-test
      Merge pull request #11109 from stonebig/patch-1
      Merge pull request #11106 from takluyver/displayobjs-metadata
      Merge pull request #11101 from MSeal/captureHookFixPy3
      What's new in 5.7
      What's new in 6.4
      Merge pull request #11119 from Carreau/wn64
      Refactor of coloring and traceback mechanism.
      Traceback printing change `in <module>()` to `in <module>`.
      Some other cleaning of ultratb
      Remove import indirection and deprecated private function.
      Fix timeti test nightly/3.7
      Update execution.py
      fix tests
      Leak less files during test.
      Fix test for Python 3.7+
      test on 3.7
      Merge pull request #11137 from joergdietrich/magic_highlighting
      Merge pull request #11131 from Carreau/fix-timeit-nightly
      Merge pull request #11135 from Carreau/fix-37
      Merge pull request #11134 from Carreau/fix-fleaks
      Merge pull request #11129 from Carreau/cleanup
      Merge pull request #11127 from Carreau/clean-tb
      Merge pull request #11128 from Carreau/non-callable-module
      Some cleanup of Pycolorize.
      Merge pull request #11130 from Carreau/less-indir
      remove unsused utitlity function
      Merge pull request #11148 from Carreau/deprecated
      Bump Deprecation to 8.0, API has not stabilized.
      Autoclose stdin/err for bg scripts, when not assigned to user_ns
      Correctly tearDown some test cases with multiple inheritance.
      Swap base class instead of redefinign the tearDown method
      add docstring, and emit deprecation warnings
      Merge pull request #11211 from dhirschfeld/patch-2
      Merge pull request #11192 from alphaCTzo7G/fix_typo_ipython_magic
      Merge pull request #11199 from mpacer/script_raise_error
      Merge pull request #11227 from yen223/11226-3-7-autocomplete-bugfix
      Upgrade pip before setuptools for Python compat.
      Switch protocol to https
      Merge pull request #11173 from Carreau/reswarn
      what's new in 6.5
      fix release instructions
      Merge pull request #11245 from Carreau/fix-release-instructions
      Update version6.rst
      Merge pull request #11243 from Carreau/whatsnew65
      Use raw string when invalid escape sequence present.
      Fix more escape sequences
      Merge pull request #11255 from alyssawhitwell/add-python-version-to-gitignore
      Merge pull request #11254 from alyssawhitwell/11206-signature-warning
      Merge pull request #11253 from parente/fix-ipynb
      Update deprecation warning message
      Merge pull request #11256 from alyssawhitwell/deprecate-imp-import
      Merge pull request #11251 from Carreau/deprecated-imports
      Prototype async REPL using IPython, take III
      fix UnboundLocalError
      titleto have doc build
      fix docs
      Load the asycn ext only on 3.5+
      fix runnign on 3.7
      remove duplicate WatsNew from bad rebase
      test with trio
      3.7 still nto on travis
      reformat with black
      doc, remove  appveyor 3.4
      move infor to the right place
      DeprecationWarning
      runblack on new tests
      att triio runner at top level now that 3.4 drop + docs
      Add pseudo sync mode
      no f-strings
      docs cleanup, reformat code, remove dead code.
      remove trio from test requirement and skipp if not there
      fix test
      Merge pull request #11271 from luzpaz/misc-typo-and-whitespaces
      Test that using wrong runner/coroutine pair does not crash.
      run tests with trio and curio on travis
      Undeprecate autoindent
      Merge remote-tracking branch 'origin/master' into inputtransformer2
      add notes some magic don't work yet
      Merge remote-tracking branch 'origin/master' into inputtransformer2
      Actually run some tests.
      Merge pull request #11041 from takluyver/inputtransformer2
      Remove Python 2 shim.
      Update test_ipdoctest.py
      Merge pull request #11277 from Carreau/map-is-lazy
      typo
      add tests
      test both old and new classes
      Add some extra metadata for PyPI (in the sidebar)
      remove unused utils function in setupbase.py
      Switch scripts cell magic default to raise if error happens.
      Merge pull request #11283 from oscar6echo/improve-autoreload
      Tool to go around sphinx limitation during developpement
      run doc fixing tool on travis
      Merge pull request #11288 from tkf/inputhook
      Merge pull request #11286 from Carreau/unused-utils-in-setupbase
      Merge pull request #11285 from Carreau/project_url
      Merge pull request #11265 from Carreau/async-repl-aug-2018
      Start to updates the what's new / changelog.
      update release process
      fix comments
      build error on traivs
      Merge pull request #11289 from minrk/async-real-coroutine
      Add debug and fix build hopefully
      Merge pull request #11291 from minrk/catch-base-exception
      typos
      Merge pull request #10974 from boeddeker/master
      Merge pull request #11290 from Carreau/wn7
      cleanup build process
      Remove include from MANIFEST
      Merge pull request #11304 from michael-k/python3.7
      Merge pull request #11300 from luzpaz/misc-typos
      Revert 3.7 AST fix
      update docs about autoawait
      Add some inputtransformer test cases
      Some propose fixes.
      Better alternative; try each transformer in a row,
      fix Paul's comments
      one more rephrasing/plural
      one more plural
      Rst formatting (typo and decrease level to not be in sidebar)
      Fix magic directive and role.
      Update autoawait.rst
      Merge pull request #11308 from yarko/patch-1
      Merge pull request #11313 from Carreau/fix-magic-directive
      update instructions: --pre apply to rc as well
      Merge pull request #11318 from blink1073/allow-setting-matchers
      Merge pull request #11315 from Carreau/instr-rc
      Fix the sphinx_ipython directive.
      Merge pull request #11321 from Carreau/fix-ipython-directive
      Merge pull request #11323 from hongshaoyang/patch-1
      release 7.0.0rc1
      back to development
      Prepare changelog for relese
      Update what's new and stats
      release 7.0.0

Michael Käufl (1):
      [travis] Run tests against Python 3.7 and a non-outdated nightly

Paul Ganssle (5):
      Improve async detection mechanism with blacklist
      Add tests for SyntaxError with top-level return
      Fix indentation problem with _asyncify
      Fix async_helpers tests
      Refactor and tweak async-in-function tests

Paul Ivanov (8):
      updated what's new
      fix typos, migrate latest to release notes
      add preliminary 7.0 release stats
      release 7.0.0b1
      back to development
      Merge pull request #11299 from Carreau/cleanup-build-process
      Merge pull request #11162 from Carreau/revert-37-ast-fix
      minor typo and style tweak

Peter Parente (1):
      Fix breaking newline

Sourav Singh (1):
      Add deprecationwarning with stacklevel

Srinivas Reddy Thatiparthy (1):
      remove deprecated variable - default_gui_banner

Steven Silvester (1):
      Allow showing dict keys only

Subhendu Ranjan Mishra (2):
      Fixes #11068 : Cleanup extra logic that handle python <=3.3 in IPython/core/extensions.py
      Issue #11068 : Cleanup extra logic that handle python <=3.3 in IPython/utils/openpy.py

Takafumi Arakaki (1):
      Pass inputhook to prompt_toolkit

Thomas A Caswell (1):
      FIX: Remove the non-interactive backends from pylabtools.backend2gui

Thomas Kluyver (109):
      Working on new input transformation machinery
      Start adding tests for inputtransformer2
      Add transformation for system assignments
      Factor out handling of line continuations
      Debugging function to see tokens
      Escaped commands
      Some more tests for escaped commands
      Transformations for 'help?' syntax
      Fix cell magic transformation
      More tests for line-based input transformers
      Start integrating new input transformation machinery into InteractiveShell
      Start adding code for checking when input is complete
      Start integrating new machinery for checking code completeness
      Fix fallback prompt & improve info on test failure
      Fix complete checking with space after backslash
      Ensure that result is defined in run_cell()
      Update test for new transformation API
      Document the new input transformation API
      Switch some references to input_splitter to input_transformer_manager
      Remove unused TerminalMagics.input_splitter
      Update sphinxext for new API
      Remove unused import
      Switch some imports from inputsplitter to inputtransformer2
      Mark inputsplitter & inputtransformer as deprecated
      Switch inputtransformer2 back to stdlib tokenize module
      Drop bundled, outdated copy of the tokenize module
      Add shell.check_complete() method
      Convert syntax warnings to errors when checking code completeness
      This is an incompatible change
      Working on new input transformation machinery
      Start adding tests for inputtransformer2
      Add transformation for system assignments
      Factor out handling of line continuations
      Debugging function to see tokens
      Escaped commands
      Some more tests for escaped commands
      Transformations for 'help?' syntax
      Fix cell magic transformation
      More tests for line-based input transformers
      Start integrating new input transformation machinery into InteractiveShell
      Start adding code for checking when input is complete
      Start integrating new machinery for checking code completeness
      Fix fallback prompt & improve info on test failure
      Fix complete checking with space after backslash
      Ensure that result is defined in run_cell()
      Update test for new transformation API
      Document the new input transformation API
      Switch some references to input_splitter to input_transformer_manager
      Remove unused TerminalMagics.input_splitter
      Update sphinxext for new API
      Remove unused import
      Switch some imports from inputsplitter to inputtransformer2
      Mark inputsplitter & inputtransformer as deprecated
      Switch inputtransformer2 back to stdlib tokenize module
      Drop bundled, outdated copy of the tokenize module
      Add shell.check_complete() method
      Convert syntax warnings to errors when checking code completeness
      This is an incompatible change
      master branch is now 7.0 dev
      Add whatsnew development doc back to toc
      Drop support for Python 3.3
      Bump up Python version on Appveyor too
      Simplify some pathlib imports
      Update error message in setup.py
      Update version numbers for Python support
      Merge pull request #11069 from minrk/upgrade-pip
      Tweak wording referring to LICENSE
      Merge pull request #11067 from Carreau/less-2-compat
      Merge pull request #11018 from souravsingh/add-cod
      Merge pull request #11071 from Carreau/rdm
      Merge pull request #10758 from fm75/master
      Disable Jedi completions by default
      Enable jedi for tests that apply to that
      Move where use_jedi is disabled again back in test
      Correct help string for config option
      Add changelog for 6.3.1
      Merge pull request #11076 from takluyver/disable-jedi
      Merge pull request #11087 from tacaswell/fix_noninteractive_backends
      Merge pull request #11093 from subhrm/11068-B
      Merge pull request #11092 from subhrm/11068-A
      Display objects should emit metadata when it exists
      Comment out jedi parts of test_omit__names
      Merge pull request #11118 from Carreau/wn57
      Merge pull request #11123 from luzpaz/fix-typos
      Merge pull request #11142 from Carreau/cleanup-pyc
      Merge pull request #11147 from charlesreid1/patch-1
      Merge pull request #11156 from Carreau/completerdeprecationbump
      Merge branch 'inputtransformer2' of github.com:takluyver/ipython into inputtransformer2
      Don't let parentheses level go below 0
      Prevent infinite loops in input transformation
      Merge pull request #11164 from minrk/os-environ-pretty
      Arrange imports, add missing, remove unused
      Remove commented method
      Fix input rewrite prompt
      No newline after output prompt
      Merge pull request #11181 from apunisal/master
      Merge pull request #11182 from spookyvision/master
      Merge pull request #11190 from infmagic2047/fix-completion-style
      Simpler mechanism to extend input transformations
      Merge branch 'master' into inputtransformer2
      Fix generating shortcuts docs
      Shortcut to transform_cell in magics code
      Update tests for input transformer raising SyntaxError
      Update transformer test in terminal test case
      Add & improve docstrings following @willingc's review
      Reformat for readability
      Add description of priority system
      Merge pull request #11235 from Carreau/https
      Merge pull request #11234 from Carreau/pip-first

Todd (1):
      Include LICENSE file in wheels

Wei Yen (3):
      Add regression test
      Ensure `try_import` does not fail on null module.__file__
      Reset sys.path after test

Yarko Tymciurak (1):
      update documentation to add beta install instructions

Yutao Yuan (1):
      Fix config option TerminalInteractiveShell.display_completions

Zi Chong Kao (2):
      Document that _repr_*_() can return metadata
      Document that _repr_*_() can return metadata

alphaCTzo7G (1):
      Fixes default value of -r<R> and updates documentation

apunisal (1):
      Changed a misspelled word in tools/git-mpr.py

dhirschf (2):
      Update test_json to account for new metadata
      Update .gitignore

gpotter2 (1):
      Add __spec__ to DummyMod

hongshaoyang (1):
      Minor edits to magics doc

luz.paz (4):
      Fix documentation typos
      Misc. typo fixes
      Whitespace fixes
      typos

meeseeksdev[bot] (1):
      Merge pull request #11273 from Carreau/re-autoindent

oscar6echo (3):
      Add new methods in update_class()
      Fix inconsequential typo in test
      Add file in whatsnew/pr

prasanth (1):
      checking weather virtualenv dir exists or not

stonebig (1):
      https for all readthedocs.io links
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment