feat: Run exec_request inside a task

[fleming79]

This PR modifies the kernel to run exec_requests inside a task using Memory object streams to queue pending tasks. This means that other shell messages can be processed while the execution request is being processed.

ref: jupyterlab/jupyterlab#17363 (comment)

New features

• Adds the start_soon method to the kernel providing a thread safe equivalent to anyio.start_soon.

Breaking changes

• set_parent & get_parent methods are removed. Instead the parent message is only set for the current execute request providing better certainty for inputs and outputs

Known issues

• Tests pass locally but occasionally fail (need help)

Example

Screenshot

Before

ipykernel.demo.before.mp4

After

ipykernel.demo.after.mp4

Notebook code

Requires:

• ipywidgets
• jupyterlab

import anyio
import time
import asyncio
from anyio import sleep
from ipywidgets import HTML, IntSlider, __version__

slider = IntSlider()
print('ipywidgets', __version__)

# async def run_till_100():
async def run_till_100():
    display(slider)
    reported_value = HTML()
    display(reported_value)
    cval = -1
    while True:
        time.sleep(0.1) # not a good idea as it blocks the main thread
        await sleep(0.1)
        if cval != slider.value:
            cval = slider.value
            reported_value.value = f"Reported value: {cval}"
            print(cval)
        if cval == 100:
            print('Done')
            await anyio.sleep(0.7)
            slider.value = 0
            await anyio.sleep(0.01)
            return 'Done'

await run_till_100()

await asyncio.create_task(run_till_100())

shell = get_ipython()
shell.kernel.start_soon(run_till_100)

[davidbrochart]

Thanks @fleming79.
That's going in the direction of akernel, so I can only approve :)
One important thing though is kernel interrupt and how execute requests are cancelled (I don't see anything around that in the code).

[fleming79]

Thanks @fleming79. That's going in the direction of akernel, so I can only approve :) One important thing though is kernel interrupt and how execute requests are cancelled (I don't see anything around that in the code).

@davidbrochart - thank you for the review and positive feedback!

I've updated the code so it will now cancel all requests where the submitted (monotonic) time is prior to the aborted time.

[fleming79]

Here is a demo video showing a forked version of ipylab using await extensively.

The modified version of ipylab defines async functions to communicate with the frontend via custom widget messages (Widget.on_msg(...)).

ipylab.async.everywhere.demo.mp4

[fleming79]

@davidbrochart - Hoping you can find time to review this PR.

Relevent:

• Subshells on 6.x branch #1387
• Meeting Notes 2025 jupyter-server/team-compass#73 (comment)

[davidbrochart]

Thanks a lot for your work in this PR @fleming79, this looks really interesting.
I don't have much time to dedicate to ipykernel these days, and given that work is ongoing in the 6.x branch, I'm not sure what to do.
I'm curious to hear @minrk's opinion on this.

[fleming79]

All tests pass locally except those that get skipped for some reason in the "inprocess" folder.

[fleming79]

@davidbrochart @minrk @krassowski @ianthomas23 @Carreau

Dear all,

I've drafted a stripped down version of ipykernel starting from this PR and am contacting you seeking feedback/advice.

For the moment I've called it asynckernel and the source is here.

The motivation was to provide an asynchronous native kernel enabling comms to pass whilst a cell is busy awaiting a result. It does this by queuing non-silent execute_requests and running them in a designated task. Silent execute requests are run in their own tasks and are non-cancellable.

To try it out you can pip install it from the source: asynckernel-0.0.0a1.tar.gz

Untitled.video.-.Made.with.Clipchamp.3.mp4

Test code

import anyio
from ipywidgets import IntSlider, HTML, __version__
import time

slider = IntSlider()
value = HTML()
print('ipywidgets', __version__)
display(slider)
display(value)

async def run_till_100():
    while True:
        await anyio.sleep(0.1)
        value.value = str(slider.value)
        if slider.value == 100:
            return 'Done'
print(await run_till_100())