Subshells on 6.x branch

[ianthomas23]

We've been planning the next release of ipykernel to be version 7 with the headline backward-incompatible changes being anyio and the addition of experimental subshell support. The anyio implementation is almost complete but it is proving difficult to finish it without breaking too much downstream, and some concerns have been expressed about the choice of anyio and if we really want to support its use going forwards.

I would like to have subshells available in an ipykernel release and although the current implementation in the main branch builds on top of anyio it does not have to do so, and hence I am proposing to try implementing on the 6.x branch so on top of tornado/asyncio to if it is possible. There was a discussion about this at yesterday's Jupyter Server/Kernel meeting which met with some support.

My plan is to submit PRs to the 6.x branch which I will clearly label with 6.x in each PR title. The initial PRs will be to fix the currently failing CI on that branch, hopefully using backported PRs from main but I may to write different fixes. When CI is passing I will attempt to implement subshells on top of that. If this works and looks reasonable, only then will there need to be a discussion of whether we actually do an ipykernel 6.30.0 release with subshells or not.

[davidbrochart]

So that we're clear, the breaking changes are not due to AnyIO per say, but to the fact that we dropped Tornado and the callback-based ZMQ streams. It would have been the same using asyncio.
AnyIO brings more guaranties than asyncio because of structured concurrency and many more design choices, while providing Trio support for free, thus I think we should keep on using it.

[ianthomas23]

The subshells PR has been merged to the 6.x branch, I will make a prerelease (6.30.0a0) so that it can be tested further.

[ccordoba12]

@ianthomas23, you said in that PR:

Although this is based on the 6.x branch there has been some discussion about whether this might be released as 6.30.0 or as 7.0.0.

Given that this is major feature (i.e. it represents a significant change in functionality), I think it should be released as 7.0.0, not 6.30.0.

In addition, the 6.x branch is quite stable, so I don't think downstream projects that have a pin in 7.0 (like Spyder-kernels) should be forced to rush and try to fix errors or issues that could appear due to this.

[ianthomas23]

The pre-release version number isn't particularly significant, the decision about whether this should be 6.30 or 7.0 has yet to be made.

Given that this is major feature (i.e. it represents a significant change in functionality), I think it should be released as 7.0.0, not 6.30.0.

Thanks for expressing an opinion. I need to hear the preferences of other interested people to justify renaming the main branch to 8.x (or anyio branch?) and renaming 6.x to main.

[ianthomas23]

ipykernel 6.30.0 has been released, from the 6.x branch, see changelog. It does not include subshells, which will be in the upcoming 7.0.0 release. Users and downstream libraries that wish to avoid subshells should pin to ipykernel < 7.

[ianthomas23]

Closing this as issues around release of ipykernel 7 with subshells is tracked in #1438.