Upgrade anyio to v3 #492

mwakaba2 · 2021-04-24T18:45:49Z

This fixes #487 (Anyio version 3 breaks AsyncContentsManager).

Changes

Upgrade anyio from v2 to v3.
Replace deprecated function run_sync_in_worker_thread with to_thread.run_sync.
Tests: Prefix contents manager fixture with jp

How to test

$  pip install -e ".[test]"
$  pip install git+https://github.com/agronholm/anyio.git@fix-264
$  python -m pytest -v jupyter_server/tests/services/contents/

This PR depends on Anyio's Refactored asyncio thread pooling .

Todo

Test against published anyio version that includes the Refactored asyncio thread pooling PR.

setup.cfg

kevin-bates · 2021-04-26T15:23:05Z

Excellent, thank you @mwakaba2 and @agronholm! I'm going to place this into DRAFT for now until Alex's anyio PR is merged and released.

ipython_genutils is complicated to package on some linux distro because of nose, and in general not useful as it mostly offered python 2/3 compat layer. Goal always has been to remove any usage of ipython_genutils

This uses an autoreformatter to reformat and fix some of the common mistakes in numpydoc. Some of this is purely visual, but other like space before colon in parameters actually have semantic values. In the case of space before colon this is to make sure that nupmydoc properly distinguish the parameter name from its type.

This change also prevents permission-related exceptions from logging the file. Co-authored-by: Shane Canon <scanon@gmail.com> Co-authored-by: Thomas Kluyver <thomas@kluyver.me.uk>

codecov-commenter · 2021-05-01T02:59:08Z

Codecov Report

Merging #492 (eff9b0a) into master (ad9ccb3) will decrease coverage by 0.03%.
The diff coverage is 93.22%.

@@            Coverage Diff             @@
##           master     #492      +/-   ##
==========================================
- Coverage   77.74%   77.71%   -0.04%     
==========================================
  Files         106      106              
  Lines        9276     9288      +12     
  Branches     1001     1001              
==========================================
+ Hits         7212     7218       +6     
- Misses       1707     1711       +4     
- Partials      357      359       +2

Impacted Files	Coverage Δ
jupyter_server/services/contents/filemanager.py	`66.52% <76.92%> (+0.21%)`	⬆️
jupyter_server/services/contents/fileio.py	`75.64% <90.90%> (+0.38%)`	⬆️
...upyter_server/services/contents/filecheckpoints.py	`63.58% <100.00%> (+0.64%)`	⬆️
...pyter_server/services/contents/largefilemanager.py	`82.97% <100.00%> (+0.56%)`	⬆️
...ter_server/tests/services/contents/test_manager.py	`97.28% <100.00%> (ø)`
jupyter_server/services/kernels/handlers.py	`61.37% <0.00%> (-1.59%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad9ccb3...eff9b0a. Read the comment docs.

kevin-bates · 2021-05-01T16:16:47Z

Good to see the quick turnaround on the anyio patch, although it looks like 3.6 still has some heartburn.

Choices...

Cap anyio < 3 for python < 3.7 and alias run_sync to run_sync_in_worker_thread on 3.6?
Disable AsyncContentsManager on 3.6 given it is EOL at EOY and probably has few consumers at this point?
Try to determine what it is about 3.6 that's causing the hang?
?

The PR for #490 will probably need option 1.

yan12125 · 2021-05-02T14:11:27Z

Disable AsyncContentsManager on 3.6 given it is EOL at EOY and probably has few consumers at this point?

FWIW, according to https://pypistats.org/packages/jupyter-server, Python 3.6 downloads account for 13%~36% on PyPI for the past 30 days.

kevin-bates · 2021-05-02T15:14:22Z

Thanks for the metric @yan12125 - that's good to know, although I suspect the percentage overriding the default contents manager to the async version is nearly zero. Nevertheless, I think the first option would be the best step forward (barring a more graceful solution) as it can also be used to accomplish #490.

mwakaba2 · 2021-05-03T02:24:43Z

Yeah I think the first option sounds good to unblock #490. I'll also look into the issue with python 3.6 and check with anyio if it's related to their new update.

agronholm · 2021-05-03T07:34:54Z

If the problem with py3.6 is caused by AnyIO, I would like to know.

mwakaba2 · 2021-05-03T14:20:40Z

@kevin-bates I'd appreciate your review on this PR.

@agronholm I created a new issue for the py3.6 error. #505 It looks like it's an anyio issue.
I'd appreciate if you can take a look. I'll be able to debug further after this Wednesday.

kevin-bates

These changes look good and allow us to proceed with #490. Thank you!

mwakaba2 · 2021-05-03T15:29:49Z

thank you @kevin-bates ! I don't have permission to merge the pull request, so if it all looks good, please feel free to merge it.

kevin-bates · 2021-05-03T15:32:21Z

Thanks Mariko - I was giving others a chance to comment, but these seem straightforward enough - so I'll go ahead.

Zsailer · 2021-05-03T20:33:11Z

@mwakaba2, I've upgraded your permissions to "write" 😎 . My apologies for not doing this sooner!

mwakaba2 · 2021-05-03T22:47:10Z

sweet! thanks so much. I look forward to contributing more! 😎

Upgrade anyio to v3

error when running unittest: /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:178: in exec_module exec(co, module.__dict__) enterprise_gateway/tests/test_gatewayapp.py:11: in <module> from enterprise_gateway.enterprisegatewayapp import EnterpriseGatewayApp enterprise_gateway/enterprisegatewayapp.py:19: in <module> from jupyter_server.serverapp import random_ports /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/jupyter_server/serverapp.py:77: in <module> from .services.contents.filemanager import AsyncFileContentsManager, FileContentsManager /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/jupyter_server/services/contents/filemanager.py:15: in <module> from anyio import run_sync_in_worker_thread E ImportError: cannot import name 'run_sync_in_worker_thread' from 'anyio' (/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/anyio/__init__.py) ref: jupyter-server/jupyter_server@678878f jupyter-server/jupyter_server#492

mwakaba2 commented Apr 24, 2021

View reviewed changes

setup.cfg Outdated Show resolved Hide resolved

hMED22 mentioned this pull request Apr 25, 2021

nbconvert handler is affected by slow filesystems and blocks event loop #490

Closed

kevin-bates marked this pull request as draft April 26, 2021 15:23

Carreau and others added 6 commits April 30, 2021 20:43

Remove some of ipython_genutils no-op.

d040e51

ipython_genutils is complicated to package on some linux distro because of nose, and in general not useful as it mostly offered python 2/3 compat layer. Goal always has been to remove any usage of ipython_genutils

Prep for Release Helper Usage

e6b46cd

Remove permissions from CI workflow

d80df5b

Fix for recursive symlink - Notebook 4670

80d9ad5

This change also prevents permission-related exceptions from logging the file. Co-authored-by: Shane Canon <scanon@gmail.com> Co-authored-by: Thomas Kluyver <thomas@kluyver.me.uk>

Merge branch 'master' of github.com:jupyter/jupyter_server

6dce4d9

mwakaba2 force-pushed the upgrade-anyio-3 branch 2 times, most recently from 280c3da to 409b5b2 Compare May 1, 2021 02:54

Upgrade anyio to v3

02d2e8b

mwakaba2 force-pushed the upgrade-anyio-3 branch from 409b5b2 to 02d2e8b Compare May 1, 2021 02:56

mwakaba2 changed the title ~~[WIP] Upgrade anyio to v3~~ Upgrade anyio to v3 May 1, 2021

mwakaba2 changed the title ~~Upgrade anyio to v3~~ [WIP] Upgrade anyio to v3 May 1, 2021

Cap anyio < 3 for python < 3.7

eff9b0a

mwakaba2 changed the title ~~[WIP] Upgrade anyio to v3~~ Upgrade anyio to v3 May 3, 2021

mwakaba2 marked this pull request as ready for review May 3, 2021 03:05

kevin-bates approved these changes May 3, 2021

View reviewed changes

kevin-bates merged commit 678878f into jupyter-server:master May 3, 2021

mwakaba2 deleted the upgrade-anyio-3 branch May 3, 2021 22:47

blink1073 mentioned this pull request May 6, 2021

Jupyter Server Notes 2021 jupyter-server/team-compass#4

Closed

hMED22 mentioned this pull request May 7, 2021

Make nbconvert root handler asynchronous #512

Merged

hMED22 pushed a commit to hMED22/jupyter_server that referenced this pull request Jan 23, 2023

Merge pull request jupyter-server#492 from mwakaba2/upgrade-anyio-3

2e89f58

Upgrade anyio to v3

luong-komorebi mentioned this pull request Sep 24, 2023

Drop support for python 3.7 and upgrade min version of some dependencies jupyter-server/enterprise_gateway#1334

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade anyio to v3 #492

Upgrade anyio to v3 #492

mwakaba2 commented Apr 24, 2021 •

edited

Loading

kevin-bates commented Apr 26, 2021

codecov-commenter commented May 1, 2021 •

edited

Loading

kevin-bates commented May 1, 2021

yan12125 commented May 2, 2021

kevin-bates commented May 2, 2021

mwakaba2 commented May 3, 2021

agronholm commented May 3, 2021

mwakaba2 commented May 3, 2021

kevin-bates left a comment

mwakaba2 commented May 3, 2021

kevin-bates commented May 3, 2021

Zsailer commented May 3, 2021

mwakaba2 commented May 3, 2021

Upgrade anyio to v3 #492

Upgrade anyio to v3 #492

Conversation

mwakaba2 commented Apr 24, 2021 • edited Loading

Changes

How to test

Todo

kevin-bates commented Apr 26, 2021

codecov-commenter commented May 1, 2021 • edited Loading

Codecov Report

kevin-bates commented May 1, 2021

yan12125 commented May 2, 2021

kevin-bates commented May 2, 2021

mwakaba2 commented May 3, 2021

agronholm commented May 3, 2021

mwakaba2 commented May 3, 2021

kevin-bates left a comment

Choose a reason for hiding this comment

mwakaba2 commented May 3, 2021

kevin-bates commented May 3, 2021

Zsailer commented May 3, 2021

mwakaba2 commented May 3, 2021

mwakaba2 commented Apr 24, 2021 •

edited

Loading

codecov-commenter commented May 1, 2021 •

edited

Loading