anyio 4.0.0 introduces breaking changes for prefect

[zzstoatzz]

changelog for anyio here detailing backwards incompatible changes to cancellation semantics

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the Prefect documentation for this issue.
• I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

anyio==4.0.0 seems to include a breaking change that includes moving start_blocking_portal

Reproduction

pip install -U prefect anyio

Error

Successfully installed anyio-4.0.0 prefect-2.11.5
❯ prefect
Traceback (most recent call last):
  File "/Users/nate/micromamba/envs/bleeding-prefect/bin/prefect", line 5, in <module>
    from prefect.cli import app
  File "/Users/nate/micromamba/envs/bleeding-prefect/lib/python3.11/site-packages/prefect/__init__.py", line 45, in <module>
    from prefect.engine import pause_flow_run, resume_flow_run
  File "/Users/nate/micromamba/envs/bleeding-prefect/lib/python3.11/site-packages/prefect/engine.py", line 98, in <module>
    from anyio import start_blocking_portal
ImportError: cannot import name 'start_blocking_portal' from 'anyio' (/Users/nate/micromamba/envs/bleeding-prefect/lib/python3.11/site-packages/anyio/__init__.py)

Versions

# prefect version does not work with the problematic env due to the ImportError
❯ pip list | grep 'prefect\|anyio'
anyio                                        4.0.0
prefect                                      2.11.5

Additional context

No response

[cicdw]

The earlier we can get this in the better - @zzstoatzz can you figure out where that function moved to?

[zzstoatzz]

The earlier we can get this in the better - @zzstoatzz can you figure out where that function moved to?

it moved to

from anyio.from_thread import start_blocking_portal

but it seems the more gnarly issue might be the backwards imcompatable changes to cancellation here

[xarasco]

We're getting this same failure when running the below command in a CI context. We haven't experienced this yet in our local machines and I will ask folks to avoid upgrading anything for now. But we would like to be able to commit new code. :)
prefect config set ...

[abhishekm96]

Downgrading anyio to 3.7.1 temporarily solves the issue.
pip install anyio==3.7.1

[ori-scala]

A question to the Prefect engineering team - Why aren't you pinning the requirements, or at least using < to avoid major version upgrades that break the product?

[Cediddi]

This is breaking PrefectHQ/actions-prefect-auth@v1 and we can't provide a requirements.txt to fix anyio==3.7.1 to that GitHub action, thus all GitHub actions are down.

[cicdw]

@ori-scala good question: as a library, we try to be as liberal as possible with our dependencies (until we know there's a specific incompatibility of course - the next release will include an upper bound for anyio). This gives us maximal install flexibility across our user base. Our users, on the other hand, are developing applications and as a matter of best practice applications should have tightly pinned dependencies.

I don't think we have this documented clearly anywhere, so this is a good reminder - we'll write up a guide in our docs and I'll link to it here when it's ready for reference.

[ori-scala]

@cicdw Thank you Chris for the detailed explanation. Going forward, we will be pinning every dependency in Prefect's requirements.txt to avoid such cases in the future.
I do believe that even as a library, using < for major versions (which usually means "breaking changes") is a good compromise between flexibility and stability.
For example for the specific case here, having:
anyio >= 3.4.0, < 4.0.0
Instead of
anyio >= 3.4.0

would have:

1. Saved dozens of applications from complete failure once anyio 4.0.0 was published.
2. Let the Prefect team work on the flexibility aspect and release a proper upgrade as part of your next release.
3. Help you avoid a quick-and-dirty hotfix with the change above.

Also note the exact same thing happen last month with PyDantic: #10143

[cicdw]

The pydantic v2 issue is a good second data point; I do see a few other references to folks wanting major version upper bounds in our issues, so I'll take this to the team and see where we land.

[peterbygrave]

The pydantic v2 issue is a good second data point; I do see a few other references to folks wanting major version upper bounds in our issues, so I'll take this to the team and see where we land.

If you decide to major version upper bounds, please can you upstream this to conda-forge too: https://github.com/conda-forge/prefect-feedstock/blob/main/recipe/meta.yaml

[JustinGuese]

appeared again today

[cicdw]

@JustinGuese can you say more? Prefect requires anyio > 4.4.0 now.