Can't use flow.serve or serve(flow.to_deployment(...)) with a 2.x client using a 3.x server

[abrookins]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the Prefect documentation for this issue.
• I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

Trying to use flow.serve() or serve(flow.to_deployment(...)) results in an API error that extra fields aren't allowed, with infra_overrides being the field. I.e. the client is sending infra_overrides and the 3.0 server doesn't know about this field.

Reproduction

# FYI, sorry, this was also test flow for job variables :D
import sys
import os

from prefect import flow, get_run_logger, serve


@flow
def test_flow():
    logger = get_run_logger()
    logger.info(f"Hello {os.environ.get('name')}")


if __name__ == '__main__':
    cmd = sys.argv[1] if len(sys.argv) > 1 else None
    if cmd == 'deploy':
        deployment = test_flow.to_deployment(
            "test-flow-deployment-infra",
            work_pool_name="test",
            job_variables={
                "user": {
                    "name": "bob"
                }
            }
        )
        serve(deployment)
    elif cmd == 'serve':
        test_flow.serve()
    else:
        print("Invalid command")
        sys.exit(1)

Error

(env) ~/src/flows main ❯ python deployment_from_flow.py serve
Traceback (most recent call last):
  File "/private/tmp/env/lib/python3.11/site-packages/prefect/deployments/runner.py", line 318, in apply
    deployment_id = await client.create_deployment(**create_payload)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/env/lib/python3.11/site-packages/prefect/client/orchestration.py", line 1678, in create_deployment
    response = await self._client.post(
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/env/lib/python3.11/site-packages/httpx/_client.py", line 1892, in post
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/env/lib/python3.11/site-packages/httpx/_client.py", line 1574, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/env/lib/python3.11/site-packages/prefect/client/base.py", line 358, in send
    response.raise_for_status()
  File "/private/tmp/env/lib/python3.11/site-packages/prefect/client/base.py", line 171, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '422 Unprocessable Entity' for url 'http://127.0.0.1:4200/api/deployments/'
Response: {'exception_message': 'Invalid request received.', 'exception_detail': [{'type': 'extra_forbidden', 'loc': ['body', 'infra_overrides'], 'msg': 'Extra inputs are not permitted', 'input': {}}], 'request_body': {'infra_overrides': {}, 'name': 'test-flow', 'flow_id': 'eb44f666-ee1c-4a6c-bb7c-9288e36c4a31', 'paused': False, 'schedules': [], 'enforce_parameter_schema': False, 'parameter_openapi_schema': {'title': 'Parameters', 'type': 'object', 'properties': {}}, 'parameters': {}, 'tags': [], 'manifest_path': None, 'work_queue_name': None, 'storage_document_id': None, 'infrastructure_document_id': None, 'schedule': None, 'description': None, 'path': '.', 'version': 'e7d81b87e286b0fcfc6f1cbac38ccb04', 'entrypoint': 'deployment_from_flow.py:test_flow'}}
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/422

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/andrew/src/flows/deployment_from_flow.py", line 27, in <module>
    test_flow.serve()
  File "/private/tmp/env/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 304, in coroutine_wrapper
    return call()
           ^^^^^^
  File "/private/tmp/env/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 432, in __call__
    return self.result()
           ^^^^^^^^^^^^^
  File "/private/tmp/env/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/env/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/private/tmp/env/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/private/tmp/env/lib/python3.11/site-packages/prefect/flows.py", line 824, in serve
    deployment_id = await runner.add_flow(
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/env/lib/python3.11/site-packages/prefect/runner/runner.py", line 291, in add_flow
    return await self.add_deployment(deployment)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/env/lib/python3.11/site-packages/prefect/runner/runner.py", line 201, in add_deployment
    deployment_id = await deployment.apply()
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/env/lib/python3.11/site-packages/prefect/deployments/runner.py", line 324, in apply
    raise DeploymentApplyError(
prefect.deployments.runner.DeploymentApplyError: Error while applying deployment: Client error '422 Unprocessable Entity' for url 'http://127.0.0.1:4200/api/deployments/'
Response: {'exception_message': 'Invalid request received.', 'exception_detail': [{'type': 'extra_forbidden', 'loc': ['body', 'infra_overrides'], 'msg': 'Extra inputs are not permitted', 'input': {}}], 'request_body': {'infra_overrides': {}, 'name': 'test-flow', 'flow_id': 'eb44f666-ee1c-4a6c-bb7c-9288e36c4a31', 'paused': False, 'schedules': [], 'enforce_parameter_schema': False, 'parameter_openapi_schema': {'title': 'Parameters', 'type': 'object', 'properties': {}}, 'parameters': {}, 'tags': [], 'manifest_path': None, 'work_queue_name': None, 'storage_document_id': None, 'infrastructure_document_id': None, 'schedule': None, 'description': None, 'path': '.', 'version': 'e7d81b87e286b0fcfc6f1cbac38ccb04', 'entrypoint': 'deployment_from_flow.py:test_flow'}}
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/422

Versions

Server:
Version:             3.0.0rc3
API version:         0.8.4
Python version:      3.11.6
Git commit:          8a59fb49
Built:               Mon, Jun 17, 2024 9:13 AM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         server
Pydantic version:    2.7.3

Client:
(env) ~/src/flows main ❯ prefect version
Version:             2.19.5
API version:         0.8.4
Python version:      3.11.6
Git commit:          538dc374
Built:               Thu, Jun 13, 2024 2:50 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         server

Additional context

No response

[alan-icebreaker]

Same for me :(

[aaazzam]

Closing this issue as it's no longer relevant for Prefect 3.0.

While Prefect Cloud is compatible with both 2.x and 3.x clients, 2.x clients are not compatible with self-hosted 3.x servers. This was an intentional choice not made lightly, but was ultimately necessary to introduce our Eventing system and a whole suite of performance improvements. This means Prefect users will need to coordinate their client and server upgrades.

To use Prefect 3.0 features:

• Use 3.x clients with 3.x servers
• 2.x clients must use 2.x servers

Check the 3.X documentation and release notes for upgrade information. For issues with matching client/server versions (e.g., 3.x client with 3.x server), open a new issue with your setup details and problem description.