Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid creating an extra client instance during result factory creation #8072

Merged
merged 2 commits into from
Jan 5, 2023

Conversation

zanieb
Copy link
Contributor

@zanieb zanieb commented Jan 5, 2023

Without this change, you can see an extra client is created during result factory creation

❯ python example.py
12:22:42.162 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////Users/mz/.prefect/orion.db
Stack (most recent call last):
  File "/Users/mz/dev/prefect/example.py", line 16, in <module>
    anyio.run(test_flow)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/asyncio/base_events.py", line 634, in run_until_complete
    self.run_forever()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
    self._run_once()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
    handle._run()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
12:22:42.208 | INFO    | prefect.engine - Created flow run 'famous-bulldog' for flow 'test-flow'
12:22:42.209 | DEBUG   | Flow run 'famous-bulldog' - Starting 'ConcurrentTaskRunner'; submitted tasks will be run concurrently...
12:22:42.210 | DEBUG   | prefect.task_runner.concurrent - Starting task runner...
12:22:42.211 | DEBUG   | prefect.client - Using ephemeral application with database at sqlite+aiosqlite:////Users/mz/.prefect/orion.db
Stack (most recent call last):
  File "/Users/mz/dev/prefect/example.py", line 16, in <module>
    anyio.run(test_flow)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/asyncio/base_events.py", line 634, in run_until_complete
    self.run_forever()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
    self._run_once()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
    handle._run()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-39/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/mz/dev/prefect/src/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/mz/dev/prefect/src/prefect/engine.py", line 237, in create_then_begin_flow_run
    state = await begin_flow_run(
  File "/Users/mz/dev/prefect/src/prefect/engine.py", line 365, in begin_flow_run
    flow_run_context.result_factory = await ResultFactory.from_flow(
  File "/Users/mz/dev/prefect/src/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/mz/dev/prefect/src/prefect/results.py", line 161, in from_flow
    return await cls.default_factory(
  File "/Users/mz/dev/prefect/src/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/mz/dev/prefect/src/prefect/results.py", line 123, in default_factory
    return await cls.from_settings(**kwargs, client=client)
  File "/Users/mz/dev/prefect/src/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/mz/dev/prefect/src/prefect/results.py", line 229, in from_settings
    storage_block_id, storage_block = await cls.resolve_storage_block(
  File "/Users/mz/dev/prefect/src/prefect/results.py", line 258, in resolve_storage_block
    or await storage_block._save(is_anonymous=True, overwrite=True)
12:22:42.289 | DEBUG   | Flow run 'famous-bulldog' - Executing flow 'test-flow' for flow run 'famous-bulldog'...

To reduce the overhead of creating additional connections to the database (when ephemeral) and new connection pools (when using a hosted API) we should avoid this wherever possible.

@netlify
Copy link

netlify bot commented Jan 5, 2023

Deploy Preview for prefect-orion ready!

Name Link
🔨 Latest commit 555189e
🔍 Latest deploy log https://app.netlify.com/sites/prefect-orion/deploys/63b72f4ab17d6b00091827cb
😎 Deploy Preview https://deploy-preview-8072--prefect-orion.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@zanieb zanieb added the enhancement An improvement of an existing feature label Jan 5, 2023
@zanieb zanieb marked this pull request as ready for review January 5, 2023 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants