Prefect 2 - How to use Secret block in local developing #6699

tonal · 2022-09-03T07:04:56Z

tonal
Sep 3, 2022

I try migrate simple flow to prefect 2 (flow_drop.py):

@task(tags=['db', 'mysql'])
def update_mysql(mysql_conf:Secret):
  conf = MySqlConf.parse_raw(mysql_conf.get())
  drop_tags(conf)


@flow(name='Drop tags')
def drop_tags_flow():
  # mysql_conf = PrefectSecret('MYSQL_MASTER')
  mysql_conf = Secret.load("epool-mysql-dt")
  update_mysql(mysql_conf)

def main():
  drop_tags_flow()

How to emulate Secret block in local execute?
Such as Environment PREFECT__CONTEXT__SECRETS__ in prefect 1?

Console out:

/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/bin/python /home/tonal/lang/projects/promsoft/epool/goods_descr_drop_tags/flow_drop.py 
13:52:13.290 | INFO    | prefect.engine - Created flow run 'optimal-tapir' for flow 'Drop tags'
13:52:13.560 | ERROR   | Flow run 'optimal-tapir' - Encountered exception during execution:
Traceback (most recent call last):
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/client.py", line 1268, in read_block_document_by_name
    response = await self._client.get(
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/httpx/_client.py", line 1751, in get
    return await self.request(
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/client.py", line 279, in send
    response.raise_for_status()
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/client.py", line 225, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url 'http://ephemeral-orion/api/block_types/slug/secret/block_documents/name/epool-mysql-dt?include_secrets=true'
Response: {'detail': 'Block document not found'}
For more information check: https://httpstatuses.com/404

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/blocks/core.py", line 606, in load
    block_document = await client.read_block_document_by_name(
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/client.py", line 1274, in read_block_document_by_name
    raise prefect.exceptions.ObjectNotFound(http_exc=e) from e
prefect.exceptions.ObjectNotFound

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/engine.py", line 595, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 57, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/tonal/lang/projects/promsoft/epool/goods_descr_drop_tags/flow_drop.py", line 20, in drop_tags_flow
    mysql_conf = Secret.load("epool-mysql-dt")
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 208, in wrapper
    return run_async_from_worker_thread(async_fn, *args, **kwargs)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 137, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/home/tonal/.pyenv/versions/3.10.6/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/tonal/.pyenv/versions/3.10.6/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/blocks/core.py", line 610, in load
    raise ValueError(
ValueError: Unable to find block document named epool-mysql-dt for block type secret
13:52:13.621 | ERROR   | Flow run 'optimal-tapir' - Finished in state Failed('Flow run encountered an exception.')
Traceback (most recent call last):
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/client.py", line 1268, in read_block_document_by_name
    response = await self._client.get(
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/httpx/_client.py", line 1751, in get
    return await self.request(
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/client.py", line 279, in send
    response.raise_for_status()
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/client.py", line 225, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url 'http://ephemeral-orion/api/block_types/slug/secret/block_documents/name/epool-mysql-dt?include_secrets=true'
Response: {'detail': 'Block document not found'}
For more information check: https://httpstatuses.com/404

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/blocks/core.py", line 606, in load
    block_document = await client.read_block_document_by_name(
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/client.py", line 1274, in read_block_document_by_name
    raise prefect.exceptions.ObjectNotFound(http_exc=e) from e
prefect.exceptions.ObjectNotFound

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tonal/lang/projects/promsoft/epool/goods_descr_drop_tags/flow_drop.py", line 39, in <module>
    main()
  File "/home/tonal/lang/projects/promsoft/epool/goods_descr_drop_tags/flow_drop.py", line 35, in main
    drop_tags_flow()
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/flows.py", line 384, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/engine.py", line 158, in enter_flow_run_engine_from_flow_call
    return anyio.run(begin_run)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/home/tonal/.pyenv/versions/3.10.6/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/tonal/.pyenv/versions/3.10.6/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/client.py", line 103, in with_injected_client
    return await fn(*args, **kwargs)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/engine.py", line 238, in create_then_begin_flow_run
    return state.result()
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/orion/schemas/states.py", line 143, in result
    raise data
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/engine.py", line 595, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 57, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/tonal/lang/projects/promsoft/epool/goods_descr_drop_tags/flow_drop.py", line 20, in drop_tags_flow
    mysql_conf = Secret.load("epool-mysql-dt")
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 208, in wrapper
    return run_async_from_worker_thread(async_fn, *args, **kwargs)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 137, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/home/tonal/.pyenv/versions/3.10.6/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/tonal/.pyenv/versions/3.10.6/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/tonal/.pyenv/versions/epool-goods_descr_drop_tags/lib/python3.10/site-packages/prefect/blocks/core.py", line 610, in load
    raise ValueError(
ValueError: Unable to find block document named epool-mysql-dt for block type secret

Process finished with exit code 1

Answered by tonal

Sep 4, 2022

Need save block before flow call. :)

def main():
  secret_block = Secret(value='my secret')
  secret_block.save('epool-mysql-dt', overwrite=True)
  drop_tags_flow()

View full answer

tonal · 2022-09-04T13:27:47Z

tonal
Sep 4, 2022
Author

Need save block before flow call. :)

def main():
  secret_block = Secret(value='my secret')
  secret_block.save('epool-mysql-dt', overwrite=True)
  drop_tags_flow()

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prefect 2 - How to use Secret block in local developing #6699

{{title}}

Replies: 1 comment

{{title}}

Select a reply

Prefect 2 - How to use Secret block in local developing #6699

tonal Sep 3, 2022

Replies: 1 comment

tonal Sep 4, 2022 Author

tonal
Sep 3, 2022

tonal
Sep 4, 2022
Author