New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
De-dupe constant tasks #1527
Comments
This is a really good call out; the original reason for this behavior is that we needed a place to track the values that are being used as Task inputs, and simply reusing Prefect's Task machinery was the most natural way to do that. In addition, if we see a non-Prefect Task as an input and simply stored its value without further introspection, we might lose dependency relationships such as: with Flow("example") as flow:
my_task(inputs=[task_a, task_b]) However, I fully agree that this is not ideal and something we should improve upon. In the meantime, one way of at least de-duping some constants is to wrap them in a Prefect Constant task: from prefect import task, Flow
from prefect.tasks.core.constants import Constant
from prefect.engine.executors import DaskExecutor
@task
def add(x, y):
return x + y
with Flow('flow') as flow:
zero = Constant(0)
for i in range(10):
add(i, zero)
flow.run(executor=DaskExecutor('localhost:8786')) I'll revisit the logic for this and see how we might improve it. |
Makes sense, thanks! One more note: the same behavior would be nice for parameters. Currently:
|
…-ui/typescript-4.6.3 Bump typescript from 4.6.2 to 4.6.3 in /orion-ui
Probably not a high priority (and maybe not feasible for some reason I haven't thought of) but I find this behavior a bit irritating:
Result: the constant
0
task is repeated every time, as is the corresponding log:Is there any kind of simple de-duping scheme that might work within a flow, maybe just for Python literals...?
Edit: possibly kind of a dupe of #891 which was already declined...? 🤷♂
The text was updated successfully, but these errors were encountered: