Raw output prefix #34
Conversation
| @@ -20,7 +20,8 @@ Note this currently does not include articles/tid-bits on how to use the Flyte p | |||
| 1. [Dynamic Tasks](recipes/dynamictasks) | |||
| 1. [Tasks without flytekit or Using arbitrary containers](recipes/rawcontainers) | |||
| 1. [Jupyter notebook as a task](recipes/notebook_tasks) | |||
| 1. [Different container per task](recipes/differentcontainers) | |||
| 1. [Different container per task](recipes/differentcontainers) | |||
| 1. [Customize Offloaded Data Location](recipes/offloaded_output_data) | |||
There was a problem hiding this comment.
instead of this, can we just make this work for all examples?
There was a problem hiding this comment.
ohh nevermind, I get it now, we are saying that all outputs will be in default storage location and you can customize if you want
|
|
||
| [platform] | ||
| url=host.docker.internal:30081 | ||
| insecure=True | ||
|
|
||
| # Deprecated. Please use the raw_output_data_prefix above |
There was a problem hiding this comment.
I think we should just delete it?
There was a problem hiding this comment.
If we want to delete the current config objects, then I need to add more code to the flytekit pr. Specifically, the new raw output config object is not currently used at run time, it’s only used at compile time (sent up to admin at launch plan registration).
Currently in that PR, if it's missing at runtime, it'll fall back to the current aws_shard/gcs_prefix config objects. If we want to remove the old configs entirely, we'll have to make it so that if Admin doesn't send it (because users are on an older version of admin), we do our own sharding based off of the raw output config at runtime, when click starts up for instance.
However, if users update flytekit, but don't add a raw output config and we remove the existing config objects, then at runtime, it won't know where to write to.
So we can,
- We can just add runtime deprecation notices and delete them later.
- Add a fallback at registration time to the existing shard formatter/prefix if the raw output prefix is missing.
I actually think I'd go with option 1 for now, at least until the different sharding methods you just added to flyteplugins are available for picking and choosing. I hate assuming migration timelines though.
There was a problem hiding this comment.
I meant delete it in the example? Set good precedence
There was a problem hiding this comment.
talked offline, I understand the reason is for backwards compatibility - Someone may run the example on an older Flyte installation.
|
I think we should update |
No description provided.