Add a step to upload artifact

[MartinForReal]

mirror issue: kubeflow/mpi-operator#342

[gaocegege]

Should we follow the design here? https://github.com/kubeflow/community/blob/master/proposals/pvc-template.md

[MartinForReal]

PVC works perfectly when it comes to storing training data, but it would be a little different when it comes to uploading output artifacts (such as model uploading) And S3 would be an more feasible option if we need to run jobs in multiple clusters and it would have less dependencies.
Also it works well with workflows because there is no way for argo controller to discover pvc provisioned by tfjob.

Do you think so? Any advice is appreciated. Thanks!

[Jeffwan]

@MartinForReal I see your request in mpi-operator as well. Trying to understand if that's possible to add your logic in tfjob? Since KFP based training jobs uses launcher and that's not possible to leverage artifact feature. If you have some thoughts, feel free to draft a proposal

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[johnugeorge]

keeping it open for more discussions

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.