[feature] Artifacts validity period

[jw-websensa]

Feature Area

/area backend
/area components

What feature would you like to see?

Pipeline run artifacts validity period.

What is the use case or pain point?

I'm using Kubeflow with on-premise Kubernetes cluster.
As we use Rancher local path provisioner for Storage Class it is crucial to control disk usage (this class does not work with pvc limits).
I wonder if it is possible to set some kind of period of validity for each artifact to periodically and automatically delete the oldest.

Is there a workaround currently?

I was thinking of preparing k8s crone job to do it manually, but I don't know specifically pipelines architecture and already noticed that deletion from minio is not enough (cacher still finds deleted artifacts available).

---

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

[zijianjoy]

Hello @jw-websensa , currently we recommend using minio rule to set expiration date: https://docs.min.io/docs/minio-bucket-lifecycle-guide.html. We will learn more about the usability and improve in the future.

[jw-websensa]

Hello @zijianjoy, thank you for your response, I am aware about minio rules, but I think it will not work very well as cacher will still find those expired artifacts as available.

[zijianjoy]

Thank you @jw-websensa for the extra context, we will consider this in an end-to-end fashion for configuring artifact lifetime in the coming versions.