[v2compat] umbrella - KFP data model based caching

[Bobgy]

internal tracker: b/181133870

Work Items estimate:

• [v2compat] caching design #5726 (size/L)
• [v2compat] generate cache key and calculate hash of the cache key #5816 (size/M)
• [v2 compat] DB and API for creating/querying cache entries #5817(size/M)
• [v2compat] add enable cache option in v2 compatible mode #5818(size/S)
• [v2compat] Fetch cache result from MLMD and put cache result into argo output artifacts/parameters #5819 (size/M)
• [v2compat] cache integration test #5820 (size/M)
• [v2compat] UI features for cached tasks (size/S) #5977

total: 4w, 1d for SDK team

[Bobgy]

/assign @capri-xiyue

[Bobgy]

Needs further TODO item break down

[juliusvonkohout]

Hi @Bobgy do you have the following in mind ? #5509
Or will there be a native way such that deleting runs from the pripelines webinterface will also delete the artifacts from minio and make the cacher aware that the files are gone?

[Bobgy]

@capri-xiyue can introduce more, but our implementation is basically what you described.

When runs are deleted, their entries will no longer be used for caching.

We haven't invented a way to delete external data for users yet, because data loss would be very frightening.

[Bobgy]

Hi @zijianjoy, I just realized you'll need to make a small change #5977 for caching.
You can coordinate with @capri-xiyue for when and how to do that

[Bobgy]

All P0s are done, great work @capri-xiyue @zijianjoy