Models deployed with ModelMesh-Serving get restarted on upgrade

[vaibhavjainwiz]

Describe the bug

KServe community follow an approach to release all repos together irrespective if there are code changes in independent repos or not.
For example, In release v0.11.2, a new tag is created for all repos(modelmesh, modelmesh-runtime-adaptor, rest-proxy etc).
Earlier we had created an image for v0.11.1 but with the new release (v0.11.2), we had created a new set of images by pointing to the v0.11.2 tag.
Now when we roll out a new modelmesh-serving(v0.11.2) controller, all deployed runtimes are restarted because the runtime sidecar container image(modelmesh, modelmesh-runtime-adaptor, rest-proxy) tag has been changed.

Problem is ModelMesh tech stack is quite stable and repos like modelmesh, modelmesh-runtime-adaptor, rest-proxy hardly see any code change between the releases. But to release complete KServe org all together we provide their new versions as well.

Expected behavior

Model runtime pod should not restarted on ModelMesh-Serving controller upgrade, if there is no change at runtime/application level.

Additional context

I am proposing every repo should be released independently only if there is a code change.
For example, On next release cycle, new tag for modelmesh-runtime-adaptor repo should only be created if there are some changes.

[Jooho]

@vaibhavjainwiz If we don't set a new tag for modelmesh images that didn't change at all, the restarting issue that all the runtime pods roll out when modelmesh-serving upgraded will be solved?
@ckadner could you please review this ticket?

[vaibhavjainwiz]

@vaibhavjainwiz If we don't set a new tag for modelmesh images that didn't change at all, the restarting issue that all the runtime pods roll out when modelmesh-serving upgraded will be solved?

yes.. if we don't build new image for ModelMesh repo where code didn't change since last build then this issue would be resolved.

[ckadner]

I am proposing every repo should be released independently only if there is a code change.
For example, On next release cycle, new tag for modelmesh-runtime-adaptor repo should only be created if there are some changes.

There usually are code changes in at least one of the 3 modelmesh repos for each release. At the very least we have a few security fixes. If one of the images changes, a new deployment would get rolled out.

We could make some changes to the MM controller to stagger the update/rollout over time to reduce the resource spike, e.g. one runtime kind at a time.