kserve/models-web-app v0.10.0 using kubeflow/kubeflow#2b4cc42e (not v1.7.0)

[axel7083]

Description

In the readme of the manifest repository, the 1.7.0 Kubeflow release stated that the KServe Models Web App used is tag v0.10.0. But it is using a very old version of the kubeflow/kubeflow repository for using the components/crud-web-apps/common shared code.

As it is visible in kserve/models-web-app/v0.10.0/Dockerfile#L9, the commit used is 2b4cc42e, and has been committed the Nov 25, 2022 between the tag v1.6.1 (Oct 10, 2022) and v1.7.0-rc.0 (Feb 2, 2023)

This make the compilation of kserve/models-web-app not compatible with 1.7.0

Reproduce

git clone https://github.com/kserve/models-web-app
cd models-web-app
sed -i 's/2b4cc42e/tags\/v1.7.0/g' .\Dockerfile # replace 2b4cc42e with tags/v1.7.0
docker build -f Dockerfile .

Current state

The current master does not have a hardcoded commit in the Dockerfile but instead use a file containing the commit. See kserve/models-web-app#65.

All of this is very confusing for maintaining versioning among app, especially when upgrading components. Synchronizing this new "COMMIT" file with tags would be a great way to keep consistency for components depending on code in the kubeflow/kubeflow repository.

Edit

Currently the master is checking out the kubeflow/kubeflow commit 046c6d3c8 (on Feb 13, 2023) so still not compatible with 1.7.0 commit (on March 29, 2023).

[juliusvonkohout]

Is this still relevant for Kubeflow 1.8 ?

[axel7083]

On the current master

kserve/models-web-app/frontend/COMMIT we have 3649e7e61 which has been merged on Mar 23 which is before release date 1.7.0 (Mar 29), so I guess yes the kserve is not up to date

[juliusvonkohout]

@axel7083 then please provide a PR such that @kimwnasptd can review.

[juliusvonkohout]

@kromanow94 can you investigate whether this is still relevant? We also need to update to kserve 0.11.2 anyway due to #2572 and #2553

[kromanow94]

Sure, I can have a look. I'll try to do it next week.

[juliusvonkohout]

#2603 (comment)

[juliusvonkohout]

We probably need a new home for the web application https://kubeflow.slack.com/archives/C9V2WT2KV/p1709142634280649?thread_ts=1708948152.539709&cid=C9V2WT2KV

[Ayush9026]

/assign @Ayush9026

[juliusvonkohout]

@kimwnasptd
It seems you are the owner of https://github.com/kserve/models-web-app/ repo. Can you confirm that? We need to update and rebuild that image for Kubeflow 1.9 release.

[ca-scribner]

@Ayush9026 are you still working on this, or do we need additional help?

[tiansiyuan]

When I try to 1.9.0-rc.0 with minikube on a mac M3, I am stuck at

% kubectl get po -A

...
kubeflow                    kserve-models-web-app-f6f4d5bd8-lj8qc                    1/2     ImagePullBackOff   0                15h
...

% bubectl describe  pod -n kubeflow                    kserve-models-web-app-f6f4d5bd8-lj8qc
...
Events:
  Type    Reason   Age                      From     Message
  ----    ------   ----                     ----     -------
  Normal  BackOff  4m39s (x425 over 6h11m)  kubelet  Back-off pulling image "docker.m.daocloud.io/kserve/models-web-app:v0.10.0"

% docker pull docker.io/kserve/models-web-app:v0.10.0     
v0.10.0: Pulling from kserve/models-web-app
no matching manifest for linux/arm64/v8 in the manifest list entries

[juliusvonkohout]

@tiansiyuan yes @rimolive and others are working on rebuilding the image. Can you help with the frontend? If yes please reach out to Ricardo.

[tiansiyuan]

Hi @juliusvonkohout @rimolive

I'd be glad if I could help. Please let me know what I need to do.

An update, I tried installing of 1.9.0-rc.1 with Minikube 1.33.1 (K8s 1.29.4) on a Mac M3 and got the same issue, image docker.io/kserve/models-web-app:v0.10.0 (for arm) is not available.

[tiansiyuan]

I managed to build the image based on the Dockerfile at https://github.com/kserve/models-web-app/blob/master/Dockerfile without any change. (any changes needed?)

As I have some difficulty to access docker.io directly, then I pushed it to the local registry of Minikube and referred it in the image definition since the imagePullPolicy isAlways. And it works.

Any more testing needed?

[rimolive]

We need to fix the CVE issues in the code. If you run npm audit in the frontend folder, you will get the CVEs that needs to be fixed. Please sync the work with @vikas-saxena02 who also volunteered to help.

In advance, thank you both for contributing in this!

[thesuperzapper]

@rimolive it is very likely that we wont be able to remove all the CVEs without a lot of reworking (not sure all packages can be updated in place).

We might be able to at least update the node version from 12 to 16, like I did in the central dashboard kubeflow/kubeflow#7578, but this is less important for the web apps because they dont actually run node in the final container (like the dashboard).

For example, we still build with Node 12 in Jupyter Web App (but we would like to update if anyone has some spare time).

[thesuperzapper]

As a side note @tiansiyuan is correct, we need to stop using imagePullPolicy: Always as it is not needed, it can be IfNotPresent to avoid pulling every time.

[juliusvonkohout]

@thesuperzapper as far is i know always means

"It's worth noting regardless of what others are saying, imagePullPolicy: Always just redownloads the digest, not the layers if they match the contents of the digest." But mabye that differs between Kubernetes distributions.

Yes, just rebuilding with a few fast fixes is already enough.

[vikas-saxena02]

there are 19 high and 6 critical vulnerabilities, which needs addressing. Some of the fixes which requires upgrade are breaking changes. I am going to deal with them at the end

[juliusvonkohout]

Please check https://github.com/kubeflow/manifests/actions/runs/9491962409/job/26158322115 for the trivy image scanning results. @vikas-saxena02 please check whether you have the same numbers for the kserve-models-web-application image.

 Summary written to: ../docs/image_lists/summary_of_severity_counts_for_WG/severity_summary_in_json_format.json
Image scanning completed. Reports are saved in ../docs/image_lists/security_scan_reports
+------------------------+--------+--------------+----------+------------+---------+
|     Working Group      | Images | Critical CVE | High CVE | Medium CVE | Low CVE |
+------------------------+--------+--------------+----------+------------+---------+
|         AutoML         |   13   |      11      |   221    |    595     |   757   |
|       Pipelines        |   45   |     129      |   1554   |    6270    |   3205  |
| Workbenches(Notebooks) |   11   |      5       |    78    |    215     |   229   |
|         Kserve         |   14   |      40      |   516    |    1894    |   1412  |
|       Manifests        |   15   |      0       |    2     |     87     |    78   |
|     Model Registry     |   3    |      17      |    71    |    102     |   147   |
|       All Images       |  101   |     202      |   2442   |    9163    |   5828  |
+------------------------+--------+--------------+----------+------------+---------+

[vikas-saxena02]

Hi guys, I am down to just 11 moderate vulnerabilities.... do we need to fix only high and critical or all of them.... also how do I ensure that I am not breaking anything?

[vikas-saxena02]

pull request kserve/models-web-app#81