hyperkube images affected by multiple CVE's

[cjschaef]

What happened: Debian hyperkube images built for the v1.10, v1.11, v1.12 and v1.13 releases contain multiple security vulnerabilities. CVE-2018-17456 CVE-2018-0732 CVE-2018-0734 CVE-2018-0735 CVE-2018-0737 CVE-2018-5407 CVE-2018-18311 CVE-2018-18312 CVE-2018-18313 CVE-2018-18314 CVE-2018-1060 CVE-2018-1061 CVE-2018-14647 CVE-2018-1000802

What you expected to happen: Latest debian hyperkube images built using community automation should have patches to address existing security vulnerabilities.

How to reproduce it (as minimally and precisely as possible): Build debian hyperkube images for the v1.10, v1.11, v1.12 and v1.13 releases and check the vulnerable versions of python, git, perl, and openssl are installed within the images.

Anything else we need to know?: Tested GCR images as well, seeing the same affect. I believe the issue is coming from the base image used to build the hyperkube images, k8s.gcr.io/debian-hyperkube-base-$(ARCH):0.10.2 and tags 0.11.0 and 0.12.0

Environment:

• Kubernetes version (use kubectl version): debian hyperkube images v1.10, v1.11, v1.12, and v1.13
• Cloud provider or hardware configuration: not applicable
• OS (e.g. from /etc/os-release): 16.04.5 LTS (Xenial Xerus)
• Kernel (e.g. uname -a): 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: build/release.sh
• Others:

/kind bug
ref: #80374

[cjschaef]

/sig release

[ixdy]

cc @immutableT

The hyperkube image has a lot of dependencies, which means it has a pretty large source of potential CVEs (as you've noticed). We update these images when there are upstream fixes from debian, but I believe none of these currently has a fix available.

In some cases, the vulnerabilities may also not be relevant, as the vulnerable binaries may not be used in your deployment.

[ixdy]

see also the discussion in #40955

[cjschaef]

I guess in this case, I do see that the relevant releases are available to address the current set of CVE's, although that would require building a new debian-hypberkube image. If that were to happen, it would be nice to see that new image version cherry picked to the previous kubernetes releases as well.

Recommended package releases for CVE's found within the debian-hyperkube images (more or less per kubernetes release):

Vulnerability ID   Policy Status   Affected Packages   How to Resolve   
CVE-2018-17456     Active          git                 Upgrade git to >= 1:2.11.0-3+deb9u4   
CVE-2018-0732      Active          openssl             Upgrade openssl to >= 1.1.0j-1~deb9u1   
CVE-2018-0734      Active          openssl             Upgrade openssl to >= 1.1.0j-1~deb9u1   
CVE-2018-0735      Active          openssl             Upgrade openssl to >= 1.1.0j-1~deb9u1   
CVE-2018-0737      Active          openssl             Upgrade openssl to >= 1.1.0j-1~deb9u1   
CVE-2018-5407      Active          openssl             Upgrade openssl to >= 1.1.0j-1~deb9u1   
CVE-2018-18311     Active          perl                Upgrade perl to >= 5.24.1-3+deb9u5   
CVE-2018-18312     Active          perl                Upgrade perl to >= 5.24.1-3+deb9u5   
CVE-2018-18313     Active          perl                Upgrade perl to >= 5.24.1-3+deb9u5   
CVE-2018-18314     Active          perl                Upgrade perl to >= 5.24.1-3+deb9u5   
CVE-2018-1060      Active          python2.7           Upgrade python2.7 to >= 2.7.13-2+deb9u3   
CVE-2018-1061      Active          python2.7           Upgrade python2.7 to >= 2.7.13-2+deb9u3   
CVE-2018-14647     Active          python2.7           Upgrade python2.7 to >= 2.7.13-2+deb9u3   
CVE-2018-1000802   Active          python2.7           Upgrade python2.7 to >= 2.7.13-2+deb9u3   

$ docker run k8s.gcr.io/debian-hyperkube-base-amd64:0.10.2 /bin/sh -c "dpkg -s git perl openssl python2.7 | grep -e Package -e Version"
Package: git
Version: 1:2.11.0-3+deb9u3
Package: perl
Version: 5.24.1-3+deb9u4
Package: openssl
Version: 1.1.0f-3+deb9u2
Package: python2.7
Version: 2.7.13-2+deb9u2

$ docker run k8s.gcr.io/debian-hyperkube-base-amd64:0.12.0 /bin/sh -c "dpkg -s git perl openssl python2.7 | grep -e Package -e Version"
Package: git
Version: 1:2.11.0-3+deb9u4
Package: perl
Version: 5.24.1-3+deb9u4
Package: openssl
Version: 1.1.0f-3+deb9u2
Package: python2.7
Version: 2.7.13-2+deb9u3

Using a locally built image:

$ docker run debian-hyperkube-base-amd64:local /bin/sh -c "dpkg -s git perl openssl python2.7 | grep -e Package -e Version"
Package: git
Version: 1:2.11.0-3+deb9u4
Package: perl
Version: 5.24.1-3+deb9u5
Package: openssl
Version: 1.1.0j-1~deb9u1
Package: python2.7
Version: 2.7.13-2+deb9u3

[rtheis]

@ixdy this appears to be fixed in 1.14 with #73714 and #73493. This is excellent. Is this something that the community would consider cherry-picking to 1.11, 1.12 and 1.13?

[rtheis]

Looks like kubernetes/enhancements#900 is the future path forward and we'll live with the 1.11, 1.12 and 1.13 image CVEs for now since I'm not aware of any actual exploits.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tpepper]

/lifecycle frozen
/area release-eng

[justaugustus]

We need to close the loop on this for 1.18.

@kubernetes/release-engineering -- This is a nice one to potentially pick up.
/help
/unassign @justaugustus @BenTheElder @hoegaarden
/milestone v1.18
/priority important-soon

[k8s-ci-robot]

@justaugustus:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

We need to close the loop on this for 1.18.

@kubernetes/release-engineering -- This is a nice one to potentially pick up.
/help
/unassign @justaugustus @BenTheElder @hoegaarden
/milestone v1.18
/priority important-soon

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[justaugustus]

Sounds good, Carlos. Let's find some time to chat next week.
/assign @cpanato

[ameukam]

@justaugustus Interested to also work on this. Like @cpanato I will need some guidance.

[cpanato]

I was kind busy with work things this last week :(. Planning to check this in the following week

[cpanato]

@ameukam will let you know the progress and status

[justaugustus]

/assign @ameukam

[justaugustus]

May potentially be fixed with #80909.
/assign @hoegaarden

[micahhausler]

For what its worth I built, uploaded, and scanned the v1.17.0 kube-proxy image in ECR and found 1 High CVE among others.

Is there an ongoing discussion or SIG for maintaining an updated base image or using a simpler base image?

[BenTheElder]

there is a custom simplified base image, but it's barely maintained, it would he helpful for people to pitch in to wg-k8s-infra to get automated community owned builds for these images/

[smourapina]

Hello @BenTheElder, @ameukam et al.
Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March, which is about 3 weeks from now. Are you still targeting being ready before then?
Note: Meant to ping this issue instead of the related PR (#80909).

[BenTheElder]

Kubeproxy contains GCC ...? These scanners go off on things that aren't really relevant to running kube-proxy in a container FWIW. Really common ones are Linux hearders and systemd, neither of which are actually running but the scanner will find some trace ...

…

On Mon, Feb 17, 2020, 09:25 Silvia P. ***@***.***> wrote: Hello @BenTheElder <https://github.com/BenTheElder>, @ameukam <https://github.com/ameukam> et al. Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March, which is about 3 weeks from now. Are you still targeting being ready before then? Note: Meant to ping this issue instead of the related PR (#80909 <#80909>). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#71719?email_source=notifications&email_token=AAHADK74PNDNWSPUQFEGPL3RDLCBDA5CNFSM4GIFNPK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL7FNRY#issuecomment-587093703>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHADK4KY7SVXUA74XFJ2FTRDLCBDANCNFSM4GIFNPKQ> .

[BenTheElder]

@smourapina I'm not assigned to or working on this, and sampling the listed CVEs supposedly affecting the image I'm not particularly concerned.

E.g. CVE-2018-0732 -- kube-proxy is using go's TLS stack, not openssl

[smourapina]

Thanks for following up on my question @BenTheElder. Would it make sense in that case to move this issue to milestone 1.19 to allow for more time?
Just want to make sure that everything included in milestone 1.18 is on track and has a chance to be finished before code freeze, which is in less than 1 week (5 March).

[BenTheElder]

definitely let's move it, I'm somewhat unconvinced that this is a priority to track at all, lacking evidence of actual impact and knowing that hyperkube has limited usage.

I'm somewhat inclined to suggest that we stop having the project officially ship hyperkube in the main repo, users could always build a custom image with the upstream binaries or it could be a kubernetes-sigs project.

[BenTheElder]

the old hyperkube was at least some sort of optimization with all the commands linked together, but IIRC we no longer do that anyhow.

[justaugustus]

We're discussing removing the debian base images in #80909 (comment) and #88603, but I'm not sure this work will land for 1.18, so:
/milestone clear

[abhiTamrakar]

more vulnerabilities around v1.15.x-v1.16.7.

CVE Details below for Severity 5 -

CVE-2018-15686, CVE-2018-1049
CVE-2019-18276
CVE-2019-3846,CVE-2019-5489,CVE-2019-9500,CVE-2019-9503,CVE-2019-10126,CVE-2019-11477,CVE-2019-11478,CVE-2019-11479,CVE-2019-11486,CVE-2019-11599,CVE-2019-11815,CVE-2019-11833,CVE-2019-11884
CVE-2015-8553,CVE-2018-5995,CVE-2018-20836,CVE-2018-20856,CVE-2019-1125,CVE-2019-3882,CVE-2019-3900,CVE-2019-10207,CVE-2019-10638,CVE-2019-10639,CVE-2019-13631,CVE-2019-13648,CVE-2019-14283,CVE-2019-14284

CVE Details below for Severity 4 - (comparatively small severity)

CVE-2019-3855,CVE-2019-3856,CVE-2019-3857,CVE-2019-3858,CVE-2019-3859,CVE-2019-3860,CVE-2019-3861,CVE-2019-3862,CVE-2019-3863
CVE-2019-13272
CVE-2018-12126,CVE-2018-12127,CVE-2018-12130,CVE-2019-11091
CVE-2019-14821,CVE-2019-14835,CVE-2019-15117,CVE-2019-15118,CVE-2019-15902

[BenTheElder]

CVE-2018-15686, CVE-2018-1049

systemd, we don't actually run this as far as I know ...

CVE-2019-18276

bash, do we use this anywhere ...? and if we do, are we using effective UID != real UID ...?

CVE-2019-3846

a kernel vulnerability in a wifi driver... these are container images, so kernel vulnerabilities are pretty irrelevant, no kernel is running from the image.

CVE-2019-5489

kernel again.

are any of these relevant to things we actually run?
this is a very long list and the hyperkube base image is a deprecated effort more or less maintained by volunteers that do not use it ...

[AshutoshNirkhe]

After adding debian stretch security updates repos and upgrading below packages resolves all of the vulnerabilites except for bash 4.4-5 (https://security-tracker.debian.org/tracker/CVE-2019-18276). However, binaries running with an effective UID of 0 are unaffected, as per the link which is the case for control plane components -

cat /etc/apt/sources.list
deb http://deb.debian.org/debian stretch main
deb-src http://deb.debian.org/debian stretch main

deb http://deb.debian.org/debian stretch-updates main
deb-src http://deb.debian.org/debian stretch-updates main

deb http://security.debian.org/debian-security/ stretch/updates main
deb-src http://security.debian.org/debian-security/ stretch/updates main

apt-get update && apt-get upgrade linux-libc-dev libsystemd0 libssh2-1 libcurl3-gnutls openssh-client && apt-get clean

Can these be taken care of in base hyperkube image to avoid everyone rebuilding it ?

[tallclair]

FYI, we're discussing removing hyperkube altogether: #88676

[AshutoshNirkhe]

Ok that will change a lot of things for us. Thanks for the info, will keep watch there.