Jessie apt-get package incomplete

[r4j4h]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): jessie, kubectl, apt-get, package

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Kubernetes version (use kubectl version): Affects all

Environment:

• Cloud provider or hardware configuration: Hardware, VM via Vagrant
• OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie)
• Kernel (e.g. uname -a): Linux jessie 3.16.0-4-amd64 Unit test coverage in Kubelet is lousy. (~30%) #1 SMP Debian 3.16.39-1 (2016-12-30) x86_64 GNU/Linux
• Install tools: apt-key and apt-get
• Others: apt-transport-https

What happened:

The apt-get repos for debian jessie add and fetch fine but no valid packages are found.

At least for core components like kubelet and kubectl.

kops's documentation says it uses a modified Debian as it's default base image in AWS. I understand there is extra work re: memory cgroups but these work with the xenial versions even. This problem is exclusively around being able to install only using the jessie apt-get repo.

root@jessie:/home/vagrant# cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
> deb http://apt.kubernetes.io/ kubernetes-jessie main
> EOF
root@jessie:/home/vagrant# apt-get clean
root@jessie:/home/vagrant# rm -rf /var/lib/apt/lists/*
root@jessie:/home/vagrant# apt-get clean
root@jessie:/home/vagrant# apt-get update -y
root@jessie:/home/vagrant# apt-get upgrade -y
Get:1 http://security.debian.org jessie/updates InRelease [63.1 kB]
....
Get:9 http://apt.kubernetes.io kubernetes-jessie InRelease [3,700 B]
Get:10 http://httpredir.debian.org jessie Release.gpg [2,373 B]                                  
Get:11 http://httpredir.debian.org jessie Release [148 kB]
Get:12 http://apt.kubernetes.io kubernetes-jessie/main amd64 Packages [23 B]
Get:13 http://httpredir.debian.org jessie/main Sources [7,056 kB]
Get:14 http://httpredir.debian.org jessie/main amd64 Packages [6,776 kB]
...
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@jessie:/home/vagrant# apt-get install -y kubelet kubeadm kubectl kubernetes-cni
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package kubelet
E: Unable to locate package kubeadm
E: Unable to locate package kubectl
E: Unable to locate package kubernetes-cni

What you expected to happen:

...
Setting up ethtool (1:3.16-1) ...
Setting up kubernetes-cni (0.3.0.1-07a8a2-00) ...
Setting up socat (1.7.2.4-2) ...
Setting up kubelet (1.5.2-00) ...
Setting up kubectl (1.5.2-00) ...
Setting up kubeadm (1.6.0-alpha.0-2074-a092d8e0f95f52-00) ...
Processing triggers for systemd (215-17+deb8u6) ...

How to reproduce it (as minimally and precisely as possible):

Start a vagrant box using debian/jessie64 and try to install Kubernetes with:

Install pre-reqs: (apt-transport-https needed for k8s, software-prop-common needed for docker)

apt-get install -y curl apt-transport-https software-properties-common
curl -fsSL https://yum.dockerproject.org/gpg | sudo apt-key add -
sudo add-apt-repository \
   "deb https://apt.dockerproject.org/repo/ \
   debian-$(lsb_release -cs) \
   main"
apt-get install -y docker-engine

Add jessie repo:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-jessie main
EOF

See it fail:

apt-get install -y kubelet kubeadm kubectl kubernetes-cni

Add xenial repo too

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF

See it succeed using both (or at least xenial and falling back to jessie for dependencies):

apt-get install -y kubelet kubeadm kubectl kubernetes-cni

Anything else we need to know:

I'm not worried about kubeadm not being present, although that would be great. But shouldn't at least kubelet and kubectl be available?

https://github.com/kubernetes/release/blob/master/debian/jessie/kubectl indicates that xenial is fine to use for kubectl, but kubectl also showed up in the failed to install list.

Here's the output of the last command given to demonstrate it succeeding using both:

root@jessie:/home/vagrant# cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
> deb http://apt.kubernetes.io/ kubernetes-xenial main
> EOF
root@jessie:/home/vagrant# apt-get clean
root@jessie:/home/vagrant# rm -rf /var/lib/apt/lists/*
root@jessie:/home/vagrant# apt-get clean
root@jessie:/home/vagrant# apt-get update -y
root@jessie:/home/vagrant# apt-get upgrade -y
Get:1 http://security.debian.org jessie/updates InRelease [63.1 kB]
...
Get:9 http://apt.kubernetes.io kubernetes-xenial InRelease [6,299 B]
Get:10 http://httpredir.debian.org jessie Release.gpg [2,373 B]
Get:11 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages [1,739 B]
...
Fetched 14.6 MB in 4s (3,393 kB/s)                         
Reading package lists... Done
...
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@jessie:/home/vagrant# apt-get install -y kubelet kubeadm kubectl kubernetes-cni
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  ebtables ethtool socat
The following NEW packages will be installed:
  ebtables ethtool kubeadm kubectl kubelet kubernetes-cni socat
0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 37.7 MB of archives.
After this operation, 261 MB of additional disk space will be used.
Get:1 http://httpredir.debian.org/debian/ jessie/main ebtables amd64 2.0.10.4-3 [103 kB]
Get:2 http://httpredir.debian.org/debian/ jessie/main ethtool amd64 1:3.16-1 [94.9 kB]                   
Get:3 http://httpredir.debian.org/debian/ jessie/main socat amd64 1.7.2.4-2 [333 kB]                     
Get:4 http://apt.kubernetes.io/ kubernetes-xenial/main kubernetes-cni amd64 0.3.0.1-07a8a2-00 [6,868 kB]
Get:5 http://apt.kubernetes.io/ kubernetes-xenial/main kubelet amd64 1.5.2-00 [15.2 MB]               
Get:6 http://apt.kubernetes.io/ kubernetes-xenial/main kubectl amd64 1.5.2-00 [7,950 kB]                                                                                                                           
Get:7 http://apt.kubernetes.io/ kubernetes-xenial/main kubeadm amd64 1.6.0-alpha.0-2074-a092d8e0f95f52-00 [7,120 kB]                                                                                               
Fetched 37.7 MB in 18s (2,077 kB/s)                                                                                                                                                                                
Selecting previously unselected package ebtables.
(Reading database ... 45564 files and directories currently installed.)
Preparing to unpack .../ebtables_2.0.10.4-3_amd64.deb ...
Unpacking ebtables (2.0.10.4-3) ...
Selecting previously unselected package ethtool.
Preparing to unpack .../ethtool_1%3a3.16-1_amd64.deb ...
Unpacking ethtool (1:3.16-1) ...
Selecting previously unselected package kubernetes-cni.
Preparing to unpack .../kubernetes-cni_0.3.0.1-07a8a2-00_amd64.deb ...
Unpacking kubernetes-cni (0.3.0.1-07a8a2-00) ...
Selecting previously unselected package socat.
Preparing to unpack .../socat_1.7.2.4-2_amd64.deb ...
Unpacking socat (1.7.2.4-2) ...
Selecting previously unselected package kubelet.
Preparing to unpack .../kubelet_1.5.2-00_amd64.deb ...
Unpacking kubelet (1.5.2-00) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../kubectl_1.5.2-00_amd64.deb ...
Unpacking kubectl (1.5.2-00) ...
Selecting previously unselected package kubeadm.
Preparing to unpack .../kubeadm_1.6.0-alpha.0-2074-a092d8e0f95f52-00_amd64.deb ...
Unpacking kubeadm (1.6.0-alpha.0-2074-a092d8e0f95f52-00) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for systemd (215-17+deb8u6) ...
Setting up ebtables (2.0.10.4-3) ...
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
Setting up ethtool (1:3.16-1) ...
Setting up kubernetes-cni (0.3.0.1-07a8a2-00) ...
Setting up socat (1.7.2.4-2) ...
Setting up kubelet (1.5.2-00) ...
Setting up kubectl (1.5.2-00) ...
Setting up kubeadm (1.6.0-alpha.0-2074-a092d8e0f95f52-00) ...
Processing triggers for systemd (215-17+deb8u6) ...

And at this point, aside from Debian things like memory cgroup control being disabled by default, I am able to fully proceed and create a cluster from here. I think that verifies enough that the system is capable, but that something is missing or out of date in the apt-get Jessie repo?

FWIW I believe jessie is using amd64 based on the apt-get output and https://packages.cloud.google.com/apt/dists/kubernetes-jessie/InRelease says it provides amd64. I don't know enough about apt-get to find out WHAT packages are provided by that release and the relevant folders don't seem to load properly in my browser. Sorry I can't provide more help!

[luxas]

cc @mikedanese ^

[dixudx]

@r4j4h There are no available kubernetes packages for Jessie on https://packages.cloud.google.com/apt/dists/kubernetes-jessie. It is an empty repo.

If you visit https://packages.cloud.google.com/apt/dists/kubernetes-jessie/main/binary-amd64/Packages, you may find an empty page. While if you visit https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-amd64/Packages, you may see the package info. Currently among all the Debian platforms, kubernetes packages are only available on xenial.

But you can still install and use kubectl on Jessie. kubectl can be installed with repo configured to https://packages.cloud.google.com/apt/dists/cloud-sdk-jessie.

[mikedanese]

I can publish kubectl to most of the distros. Supporting kubelet is significantly harder....

[r4j4h]

If it's at all difficult I don't think you should put yourself through extra work trying to publish to the extra distros at this point.

I believe I changed some lines from xenial to jessie thinking it would be safer, and others might too, so warning that we're purposely using xenial even in other distros in the docs would probably be just as effective and easier.

I apologize for straying from the docs since it caused this problem. Thanks @dixudx for explaining what I seeing while browsing. cloud-sdk does get kubectl but I ideally want other core components and have less need for the other elements of the cloud-sdk so I'll keep using xenial for now.

I'll leave it open to you guys as to whether doing nothing, docs, or publishing is the right way to go. 👍

[mikedanese]

The problem with jessie and kubelet is that they have chosen not to enable memcg by default and to enable requires a kernel boot param, so we would have to muck with grub configs and restart the machine. No one has had time to sketch the user experience around that yet. Publishing kubectl everywhere would not be tricky.

[r4j4h]

Yep, good point! Backports provides newer kernels for more, but also requires restarting the machine. What an annoying situation!

[k8s-github-robot]

@r4j4h There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
(2) specifying the label manually: /sig <label>

Note: method (1) will trigger a notification to the team. You can find the team list here.

[luxas]

/sig cluster-lifecycle

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close