Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test with Raspbian 10 / Buster / Pi 4 model B #156

Closed
geerlingguy opened this issue Jun 24, 2019 · 22 comments

Comments

@geerlingguy
Copy link
Owner

commented Jun 24, 2019

With the announcement of the Pi 4 today, the Pi Foundation also announced a new version of the OS based on Buster (which is going to officially be released in a couple weeks). Would like to start testing a new build based on it to see if anything needs changing. Probably at least the Docker install... :/

@geerlingguy geerlingguy changed the title Test with Raspbian 10 / Buster Test with Raspbian 10 / Buster / Pi 4 model B Jul 6, 2019

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 6, 2019

I have my Raspberry Pi 4 model B's in (2 GB model since 4 GB are hard to acquire), and I'm beginning work on them.

First challenge I've hit—note that networking and all that seems to work fine, and half the install as well—is that Docker doesn't yet have an ARM build published for Buster / Debian/ARM / 10...

E: The repository 'https://download.docker.com/linux/raspbian 10 Release' does not have a Release file.
@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 6, 2019

Upstream Docker issue: docker/for-linux#709

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 6, 2019

Temporary workaround (as mentioned in the issue linked above) is to install from stretch sources, so manually testing that now, will have to devise a temporary workaround :/

sudo rm /etc/apt/sources.list.d/docker.list; curl -sL get.docker.com | sed 's/9)/10)/' | sh
@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 6, 2019

During Docker Compose install I got:

        Running setup.py bdist_wheel for cffi: finished with status 'error'
        Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-cMxyHh/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-SaF2_X --python-tag cp27:
        Package libffi was not found in the pkg-config search path.
        Perhaps you should add the directory containing `libffi.pc'
        to the PKG_CONFIG_PATH environment variable
        No package 'libffi' found

I believe this is related to #155 — merging that and testing again.

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 6, 2019

That seemed to help, but I'm hitting some other issue with the Docker Compose install, I'll have to run it interactively to see what's happening. Too late for tonight.

geerlingguy added a commit that referenced this issue Jul 6, 2019

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 6, 2019

Another oddity; I use ansible to run shutdown -h now, and they shut down... but then 5 seconds later they boot back up, as if I was running reboot. Maybe a weird firmware thing with the PoE boards?

@mrconnerton

This comment has been minimized.

Copy link

commented Jul 6, 2019

This seems to be known issue with no resolution yet https://www.raspberrypi.org/forums/viewtopic.php?t=244209

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 6, 2019

Now during install of kubernetes* related apt packages:

    E: Could not get lock /var/cache/apt/archives/lock - open (11: Resource temporarily unavailable)
    E: Unable to lock directory /var/cache/apt/archives/
@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 6, 2019

Apparently the Pis were in the middle of a daily apt update:

$ ps aux
...
root       718  0.0  0.0   1940   396 ?        Ss   16:14   0:00 /bin/sh /usr/lib/apt/apt.systemd.daily update
root       722  0.0  0.0   1940  1192 ?        S    16:14   0:00 /bin/sh /usr/lib/apt/apt.systemd.daily lock_is_held upd
root       901  0.8  2.3  55776 47252 ?        S    16:14   0:01 apt-get -qq -y -d dist-upgrade

Tried again and Kubernetes installation proceeded normally.

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 6, 2019

@mrconnerton - Interesting! Thanks for the link.

Also...

root@kube1:/home/pi# kubectl get nodes
NAME    STATUS   ROLES    AGE    VERSION
kube1   Ready    master   105s   v1.13.5
kube2   Ready    <none>   87s    v1.13.5
kube3   Ready    <none>   90s    v1.13.5
kube4   Ready    <none>   89s    v1.13.5

So the only things I needed to change so far are:

  1. Don't install Docker Compose (don't need it, this is K8s!)
  2. Manually install/monkey-patch Docker itself since the script doesn't account for Buster/10, and there's no apt repo for Buster yet...

For #1, I have a patch incoming.

geerlingguy added a commit that referenced this issue Jul 6, 2019

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 6, 2019

Hmm... for the Drupal and MySQL pods though, I'm getting:

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  59s (x7 over 2m29s)  default-scheduler  pod has unbound immediate PersistentVolumeClaims (repeated 3 times)

Looks like the PVCs aren't getting created successfully. Getting the drupal-files PVC:

Events:
  Type       Reason                Age                 From                         Message
  ----       ------                ----                ----                         -------
  Normal     ExternalProvisioning  6s (x14 over 3m6s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "pidramble.com/nfs" or manually created by system administrator

Time's up for the morning debugging session, I'll look into this later today.

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 6, 2019

Ah...

root@kube1:/home/pi# kubectl get pods
NAME                                      READY   STATUS             RESTARTS   AGE
nfs-client-provisioner-78fd5ffbd8-jjljh   0/1     CrashLoopBackOff   4          5m13s

root@kube1:/home/pi# kubectl logs nfs-client-provisioner-78fd5ffbd8-jjljh
F0706 15:29:22.045981       1 provisioner.go:180] Error getting server version: Get https://10.96.0.1:443/version?timeout=32s: dial tcp 10.96.0.1:443: i/o timeout
@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 6, 2019

I'm going to try upgrading to K8s 1.15, see if anything changes. Everything will probably break instead, but YOLO! geerlingguy/ansible-role-kubernetes#54

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 6, 2019

Time lapse I recorded upgrading the hardware for the cluster, for those curious: https://youtu.be/iXQAo_hMDWk

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 8, 2019

I'm also updating my Debian 10 test box for Vagrant (https://app.vagrantup.com/geerlingguy/boxes/debian10), and the test container image for Docker (https://hub.docker.com/r/geerlingguy/docker-debian10-ansible).

Testing things locally, working on the hack to make the convenience script work on Debian 10 by forcing it to use stretch for Debian 10.

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 8, 2019

Also tagged a new release of the docker_arm role since it needed that script hack to work correctly on Stretch. I'm opening a follow-up ticket in that repo to remove the hack once Buster is supported properly.

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 8, 2019

The metrics-server pod is also failing due to Flannel network issues:

NetworkPlugin cni failed to set up pod "metrics-server-d7c6dc6bc-lpn9n_kube-system" network: open /run/flannel/subnet.env: no such file or directory
@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 8, 2019

Hmm, flannel is a distraction, it seems like it's actually related to kubernetes/kubernetes#71305

Checking the version on Buster:

root@kube1:/home/vagrant# iptables --version
iptables v1.8.2 (nf_tables)
@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 8, 2019

Well then... I just ran the following command on each of the nodes, and all is well:

update-alternatives --set iptables /usr/sbin/iptables-legacy

Seems like the iptables / nftables bits of Debian 10 / Buster are the cause of all my networking consternation today. Never trust a firewall whose rules you can't comprehend.

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 8, 2019

K, that hurdle was passed. Tidying up a few things and I'm going to maybe add in this as a (hopefully-temporary?) hack to make sure iptables-legacy is used in Buster... for now at least.

geerlingguy added a commit that referenced this issue Jul 8, 2019

@geerlingguy

This comment has been minimized.

Copy link
Owner Author

commented Jul 8, 2019

Just released 4.0.0, which I'm running through more testing on my cluster, but already have running quite nicely in the local Vagrant environment. Woohoo!

@kailin4u

This comment has been minimized.

Copy link

commented Aug 6, 2019

pi@raspberrypi:~ $ iptables
iptables/1.8.2 Failed to initialize nft: Protocol not supported

it seems iptables dosn't work on my buster raspbian for pi 4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.