Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump istio to 1.0.1+ #230

Closed
AntonOfTheWoods opened this issue Dec 7, 2018 · 6 comments

Comments

Projects
None yet
2 participants
@AntonOfTheWoods
Copy link

commented Dec 7, 2018

There is a performance issue in 1.0.0 that is fixed in subsequent versions istio/istio#6962. I would have submitted a PR but the instructions allowing reuse of the downloaded resources for building the snap don't work, meaning I would have to find a way to clean and rebuild downloading everything again (I have only a slow wireless connection with a bandwidth cap) and snap is still very broken for rebuilding (see https://bugs.launchpad.net/snapcraft/+bug/1582469).

I already downloaded and set up a VM with 16.04 because trying to build on 18.04, even using cleanbuild, didn't work either (though admittedly I have lots of potentially conflicting stuff installed - docker, microk8s, then lxd...).

FYI,

anton@ubv:~/dev/microk8s$ ISTIO_VERSION='v1.0.4' snapcraft
Skipping pull libnftnl (already ran)
Skipping pull iptables (already ran)
Failed to reuse files from previous run: The 'pull' step of 'docker' is out of date:
The source has changed on disk.
To continue, clean that part's 'pull' step by running `snapcraft clean docker -s pull`.```

which then asks to also clean `microk8s` but I'm not sure that those clean instructions are actually doing anything, because

```anton@ubv:~/dev/microk8s$ ISTIO_VERSION='v1.0.4' snapcraft clean docker -s pull
Cleaning pull step (and all subsequent steps) for docker
Cleaned 'docker', which makes the following part out of date: 'microk8s'
anton@ubv:~/dev/microk8s$ ISTIO_VERSION='v1.0.4' snapcraft clean microk8s -s pull
Cleaning pull step (and all subsequent steps) for microk8s
anton@ubv:~/dev/microk8s$ ISTIO_VERSION='v1.0.4' snapcraft
Skipping pull libnftnl (already ran)
Skipping pull iptables (already ran)
Hit http://jp.archive.ubuntu.com/ubuntu xenial InRelease                                                                                                     
Get:1 http://jp.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]                                                                                  
Hit http://archive.canonical.com/ubuntu xenial InRelease                                                                                                     
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]                                                                                   
Get:3 http://jp.archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]                                                                                
Fetched 323 kB in 0s (0 B/s)                                                                                                                                 
Pulling docker 
Skipping pull libnvidia (already ran)
Skipping pull libnvidia-tools (already ran)
'microk8s' has dependencies that need to be staged: docker
Skipping pull docker (already ran)
Building docker 
Staging docker 
Hit http://jp.archive.ubuntu.com/ubuntu xenial InRelease                                                                                                     
Get:1 http://jp.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]                                                                                  
Hit http://archive.canonical.com/ubuntu xenial InRelease                                                                                                     
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]                                                                                   
Get:3 http://jp.archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]                                                                                
Fetched 323 kB in 0s (0 B/s)                                                                                                                                 
Unable to open /home/anton/dev/microk8s/parts/microk8s/install/etc/sudoers.d/zfs for writing: [Errno 13] Permission denied: '/home/anton/dev/microk8s/parts/microk8s/install/etc/sudoers.d/zfs'
Unable to open /home/anton/dev/microk8s/parts/microk8s/install/etc/sudoers.d/zfs for writing: [Errno 13] Permission denied: '/home/anton/dev/microk8s/parts/microk8s/install/etc/sudoers.d/zfs'
Pulling microk8s 
...
Preparing kube-proxy
Preparing kubelet
Preparing user config
Creating commands and wrappers
Preparing istio
mv: cannot stat '/home/anton/dev/microk8s/kube_bins/v1.13.0//istio-yaml': No such file or directory
Failed to run 'override-build': Exit code was 1.

I tried everything I could think of but it looks like I might have to dive into the code of snapcraft in order to build microk8s. Not a wonderful experience for contributors!

Edit: I ended up doing a full clean and managed to produce a snap. Unfortunately, it looks like something else happened and the package that was produced was missing libreadline.so.6, so several of the services refused to start.

@ktsakalozos

This comment has been minimized.

Copy link
Collaborator

commented Dec 10, 2018

Hi @AntonOfTheWoods

Thank you for opening this issue. Apologies for the poor experience you had in building the snap with pre-fetched binaries. The issue is addressed in this PR: #237 Most of our builds are now done through travis and launchpad builders so we haven't tested this approach for a long time. Regardless, it should have worked.

I have never seen this "missing libreadline.so.6" error, could you share the full build process (steps and output) you are following?

We appreciate your time and effort to improve MicroK8s and we are here to assist in any way we can.

Thanks

@ktsakalozos

This comment has been minimized.

Copy link
Collaborator

commented Dec 10, 2018

@AntonOfTheWoods you must have already seen that the istio update is a single line patch https://github.com/ubuntu/microk8s/blob/master/build-scripts/prepare-env.sh#L16

@AntonOfTheWoods

This comment has been minimized.

Copy link
Author

commented Dec 10, 2018

Certainly @ktsakalozos , I guess I'm just a little old fashioned in that I like to test my patches before submitting them :-). I was also pretty keen on actually using it. I am pretty busy this week but will try and get you the build process asap.

@AntonOfTheWoods

This comment has been minimized.

Copy link
Author

commented Dec 11, 2018

Ok, so I did a completely fresh pull of microk8s and rebuilt with
$ ISTIO_VERSION='1.0.4' snapcraft
The build output is build.output.txt
I then attempted to install on Ubuntu 16.04 (a copy of the VM used to build) and a brand-spanking new Ubuntu 18.04 virtualbox VM. I had exactly the same problem as I did with my laptop host - complaining about a missing libreadline.so.6. Here is the journald for the apiserver

root@ub18:~# journalctl -u snap.microk8s.daemon-apiserver
-- Logs begin at Tue 2018-12-11 15:03:49 CST, end at Tue 2018-12-11 16:18:36 CST. --
Dec 11 16:09:17 ub18 systemd[1]: Started Service for snap application microk8s.daemon-apiserver.
Dec 11 16:09:17 ub18 microk8s.daemon-apiserver[1174]: gawk: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such fi
Dec 11 16:09:17 ub18 microk8s.daemon-apiserver[1174]: gawk: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such fi
Dec 11 16:09:17 ub18 microk8s.daemon-apiserver[1174]: : error fetching interface information: Device not found
Dec 11 16:09:17 ub18 systemd[1]: snap.microk8s.daemon-apiserver.service: Main process exited, code=exited, status=139/n/a
Dec 11 16:09:17 ub18 systemd[1]: snap.microk8s.daemon-apiserver.service: Failed with result 'exit-code'.
Dec 11 16:09:18 ub18 systemd[1]: snap.microk8s.daemon-apiserver.service: Service hold-off time over, scheduling restart.
Dec 11 16:09:18 ub18 systemd[1]: snap.microk8s.daemon-apiserver.service: Scheduled restart job, restart counter is at 1.
Dec 11 16:09:18 ub18 systemd[1]: Stopped Service for snap application microk8s.daemon-apiserver.
Dec 11 16:09:18 ub18 systemd[1]: Started Service for snap application microk8s.daemon-apiserver.
Dec 11 16:09:18 ub18 microk8s.daemon-apiserver[1330]: gawk: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such fi
Dec 11 16:09:18 ub18 microk8s.daemon-apiserver[1330]: gawk: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such fi
Dec 11 16:09:18 ub18 microk8s.daemon-apiserver[1330]: : error fetching interface information: Device not found
Dec 11 16:09:18 ub18 systemd[1]: snap.microk8s.daemon-apiserver.service: Main process exited, code=exited, status=139/n/a
Dec 11 16:09:18 ub18 systemd[1]: snap.microk8s.daemon-apiserver.service: Failed with result 'exit-code'.
Dec 11 16:09:18 ub18 systemd[1]: snap.microk8s.daemon-apiserver.service: Service hold-off time over, scheduling restart.
Dec 11 16:09:18 ub18 systemd[1]: snap.microk8s.daemon-apiserver.service: Scheduled restart job, restart counter is at 2.
Dec 11 16:09:18 ub18 systemd[1]: Stopped Service for snap application microk8s.daemon-apiserver.
Dec 11 16:09:18 ub18 systemd[1]: Started Service for snap application microk8s.daemon-apiserver.
...

Looks very much like it is expecting a shared lib to be present somewhere and it's not. It does appear to be present in the core at /snap/core/6034/lib/x86_64-linux-gnu/libreadline.so.6 so maybe the snap is just missing this directory in the so path?

I had a completely different problem on the Ubuntu 16.04 VM - it was refusing to microk8s.enable anything because
Warning FailedScheduling 2s (x9 over 26s) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
Weird but whatever (I didn't get this with the normal snap installed version on 18.04) - the interwebs says this should be fixed by
anton@ubvv:~$ microk8s.kubectl taint nodes --all node-role.kubernetes.io/master-
But instead I get
error: taint "node-role.kubernetes.io/master:" not found

I am not sure what use I can be without really deeply diving into how snap and microk8s works. I'll confess that I've already manually installed istio (+ manually activated RBAC) and it seems to be working well... so my itch-scratching level is pretty low on this one now :-). If there is anything particular I can do let me know, if I can help with low time investment I'm definitely keen!

@ktsakalozos

This comment has been minimized.

Copy link
Collaborator

commented Dec 11, 2018

@AntonOfTheWoods I could not reproduce the missing libreadline error. I believe the problem is with snapcraft build process that does not pull the expected dependencies in the snap. The snapcraft team is aware of such issues and therefore are pushing towards builds inside lxc containers. This should be transparent to you.

Could you try this:
Update the istio version here: https://github.com/ubuntu/microk8s/blob/master/build-scripts/prepare-env.sh#L16 Then instead of snapcraft use snapcraft cleanbuild. If you haven't already done so you will be prompted to initialise lxd. Here is a build process on a 18.04 VM on AWS:

ubuntu@ip-172-31-13-57:~$ git clone https://github.com/ubuntu/microk8s
Cloning into 'microk8s'...
remote: Enumerating objects: 65, done.
remote: Counting objects: 100% (65/65), done.
remote: Compressing objects: 100% (62/62), done.
remote: Total 1698 (delta 26), reused 22 (delta 3), pack-reused 1633
Receiving objects: 100% (1698/1698), 365.37 KiB | 20.30 MiB/s, done.
Resolving deltas: 100% (1085/1085), done.
ubuntu@ip-172-31-13-57:~$ cd microk8s/

ubuntu@ip-172-31-13-57:~/microk8s$ sudo snap install snapcraft --classic
snapcraft 3.0.1 from Canonical✓ installed

ubuntu@ip-172-31-13-57:~/microk8s$ snapcraft cleanbuild
If this is your first time running LXD on this machine, you should also run: lxd init
To start your first container, try: lxc launch ubuntu:16.04

Creating snapcraft-idiomorphically-subumbellar-stephany
Error: Failed container creation: No storage pool found. Please create a new storage pool.
Failed to create container: a new LXD container could not be created.
Refer to the documentation at https://linuxcontainers.org/lxd/getting-started-cli.

ubuntu@ip-172-31-13-57:~/microk8s$ lxd init
Would you like to use LXD clustering? (yes/no) [default=no]:
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]:
Name of the storage backend to use (btrfs, dir) [default=btrfs]:
Create a new BTRFS pool? (yes/no) [default=yes]:
Would you like to use an existing block device? (yes/no) [default=no]:
Size in GB of the new loop device (1GB minimum) [default=15GB]:
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]:
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
Would you like LXD to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

ubuntu@ip-172-31-13-57:~/microk8s$ snapcraft cleanbuild
Creating snapcraft-pronouncedly-sparkish-deeanna
Retrieving image: metadata: 100% (604.59MB/s)

On the taint errors, could you see if the node is under disk or memory pressure? Try: microk8s.kubectl describe no

What exactly did you have to do to enable RBAC, you mentioned some manual steps, could you share those?

Your feedback is very important to us. Keep opening issues for any big or small thing you find. We appreciate your time and effort you spent in improving MicroK8s. Thank you.

@AntonOfTheWoods

This comment has been minimized.

Copy link
Author

commented Dec 11, 2018

>Could you try this:
>...
> Retrieving image: metadata: 100% (604.59MB/s)
> 

Sorry, I tried a couple more times but really my bandwidth is precious and my connection flakey - I am in China and some stuff (notably everything @google) simply won't work if it doesn't go over a VPN, I have only an (often slow) 4G connection and a flakey wifi router (an old Android phone...). I got to the download of the kube elements but the VPN connection got lost and it wouldn't come back, meaning starting everything again in a new lxd container...
Stuff that would have taken 15 seconds back when I was in France can take an hour here...

On the taint errors, could you see if the node is under disk or memory pressure? Try: microk8s.kubectl describe no

Well spotted :-). Sorry, I'm still a Kube-n00b and hadn't noticed that the VM was almost out of disk space. Removing failed build attempts solved the issue for Ubuntu 16.04.

What exactly did you have to do to enable RBAC, you mentioned some manual steps, could you share those?

I just followed #84 (comment). I'll confess I didn't check things like the dashboard afterwards (#220 would suggest it gets broken) but I was able to follow the instructions to install helm with RBAC (https://github.com/helm/helm/blob/master/docs/rbac.md) and then install istio with that helm (https://istio.io/docs/setup/kubernetes/helm-install/). I was then able to get to some pods I had spun up via an istio gateway, so for me that was sufficient to think "it works". There may well be issues down the road.

Your feedback is very important to us. Keep opening issues for any big or small thing you find. We appreciate your time and effort you spent in improving MicroK8s. Thank you.

If my connection weren't so bad I'd love to do more! Thanks for your great support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.