Fix lxd ipv6 #6916

Merged
merged 3 commits into from Feb 7, 2017

Conversation

Projects
None yet
6 participants
Contributor

perrito666 commented Feb 3, 2017

Detect when IPV6.address is empty in lxd config and fallback to IPV4 when 6 is disabled in kernel.

This change prevents juju lxd provider from failing when running in a machine with ipv6 disabled in different possible ways.

QA steps

Try reproduction steps in bugs:

Bug reference

https://bugs.launchpad.net/juju/+bug/1656205
https://bugs.launchpad.net/juju/+bug/1633362

Contributor

battlemidget commented Feb 3, 2017

My test output with this PR:

kernel param
ubuntu@bobbybooshay:~$ cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-4.4.0-62-generic root=UUID=3755a423-99c8-4fdd-b5a4-63948f25b8d2 ro net.ifnames=0 ipv6.disable=1

lxd bridge
ubuntu@bobbybooshay:~$ cat /etc/default/lxd-bridge 
# WARNING: This file is generated by a debconf template!
# It is recommended to update it by using "dpkg-reconfigure -p medium lxd"

# Whether to setup a new bridge or use an existing one
USE_LXD_BRIDGE="true"

# Bridge name
# This is still used even if USE_LXD_BRIDGE is set to false
# set to an empty value to fully disable
LXD_BRIDGE="lxdbr0"

# Update the "default" LXD profile
UPDATE_PROFILE="true"

# Path to an extra dnsmasq configuration file
LXD_CONFILE=""

# DNS domain for the bridge
LXD_DOMAIN="lxd"

# IPv4
## IPv4 address (e.g. 10.0.8.1)
LXD_IPV4_ADDR="10.138.179.1"

## IPv4 netmask (e.g. 255.255.255.0)
LXD_IPV4_NETMASK="255.255.255.0"

## IPv4 network (e.g. 10.0.8.0/24)
LXD_IPV4_NETWORK="10.138.179.1/24"

## IPv4 DHCP range (e.g. 10.0.8.2,10.0.8.254)
LXD_IPV4_DHCP_RANGE="10.138.179.2,10.138.179.254"

## IPv4 DHCP number of hosts (e.g. 250)
LXD_IPV4_DHCP_MAX="252"

## NAT IPv4 traffic
LXD_IPV4_NAT="true"

# IPv6
## IPv6 address (e.g. 2001:470:b368:4242::1)
LXD_IPV6_ADDR=""

## IPv6 CIDR mask (e.g. 64)
LXD_IPV6_MASK=""

## IPv6 network (e.g. 2001:470:b368:4242::/64)
LXD_IPV6_NETWORK=""

## NAT IPv6 traffic
LXD_IPV6_NAT="false"

# Run a minimal HTTP PROXY server
LXD_IPV6_PROXY="false"

juju bootstrap working
19:25:29 INFO  juju.cmd supercommand.go:63 running juju [2.2-alpha1 gc go1.7.2]
19:25:29 DEBUG juju.cmd supercommand.go:64   args: []string{"./juju", "bootstrap", "lxd/localhost", "--debug"}
19:25:29 DEBUG juju.tools.lxdclient client.go:203 connecting to LXD remote "local": "unix:///var/lib/lxd/unix.socket"
19:25:29 INFO  juju.tools.lxdclient client.go:251 using LXD API version "1.0"
19:25:29 INFO  juju.tools.lxdclient client.go:305 LXD "default" profile uses network bridge "lxdbr0"
19:25:29 INFO  cmd cmd.go:141 Resolved LXD host address on bridge lxdbr0: 10.138.179.1
19:25:29 INFO  cmd cmd.go:141 no credentials found, checking environment
19:25:31 DEBUG juju.tools.lxdclient client.go:203 connecting to LXD remote "local": "unix:///var/lib/lxd/unix.socket"
19:25:31 INFO  juju.tools.lxdclient client.go:251 using LXD API version "1.0"
19:25:31 INFO  juju.tools.lxdclient client.go:305 LXD "default" profile uses network bridge "lxdbr0"
19:25:31 DEBUG juju.cmd.juju.commands bootstrap.go:783 authenticating with region "localhost" and credential "default" ()
19:25:31 DEBUG juju.cmd.juju.commands bootstrap.go:895 provider attrs: map[]
19:25:31 INFO  cmd cmd.go:141 Adding contents of "/home/ubuntu/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
19:25:31 DEBUG juju.cmd.juju.commands bootstrap.go:951 preparing controller with config: map[transmit-vendor-metrics:true apt-mirror: agent-stream:released http-proxy: proxy-ssh:false provisioner-harvest-mode:destroyed test-mode:false logging-config: apt-https-proxy: apt-ftp-proxy: ftp-proxy: apt-http-proxy: enable-os-refresh-update:true image-stream:released logforward-enabled:false automatically-retry-hooks:true image-metadata-url: firewall-mode:instance enable-os-upgrade:true type:lxd authorized-keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQClwKu50uGyfbOuNoJo/BmEgCSZgTqKEe502SvN5kOk4G9Qtg2FBkfT5nFtn7Ayyihpijo83prq68zNs+qF/9c7UwaUuorTM7dpnIHmgCZglJER6Syt6ZUQDZs6OOf1vyl5RiF2tQ36HbDJJ9YIsN/ZSdUmzf48jeAOa2sZz4Z2KeyvOuxmUhVXlCW14bv7f775P88y8BrvbdM8lRcciulIsaCBmDSpGW1ZV/RXjCxS+1ty6C9h23I6iGqlIWvGU+j6d8l3qoVroDaifigNJ/OfdGjJonusSx9vOmTijVJKFaRCAWU1ncFqb6YO7KtArEizu4RgfGA7yeBlF2i0j1a5 juju-client-key
 resource-tags: default-series:xenial ssl-hostname-verification:true disable-network-management:false name:controller https-proxy: uuid:1d004626-8636-4169-899d-30bdea6e5d57 development:false net-bond-reconfigure-delay:17 agent-metadata-url: no-proxy: ignore-machine-addresses:false]
19:25:32 DEBUG juju.tools.lxdclient client.go:203 connecting to LXD remote "local": "unix:///var/lib/lxd/unix.socket"
19:25:32 INFO  juju.tools.lxdclient client.go:251 using LXD API version "1.0"
19:25:32 INFO  juju.tools.lxdclient client.go:305 LXD "default" profile uses network bridge "lxdbr0"
19:25:34 INFO  cmd cmd.go:129 Creating Juju controller "localhost-localhost" on lxd/localhost
19:25:34 INFO  juju.cmd.juju.commands bootstrap.go:529 combined bootstrap constraints: 
19:25:34 DEBUG juju.environs.bootstrap bootstrap.go:199 model "controller" supports service/machine networks: false
19:25:34 DEBUG juju.environs.bootstrap bootstrap.go:201 network management by juju enabled: true
19:25:34 INFO  cmd cmd.go:141 Loading image metadata
19:25:34 INFO  cmd cmd.go:129 Looking for packaged Juju agent version 2.2-alpha1 for amd64
19:25:34 INFO  juju.environs.bootstrap tools.go:72 looking for bootstrap agent binaries: version=2.2-alpha1
19:25:34 INFO  juju.environs.tools tools.go:101 finding agent binaries in stream "devel"
19:25:34 INFO  juju.environs.tools tools.go:103 reading agent binaries with major.minor version 2.2
19:25:34 INFO  juju.environs.tools tools.go:111 filtering agent binaries by version: 2.2-alpha1
19:25:34 INFO  juju.environs.tools tools.go:117 filtering agent binaries by architecture: amd64
19:25:34 DEBUG juju.environs.tools urls.go:109 trying datasource "keystone catalog"
19:25:35 DEBUG juju.environs.simplestreams simplestreams.go:683 using default candidate for content id "com.ubuntu.juju:devel:tools" are {20161007 mirrors:1.0 content-download streams/v1/cpc-mirrors.sjson []}
^C19:25:35 INFO  cmd cmd.go:129 Interrupt signalled: waiting for bootstrap to exit

stgraber commented Feb 4, 2017

Looks reasonable to me, though I'd recommend you only check if "address family not supported by protocol" is in the error string rather than matching the whole thing as there's no guarantee we won't be changing the error prefix for this in a future LXD bugfix release.

A couple small changes.

I'm ok in principle, though I'd really like some sort of test so that we have a chance of noticing when something in the environment changes an error string (Go, LXD, or something else.)

tools/lxdclient/utils.go
@@ -65,6 +66,8 @@ func IsRunningLocally() (bool, error) {
return running, nil
}
+const errIPV6NotSupported = `cannot listen on https socket: listen tcp \[::\]:\d*: socket: address family not supported by protocol`
@jameinel

jameinel Feb 5, 2017

Owner

I don't have a particularly better regex to use. This feels a little brittle (version of Go, version of LXD API, platform Trusty vs Xenial, etc.)
Can we get some sort of Functional test so that we have an idea when this string is no longer valid and can update it?

@stgraber

stgraber Feb 5, 2017

The "socket: address family not supported by protocol" part is effectively considered Linux API (glibc) and so is very unlikely to change so long as LXD doesn't mask the error.

The rest of the string comes from a mix of underlying golang packages and LXD, and I'd strongly recommend not relying on that part. So checking if "socket: address family not supported by protocol" is part of the error string should work fine and be much more reliable.

tools/lxdclient/utils.go
+ err := client.SetServerConfig("core.https_address", "[::]")
+
+ if err != nil {
+ // if the error hints that the problem might be a protocol unsopported
@jameinel

jameinel Feb 5, 2017

Owner

unsupported

tools/lxdclient/utils.go
+ cause := errors.Cause(err)
+ matched, merr := regexp.MatchString(errIPV6NotSupported, cause.Error())
+ if merr != nil {
+ logger.Errorf("cant match error: %v", merr)
@jameinel

jameinel Feb 5, 2017

Owner

failed to match?

at the very least "can't"

Owner

jameinel commented Feb 5, 2017

Also, should this be targetting 2.1 instead of 2.2? That seems the more relevant place to get the fix.

Contributor

perrito666 commented Feb 6, 2017

@jameinel ptal again

Member

anastasiamac commented Feb 7, 2017

Please re-target for 2.1 \o/

@perrito666 perrito666 changed the base branch from develop to 2.1 Feb 7, 2017

@perrito666 perrito666 changed the base branch from 2.1 to develop Feb 7, 2017

Owner

jameinel commented Feb 7, 2017

you'll need to cherry-pick now to get to 2.1 since it includes all the extra 'develop' changes.

overall I think its good enough. I definitely worry about the fragility of our detection, but we can add to it as we get more understanding.

tools/lxdclient/utils.go
@@ -65,6 +66,8 @@ func IsRunningLocally() (bool, error) {
return running, nil
}
+const errIPV6NotSupported = `socket: address family not supported by protocol`
@jameinel

jameinel Feb 7, 2017

Owner

a comment here about how we arrived at this string would be useful.
did you trace where the actual string was generated to know if it is stable?

tools/lxdclient/utils.go
+ err := client.SetServerConfig("core.https_address", "[::]")
+
+ if err != nil {
+ // if the error hints that the problem might be a protocol unsoported
@jameinel

jameinel Feb 7, 2017

Owner

typo: unsupported

Contributor

perrito666 commented Feb 7, 2017

$$merge$$

Contributor

jujubot commented Feb 7, 2017

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

@jujubot jujubot merged commit 78b1fbf into juju:develop Feb 7, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment