LXD Static IP configuration - clear + working documentation seems scarce #2534

davidfavor · 2016-10-20T04:06:06Z

The template below is mostly useful for bug reports and support questions.
Feel free to remove anything which doesn't apply to you and add more information where it makes sense.

Required information

Distribution: Ubuntu
Distribution version: 16.10
The output of "lxc info" or if that fails:
- Kernel version: 4.8.0-22-generic
- LXC version: 2.4.1
- LXD version: 2.4.1
- Storage backend in use: dir

Issue description

Goal is to have LXD containers with static IPs which can communication with host + other containers.

Steps to reproduce

Simplest approach seems to be setting /etc/default/lxd-bridge LXD_CONFILE to a container,IP pairs + Ubuntu 16.10 seems to have removed this file.

I have 100s of LXC container,IP pairs to port to LXD + prefer a solution that avoids the old iptables nat rule approach.

None of the #2083 approaches seem to produce useful results.

The

echo -e "lxc.network.0.ipv4 = 144.217.33.224\nlxc.network.0.ipv4.gateway = 149.56.27.254\n" | lxc config set template-yakkety raw.lxc -

comes close, as my test container does end up with the correct IP assigned.

Maybe this is the correct approach, along with setting up the host base interface (eth2) in my case, to use br0, rather than eth2 + somehow bridging lxdbr0 to br0.

Suggestions appreciated, as all the Ubuntu docs seem wrong + the LXD 2.0 Introduction series seems to be missing basic networking examples for large scale LXD deployments.

Once I have a working approach, I'll publish all steps back here, so others can accomplish this easier.

Thanks.

davidfavor · 2016-10-20T15:12:02Z

Seems like the most robust way to accomplish this is to...

convert host eth2 (base interface) from static to br0.

Change host /etc/network/interfaces + reboot (works).

configure container IP/Gateway

lxc config set template-yakkety raw.lxc 'lxc.network.0.ipv4 = 144.217.33.224'
lxc config set template-yakkety raw.lxc 'lxc.network.0.ipv4.gateway = 149.56.27.254'

connect container IP link to host br0 which fails...

lxc config set template-yakkety raw.lxc 'lxc.network.0.ipv4.link = br0'
error: Only interface-specific ipv4/ipv6 lxc.network keys are allowed

It appears the only outstanding issue to fix this is how to associate container IP address with host br0 interface.

Suggestions?

stgraber · 2016-10-20T15:28:35Z

lxc network device add template-yakkety eth0 nic nictype=bridged parent=br0 name=eth0
cat | lxc config set template-yakkety raw.lxc - << EOF
lxc.network.0.ipv4 = 144.217.33.224/24
lxc.network.0.ipv4.gateway = 149.56.27.254
EOF

stgraber · 2016-10-20T15:31:03Z

Though, note that the preferred way to do this is through your Linux distribution's own configuration mechanism rather than pre-configure things through raw.lxc.

For Ubuntu, that'd be through some cloud-init configuration of some sort, that said, if raw.lxc works for you, that's fine too :)

davidfavor · 2016-10-20T16:36:02Z

I believe the first problem to be resolved is to destroy lxdbr0 + unfortunately dpkg-reconfigure no longer works... meaning the networking configuration stage never starts.

dpkg-reconfigure -p medium lxd
Warning: Stopping lxd.service, but it can still be activated by:
lxd.socket

Short of purging/reinstalling LXD completely.

I've removed all my containers.

Let me know how to reconfigure LXD to no longer use lxdbr0.

Thanks.

stgraber · 2016-10-20T16:45:02Z

lxc profile edit default

davidfavor · 2016-10-20T16:55:55Z

lxc profile edit default == only changes a text file.

Still running...

dnsmasq -u root --strict-order --bind-interfaces --pid-file=/var/lib/lxd/networks/lxdbr0/dnsmasq.pid --except-interface=lo --interface=lxdbr0 --listen-address=10.119.167.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/lib/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/lib/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-range 10.119.167.2,10.119.167.254 --listen-address=fd42:88bd:c65f:d720::1 --dhcp-range fd42:88bd:c65f:d720::2,fd42:88bd:c65f:d720:ffff:ffff:ffff:ffff,ra-stateless,ra-names -s lxd -S /lxd/

Also ifconfig still shows lxdbr0...

service lxd/lxc-containers restart has no effect on dnsmasq or ifconfig listing.

Let me how to de-entangle dnsmasq from attempting to handle lxdbr0 + to delete lxdbr0 so it's completely gone... while leaving LXD intact/running.

Thanks.

stgraber · 2016-10-20T17:03:26Z

Ah right, to destroy the bridge, you'll want "lxc network delete lxdbr0"

davidfavor · 2016-10-20T18:22:19Z

Just so the exact process is documented.

If you have currently defined containers + wish to remove the associated bridge interface...

You must first change all active profiles which reference the bridge. In my case, I'm only using the default profile... so...

lxc profile edit default -> change lxdbr0 to br0

Destroy lxdbr0

lxc network delete lxdbr0

At this point... the lxdbr0 interface is destroyed (pruned from ifconfig output + all networking tables) + also dnsmasq terminates.

Whew... stgrabber == my hero. 👍

stgraber · 2016-10-20T18:29:31Z

Cool, sounds like you got it all sorted out, Closing

davidfavor · 2016-10-20T18:29:38Z

My version of lxc has no "lxc device add" syntax.

Let me know if this looks correct...

lxc network attach br0 yakkety eth0

Which looks to mean... attach the host's br0 to the yakkety container's eth0.

stgraber · 2016-10-20T18:35:02Z

@davidfavor it's "lxc config device add"

And yes, your "lxc network attach" is fine, you need LXD 2.3 or higher for that, but I'm assuming that's what you have.

davidfavor · 2016-10-20T18:36:04Z

Using LXD-2.4.1.

Hum... After a issuing the above command, now...

lxc config get --verbose yakkety raw.lxc
error: Failed to load raw.lxc

So can't get or set raw.lxc anymore.

stgraber · 2016-10-20T18:37:16Z

@davidfavor "lxc config show yakkety"

davidfavor · 2016-10-20T18:38:52Z

net12 # lxc config show yakkety
error: Failed to load raw.lxc

Ugh... inotifywait shows that raw.lxc seems to live in lxc.db so this data lives in sqlite3 land.

stgraber · 2016-10-20T18:40:17Z

ah right, you wouldn't be able to show the container config either. It's a bug I fixed a few days ago, it shouldn't have let you set an invalid raw.lxc to begin with...

sudo sqlite3 /var/lib/lxd/lxd.db "SELECT value FROM containers_config WHERE key='raw.lxc';"

davidfavor · 2016-10-20T18:40:51Z

net12 # sqlite3 /var/lib/lxd/lxd.db .dump | grep raw
INSERT INTO "containers_config" VALUES(310,2,'raw.lxc','lxc.network.0.ipv4 = 144.217.33.224/24\nlxc.network.0.ipv4.gateway = 149.56.27.254\n

net12 # sqlite3 /var/lib/lxd/lxd.db "SELECT value FROM containers_config WHERE key='raw.lxc';"
lxc.network.0.ipv4 = 144.217.33.224/24\nlxc.network.0.ipv4.gateway = 149.56.27.254\n

Looks correct.

Let me know if I can just delete this line or if that will screw up something else.

stgraber · 2016-10-20T18:42:36Z

ok, that looks fine, so long as the container has a network device attached to it, otherwise those two entries will be invalid since they apply to a network device that's not defined...

Does running:

lxc network attach-profile br0 default eth0

Fix the problem? That should add the br0 bridge to your container by adding it to the profile it depends on, which should avoid the error you've been getting so far.

davidfavor · 2016-10-20T18:43:04Z

net12 # lxc network attach-profile br0 default eth0
error: device already exists

davidfavor · 2016-10-20T18:44:08Z

Profile seems correct...

net12 # lxc profile show default
name: default
config: {}
description: Default LXD profile
devices:
eth0:
nictype: bridged
parent: br0
type: nic
usedby:

/1.0/containers/yakkety

stgraber · 2016-10-20T18:44:09Z

hmm, alright, well, lets just fix the DB then:

sudo sqlite3 /var/lib/lxd/lxd.db "DELETE FROM containers_config WHERE key='raw.lxc';"

Then paste the output of:

lxc config show yakkety --expanded

davidfavor · 2016-10-20T18:44:46Z

net12 # lxc config show yakkety --expanded

name: yakkety
profiles:
- default
config:
  volatile.base_image: 687c1a6a81e8ce42114796f162b4b872e53c4cf5821d295f8a9eb1c0fe696389
  volatile.eth0.hwaddr: 00:16:3e:36:00:18
  volatile.eth0.name: eth0
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":165536,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.root.hwaddr: 00:16:3e:d6:88:e0
  volatile.root.name: eth1
devices:
  eth0:
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    type: disk
ephemeral: false

Hum... If you know the Markdown to use to format this in pre tags, let me know.

stgraber · 2016-10-20T18:47:03Z

Hmm, so that all looks correct.

What do you have in /var/log/lxd/yakkety/lxc.log* with a bit of luck one of the log files will tell you what the parser thought was wrong with your raw.lxc

stgraber · 2016-10-20T18:47:44Z

My guess is that it may be upset about the gateway being outside of your IP's mask, hopefully it logs that kind of problem.

davidfavor · 2016-10-20T18:53:18Z

Okay...

After the sqlite3 deletion...

echo -e "lxc.network.0.ipv4 = 144.217.33.224\nlxc.network.0.ipv4.gateway = 149.56.27.254\n" | lxc config set yakkety raw.lxc -

lxc start yakkety -> works + IP assigned + IP isn't pingable.

Maybe I have to do some of the network attach magic again.

Maybe this?

lxc network attach-profile br0 default yakkety eth0

stgraber · 2016-10-20T18:54:55Z

"ip -4 route show" and "ip -4 addr show" in the container would probably help figure out what's going on.

davidfavor · 2016-10-20T18:55:33Z

net12 # ip -4 route show
default via 149.56.27.254 dev br0 onlink
149.56.27.0/24 dev br0 proto kernel scope link src 149.56.27.129

net12 # ip -4 addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 149.56.27.129/24 brd 149.56.27.255 scope global br0
valid_lft forever preferred_lft forever

stgraber · 2016-10-20T18:59:55Z

I think you missed the "in the container" part :)

davidfavor · 2016-10-20T19:00:08Z

Maybe this is the problem.

Seems like 28: eth0@if29 is wrong.

net12 # lxc exec yakkety -- ip -4 route show
default via 149.56.27.254 dev eth0
149.56.27.254 dev eth0 scope link

net12 # lxc exec yakkety -- ip -4 addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
28: eth0@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link-netnsid 0
inet 144.217.33.224/0 brd 255.255.255.255 scope global eth0
valid_lft forever preferred_lft forever

stgraber · 2016-10-20T19:01:00Z

Seems to match what you requested LXC to do

stgraber · 2016-10-20T19:41:36Z

Hmm, so that got ignored somehow, weird

davidfavor · 2016-10-20T19:42:06Z

Hum...

I think the container's /etc/network/interfaces has to have something to include /etc/network/interfaces.d/* files.

Let me know how you do this.

davidfavor · 2016-10-20T19:48:48Z

Change container's /etc/network/interfaces...

auto eth0
iface eth0 inet dhcp

to have last line of...

source /etc/network/interfaces.d/*.cfg

Sound right? Yes?

davidfavor · 2016-10-20T19:50:31Z

net12 # lxc restart yakkety

Now produces...

net12 # lxc list

+---------+---------+--------------------------------+---------------------------------------------+------------+-----------+
|  NAME   |  STATE  |              IPV4              |                    IPV6                     |    TYPE    | SNAPSHOTS |
+---------+---------+--------------------------------+---------------------------------------------+------------+-----------+
| yakkety | RUNNING | 10.0.119.85 (eth0)             | fd42:9a2b:d62f:7e98:216:3eff:fe36:18 (eth0) | PERSISTENT | 0         |
|         |         | 144.217.33.224 (eth0)          |                                             |            |           |
+---------+---------+--------------------------------+---------------------------------------------+------------+-----------+

davidfavor · 2016-10-20T19:52:21Z

Then...

ip -4 route add 144.217.33.0/24 dev lxdbr0

To make 144.217.33.0/27 addresses pingable...

stgraber · 2016-10-20T19:53:15Z

Oh right, I thought cloud-init would add that source statement automatically, not sure why you're missing it.

stgraber · 2016-10-20T19:54:08Z

Hmm, that ip route is a bit wrong, you want your /27 instead of that /24.

So:

ip -4 route add 144.217.33.224/27 dev lxdbr0

davidfavor · 2016-10-20T19:54:14Z

Whoa! Appears...

Host can ping container.

Container can ping host.

Container IP pingable from outside machine.

Geez! Might be working.

stgraber · 2016-10-20T19:55:06Z

(The /24 will obviously work, but you're also accidentally routing a bunch of IPs that don't belong to you :))

davidfavor · 2016-10-20T19:57:27Z

Got it... So for manual/static routes, there will be one static route/IP... of the form...

ip -4 route add 144.217.33.$addr/27 dev lxdbr0

Where $addr is each active IP address.

Yes?

stgraber · 2016-10-20T20:16:39Z

nope, just "ip -4 route add 144.217.33.0/27 dev lxdbr0" will cover your whole subnet, no need to do it per container.

davidfavor · 2016-10-20T20:36:44Z

Right .0 rather than .IP will get them all.

Got it.

Dude! You're a life saver!

I'm hosting 100s of high traffic client sites + like to start converting them all from LXC to LXD.

Thanks for your huge investment of time today.

After I roll all this info into a simple step-by-step guide, I'll drop the link here.

stgraber · 2016-10-20T20:44:44Z

Oh, btw, my command earlier was wrong, you want:

ip -4 route add 144.217.33.224/27 dev lxdbr0

Since .224 is the first address. If you use .0/27, then it will do from 144.217.33.0 to to 33.31 which is not what you want :)

davidfavor · 2016-10-20T21:36:47Z

Right... BASE-IP/27

Thanks.

davidfavor · 2016-10-22T16:42:29Z

Testing shows all packet flow working as expected.

One final question about runtime IPs.

For each container to support a static/public IP, each container will have two IPs. One internal + one external (static/public).

Let me know if this looks correct to you, based on your OVH setup.

Thanks.

net12 # lxc list

+---------+---------+--------------------------------+---------------------------------------------+------------+-----------+
|  NAME   |  STATE  |              IPV4              |                    IPV6                     |    TYPE    | SNAPSHOTS |
+---------+---------+--------------------------------+---------------------------------------------+------------+-----------+
| yakkety | RUNNING | 10.0.119.85 (eth0)             | fd42:9a2b:d62f:7e98:216:3eff:fe36:18 (eth0) | PERSISTENT | 0         |
|         |         | 144.217.33.224 (eth0)          |                                             |            |           |
+---------+---------+--------------------------------+---------------------------------------------+------------+-----------+

davidfavor · 2016-10-23T00:14:07Z

Looks like you already answered my question above... where you said...

Confirm it's got both a private (10.x.x.x) address and its public IP listed in "lxc list"

davidfavor · 2016-12-07T15:35:40Z

Problem source identified.

Request for best way to resolve.

At boot time, /etc/rc.local runs + executes the following command to route a public ip range to the LXD bridge interface.

ip -4 route add 144.217.33.224/27 dev lxdbr0

During host level upgrade of LXD somehow this additional route is lost.

Running the above command on the command line fixes problem.

Question is, what's the best way to associate the above command with lxdbr0 interface stops/restarts.

For physical interfaces, host level /etc/network/interfaces is where post-up commands are added.

Someone let me know the correct way to associate a post-up command with the LXD interface.

Thanks.

stgraber · 2016-12-07T15:51:23Z

Hmm, so I think the best way to deal with this would be through a new LXD network config key.

Until then, you could define a systemd unit which starts "After=lxd.service" and runs the command you need. That way, whenever the daemon is started/restarted, your command is run again.

spyderdyne · 2017-02-13T15:56:07Z

It seems like every 6 months you are invalidating your own documentation and not updating it afterward. I am constantly following instructions in your insights pages only to find important commands and config files completely missing. I will read through this to re-learn how to set up LXD on 16.10 with a bridge to the LAN for my controller node since the instructions I followed 2 weeks ago for 16.04 no longer work and the /etc/defaults/lxd-bridge file has been completely removed now.

My current state is Ubuntu 16.10 server with br0 bridge, static Ip assignment, MaaS rack controller running as local LXD instance, static controller ip assignment, static MaaS rack controller Ip assignment, lxd br0 bridge taking over the machine's bridge that is configured in /etc/network/interfaces to static class C RFC1918 and replacing it at boot time with dynamic class A RFC1918 addresses.

I am reserving the first 10 ip addresses in MaaS for core services (MaaS Rack and Region controllers, Juju bootstrap, Cloudify, and Network Devices) and providing a dynamic range for Openstack core services via the rack controller DHCP service on this LAN.

I should probably not need to have multiple LXD network config sets just to run the current version of Openstack without backports. I cant help but feel that this project would benefit from having more discipline as to what can be ripped out/replaced versus what should be immutable to prevent users from having to replace all their documentation and config automations every 6 months or so.

CarltonSemple · 2017-03-16T15:15:53Z

@stgraber Is there something I'm missing from #2534 (comment) ?

I verified that I can attach any IP from the subnet to my host using ip addr add 169.53.244.xx/27 broadcast 169.53.244.63 dev eth1, and then removed them.
I'm using LXD version 2.11, and followed the instructions at https://stgraber.org/2016/10/27/network-management-with-lxd-2-3/ to create a network:

lxc network create testbr0 ipv6.address=none ipv4.address=10.0.3.1/24 ipv4.nat=true
lxc network attach-profile testbr0 default eth0

Then I push a file to the container's /etc/network/interfaces.d/eth0-static.cfg containing

auto eth0:1
iface eth0:1 inet static
    address 169.53.244.36
    netmask 255.255.255.255
    broadcast 169.53.244.63
    gateway 169.53.244.33

and then I add the route to the LXD bridge, after which I restart my container.
ip -4 route add 169.53.244.32/27 dev testbr0

169.53.244.32 is the first IP in the subnet

stgraber · 2017-03-16T21:05:13Z

@CarltonSemple that should be okay, though note that the route may get flushed when LXD restarts (during upgrade or such). That's why I added ipv4.routes a few releases back.

CarltonSemple · 2017-03-17T01:37:49Z

@stgraber Ah, okay. I just followed #2701. However, I still can't reach my containers from outside of the host. Is there some ip addr add command as well?

stgraber · 2017-03-17T09:02:20Z

Nope, you should only have the route setup on the host, not have the address defined there.

A firewall could be blocking incoming packets for those IPs or you may have the traffic back out from those containers get NATed somehow breaking things.

Best way to debug those kind of issues is to run tcpdump inside the container and see what's making it there.

CarltonSemple · 2017-03-17T16:52:05Z

@stgraber I'm wondering if it could be the different infrastructure. I'm using Softlayer, with a "routed to VLAN" portable IP block

CarltonSemple · 2017-03-17T16:54:40Z

With a tcpdump inside the container, nothing appears to reach the container. I see the correct ICMP echo requests only when I ping the public IP from the host VM. I did see
16:39:54.584808 IP gateway > s236: ICMP time exceeded in-transit, length 68,
but it has only showed up twice.

Conjohnsonjohnson · 2017-06-09T04:33:01Z

@davidfavor
"After I roll all this info into a simple step-by-step guide, I'll drop the link here."

Is this step-by-step guide available? I suggest this issue is still relevant. I'm having no success assigning an IP address of my own choosing to a container. I have several questions that I'm not able to answer for myself from this interesting discussion:

Why is dhcp required for this use case? The title is "LXD Static IP configuration".
What network configuration is required on the host?
What network configuration is required via the lxc command line?
What network configuration is required within the container?

I'm hoping to find answers to these questions in generic terms, not Ubuntu-centric, so the information is equally valid across the spectrum of linux hosts.

davidfavor · 2017-06-24T16:43:26Z

Conjohnsonjohnson wrote:

@davidfavor <http://links.davidfavor.com/wf/click?upn=npdP0-2FMHcGNgMeleDP-2B5C31vwdBQ6x6yfJQ45N4fzrZ5kdjmPYI81a4Peedu3dTv_s5UH8bhnBaopYjthcNSRcaC8fv1GYJ42u9dW4Ymps68fQF2Bwu68aaBpwvP0gTDdFOPj1wREGu0weJoJOtBs0DojgwEcKC3qXEKGq3tJBtD-2FKlv05s-2FnWFtQ47rf1ZMG4XKLl5fVSOCAb96N8k89x5zCtrYw7RnR8a-2FQ1FES8v-2Br5N-2BCc82g1i0r2GexAWLjU1SW-2BEByFiKyW9RQEwvUpgWD2ZABtKnAycJ5lw9fZrmQZL3os-2BPj11XWskBUDeHstLk5uE47asDZwqvJC23ECwpeUPNbmLoBLgx4IC534Zr6jcNexYNJUgDKAiShTqnBLoiRXklfk-2BPXUOMc4j5Zew-3D-3D> "After I roll all this info into a simple step-by-step guide, I'll drop the link here." Is this step-by-step guide available? I suggest this issue is still relevant. I'm having no success assigning an IP address of my own choosing to a container. I have several questions that I'm not able to answer for myself from this interesting discussion: 1. Why is dhcp required for this use case? The title is "LXD Static IP configuration". 2. What network configuration is required on the host? 3. What network configuration is required via the lxc command line? 4. What network configuration is required within the container? I'm hoping to find answers to these questions in generic terms, not Ubuntu-centric, so the information is equally valid across the spectrum of linux hosts.

Rather than answering your questions individually... http://links.davidfavor.com/wf/click?upn=npdP0-2FMHcGNgMeleDP-2B5CwDT8yGN7SPfWqbXTvyBZ-2Bok79XNVC27qhCKP5ZEg9iC_s5UH8bhnBaopYjthcNSRcaC8fv1GYJ42u9dW4Ymps68fQF2Bwu68aaBpwvP0gTDdFOPj1wREGu0weJoJOtBs0DojgwEcKC3qXEKGq3tJBtD-2FKlv05s-2FnWFtQ47rf1ZMG4XKLl5fVSOCAb96N8k89x5zCtrYw7RnR8a-2FQ1FES8v-2Br5N-2BCc82g1i0r2GexAWLjKqi-2BaArxsFXi9sgWiPPyKle67P4zKv-2B0nO0SDMi4DHp0QZiYdQxfELvfuI-2F3DfAethC-2B-2BXxVv8bByG-2BsU1fmrMGHLSYuILV22-2FW3kdfgsOUePkpD2ke5zPFHMitupNY42hJkm8dVCTH4CWoB3zs3SQ-3D-3D provides clear coverage of setting up bulletproof LXD container networking.

itisnotdone · 2017-09-05T01:03:58Z

http://cloudinit.readthedocs.io/en/latest/topics/network-config.html#network-configuration-sources
If you prefer to work with cloud-init.
If you also prefer to work with MAAS, you can create containers with cloud-init network configuration and reserve the IPs with its hostname using MAAS API so you can access them using domain(or FQDN) provided by MAAS.

stgraber closed this as completed Oct 20, 2016

davidfavor mentioned this issue Nov 6, 2016

How to connect to container which is behind iptables rule #2584

Closed

davidfavor mentioned this issue Dec 7, 2016

Make it possible to route additional subnets to LXD managed bridges #2701

Closed

LXD Static IP configuration - clear + working documentation seems scarce #2534

LXD Static IP configuration - clear + working documentation seems scarce #2534

Comments

davidfavor commented Oct 20, 2016

Required information

Issue description

Steps to reproduce

davidfavor commented Oct 20, 2016

stgraber commented Oct 20, 2016

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016 • edited

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016 • edited

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016 • edited

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016 • edited

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016

davidfavor commented Oct 20, 2016

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016 • edited

stgraber commented Oct 20, 2016

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016 • edited

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016

stgraber commented Oct 20, 2016

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016 • edited

davidfavor commented Oct 20, 2016 • edited

davidfavor commented Oct 20, 2016 • edited

davidfavor commented Oct 20, 2016

stgraber commented Oct 20, 2016

stgraber commented Oct 20, 2016 • edited

davidfavor commented Oct 20, 2016

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016

stgraber commented Oct 20, 2016

davidfavor commented Oct 20, 2016

davidfavor commented Oct 22, 2016 • edited

davidfavor commented Oct 23, 2016 • edited

davidfavor commented Dec 7, 2016

stgraber commented Dec 7, 2016

spyderdyne commented Feb 13, 2017

CarltonSemple commented Mar 16, 2017 • edited

stgraber commented Mar 16, 2017

CarltonSemple commented Mar 17, 2017

stgraber commented Mar 17, 2017

CarltonSemple commented Mar 17, 2017

CarltonSemple commented Mar 17, 2017

Conjohnsonjohnson commented Jun 9, 2017 • edited

davidfavor commented Jun 24, 2017

itisnotdone commented Sep 5, 2017

davidfavor commented Oct 20, 2016 •

edited

davidfavor commented Oct 20, 2016 •

edited

davidfavor commented Oct 20, 2016 •

edited

davidfavor commented Oct 20, 2016 •

edited

davidfavor commented Oct 20, 2016 •

edited

davidfavor commented Oct 20, 2016 •

edited

davidfavor commented Oct 20, 2016 •

edited

davidfavor commented Oct 20, 2016 •

edited

davidfavor commented Oct 20, 2016 •

edited

stgraber commented Oct 20, 2016 •

edited

davidfavor commented Oct 22, 2016 •

edited

davidfavor commented Oct 23, 2016 •

edited

CarltonSemple commented Mar 16, 2017 •

edited

Conjohnsonjohnson commented Jun 9, 2017 •

edited