Configurable grid subnet, supernet #1323

SpComb · 2016-11-15T17:53:35Z

Fixes #1304

Server

Add Grid subnet, supernet to model, JSON views
Add HostNode overlay_ip to models, JSON views

Uses IPAddr to compute the offset, allowing the use of node_number > 255 once any existing 10.81.1.X service containers are migrated over to the new 10.81.128.X range.
Support optional POST /grids {"subnet": "...", "supernet": "..."}
Migrate existing girds to use the current default subnet/supernet

CLI

Render grid subnet, host node overlay_ip from server JSON
Render kontena grid cloud-config --insecure-registry= from the grid subnet
Support kontena grid create --subnet= --supernet=

Agent

Add IPAddr helpers
Use node_info['grid']['subnet'] and node_info['overlay_ip'] for etcd, weave launchers
Use node_info['grid']['supernet'] for the IPAM launcher
Use node_info['grid]['subnet'] for the default kontena network

Automatically compute the iprange from the upper half of the grid subnet.

SpComb · 2016-11-21T15:21:06Z

Rebased for 0.17 git master

SpComb · 2016-11-30T15:30:27Z

Rebased for 1.0 git master.

TODO on fixing up the WeaveWorker#attach_overlay 0.16 container migration logic, which is currently hardcoded to assume the prior 10.81.0.0/19: #1336 (comment)

Yes, this should ideally check that the container.overlay_addr.network from the container label overlaps with the expected grid subnet, such that the existing container.overlay_addr is within the new grid subnet.
if container.overlay_cidr.network != grid_subnet && grid_subnet.include?(container.overlay_ip)
For 0.16 -> 0.17 it can be implicitly assumed that both of those are within the 10.81.0.0/#{OVERLAY_SUFFIX}. With #1323 the WeaveWorker will need access to the node_info['grid']['subnet'], and better IPAddr logic for this comparison.

This could be used to allow some limited form of grid subnet changes in the future.

We can't attach containers until we have the grid configuration, which changes the WeaveWorker` startup ordering again.

SpComb · 2016-12-07T08:42:45Z

Assigning to me as I still need to work on the configurable subnet migration.

jakolehm · 2016-12-28T10:27:35Z

Is this progressing?

SpComb · 2016-12-28T11:21:52Z

Is this progressing?

This is blocked on the hardcoded /16 subnet mask used for <1.0 container network migration in the WeaveWorker. Each attach event needs access to the grid/node info from the server, which only exists in the Celluloid notifications atm.

jakolehm · 2016-12-28T13:10:29Z

But this can be only configured when grid is created, right? I can't see how this affects <1.0 migration?

SpComb · 2016-12-28T13:13:24Z

The WeaveWorker needs to know if the io.kontena.container.overlay_cidr label on the container contains an old pre-migration subnet mask. If it does, then it rewrites it to the desired grid overlay subnet mask. For that to happen, it needs to know what the current actual grid overlay subnet mask is, and you can't update that into the pre-existing container labels.

For new service containers being deployed, it's indeed not an issue; the server assigns them an overlay_cidr and the agent just sets and uses that label.

SpComb · 2017-01-12T12:01:43Z

Rebased to current git master.

Travis is failing on the flaky server specs.

SpComb · 2017-01-23T13:35:19Z

The CLI/server parts of this are pretty straightforward, but the agent bits are pending on a major refactoring to clean up the agent:node_info notifications and weave actors.

jnummelin

Some minor comments on code, overall looks good.

Documentation should be updated, especially regarding the supernet/subnet relation. That might not be really intuitive for all the Kontena users.

Some quick testing locally:

$ k grid create --supernet 10.90.0.0/12 --subnet 10.90.0.0/17 subnet-test

# ip addr show dev weave
33: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1410 qdisc noqueue state UP qlen 1000
    link/ether de:58:13:a5:1e:ce brd ff:ff:ff:ff:ff:ff
    inet 10.90.0.1/17 scope global weave
       valid_lft forever preferred_lft forever
    inet6 fe80::dc58:13ff:fea5:1ece/64 scope link 
       valid_lft forever preferred_lft forever

$ k service create web nginx && k service deploy web
$ k service show web | grep ip:
      ip: 10.90.64.65

Everything works as expected.

jnummelin · 2017-01-26T08:54:12Z

agent/lib/kontena/workers/weave_worker.rb

-        # override label for existing containers that may need to be migrated
+      if container.overlay_network.nil?
+        # overlay network migration for 0.16 compat
+        # override overlay network /12 -> /16 suffix for existing containers that may need to be migrated


/12 network? Do you mean the old (< 1.0) /19 subnet?

Fixed. I thought I rgrep'd for all of these; I confused the grid supernet with the old subnet :)

jnummelin · 2017-01-26T08:58:14Z

agent/spec/lib/kontena/launchers/etcd_launcher_spec.rb

        'grid' => {
-          'initial_size' => 3
+          'initial_size' => 3,
+          'subnet' => '10.81.0.0/16',


node_info is repeating quite often in the test cases, maybe do a let(:node_info)... over it?

Refactored specs to use context node_info

jnummelin · 2017-01-26T09:18:07Z

agent/lib/ipaddr_helpers.rb

@@ -0,0 +1,48 @@
+class IPAddr


We're doing IPAddr patching in many places now, maybe it's time to roll these as a separate gem? (not maybe part of this PR, but overall)

Yes, the stdlib IPAddr is really inadequate. I figure these should be replaced with the ipaddress gem.

The worst part is the crazy-looking IPAddr operations on the server, like

(IPAddr.new(self.grid.subnet) | self.node_number).to_s

This would be make somewhat more sense with the agent IPAddr patching (IPAddr.new(self.grid.subnet)[self.node_number]), but the way the kontena repo is structured, there's no easy way to share code between across all three sub-components...

Should I fix all the server and agent stuff to start using ipaddress in this PR, or later?

I think we should do it later as separate exercise.

SpComb · 2017-01-26T11:40:13Z

The plugins will also need to fixed to use the grid['subnet'] for the docker --insecure-registry=

SpComb · 2017-01-26T12:24:48Z

Documentation should be updated, especially regarding the supernet/subnet relation. That might not be really intuitive for all the Kontena users.

Added Kontena Network Model docs. Not sure what other docs should mention this? It's a pretty niche thing... until we have to start dealing with platforms/providers using 10.81.0.0/16 (like Packet, see #1304).

SpComb · 2017-01-26T14:04:05Z

@kke IMO this should be mostly okay for 1.1, if we want to. Need to fix the plugins to support this after merging, though.

jnummelin · 2017-01-26T21:25:09Z

docs/core-concepts/networking.md

+Both of these private RFC1918 IP address spaces are used for internal overlay networking within each grid.
+These overlay network addresses are only unique within a grid; different grids can (and will) use the same overlay network IP addresses.
+
+The grid subnet (`10.81.0.0/16`) is used to provide overlay networking addresses for both host nodes and service contaienrs.


jnummelin · 2017-01-26T21:44:11Z

I think the grid command reference should be updated to reflect this in /docs/using-kontena/grids.md

SpComb · 2017-01-27T08:40:04Z

I think the grid command reference should be updated to reflect this in /docs/using-kontena/grids.md

Fixed several other things at the same time, see #1727

* docs grids: document grid options, initial nodes * edit, fix links * remove references to #1323

… a io.kontena.container.overlay_network label

SpComb · 2017-02-21T07:20:44Z

Rebased for latest 1.1.x master, pruned the /16 validation stuff, and re-added the grid subnet docs left out of #1727

jakolehm added agent cli enhancement server labels Nov 16, 2016

jakolehm added this to the 1.0.0 milestone Nov 16, 2016

This was referenced Nov 18, 2016

Update default Kontena subnet mask across docs, packaging, cloud-config #1332

Merged

Agent: migrate container weave subnet netmask #1336

Merged

SpComb force-pushed the grid-subnet-supernet branch 2 times, most recently from d6019b3 to 26cbf4a Compare November 21, 2016 15:20

jakolehm modified the milestones: 1.0.0, 0.17.0, 1.1.0 Nov 24, 2016

SpComb force-pushed the grid-subnet-supernet branch from 26cbf4a to 91cec77 Compare November 30, 2016 15:29

SpComb self-assigned this Dec 7, 2016

SpComb force-pushed the grid-subnet-supernet branch from 91cec77 to c7f9a27 Compare January 12, 2017 11:43

jakolehm modified the milestones: 1.2.0, 1.1.0 Jan 23, 2017

SpComb force-pushed the grid-subnet-supernet branch from 8ab7b54 to 7078b06 Compare January 25, 2017 09:01

jnummelin reviewed Jan 26, 2017

View reviewed changes

SpComb removed their assignment Jan 26, 2017

jnummelin reviewed Jan 26, 2017

View reviewed changes

SpComb mentioned this pull request Jan 27, 2017

Docs: Improve grids options docs, document initial nodes #1727

Merged

SpComb pushed a commit that referenced this pull request Feb 1, 2017

remove references to #1323

a95f6ab

kke pushed a commit that referenced this pull request Feb 2, 2017

Docs: Improve grids options docs, document initial nodes (#1727)

ef9d7fc

* docs grids: document grid options, initial nodes * edit, fix links * remove references to #1323

Tero Marttila added 8 commits February 21, 2017 09:19

server: grid subnet, supernet; host node overlay_ip

c91875f

cli: use grid subnet, supernet, node overlay_ip

963617b

agent: add IPAddr helpers

4b24993

agent: use grid subnet, supernet; host node overlay_ip

803f6a3

kontena grid create --subnet= --supernet=

66e528b

agent: only migrate overlay_cidr for 0.16 containers that do not have…

089a8af

… a io.kontena.container.overlay_network label

agent: refactor etcd launcher specs to use context node_info

c34df48

docs update for grid subnet, supernet

0195cde

SpComb force-pushed the grid-subnet-supernet branch from 95df01b to 0195cde Compare February 21, 2017 07:19

Document grid subnet options

cc60f79

SpComb requested review from jakolehm and jnummelin February 27, 2017 08:01

SpComb added the status/code-review label Feb 27, 2017

SpComb mentioned this pull request Feb 27, 2017

Refactor agent to pull services desired state from the master #1873

Merged

jnummelin approved these changes Feb 28, 2017

View reviewed changes

SpComb merged commit e38e1ec into master Feb 28, 2017

SpComb deleted the grid-subnet-supernet branch February 28, 2017 10:09

SpComb mentioned this pull request Feb 28, 2017

Display service logs correctly when giving instance option #1874

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configurable grid subnet, supernet #1323

Configurable grid subnet, supernet #1323

SpComb commented Nov 15, 2016 •

edited

SpComb commented Nov 21, 2016

SpComb commented Nov 30, 2016 •

edited

SpComb commented Dec 7, 2016

jakolehm commented Dec 28, 2016

SpComb commented Dec 28, 2016

jakolehm commented Dec 28, 2016

SpComb commented Dec 28, 2016

SpComb commented Jan 12, 2017 •

edited

SpComb commented Jan 23, 2017 •

edited

jnummelin left a comment

jnummelin Jan 26, 2017

SpComb Jan 26, 2017

jnummelin Jan 26, 2017

SpComb Jan 26, 2017

jnummelin Jan 26, 2017

SpComb Jan 26, 2017 •

edited

SpComb Jan 26, 2017

jnummelin Jan 26, 2017

SpComb commented Jan 26, 2017 •

edited

SpComb commented Jan 26, 2017

SpComb commented Jan 26, 2017

jnummelin Jan 26, 2017

jnummelin commented Jan 26, 2017 •

edited

SpComb commented Jan 27, 2017

SpComb commented Feb 21, 2017 •

edited

Configurable grid subnet, supernet #1323

Configurable grid subnet, supernet #1323

Conversation

SpComb commented Nov 15, 2016 • edited

Server

CLI

Agent

SpComb commented Nov 21, 2016

SpComb commented Nov 30, 2016 • edited

SpComb commented Dec 7, 2016

jakolehm commented Dec 28, 2016

SpComb commented Dec 28, 2016

jakolehm commented Dec 28, 2016

SpComb commented Dec 28, 2016

SpComb commented Jan 12, 2017 • edited

SpComb commented Jan 23, 2017 • edited

jnummelin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SpComb Jan 26, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SpComb commented Jan 26, 2017 • edited

SpComb commented Jan 26, 2017

SpComb commented Jan 26, 2017

Choose a reason for hiding this comment

jnummelin commented Jan 26, 2017 • edited

SpComb commented Jan 27, 2017

SpComb commented Feb 21, 2017 • edited

SpComb commented Nov 15, 2016 •

edited

SpComb commented Nov 30, 2016 •

edited

SpComb commented Jan 12, 2017 •

edited

SpComb commented Jan 23, 2017 •

edited

SpComb Jan 26, 2017 •

edited

SpComb commented Jan 26, 2017 •

edited

jnummelin commented Jan 26, 2017 •

edited

SpComb commented Feb 21, 2017 •

edited