By default we should seek a hyper-converged architecture

[marcoceppi]

If possible, the following components should be placed in LXD machines:

• kubernetes-master
• etcd
• elasticsearch
• easyrsa

The cost of getting three compute nodes is too high in today's bundle, however, not every cloud (not many) support address-ability of containers so public accessible components, kube-api-proxy, kibana need to be on "metal", along with the workers (since k8s + docker don't operate in lxd /yet/).

But a footprint of 5 machines for 3 compute instead of 9 machines for 3 compute seem much more tolerable. This will require intensive testing on public, private, and metal to insure no regressions

[mbruzek]

@marcoceppi We could get even more converged if we released a kubernetes-core bundle with our new architecture for those who don't need the ElasticCo stuff.

[marcoceppi]

I agree, I'll discuss this to see what the right direction is

[chuckbutler]

As an update to this issue, we're exploring making a core bundle, and doing a level of inheritance

[Cynerva]

Testing on openstack, where LXD containers are -not- addressable.

When deploying a modified bundle with the following mapping:

application	machine
kubeapi-load-balancer	0
easyrsa	lxd:0
etcd	lxd:0
kubernetes-master	lxd:0
kubernetes-worker	1
flannel	(subordinate)

The flannel unit attached to kubernetes-worker (machine 1) fails to reach etcd (non-addressable container on machine 0):

2016-10-07 18:50:18 INFO etcd-relation-joined Traceback (most recent call last):
2016-10-07 18:50:18 INFO etcd-relation-joined   File "/var/lib/juju/agents/unit-flannel-0/charm/hooks/etcd-relation-joined", line 19, in <module>
2016-10-07 18:50:18 INFO etcd-relation-joined     main()
2016-10-07 18:50:18 INFO etcd-relation-joined   File "/usr/local/lib/python3.5/dist-packages/charms/reactive/__init__.py", line 78, in main
2016-10-07 18:50:18 INFO etcd-relation-joined     bus.dispatch()
2016-10-07 18:50:18 INFO etcd-relation-joined   File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 434, in dispatch
2016-10-07 18:50:18 INFO etcd-relation-joined     _invoke(other_handlers)
2016-10-07 18:50:18 INFO etcd-relation-joined   File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 417, in _invoke
2016-10-07 18:50:18 INFO etcd-relation-joined     handler.invoke()
2016-10-07 18:50:18 INFO etcd-relation-joined   File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 291, in invoke
2016-10-07 18:50:18 INFO etcd-relation-joined     self._action(*args)
2016-10-07 18:50:18 INFO etcd-relation-joined   File "/var/lib/juju/agents/unit-flannel-0/charm/reactive/flannel.py", line 180, in render_flannel_config
2016-10-07 18:50:18 INFO etcd-relation-joined     _initialize_etcd_with_data(etcd)
2016-10-07 18:50:18 INFO etcd-relation-joined   File "/var/lib/juju/agents/unit-flannel-0/charm/reactive/flannel.py", line 99, in _initialize_etcd_with_data
2016-10-07 18:50:18 INFO etcd-relation-joined     client.write('/coreos.com/network/config', json_data)
2016-10-07 18:50:18 INFO etcd-relation-joined   File "/usr/local/lib/python3.5/dist-packages/etcd/client.py", line 461, in write
2016-10-07 18:50:18 INFO etcd-relation-joined     response = self.api_execute(path, method, params=params)
2016-10-07 18:50:18 INFO etcd-relation-joined   File "/usr/local/lib/python3.5/dist-packages/etcd/client.py", line 834, in wrapper
2016-10-07 18:50:18 INFO etcd-relation-joined     cause=e
2016-10-07 18:50:18 INFO etcd-relation-joined etcd.EtcdConnectionFailed: Connection to etcd failed due to MaxRetryError("HTTPSConnectionPool(host='10.0.0.158', port=2379): Max retries exceeded with url: /v2/keys/coreos.com/network/config (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7fabf479dac8>, 'Connection to 10.0.0.158 timed out. (connect timeout=60)'))",)
2016-10-07 18:50:18 ERROR juju.worker.uniter.operation runhook.go:107 hook "etcd-relation-joined" failed: exit status 1

Will do additional testing to see what else we hit.

[chuckbutler]

Chicken/egg. with ETCD in the lxd container, you will either need addressability into that container - via some means of proxy like socat, or the etcd unit has to be in the principal space. Another option would be to deploy an etcd proxy subordinate to bind on the principal unit. Which I know the project calico team has pioneered.

https://github.com/projectcalico/layer-etcd-proxy/blob/master/metadata.yaml#L26

I think if we swap that out for a True, build this and relate it both to the etcd container in lxd, and attach the sub to the kubeapi-load-balancer principal unit, we're in business with converging this and giving the flannel units access to the converged etcd application.

[Cynerva]

@chuckbutler Thanks, I'll give that a go later today or tomorrow.

For now I'm continuing to look for other problems. With etcd moved to a top-level machine:

application	machine
kubeapi-load-balancer	0
easyrsa	lxd:0
kubernetes-master	lxd:0
kubernetes-worker	1
etcd	2
flannel	(subordinate)

The deployment seems to stall with this status:

UNIT                     WORKLOAD     AGENT  MACHINE  PUBLIC-ADDRESS  PORTS     MESSAGE
easyrsa/0                active       idle   0/lxd/0  10.0.0.241                Certificate Authority connected.
etcd/0                   active       idle   2        10.246.67.21    2379/tcp  Healthy with 1 known peers. (leader)
kubeapi-load-balancer/0  active       idle   0        10.246.67.4     443/tcp   Loadbalancer ready.
kubernetes-master/0      maintenance  idle   0/lxd/1  10.0.0.198                Rendering authentication templates.
  flannel/1              waiting      idle            10.0.0.198                Flannel is starting up.
kubernetes-worker/0      waiting      idle   1        10.246.67.24              Waiting for cluster-manager to initiate start.
  flannel/0              active       idle            10.246.67.24              Flannel subnet 10.1.71.1/24

On 0/lxd/1, flannel service looks like it's having fun:

$ journalctl -o cat -u flannel
E1010 06:55:26.683050 13859 network.go:106] failed to register network: open /proc/sys/net/ipv4/neigh/flannel.1/app_solicit: no such file or directory
E1010 06:55:27.689838 13859 network.go:106] failed to register network: open /proc/sys/net/ipv4/neigh/flannel.1/app_solicit: no such file or directory
E1010 06:55:28.692227 13859 network.go:106] failed to register network: open /proc/sys/net/ipv4/neigh/flannel.1/app_solicit: no such file or directory
...
$ ls -al /proc/sys/net/ipv4/neigh/flannel.1/
total 0
dr-xr-xr-x 1 root root 0 Oct  7 21:23 .
dr-xr-xr-x 1 root root 0 Oct  7 21:23 ..
$

Unit logs:

• unit-kubernetes-master-0.log
• unit-flannel-1.log

I'll leave this deployment up in case there's more to look into.

[Cynerva]

Looks like the missing files in /proc/sys/net/ipv4/neigh/*/ are indeed what's holding that up.

After manually enabling security.privileged on the machine 0/lxd/1, those files now exist, and the deployment continues. However, nginx-ingress-controller appears to be in some kind of crash loop: https://gist.github.com/Cynerva/7114ab7d3f2b5f828c8d52a5aa7dc4a3

Hurraaay rabbit hole! I'm going to shift my attention back to etcd now.

[Cynerva]

Another option would be to deploy an etcd proxy subordinate to bind on the principal unit.

If we're willing to deploy etcd-proxy directly on the host machine, would we also be willing to deploy etcd directly on the host machine?

I don't think it needs to be a subordinate - there's nothing stopping us from deploying multiple applications to one machine. This deployment, for example, does not show any early problems:

application	machine
kubeapi-load-balancer	0
etcd	0
easyrsa	lxd:0
kubernetes-master	1
kubernetes-worker	2
flannel	(subordinate)

[Cynerva]

For now, I propose that we test with this bundle:

• Removed elasticsearch and friends
• Moved easyrsa to LXD
• Reduced num_units to 1 for etcd and kubernetes-worker

4 machines for 1 compute - a bit short of our early hopes.

We can reduce that by 1 more if we shove etcd and kubeapi-load-balancer on the same machine, or etcd-proxy -> etcd on LXD, but something about that doesn't sit right with me. Risk of charm collisions I suppose.

[mbruzek]

+1 to this latest suggestion. Should we create a new bundle with a different name in the same repository or overwrite the original bundle? @chuckbutler @wwwtyro @marcoceppi

[chuckbutler]

@Cynerva Good investigative work. I wanted to follow up that i have a similar bundle deployed in our lab environment (manually provisioned)

ubuntu@charmbox:~$ juju status -m container-lab:default
MODEL    CONTROLLER     CLOUD/REGION  VERSION
default  container-lab  manual        2.0-rc1

APP                VERSION        STATUS   SCALE  CHARM              STORE       REV  OS      NOTES
easyrsa            3.0.1          active       1  easyrsa            local         0  ubuntu
etcd               2.2.5          active       1  etcd               jujucharms   13  ubuntu
flannel            0.6.1          active       2  flannel            jujucharms    2  ubuntu
jenkins                           unknown      1  jenkins            jujucharms    6  ubuntu  exposed
kubernetes-master  1.4.0-beta.11  active       1  kubernetes-master  jujucharms    3  ubuntu
kubernetes-worker  1.4.0-beta.11  active       1  kubernetes-worker  jujucharms    3  ubuntu
workspace                         active       1  jenkins-workspace  jujucharms    2  ubuntu

UNIT                 WORKLOAD  AGENT      MACHINE  PUBLIC-ADDRESS  PORTS           MESSAGE
easyrsa/0            active    executing  0/lxd/0  10.0.0.221                      (update-status) Certificate Authority connected.
etcd/0               active    idle       0        67.205.134.156  2379/tcp        Healthy with 1 known peers. (leader)
jenkins/1            unknown   idle       3        198.211.116.73  8080/tcp
  workspace/0        active    idle                198.211.116.73                  Workspace uncompressed, ready to build.
kubernetes-master/0  active    idle       0        67.205.134.156  6443/tcp        Kubernetes master running.
  flannel/1          active    idle                67.205.134.156                  Flannel subnet 10.1.5.1/24
kubernetes-worker/0  active    idle       1        67.205.134.169  80/tcp,443/tcp  Kubernetes worker running.
  flannel/0          active    idle                67.205.134.169                  Flannel subnet 10.1.90.1/24

MACHINE  STATE    DNS             INS-ID                 SERIES  AZ
0        started  67.205.134.156  manual:67.205.134.156  xenial
0/lxd/0  started  10.0.0.221      juju-162852-0-lxd-0    xenial
1        started  67.205.134.169  manual:67.205.134.169  xenial
3        started  198.211.116.73  manual:198.211.116.73  trusty

RELATION      PROVIDES           CONSUMES           TYPE
certificates  easyrsa            kubernetes-master  regular
certificates  easyrsa            kubernetes-worker  regular
cluster       etcd               etcd               peer
etcd          etcd               flannel            regular
etcd          etcd               kubernetes-master  regular
sdn-plugin    flannel            kubernetes-master  regular
sdn-plugin    flannel            kubernetes-worker  regular
jenkins-host  jenkins            workspace          subordinate
host          kubernetes-master  flannel            subordinate
kube-dns      kubernetes-master  kubernetes-worker  regular
host          kubernetes-worker  flannel            subordinate

This seems to be a fairly reasonable deployment, but we'll also need to ensure we're cleaning up after charm removal with this pattern, as its co-locating on a pricnipal unit. There are certain members of our community that extremely dislike this pattern as it breaks encapsulation.

I'll leave it up to a thought exercise on our team if we can reasonably pull this off or if we will need to revisit deployment in LXD containers, to isolate the applications.

The subordinate of 'etcd-proxy' would give you the added benefit of making a quorem on a single host using multiple lxd containers. Allowing users to use a proper etd cluster with replication, but still suffering from a SPOF until we get routeable LXD addressing on all clouds, which then removes the need for the proxy.

[Cynerva]

Closing this, I don't think there's anything left to do here.