How to join an LXC container host?

[bradh352]

I've just spun up a cloudstack 4.20.1 8-node environment. All hosts are configured identically (regardless if their purpose is for KVM or LXC).

I joined the first 5 hosts as KVM hosts and that works fine.

Then I created a new cluster under the same zone via the UI for my LXC hosts with the intention of joining the remaining 3 nodes as LXC nodes but it fails (after a long wait).

Looking at the agent logs, all I see is:

DEBUG:root:execute:uname -r
DEBUG:root:execute:uname -m
DEBUG:root:execute:hostname -f
DEBUG:root:execute:kvm-ok
DEBUG:root:execute:awk '/MemTotal/ { printf "%.3f \n", $2/1024 }' /proc/meminfo
DEBUG:root:execute:ip a | grep "^\w" | grep -iv "^lo" | wc -l
DEBUG:root:execute:service apparmor status
DEBUG:root:execute:apparmor_status |grep libvirt
DEBUG:root:Failed to execute:
DEBUG:root:cloudbr0 is not a network device, is it down?
DEBUG:root:execute:sudo /usr/sbin/service network-manager status
DEBUG:root:Failed to execute:Unit network-manager.service could not be found.
DEBUG:root:execute:ip route show default | awk '{print $3,$5}'
DEBUG:root:execute:ifconfig public
DEBUG:root:Failed to execute:/bin/sh: 1: ifconfig: not found
DEBUG:root:Failed to get address from ifconfig
DEBUG:root:execute:sudo update-rc.d -f apparmor remove
DEBUG:root:execute:sudo update-rc.d -f apparmor defaults
DEBUG:root:execute:sudo /usr/sbin/service apparmor status
DEBUG:root:execute:sudo /usr/sbin/service apparmor start
DEBUG:root:execute:sudo /usr/sbin/service apparmor status
DEBUG:root:execute:sudo /usr/sbin/service apparmor start
DEBUG:root:execute:sudo update-rc.d -f network-manager remove
DEBUG:root:execute:sudo update-rc.d -f network-manager defaults
DEBUG:root:Failed to execute:update-rc.d: error: unable to read /etc/init.d/network-manager
DEBUG:root:execute:sudo /usr/sbin/service network-manager status
DEBUG:root:Failed to execute:Unit network-manager.service could not be found.
DEBUG:root:execute:sudo /usr/sbin/service network-manager start
DEBUG:root:Failed to execute:Failed to start network-manager.service: Unit network-manager.service not found.
DEBUG:root:execute:sudo /usr/sbin/service network-manager status
DEBUG:root:Failed to execute:Unit network-manager.service could not be found.
DEBUG:root:execute:sudo /usr/sbin/service network-manager start
DEBUG:root:Failed to execute:Failed to start network-manager.service: Unit network-manager.service not found.
DEBUG:root:execute:/etc/init.d/networking stop
DEBUG:root:Failed to execute:/bin/sh: 1: /etc/init.d/networking: not found
DEBUG:root:execute:/etc/init.d/networking start
DEBUG:root:Failed to execute:/bin/sh: 1: /etc/init.d/networking: not found

Looking at the command history, I see:

Jun  9 21:05:52 node7.testenv.bradhouse.dev sudo: cloudstack : PWD=/home/cloudstack ; USER=root ; COMMAND=/usr/bin/cloudstack-setup-agent -m 10.10.100.2,10.10.100.3,10.10.100.4 -z 2 -p 2 -c 5 -g 78999d33-6584-340b-a4bc-8c19b52aa195 -a -s --pubNic=cloudbr0 --prvNic=cloudbr0 --guestNic=cloudbr0 --hypervisor=lxc

compared to KVM:

Jun  9 20:43:31 node4.testenv.bradhouse.dev sudo: cloudstack : PWD=/home/cloudstack ; USER=root ; COMMAND=/usr/bin/cloudstack-setup-agent -m 10.10.100.2,10.10.100.3,10.10.100.4 -z 2 -p 2 -c 2 -g 044ed14f-03ec-31b4-a02a-c842cdcbba1b -a -s --pubNic=public --prvNic=hypervisor --guestNic=public --hypervisor=kvm

The thing that stands out at me is the pubNic, prvNic and guestNic ... the LXC should match what is passed for the KVM nodes, but instead are reverting to the default 'cloudbr0' which I don't have as I named my bridges more useful names. Any idea why these aren't passed properly?

Perhaps I missed something in the new cluster? I thought the network settings were zone-wide.

[bradh352]

Oh, and a secondary odd thing, when adding a host for LXC, the option for the ssh key is gone, it seems like it can only use a username/password? When looking at the API itself I'm not seeing that limitation, and not finding any discussion on that in the docs.

[bradh352]

In case its relevant, here's some cloudmonkey output with settings:

root@node3:~# cmk list clusters
{
  "cluster": [
    {
      "allocationstate": "Enabled",
      "arch": "x86_64",
      "clustertype": "CloudManaged",
      "cpuovercommitratio": "4.0",
      "hasannotations": false,
      "hypervisortype": "KVM",
      "id": "2c2425fe-a9cc-41e7-a60a-8f806fd0712a",
      "managedstate": "Managed",
      "memoryovercommitratio": "2.0",
      "name": "vm",
      "podid": "c758118b-1503-4d28-a10e-5d314760ec44",
      "podname": "pod",
      "resourcedetails": {
        "cpuOvercommitRatio": "4.0",
        "memoryOvercommitRatio": "2.0"
      },
      "zoneid": "e3b58c99-67bf-4dde-958d-bdda7e2ab667",
      "zonename": "testlab"
    },
    {
      "allocationstate": "Enabled",
      "arch": "x86_64",
      "clustertype": "CloudManaged",
      "cpuovercommitratio": "4.0",
      "hasannotations": false,
      "hypervisortype": "LXC",
      "id": "2cd66273-5698-48c5-9c20-0e21e4d174cc",
      "managedstate": "Managed",
      "memoryovercommitratio": "2.0",
      "name": "lxc",
      "podid": "c758118b-1503-4d28-a10e-5d314760ec44",
      "podname": "pod",
      "resourcedetails": {
        "cpuOvercommitRatio": "4.0",
        "memoryOvercommitRatio": "2.0"
      },
      "zoneid": "e3b58c99-67bf-4dde-958d-bdda7e2ab667",
      "zonename": "testlab"
    }
  ],
  "count": 2
}

root@node3:~# cmk list physicalnetworks
{
  "count": 3,
  "physicalnetwork": [
    {
      "broadcastdomainrange": "ZONE",
      "id": "a2feeaae-abe0-40c0-9501-55b5fbc4556f",
      "isolationmethods": "VLAN",
      "name": "mgmt-net",
      "state": "Enabled",
      "zoneid": "e3b58c99-67bf-4dde-958d-bdda7e2ab667",
      "zonename": "testlab"
    },
    {
      "broadcastdomainrange": "ZONE",
      "id": "e27bd3b0-da2b-4638-8ada-0d89e778c673",
      "isolationmethods": "VLAN",
      "name": "public-net",
      "state": "Enabled",
      "zoneid": "e3b58c99-67bf-4dde-958d-bdda7e2ab667",
      "zonename": "testlab"
    },
    {
      "broadcastdomainrange": "ZONE",
      "id": "afcfc718-06e9-44c4-9645-85b1e44c5506",
      "isolationmethods": "VXLAN",
      "name": "guest-net",
      "state": "Enabled",
      "vlan": "5000-10000",
      "zoneid": "e3b58c99-67bf-4dde-958d-bdda7e2ab667",
      "zonename": "testlab"
    }
  ]
}

root@node3:~# cmk list traffictypes physicalnetworkid=a2feeaae-abe0-40c0-9501-55b5fbc4556f
{
  "count": 2,
  "traffictype": [
    {
      "id": "798b1608-be32-4c6d-819b-629ea9331164",
      "kvmnetworklabel": "hypervisor",
      "physicalnetworkid": "a2feeaae-abe0-40c0-9501-55b5fbc4556f",
      "traffictype": "Management"
    },
    {
      "id": "31853d60-5ac4-495b-85fe-0546a3c46a35",
      "kvmnetworklabel": "hypervisor",
      "physicalnetworkid": "a2feeaae-abe0-40c0-9501-55b5fbc4556f",
      "traffictype": "Storage"
    }
  ]
}
root@node3:~# cmk list traffictypes physicalnetworkid=e27bd3b0-da2b-4638-8ada-0d89e778c673
{
  "count": 1,
  "traffictype": [
    {
      "id": "ff2303c6-7374-402f-a0b2-322b89da621e",
      "kvmnetworklabel": "public",
      "physicalnetworkid": "e27bd3b0-da2b-4638-8ada-0d89e778c673",
      "traffictype": "Public"
    }
  ]
}
root@node3:~# cmk list traffictypes physicalnetworkid=afcfc718-06e9-44c4-9645-85b1e44c5506
{
  "count": 1,
  "traffictype": [
    {
      "id": "65b0f74c-4f7a-42b2-98d5-514c5988d154",
      "kvmnetworklabel": "public",
      "physicalnetworkid": "afcfc718-06e9-44c4-9645-85b1e44c5506",
      "traffictype": "Guest"
    }
  ]
}

[weizhouapache]

@bradh352
I do not remember anyone else uses LXC
maybe add the host to a KVM host to see if it works ?

[bradh352]

yes, if I join it to the kvm cluster it joins fine. this is an lxc-specific issue. are you saying lxc is no longer supported?

[weizhouapache]

I would say , it is not tested and well-maintained

[DaanHoogland]

@bradh352 there is a deploy anything feature in the making, maybe you can use that for LXC? (see #9752)