Few containers fail to start: iptables v1.4.21: can't initialize iptables table `filter' #82

yprokule · 2019-07-29T15:14:46Z

Few ironic* containers fail to start as part of bmo pod:

$ oc get po -n openshift-machine-api | grep baremet
metal3-baremetal-operator-74fdb86688-pw6c4    4/8     CrashLoopBackOff    152        3h5m

$ oc logs po/metal3-baremetal-operator-74fdb86688-pw6c4 -n openshift-machine-api -c ironic-dnsmasq
iptables v1.4.21: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
$ oc logs po/metal3-baremetal-operator-74fdb86688-pw6c4 -n openshift-machine-api -c ironic-api
iptables v1.4.21: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
$ oc logs po/metal3-baremetal-operator-74fdb86688-pw6c4 -n openshift-machine-api -c ironic-httpd
iptables v1.4.21: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

yprokule · 2019-07-29T15:15:18Z

/cc @dtantsur @derekhiggins

juliakreger · 2019-07-29T15:28:05Z

@yprokule Are they being launched by the BMO as privileged containers with host networking?

yprokule · 2019-07-29T16:22:32Z

@juliakreger not sure how to check it, but here is containers' description from pod:

  baremetal-operator:
    Container ID:  cri-o://ba633c4366c33bb0b4a8b457e891b88a78c93fb0f4d27b18ed35262bbfb2a1c9
    Image:         quay.io/metal3-io/baremetal-operator:master
    Image ID:      quay.io/metal3-io/baremetal-operator@sha256:9723f8bc650e83e135e1be3387c906ec4c5fc5213030a043d8f955b2f8f22638
    Port:          60000/TCP
    Host Port:     60000/TCP
    Command:
      /baremetal-operator
    State:          Running
      Started:      Mon, 29 Jul 2019 12:17:38 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      WATCH_NAMESPACE:            openshift-machine-api (v1:metadata.namespace)
      POD_NAME:                   metal3-baremetal-operator-74fdb86688-pw6c4 (v1:metadata.name)
      OPERATOR_NAME:              baremetal-operator
      DEPLOY_KERNEL_URL:          <set to the key 'deploy_kernel_url' of config map 'ironic-bmo-configmap'>          Optional: false
      DEPLOY_RAMDISK_URL:         <set to the key 'deploy_ramdisk_url' of config map 'ironic-bmo-configmap'>         Optional: false
      IRONIC_ENDPOINT:            <set to the key 'ironic_endpoint' of config map 'ironic-bmo-configmap'>            Optional: false
      IRONIC_INSPECTOR_ENDPOINT:  <set to the key 'ironic_inspector_endpoint' of config map 'ironic-bmo-configmap'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from metal3-baremetal-operator-token-9s57b (ro)
  ironic-dnsmasq:
    Container ID:  cri-o://9cbc212cc120330dae1c053a04f7ef9593222642bb89446c63874efcf80c33e9
    Image:         quay.io/metal3-io/ironic:master
    Image ID:      quay.io/metal3-io/ironic@sha256:187b0a2918d4ec3c79f2aa2ffd8cfbae19603c75266a675789e5b014936088bb
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/rundnsmasq
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    3
      Started:      Mon, 29 Jul 2019 16:09:59 +0000
      Finished:     Mon, 29 Jul 2019 16:09:59 +0000
    Ready:          False
    Restart Count:  49
    Environment:
      HTTP_PORT:               <set to the key 'http_port' of config map 'ironic-bmo-configmap'>               Optional: false
      PROVISIONING_INTERFACE:  <set to the key 'provisioning_interface' of config map 'ironic-bmo-configmap'>  Optional: false
      DHCP_RANGE:              <set to the key 'dhcp_range' of config map 'ironic-bmo-configmap'>              Optional: false
    Mounts:
      /shared from ironic-data-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metal3-baremetal-operator-token-9s57b (ro)
  ironic-httpd:
    Container ID:  cri-o://8d567cb6e0382b1d48b9db032a04ec9585ba8aae86011ac5da0f439f2c6d3ab2
    Image:         quay.io/metal3-io/ironic:master
    Image ID:      quay.io/metal3-io/ironic@sha256:187b0a2918d4ec3c79f2aa2ffd8cfbae19603c75266a675789e5b014936088bb
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/runhttpd
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    3
      Started:      Mon, 29 Jul 2019 16:10:03 +0000
      Finished:     Mon, 29 Jul 2019 16:10:03 +0000
    Ready:          False
    Restart Count:  49
    Environment:
      HTTP_PORT:               <set to the key 'http_port' of config map 'ironic-bmo-configmap'>               Optional: false
      PROVISIONING_INTERFACE:  <set to the key 'provisioning_interface' of config map 'ironic-bmo-configmap'>  Optional: false
    Mounts:
      /shared from ironic-data-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metal3-baremetal-operator-token-9s57b (ro)
  ironic-api:
    Container ID:  cri-o://b1b3e604f303bf93fedd0d5b43f8116917a3392568598443ff1ac56ad1c78d33
    Image:         quay.io/metal3-io/ironic:master
    Image ID:      quay.io/metal3-io/ironic@sha256:187b0a2918d4ec3c79f2aa2ffd8cfbae19603c75266a675789e5b014936088bb
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/runironic-api
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    3
      Started:      Mon, 29 Jul 2019 16:11:05 +0000
      Finished:     Mon, 29 Jul 2019 16:11:05 +0000
    Ready:          False
    Restart Count:  49
    Environment:
      MARIADB_PASSWORD:        <set to the key 'password' in secret 'mariadb-password'>                        Optional: false
      HTTP_PORT:               <set to the key 'http_port' of config map 'ironic-bmo-configmap'>               Optional: false
      PROVISIONING_INTERFACE:  <set to the key 'provisioning_interface' of config map 'ironic-bmo-configmap'>  Optional: false
    Mounts:
      /shared from ironic-data-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metal3-baremetal-operator-token-9s57b (ro)

sadasu · 2019-07-29T16:54:08Z

@imain @dhellmann All containers except the baremetal-operator have privileged set to True. https://github.com/openshift-metal3/dev-scripts/blob/master/operator_ironic.yaml

juliakreger · 2019-07-29T17:18:49Z

@yprokule Are your steps to reproduce purely fire-up an install with openshift-metal3/dev-scripts? Or are you taking other steps?

juliakreger · 2019-07-29T19:22:37Z

I suspect the only option here is to try ingress policy filters for the pod....

In other words, remove the ip tables statements and we update the yaml loading the pod. Thoughts? @dtantsur @derekhiggins

see: https://kubernetes.io/docs/concepts/services-networking/network-policies/

dtantsur · 2019-07-30T08:17:23Z

As long as it works - I'm good with it :) I iptables might have been a temporary solution while we were launching ironic directly via podman.

derekhiggins · 2019-07-30T10:11:18Z

I thought this error could be ignored but maybe i'm wrong,

hmm, in the pod description(above) i don't see mention of privileged (as in https://github.com/metal3-io/baremetal-operator/blob/8169897aeb6580c27671f1b78d6217a541c8e079/deploy/operator_ironic.yaml#L60)
securityContext:
privileged: true

not sure if its relevant tbh..

derekhiggins · 2019-07-30T11:08:28Z

#83

dhellmann · 2019-07-31T13:11:08Z

I think #83 fixes this.

imain · 2019-07-31T16:44:15Z

What environment are you running this on? Is it a baremetal deployment? I'm wondering if 'provisioning interface' is set correctly?

yprokule · 2019-08-01T15:09:31Z

What environment are you running this on? Is it a baremetal deployment? I'm wondering if 'provisioning interface' is set correctly?

Virtual, deployed with make script from dev-scripts

dhellmann closed this as completed Jul 31, 2019

yprokule mentioned this issue Jul 31, 2019

ironic containers running under the metal3-baremetal-operator pod are crash-looping openshift-metal3/dev-scripts#703

Closed

dhellmann mentioned this issue Jul 31, 2019

minimize iptables rule in ironic images #84

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Few containers fail to start: iptables v1.4.21: can't initialize iptables table `filter' #82

Few containers fail to start: iptables v1.4.21: can't initialize iptables table `filter' #82

yprokule commented Jul 29, 2019

yprokule commented Jul 29, 2019

juliakreger commented Jul 29, 2019

yprokule commented Jul 29, 2019

sadasu commented Jul 29, 2019

juliakreger commented Jul 29, 2019

juliakreger commented Jul 29, 2019 •

edited

Loading

dtantsur commented Jul 30, 2019

derekhiggins commented Jul 30, 2019

derekhiggins commented Jul 30, 2019

dhellmann commented Jul 31, 2019

imain commented Jul 31, 2019

yprokule commented Aug 1, 2019

Few containers fail to start: iptables v1.4.21: can't initialize iptables table `filter' #82

Few containers fail to start: iptables v1.4.21: can't initialize iptables table `filter' #82

Comments

yprokule commented Jul 29, 2019

yprokule commented Jul 29, 2019

juliakreger commented Jul 29, 2019

yprokule commented Jul 29, 2019

sadasu commented Jul 29, 2019

juliakreger commented Jul 29, 2019

juliakreger commented Jul 29, 2019 • edited Loading

dtantsur commented Jul 30, 2019

derekhiggins commented Jul 30, 2019

derekhiggins commented Jul 30, 2019

dhellmann commented Jul 31, 2019

imain commented Jul 31, 2019

yprokule commented Aug 1, 2019

juliakreger commented Jul 29, 2019 •

edited

Loading