Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

calico-node never becomes healthy #2901

Closed
thommay opened this issue Oct 2, 2019 · 7 comments
Closed

calico-node never becomes healthy #2901

thommay opened this issue Oct 2, 2019 · 7 comments

Comments

@thommay
Copy link

@thommay thommay commented Oct 2, 2019

Expected Behavior

The calico-node should become healthy

Current Behavior

> kubectl describe -n kube-system pods calico-node-8k2tm
  Warning  Unhealthy  25s (x61 over 25m)    kubelet, upsilon   Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
click for kubectl logs output
> kubectl logs --namespace kube-system  calico-node-8k2tm
	/go/src/github.com/projectcalico/node/cmd/calico-node/main.go:100 +0x405
2019-10-02 20:20:13.305 [INFO][25423] logutils.go 82: Early screen log level set to info
2019-10-02 20:20:13.306 [INFO][25423] daemon.go 146: Felix starting up GOMAXPROCS=12 builddate="7a042b762bc3a20b28a712e22fcc72160e4b5ddf" gitcommit="2019-09-26T17:59:26+0000" version="v3.9.1"
2019-10-02 20:20:13.306 [INFO][25423] daemon.go 164: Loading configuration...
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "ipv6support"="false"
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "felixhostname"="upsilon"
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "etcdcertfile"=""
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "logseverityscreen"="info"
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "etcdscheme"=""
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "defaultendpointtohostaction"="ACCEPT"
2019-10-02 20:20:13.307 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "etcdaddr"=""
2019-10-02 20:20:13.308 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "datastoretype"="kubernetes"
2019-10-02 20:20:13.308 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "etcdkeyfile"=""
2019-10-02 20:20:13.308 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "healthenabled"="true"
2019-10-02 20:20:13.308 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "ipinipmtu"="1440"
2019-10-02 20:20:13.308 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "etcdendpoints"=""
2019-10-02 20:20:13.308 [INFO][25423] env_var_loader.go 40: Found felix environment variable: "etcdcafile"=""
2019-10-02 20:20:13.308 [INFO][25423] daemon.go 187: Loading config file: /etc/calico/felix.cfg
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 253: Merging in config from environment variable: map[datastoretype:kubernetes defaultendpointtohostaction:ACCEPT etcdaddr: etcdcafile: etcdcertfile: etcdendpoints: etcdkeyfile: etcdscheme: felixhostname:upsilon healthenabled:true ipinipmtu:1440 ipv6support:false logseverityscreen:info]
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 262: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdendpoints" source=environment variable
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 262: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdcafile" source=environment variable
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 262: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdscheme" source=environment variable
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 262: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdaddr" source=environment variable
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 262: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdkeyfile" source=environment variable
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 262: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdcertfile" source=environment variable
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 334: Parsing value for LogSeverityScreen: info (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 370: Parsed value for LogSeverityScreen: INFO (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 334: Parsing value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 370: Parsed value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 334: Parsing value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 370: Parsed value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 334: Parsing value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 370: Parsed value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.309 [INFO][25423] config_params.go 334: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 253: Merging in config from config file: map[LogFilePath:None LogSeverityFile:None LogSeveritySys:None MetadataAddr:None]
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 334: Parsing value for LogSeverityScreen: info (from environment variable)
2019-10-02 20:20:13.310 [INFO][25423] config_params.go 370: Parsed value for LogSeverityScreen: INFO (from environment variable)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 334: Parsing value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 370: Parsed value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 334: Parsing value for MetadataAddr: None (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 370: Parsed value for MetadataAddr:  (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 334: Parsing value for LogFilePath: None (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 370: Parsed value for LogFilePath:  (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 334: Parsing value for LogSeverityFile: None (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 370: Parsed value for LogSeverityFile:  (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 334: Parsing value for LogSeveritySys: None (from config file)
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 370: Parsed value for LogSeveritySys:  (from config file)
2019-10-02 20:20:13.311 [INFO][25423] health.go 233: Health enabled.  Starting server. host="localhost" port=9099
2019-10-02 20:20:13.311 [INFO][25423] config_params.go 455: Encap disabled, disabling node poll (if KDD is in use).
2019-10-02 20:20:13.311 [INFO][25423] daemon.go 219: Connecting to datastore datastore="kubernetes"
2019-10-02 20:20:13.314 [INFO][25423] k8s.go 228: Using Calico IPAM
2019-10-02 20:20:13.314 [INFO][25423] daemon.go 226: Created datastore client
2019-10-02 20:20:13.328 [INFO][25423] daemon.go 800: Loaded ready flag kind="ClusterInformation" name="default" ready=true
2019-10-02 20:20:13.335 [INFO][25423] daemon.go 780: No config of this type kind="FelixConfiguration" name="node.upsilon"
2019-10-02 20:20:13.340 [INFO][25423] config_params.go 253: Merging in config from datastore (global): map[CalicoVersion:v3.9.1 ClusterGUID:76ffcce211b64b55b8650bdf51ce73ae ClusterType:k8s,bgp,kdd IpInIpEnabled:true LogSeverityScreen:Info ReportingIntervalSecs:0]
2019-10-02 20:20:13.340 [INFO][25423] config_params.go 334: Parsing value for LogSeverityScreen: info (from environment variable)
2019-10-02 20:20:13.340 [INFO][25423] config_params.go 370: Parsed value for LogSeverityScreen: INFO (from environment variable)
2019-10-02 20:20:13.340 [INFO][25423] config_params.go 334: Parsing value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.340 [INFO][25423] config_params.go 370: Parsed value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for LogSeverityFile: None (from config file)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for LogSeverityFile:  (from config file)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for LogSeveritySys: None (from config file)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 370: Parsed value for LogSeveritySys:  (from config file)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 334: Parsing value for MetadataAddr: None (from config file)
2019-10-02 20:20:13.341 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for MetadataAddr:  (from config file)
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for LogFilePath: None (from config file)
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for LogFilePath:  (from config file)
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for ClusterType: k8s,bgp,kdd (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for ClusterType: k8s,bgp,kdd (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for CalicoVersion: v3.9.1 (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for CalicoVersion: v3.9.1 (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for LogSeverityScreen: Info (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for LogSeverityScreen: INFO (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 373: Skipping config value for LogSeverityScreen from datastore (global); already have a value from environment variable
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for IpInIpEnabled: true (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for IpInIpEnabled: true (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for ReportingIntervalSecs: 0 (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for ReportingIntervalSecs: 0s (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 334: Parsing value for ClusterGUID: 76ffcce211b64b55b8650bdf51ce73ae (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 370: Parsed value for ClusterGUID: 76ffcce211b64b55b8650bdf51ce73ae (from datastore (global))
2019-10-02 20:20:13.342 [INFO][25423] config_params.go 253: Merging in config from datastore (per-host): map[IpInIpTunnelAddr:192.168.33.128]
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for HealthEnabled: true (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for Ipv6Support: false (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for IpInIpMtu: 1440 (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for FelixHostname: upsilon (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for LogSeverityScreen: info (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for LogSeverityScreen: INFO (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for DatastoreType: kubernetes (from environment variable)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 334: Parsing value for MetadataAddr: None (from config file)
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.343 [INFO][25423] config_params.go 370: Parsed value for MetadataAddr:  (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for LogFilePath: None (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for LogFilePath:  (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for LogSeverityFile: None (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for LogSeverityFile:  (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for LogSeveritySys: None (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 351: Value set to 'none', replacing with zero-value: "".
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for LogSeveritySys:  (from config file)
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for IpInIpTunnelAddr: 192.168.33.128 (from datastore (per-host))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for IpInIpTunnelAddr: 192.168.33.128 (from datastore (per-host))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for ClusterGUID: 76ffcce211b64b55b8650bdf51ce73ae (from datastore (global))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for ClusterGUID: 76ffcce211b64b55b8650bdf51ce73ae (from datastore (global))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for ClusterType: k8s,bgp,kdd (from datastore (global))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for ClusterType: k8s,bgp,kdd (from datastore (global))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 334: Parsing value for CalicoVersion: v3.9.1 (from datastore (global))
2019-10-02 20:20:13.344 [INFO][25423] config_params.go 370: Parsed value for CalicoVersion: v3.9.1 (from datastore (global))
2019-10-02 20:20:13.345 [INFO][25423] config_params.go 334: Parsing value for LogSeverityScreen: Info (from datastore (global))
2019-10-02 20:20:13.345 [INFO][25423] config_params.go 370: Parsed value for LogSeverityScreen: INFO (from datastore (global))
2019-10-02 20:20:13.345 [INFO][25423] config_params.go 373: Skipping config value for LogSeverityScreen from datastore (global); already have a value from environment variable
2019-10-02 20:20:13.345 [INFO][25423] config_params.go 334: Parsing value for IpInIpEnabled: true (from datastore (global))
2019-10-02 20:20:13.346 [INFO][25423] config_params.go 370: Parsed value for IpInIpEnabled: true (from datastore (global))
2019-10-02 20:20:13.346 [INFO][25423] config_params.go 334: Parsing value for ReportingIntervalSecs: 0 (from datastore (global))
2019-10-02 20:20:13.346 [INFO][25423] config_params.go 370: Parsed value for ReportingIntervalSecs: 0s (from datastore (global))
2019-10-02 20:20:13.348 [INFO][25423] k8s.go 228: Using Calico IPAM
2019-10-02 20:20:13.348 [INFO][25423] health.go 133: Health of component changed lastReport=health.HealthReport{Live:true, Ready:false} name="felix-startup" newReport=&health.HealthReport{Live:true, Ready:true}
2019-10-02 20:20:13.348 [INFO][25423] health.go 230: Health enabled.  Server is already running. host="localhost" port=9099
2019-10-02 20:20:13.348 [INFO][25423] daemon.go 294: Successfully loaded configuration. GOMAXPROCS=12 builddate="7a042b762bc3a20b28a712e22fcc72160e4b5ddf" config=&config.Config{UseInternalDataplaneDriver:true, DataplaneDriver:"calico-iptables-plugin", DatastoreType:"kubernetes", FelixHostname:"upsilon", EtcdAddr:"127.0.0.1:2379", EtcdScheme:"http", EtcdKeyFile:"", EtcdCertFile:"", EtcdCaFile:"", EtcdEndpoints:[]string(nil), TyphaAddr:"", TyphaK8sServiceName:"", TyphaK8sNamespace:"kube-system", TyphaReadTimeout:30000000000, TyphaWriteTimeout:10000000000, TyphaKeyFile:"", TyphaCertFile:"", TyphaCAFile:"", TyphaCN:"", TyphaURISAN:"", Ipv6Support:false, IgnoreLooseRPF:false, IptablesBackend:"legacy", RouteRefreshInterval:90000000000, DeviceRouteSourceAddress:net.IP(nil), DeviceRouteProtocol:3, RemoveExternalRoutes:true, IptablesRefreshInterval:90000000000, IptablesPostWriteCheckIntervalSecs:1000000000, IptablesLockFilePath:"/run/xtables.lock", IptablesLockTimeoutSecs:0, IptablesLockProbeIntervalMillis:50000000, IpsetsRefreshInterval:10000000000, MaxIpsetSize:1048576, XDPRefreshInterval:90000000000, PolicySyncPathPrefix:"", NetlinkTimeoutSecs:10000000000, MetadataAddr:"", MetadataPort:8775, OpenstackRegion:"", InterfacePrefix:"cali", InterfaceExclude:[]*regexp.Regexp{(*regexp.Regexp)(0xc0000d9560)}, ChainInsertMode:"insert", DefaultEndpointToHostAction:"ACCEPT", IptablesFilterAllowAction:"ACCEPT", IptablesMangleAllowAction:"ACCEPT", LogPrefix:"calico-packet", LogFilePath:"", LogSeverityFile:"", LogSeverityScreen:"INFO", LogSeveritySys:"", VXLANEnabled:false, VXLANPort:4789, VXLANVNI:4096, VXLANMTU:1410, IPv4VXLANTunnelAddr:net.IP(nil), VXLANTunnelMACAddr:"", IpInIpEnabled:true, IpInIpMtu:1440, IpInIpTunnelAddr:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xc0, 0xa8, 0x21, 0x80}, ReportingIntervalSecs:0, ReportingTTLSecs:90000000000, EndpointReportingEnabled:false, EndpointReportingDelaySecs:1000000000, IptablesMarkMask:0xffff0000, DisableConntrackInvalidCheck:false, HealthEnabled:true, HealthPort:9099, HealthHost:"localhost", PrometheusMetricsEnabled:false, PrometheusMetricsHost:"", PrometheusMetricsPort:9091, PrometheusGoMetricsEnabled:true, PrometheusProcessMetricsEnabled:true, FailsafeInboundHostPorts:[]config.ProtoPort{config.ProtoPort{Protocol:"tcp", Port:0x16}, config.ProtoPort{Protocol:"udp", Port:0x44}, config.ProtoPort{Protocol:"tcp", Port:0xb3}, config.ProtoPort{Protocol:"tcp", Port:0x94b}, config.ProtoPort{Protocol:"tcp", Port:0x94c}, config.ProtoPort{Protocol:"tcp", Port:0x1a0a}, config.ProtoPort{Protocol:"tcp", Port:0x1a0b}}, FailsafeOutboundHostPorts:[]config.ProtoPort{config.ProtoPort{Protocol:"udp", Port:0x35}, config.ProtoPort{Protocol:"udp", Port:0x43}, config.ProtoPort{Protocol:"tcp", Port:0xb3}, config.ProtoPort{Protocol:"tcp", Port:0x94b}, config.ProtoPort{Protocol:"tcp", Port:0x94c}, config.ProtoPort{Protocol:"tcp", Port:0x1a0a}, config.ProtoPort{Protocol:"tcp", Port:0x1a0b}}, KubeNodePortRanges:[]numorstring.Port{numorstring.Port{MinPort:0x7530, MaxPort:0x7fff, PortName:""}}, NATPortRange:numorstring.Port{MinPort:0x0, MaxPort:0x0, PortName:""}, NATOutgoingAddress:net.IP(nil), UsageReportingEnabled:true, UsageReportingInitialDelaySecs:300000000000, UsageReportingIntervalSecs:86400000000000, ClusterGUID:"76ffcce211b64b55b8650bdf51ce73ae", ClusterType:"k8s,bgp,kdd", CalicoVersion:"v3.9.1", ExternalNodesCIDRList:[]string(nil), DebugMemoryProfilePath:"", DebugCPUProfilePath:"/tmp/felix-cpu-<timestamp>.pprof", DebugDisableLogDropping:false, DebugSimulateCalcGraphHangAfter:0, DebugSimulateDataplaneHangAfter:0, sourceToRawConfig:map[config.Source]map[string]string{0x1:map[string]string{"CalicoVersion":"v3.9.1", "ClusterGUID":"76ffcce211b64b55b8650bdf51ce73ae", "ClusterType":"k8s,bgp,kdd", "IpInIpEnabled":"true", "LogSeverityScreen":"Info", "ReportingIntervalSecs":"0"}, 0x2:map[string]string{"IpInIpTunnelAddr":"192.168.33.128"}, 0x3:map[string]string{"LogFilePath":"None", "LogSeverityFile":"None", "LogSeveritySys":"None", "MetadataAddr":"None"}, 0x4:map[string]string{"datastoretype":"kubernetes", "defaultendpointtohostaction":"ACCEPT", "felixhostname":"upsilon", "healthenabled":"true", "ipinipmtu":"1440", "ipv6support":"false", "logseverityscreen":"info"}}, rawValues:map[string]string{"CalicoVersion":"v3.9.1", "ClusterGUID":"76ffcce211b64b55b8650bdf51ce73ae", "ClusterType":"k8s,bgp,kdd", "DatastoreType":"kubernetes", "DefaultEndpointToHostAction":"ACCEPT", "FelixHostname":"upsilon", "HealthEnabled":"true", "IpInIpEnabled":"true", "IpInIpMtu":"1440", "IpInIpTunnelAddr":"192.168.33.128", "Ipv6Support":"false", "LogFilePath":"None", "LogSeverityFile":"None", "LogSeverityScreen":"info", "LogSeveritySys":"None", "MetadataAddr":"None", "ReportingIntervalSecs":"0"}, Err:error(nil), IptablesNATOutgoingInterfaceFilter:"", SidecarAccelerationEnabled:false, XDPEnabled:true, GenericXDPEnabled:false, loadClientConfigFromEnvironment:(func() (*apiconfig.CalicoAPIConfig, error))(0x10477e0), useNodeResourceUpdates:false} gitcommit="2019-09-26T17:59:26+0000" version="v3.9.1"
2019-10-02 20:20:13.350 [INFO][25423] driver.go 43: Using internal (linux) dataplane driver.
2019-10-02 20:20:13.350 [INFO][25423] driver.go 85: Calculated iptables mark bits acceptMark=0x10000 endpointMark=0xfff00000 endpointMarkNonCali=0x100000 passMark=0x20000 scratch0Mark=0x40000 scratch1Mark=0x80000
2019-10-02 20:20:13.350 [INFO][25423] int_dataplane.go 240: Creating internal dataplane driver. config=intdataplane.Config{Hostname:"upsilon", IPv6Enabled:false, RuleRendererOverride:rules.RuleRenderer(nil), IPIPMTU:1440, VXLANMTU:1410, IgnoreLooseRPF:false, MaxIPSetSize:1048576, IptablesBackend:"legacy", IPSetsRefreshInterval:10000000000, RouteRefreshInterval:90000000000, DeviceRouteSourceAddress:net.IP(nil), DeviceRouteProtocol:3, RemoveExternalRoutes:true, IptablesRefreshInterval:90000000000, IptablesPostWriteCheckInterval:1000000000, IptablesInsertMode:"insert", IptablesLockFilePath:"/run/xtables.lock", IptablesLockTimeout:0, IptablesLockProbeInterval:50000000, XDPRefreshInterval:90000000000, NetlinkTimeout:10000000000, RulesConfig:rules.Config{IPSetConfigV4:(*ipsets.IPVersionConfig)(0xc0000be7d0), IPSetConfigV6:(*ipsets.IPVersionConfig)(0xc0000be910), WorkloadIfacePrefixes:[]string{"cali"}, IptablesMarkAccept:0x10000, IptablesMarkPass:0x20000, IptablesMarkScratch0:0x40000, IptablesMarkScratch1:0x80000, IptablesMarkEndpoint:0xfff00000, IptablesMarkNonCaliEndpoint:0x100000, KubeNodePortRanges:[]numorstring.Port{numorstring.Port{MinPort:0x7530, MaxPort:0x7fff, PortName:""}}, KubeIPVSSupportEnabled:false, OpenStackMetadataIP:net.IP(nil), OpenStackMetadataPort:0x2247, OpenStackSpecialCasesEnabled:false, VXLANEnabled:false, VXLANPort:4789, VXLANVNI:4096, IPIPEnabled:true, IPIPTunnelAddress:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xc0, 0xa8, 0x21, 0x80}, VXLANTunnelAddress:net.IP(nil), IptablesLogPrefix:"calico-packet", EndpointToHostAction:"ACCEPT", IptablesFilterAllowAction:"ACCEPT", IptablesMangleAllowAction:"ACCEPT", FailsafeInboundHostPorts:[]config.ProtoPort{config.ProtoPort{Protocol:"tcp", Port:0x16}, config.ProtoPort{Protocol:"udp", Port:0x44}, config.ProtoPort{Protocol:"tcp", Port:0xb3}, config.ProtoPort{Protocol:"tcp", Port:0x94b}, config.ProtoPort{Protocol:"tcp", Port:0x94c}, config.ProtoPort{Protocol:"tcp", Port:0x1a0a}, config.ProtoPort{Protocol:"tcp", Port:0x1a0b}}, FailsafeOutboundHostPorts:[]config.ProtoPort{config.ProtoPort{Protocol:"udp", Port:0x35}, config.ProtoPort{Protocol:"udp", Port:0x43}, config.ProtoPort{Protocol:"tcp", Port:0xb3}, config.ProtoPort{Protocol:"tcp", Port:0x94b}, config.ProtoPort{Protocol:"tcp", Port:0x94c}, config.ProtoPort{Protocol:"tcp", Port:0x1a0a}, config.ProtoPort{Protocol:"tcp", Port:0x1a0b}}, DisableConntrackInvalid:false, NATPortRange:numorstring.Port{MinPort:0x0, MaxPort:0x0, PortName:""}, IptablesNATOutgoingInterfaceFilter:"", NATOutgoingAddress:net.IP(nil)}, IfaceMonitorConfig:ifacemonitor.Config{InterfaceExcludes:[]*regexp.Regexp{(*regexp.Regexp)(0xc0000d9560)}}, StatusReportingInterval:0, ConfigChangedRestartCallback:(func())(0x15f81e0), PostInSyncCallback:(func())(0x15dced0), HealthAggregator:(*health.HealthAggregator)(0xc000540000), DebugSimulateDataplaneHangAfter:0, ExternalNodesCidrs:[]string(nil), XDPEnabled:true, XDPAllowGeneric:false, SidecarAccelerationEnabled:false, LookPathOverride:(func(string) (string, error))(nil)}
2019-10-02 20:20:13.350 [INFO][25423] rule_defs.go 294: Creating rule renderer. config=rules.Config{IPSetConfigV4:(*ipsets.IPVersionConfig)(0xc0000be7d0), IPSetConfigV6:(*ipsets.IPVersionConfig)(0xc0000be910), WorkloadIfacePrefixes:[]string{"cali"}, IptablesMarkAccept:0x10000, IptablesMarkPass:0x20000, IptablesMarkScratch0:0x40000, IptablesMarkScratch1:0x80000, IptablesMarkEndpoint:0xfff00000, IptablesMarkNonCaliEndpoint:0x100000, KubeNodePortRanges:[]numorstring.Port{numorstring.Port{MinPort:0x7530, MaxPort:0x7fff, PortName:""}}, KubeIPVSSupportEnabled:false, OpenStackMetadataIP:net.IP(nil), OpenStackMetadataPort:0x2247, OpenStackSpecialCasesEnabled:false, VXLANEnabled:false, VXLANPort:4789, VXLANVNI:4096, IPIPEnabled:true, IPIPTunnelAddress:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xc0, 0xa8, 0x21, 0x80}, VXLANTunnelAddress:net.IP(nil), IptablesLogPrefix:"calico-packet", EndpointToHostAction:"ACCEPT", IptablesFilterAllowAction:"ACCEPT", IptablesMangleAllowAction:"ACCEPT", FailsafeInboundHostPorts:[]config.ProtoPort{config.ProtoPort{Protocol:"tcp", Port:0x16}, config.ProtoPort{Protocol:"udp", Port:0x44}, config.ProtoPort{Protocol:"tcp", Port:0xb3}, config.ProtoPort{Protocol:"tcp", Port:0x94b}, config.ProtoPort{Protocol:"tcp", Port:0x94c}, config.ProtoPort{Protocol:"tcp", Port:0x1a0a}, config.ProtoPort{Protocol:"tcp", Port:0x1a0b}}, FailsafeOutboundHostPorts:[]config.ProtoPort{config.ProtoPort{Protocol:"udp", Port:0x35}, config.ProtoPort{Protocol:"udp", Port:0x43}, config.ProtoPort{Protocol:"tcp", Port:0xb3}, config.ProtoPort{Protocol:"tcp", Port:0x94b}, config.ProtoPort{Protocol:"tcp", Port:0x94c}, config.ProtoPort{Protocol:"tcp", Port:0x1a0a}, config.ProtoPort{Protocol:"tcp", Port:0x1a0b}}, DisableConntrackInvalid:false, NATPortRange:numorstring.Port{MinPort:0x0, MaxPort:0x0, PortName:""}, IptablesNATOutgoingInterfaceFilter:"", NATOutgoingAddress:net.IP(nil)}
2019-10-02 20:20:13.350 [INFO][25423] rule_defs.go 304: Workload to host packets will be accepted.
2019-10-02 20:20:13.350 [INFO][25423] rule_defs.go 318: filter table allowed packets will be accepted immediately.
2019-10-02 20:20:13.351 [INFO][25423] rule_defs.go 326: mangle table allowed packets will be accepted immediately.
2019-10-02 20:20:13.353 [INFO][25423] feature_detect.go 110: Updating detected iptables features features=iptables.Features{SNATFullyRandom:true, MASQFullyRandom:true, RestoreSupportsLock:true} iptablesVersion=1.8.2 kernelVersion=5.0.0
2019-10-02 20:20:13.354 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-restore", "iptables-restore"} command="iptables-legacy-restore" ipVersion=0x4 saveOrRestore="restore"
2019-10-02 20:20:13.354 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-save", "iptables-save"} command="iptables-legacy-save" ipVersion=0x4 saveOrRestore="save"
2019-10-02 20:20:13.354 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-restore", "iptables-restore"} command="iptables-legacy-restore" ipVersion=0x4 saveOrRestore="restore"
2019-10-02 20:20:13.354 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-save", "iptables-save"} command="iptables-legacy-save" ipVersion=0x4 saveOrRestore="save"
2019-10-02 20:20:13.354 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-restore", "iptables-restore"} command="iptables-legacy-restore" ipVersion=0x4 saveOrRestore="restore"
2019-10-02 20:20:13.354 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-save", "iptables-save"} command="iptables-legacy-save" ipVersion=0x4 saveOrRestore="save"
2019-10-02 20:20:13.355 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-restore", "iptables-restore"} command="iptables-legacy-restore" ipVersion=0x4 saveOrRestore="restore"
2019-10-02 20:20:13.355 [INFO][25423] table.go 444: Looked up iptables command backendMode="legacy" candidates=[]string{"iptables-legacy-save", "iptables-save"} command="iptables-legacy-save" ipVersion=0x4 saveOrRestore="save"
2019-10-02 20:20:13.355 [INFO][25423] int_dataplane.go 355: Checking if we need to clean up the VXLAN device
2019-10-02 20:20:13.355 [WARNING][25423] int_dataplane.go 357: Failed to query VXLAN device error=Link not found
2019-10-02 20:20:13.357 [INFO][25423] int_dataplane.go 377: XDP acceleration enabled.
2019-10-02 20:20:13.362 [INFO][25423] route_table.go 170: Calculated interface name regexp regex="^cali.*"
2019-10-02 20:20:13.362 [INFO][25423] ipsets.go 119: Queueing IP set for creation family="inet" setID="all-ipam-pools" setType="hash:net"
2019-10-02 20:20:13.362 [INFO][25423] ipsets.go 119: Queueing IP set for creation family="inet" setID="masq-ipam-pools" setType="hash:net"
2019-10-02 20:20:13.362 [INFO][25423] int_dataplane.go 539: Registering to report health.
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-failsafe-in" ipVersion=0x4 table="raw"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-failsafe-out" ipVersion=0x4 table="raw"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-PREROUTING" ipVersion=0x4 table="raw"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-OUTPUT" ipVersion=0x4 table="raw"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-FORWARD" ipVersion=0x4 table="filter"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-INPUT" ipVersion=0x4 table="filter"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-wl-to-host" ipVersion=0x4 table="filter"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-failsafe-in" ipVersion=0x4 table="filter"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-OUTPUT" ipVersion=0x4 table="filter"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-failsafe-out" ipVersion=0x4 table="filter"
2019-10-02 20:20:13.363 [INFO][25423] int_dataplane.go 697: IPIP enabled, starting thread to keep tunnel configuration in sync.
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-PREROUTING" ipVersion=0x4 table="nat"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-POSTROUTING" ipVersion=0x4 table="nat"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-OUTPUT" ipVersion=0x4 table="nat"
2019-10-02 20:20:13.363 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-failsafe-in" ipVersion=0x4 table="mangle"
2019-10-02 20:20:13.363 [INFO][25423] ipip_mgr.go 85: IPIP thread started.
2019-10-02 20:20:13.364 [INFO][25423] table.go 475: Queueing update of chain. chainName="cali-PREROUTING" ipVersion=0x4 table="mangle"
2019-10-02 20:20:13.366 [WARNING][25423] int_dataplane.go 728: failed to set XDP failsafe ports, disabling XDP: failed to create map (calico_failsafe_ports_v1): exit status 255
Error: map create failed: Operation not permitted

2019-10-02 20:20:13.436 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-020661529): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-020661529'
Error: failed to load object file
 try=0
2019-10-02 20:20:13.504 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-216357796): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-216357796'
Error: failed to load object file
 try=1
2019-10-02 20:20:13.560 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-340555443): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-340555443'
Error: failed to load object file
 try=2
2019-10-02 20:20:13.624 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-394035830): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-394035830'
Error: failed to load object file
 try=3
2019-10-02 20:20:13.692 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-882563421): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-882563421'
Error: failed to load object file
 try=4
2019-10-02 20:20:13.760 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-573287704): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-573287704'
Error: failed to load object file
 try=5
2019-10-02 20:20:13.832 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-399081367): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-399081367'
Error: failed to load object file
 try=6
2019-10-02 20:20:13.896 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-183507978): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-183507978'
Error: failed to load object file
 try=7
2019-10-02 20:20:13.960 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-851860449): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-851860449'
Error: failed to load object file
 try=8
2019-10-02 20:20:14.016 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-524674508): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-524674508'
Error: failed to load object file
 try=9
2019-10-02 20:20:14.016 [PANIC][25423] int_dataplane.go 784: Failed to wipe the XDP state after 10 tries
panic: (*logrus.Entry) (0x19b44a0,0xc0006637c0)

goroutine 1 [running]:
github.com/sirupsen/logrus.Entry.log(0xc0000be050, 0xc00035fa10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7f6700000000, ...)
	/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180627202928-fc9bbf2f57995271c5cd6911ede7a2ebc5ea7c6f/entry.go:112 +0x2d2
github.com/sirupsen/logrus.(*Entry).Panic(0xc000663180, 0xc0003d2290, 0x1, 0x1)
	/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180627202928-fc9bbf2f57995271c5cd6911ede7a2ebc5ea7c6f/entry.go:182 +0x103
github.com/sirupsen/logrus.(*Entry).Panicf(0xc000663180, 0x1a336bb, 0x2b, 0xc0003d2340, 0x1, 0x1)
	/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180627202928-fc9bbf2f57995271c5cd6911ede7a2ebc5ea7c6f/entry.go:230 +0xd4
github.com/sirupsen/logrus.(*Logger).Panicf(0xc0000be050, 0x1a336bb, 0x2b, 0xc0003d2340, 0x1, 0x1)
	/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180627202928-fc9bbf2f57995271c5cd6911ede7a2ebc5ea7c6f/logger.go:173 +0x86
github.com/sirupsen/logrus.Panicf(...)
	/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180627202928-fc9bbf2f57995271c5cd6911ede7a2ebc5ea7c6f/exported.go:145
github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).shutdownXDPCompletely(0xc00031f200)
	/go/pkg/mod/github.com/projectcalico/felix@v0.0.0-20190910213021-a2d8a80b2ace/dataplane/linux/int_dataplane.go:784 +0x2cd
github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).doStaticDataplaneConfig(0xc00031f200)
	/go/pkg/mod/github.com/projectcalico/felix@v0.0.0-20190910213021-a2d8a80b2ace/dataplane/linux/int_dataplane.go:729 +0xbaa
github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).Start(0xc00031f200)
	/go/pkg/mod/github.com/projectcalico/felix@v0.0.0-20190910213021-a2d8a80b2ace/dataplane/linux/int_dataplane.go:592 +0x2f
github.com/projectcalico/felix/dataplane.StartDataplaneDriver(0xc0005ccf00, 0xc000540000, 0xc000607100, 0x1, 0xc00039d7d8, 0x0)
	/go/pkg/mod/github.com/projectcalico/felix@v0.0.0-20190910213021-a2d8a80b2ace/dataplane/driver.go:186 +0xf09
github.com/projectcalico/felix/daemon.Run(0x1a05d48, 0x15, 0x1cbbc38, 0x6, 0x1d106e0, 0x28, 0x1ce87a0, 0x18)
	/go/pkg/mod/github.com/projectcalico/felix@v0.0.0-20190910213021-a2d8a80b2ace/daemon/daemon.go:305 +0x1759
main.main()
	/go/src/github.com/projectcalico/node/cmd/calico-node/main.go:100 +0x405

Steps to Reproduce (for bugs)

(following https://docs.projectcalico.org/v3.9/getting-started/kubernetes/ )

  1. Install cri-o from Ubuntu PPA: https://launchpad.net/~projectatomic/+archive/ubuntu/ppa
  2. kubeadm init --pod-network-cidr=10.100.0.1/24
  3. kubectl apply -f https://docs.projectcalico.org/v3.9/manifests/calico.yaml

Context

Trying to install Calico on a completely fresh kube install.

Your Environment

  • Calico version 3.9.1
  • Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes 1.16.0, cri-o 1.13
  • Operating System and version: Ubunto 19.04
@caseydavenport

This comment has been minimized.

Copy link
Member

@caseydavenport caseydavenport commented Oct 2, 2019

@thommay could you share the calico/node logs? That should show why Felix isn't becoming ready.

@thommay

This comment has been minimized.

Copy link
Author

@thommay thommay commented Oct 3, 2019

@caseydavenport they're in the "click for kubectl logs output" bit.

@fasaxc

This comment has been minimized.

Copy link
Member

@fasaxc fasaxc commented Oct 3, 2019

I had a go at setting up a CRIO system but went down a rabbit-hole and failed to find a working set of instructions. Can you share the exact steps you used?

@thommay

This comment has been minimized.

Copy link
Author

@thommay thommay commented Oct 4, 2019

@fasaxc sure:

sudo add-apt-repository ppa:projectatomic/ppa
sudo apt install cri-o-1.13
sudoedit /etc/crio/crio.conf 
 # go to line 258 and uncomment the registries stanza
sudo systemctl start crio 

follow https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm

At https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-control-plane-node
the cgroup driver is systemd.

kubeadm should now automatically use cri-o.

@caseydavenport

This comment has been minimized.

Copy link
Member

@caseydavenport caseydavenport commented Oct 7, 2019

2019-10-02 20:20:13.436 [WARNING][25423] int_dataplane.go 781: failed to wipe the XDP state error=failed to load BPF program (/tmp/felix-bpf-020661529): stat /sys/fs/bpf/calico/xdp/prefilter_v1_calico_tmp_A: no such file or directory
libbpf: Error in bpf_object__probe_name():Operation not permitted(1). Couldn't load basic 'r0 = 0' BPF program.
libbpf: failed to load object '/tmp/felix-bpf-020661529'
Error: failed to load object file
 try=0

Looks like we're seeing the same thing as in this issue: #2904

@tmjd tmjd added kind/bug priority/P2 and removed kind/support labels Oct 8, 2019
@tmjd

This comment has been minimized.

Copy link
Member

@tmjd tmjd commented Nov 27, 2019

The fix for this should be in v3.11. projectcalico/felix#2165

@tmjd tmjd closed this Nov 27, 2019
@54853315

This comment has been minimized.

Copy link

@54853315 54853315 commented Dec 11, 2019

这个问题其实在安装完kubeadm init时就已经提示过:

[init] Using Kubernetes version: v1.16.3
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster

大家容易忽略。

我把解决办法写出来了,创建一个文件,kubele会去读的:
vim /etc/default/kubele
写入:
Environment=KUBELET_EXTRA_ARGS=--cgroup-driver=systemd

然后删除calico-node-[name]那个pod容器,等它几秒重新创建完,ok!问题解决。

image

Guys, just follow the code zone ,It will finish your problem.

Have a nice day .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.