Crashloopbackoff on v.0.6.1 with *-counter.json

[Hashfyre]

Env:

• AWS + Kops
• Ubuntu-16.04 hosts
• Deployed with
  helm upgrade --install node-problem-detector -f node-problem-detector/helm/values/dev/values.yaml --version '1.0' --namespace=kube-system stable/node-problem-detector
• values:

---
settings:
  log_monitors:
    - /config/abrt-adaptor.json
    - /config/docker-monitor.json
    - /config/docker-monitor-filelog.json
    - /config/docker-monitor-counter.json
    - /config/kernel-monitor.json
    - /config/kernel-monitor-filelog.json
    - /config/kernel-monitor-counter.json
    - /config/network-problem-monitor.json
    - /config/systemd-monitor-counter.json

image:
  repository: k8s.gcr.io/node-problem-detector
  tag: v0.6.1
  pullPolicy: IfNotPresent

nameOverride: "node-problem-detector"
fullnameOverride: "node-problem-detector"

rbac:
  create: true

resources: {}

annotations:
  sidecar.istio.io/inject: "false"

tolerations:
  - effect: NoSchedule
    operator: Exists
  - key: CriticalAddonsOnly
    operator: Exists

serviceAccount:
  # Specifies whether a ServiceAccount should be created
  create: true
  # The name of the ServiceAccount to use.
  # If not set and create is true, a name is generated using the fullname template
  name: "node-problem-detector"

affinity: {}

Logs:

I1221 12:13:10.771074       1 log_monitor.go:64] Finish parsing log monitor config file: {WatcherConfig:{Plugin:journald PluginConfig:map[source:abrt-notification] LogPath:/var/log/journal Lookback:5m Delay:} BufferSize:10 Source:abrt-adaptor DefaultConditions:[] Rules:[{Type:temporary Condition: Reason:CCPPCrash Pattern:Process \d+ \(\S+\) crashed in .*} {Type:temporary Condition: Reason:UncaughtException Pattern:Process \d+ \(\S+\) of user \d+ encountered an uncaught \S+ exception} {Type:temporary Condition: Reason:XorgCrash Pattern:Display server \S+ crash in \S+} {Type:temporary Condition: Reason:VMcore Pattern:System encountered a fatal error in \S+} {Type:temporary Condition: Reason:Kerneloops Pattern:System encountered a non-fatal error in \S+}]}
I1221 12:13:10.771124       1 log_watchers.go:40] Use log watcher of plugin "journald"
I1221 12:13:10.771202       1 log_monitor.go:64] Finish parsing log monitor config file: {WatcherConfig:{Plugin:journald PluginConfig:map[source:dockerd] LogPath:/var/log/journal Lookback:5m Delay:} BufferSize:10 Source:docker-monitor DefaultConditions:[] Rules:[{Type:temporary Condition: Reason:CorruptDockerImage Pattern:Error trying v2 registry: failed to register layer: rename /var/lib/docker/image/(.+) /var/lib/docker/image/(.+): directory not empty.*}]}
I1221 12:13:10.771215       1 log_watchers.go:40] Use log watcher of plugin "journald"
I1221 12:13:10.771281       1 log_monitor.go:64] Finish parsing log monitor config file: {WatcherConfig:{Plugin:filelog PluginConfig:map[message:msg="([^
]*)" timestampFormat:2006-01-02T15:04:05.999999999-07:00 timestamp:^time="(\S*)"] LogPath:/var/log/docker.log Lookback:5m Delay:} BufferSize:10 Source:docker-monitor DefaultConditions:[] Rules:[{Type:temporary Condition: Reason:CorruptDockerImage Pattern:Error trying v2 registry: failed to register layer: rename /var/lib/docker/image/(.+) /var/lib/docker/image/(.+): directory not empty.*}]}
I1221 12:13:10.771294       1 log_watchers.go:40] Use log watcher of plugin "filelog"
F1221 12:13:10.771364       1 log_monitor.go:56] Failed to unmarshal configuration file "/config/docker-monitor-counter.json": json: cannot unmarshal number into Go struct field MonitorConfig.pluginConfig of type string

[Hashfyre]

most monitors with *-counter suffix fails with the same unmarshalling error.

F1221 12:26:33.696942       1 log_monitor.go:56] Failed to unmarshal configuration file "/config/systemd-monitor-counter.json": json: cannot unmarshal number into Go struct field MonitorConfig.pluginConfig of type string

Leads me to suspect this has something to do with the stable helm chart not configuring something correctly.

[Hashfyre]

Could the issue be with non-quoted numerics in the said JSONs?

https://github.com/kubernetes/node-problem-detector/blob/master/config/systemd-monitor-counter.json#L6

[Hashfyre]

To back that up pluginConfig is a map of strings

node-problem-detector/pkg/systemlogmonitor/logwatchers/types/log_watcher.go

Line 38 in e807235

PluginConfig map[string]string `json:"pluginConfig,omitempty"`

[andyxning]

Actually all *-counter plugins are custom plugins. I suspect that you have configured NPD's command line argument wrongly. Check your --custom-plugin-monitors and --system-log-monitors settings.

[Hashfyre]

@andyxning I'm using the official helm-chart, I'll take your point into account and check my config.

The chart doesn't yet seem to support --custom-plugin-monitors flag. I'll fork, test and raise a PR may be.

[Hashfyre]

Tried running

   spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - exec /node-problem-detector --logtostderr --system-log-monitors=/config/abrt-adaptor.json,/config/docker-monitor.json,/config/docker-monitor-filelog.json,/config/kernel-monitor.json,/config/kernel-monitor-filelog.json
          --custom-plugin-monitors=/config/docker-monitor-counter.json

with --custom-plugin-monitors flag.
Fails with:

F0107 11:18:40.816070       1 custom_plugin_monitor.go:64] Failed to validate custom plugin config {Plugin:custom PluginGlobalConfig:{InvokeIntervalString:0xc0002933c0 TimeoutString:0xc0002933d0 InvokeInterval:5m0s Timeout:1m0s MaxOutputLength:0xc000133728 Concurrency:0xc000133738} Source:docker-monitor DefaultConditions:[{Type:CorruptDockerOverlay2 Status: Transition:0001-01-01 00:00:00 +0000 UTC Reason:NoCorruptDockerOverlay2 Message:docker overlay2 is functioning properly}] Rules:[0xc00038d2d0]}: rule path "/home/kubernetes/bin/log-counter" does not exist. Rule: &{Type:permanent Condition:CorruptDockerOverlay2 Reason:CorruptDockerOverlay2 Path:/home/kubernetes/bin/log-counter Args:[--journald-source=dockerd --log-path=/var/log/journal --lookback=5m --count=10 --pattern=returned error: readlink /var/lib/docker/overlay2.*: invalid argument.*] TimeoutString:0xc000293420 Timeout:1m0s}

[dashpole]

/cc @dashpole

[dashpole]

I don't think NPD has an official helm chart... Do we know who owns it?

[andyxning]

I don't think NPD has an official helm chart... Do we know who owns it?

Actually we do have an official helm chart in https://github.com/helm/charts/tree/master/stable/node-problem-detector. It is referenced in #217. It is owned by @max-rocket-internet

[andyxning]

@Hashfyre Has you resolved this Problem?