mountstats error: invalid NFS per-operations stats

[weikinhuang]

Host operating system: output of uname -a

Linux n001 5.3.0-18-generic #19-Ubuntu SMP Tue Oct 8 20:14:06 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

v0.18.1

node_exporter command line flags

node_exporter --path.procfs=/host/proc --path.sysfs=/host/sys --web.listen-address=0.0.0.0:9100 --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/rkt/pods/.+)($|/) --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$ --collector.mountstats

Are you running node_exporter in Docker?

yes

What did you do that produced an error?

Just trying to get stats on my nfs volume mounts (k8s).

What did you expect to see?

No errors, and stats in grafana.

What did you see instead?

ERROR: mountstats collector failed after 0.003237s: failed to parse mountstats: invalid NFS per-operations stats: [NULL: 1 1 0 44 24 0 0 0 0]

I'm using the latest prometheus-operator chart here and noticed recently that my nfs volume stats are missing. I changed my base OS from container-linux to ubuntu 19.10. I'm also running a single instance of ubuntu 18.04 and that is also not reporting stats.

Example output from /proc/self/mountstats:

device fs.example.com:/volume4/apps/home-automation/node-red-data mounted on /var/lib/kubelet/pods/1c2215a7-0d92-4df5-83ce-a807bcc2f8c8/volumes/kubernetes.io~nfs/home-automation--node-red-data--pv0001 with fstype nfs4 statvers=1.1
        opts:   rw,vers=4.1,rsize=131072,wsize=131072,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.191,local_lock=none
        age:    83520
        impl_id:        name='',domain='',date='0,0'
        caps:   caps=0x3fff7,wtmult=512,dtsize=32768,bsize=0,namlen=255
        nfsv4:  bm0=0xfdffafff,bm1=0xf9be3e,bm2=0x800,acl=0x0,sessions,pnfs=not configured,lease_time=90,lease_expired=0
        sec:    flavor=1,pseudoflavor=1
        events: 52472 472680 16671 57552 2104 9565 749555 9568641 168 24103 1 267134 3350 20097 116581 18214 43757 111141 0 28 9563845 34 0 0 0 0 0 
        bytes:  2021340783 39056395530 0 0 1788561151 39087991255 442605 9557343 
        RPC iostats version: 1.1  p/v: 100003/4 (nfs)
        xprt:   tcp 940 0 2 0 1 938505 938504 0 12756069 0 32 254729 10823602
        per-op statistics
                NULL: 1 1 0 44 24 0 0 0 0
                READ: 34096 34096 0 7103096 1792122744 2272 464840 467945 0
               WRITE: 322308 322308 0 39161277084 56725504 401718334 10139998 411864389 0
              COMMIT: 12541 12541 0 2709896 1304264 342 7179 7819 0
                OPEN: 12637 12637 0 3923256 4659940 871 57185 58251 394
        OPEN_CONFIRM: 0 0 0 0 0 0 0 0 0
         OPEN_NOATTR: 98741 98741 0 25656212 31630800 3366 77710 82693 0
        OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0 0
               CLOSE: 87075 87075 0 18778608 15308496 2026 49131 52399 116
             SETATTR: 24576 24576 0 5825876 6522260 643 34384 35650 0
              FSINFO: 1 1 0 168 152 0 0 0 0
               RENEW: 0 0 0 0 0 0 0 0 0
         SETCLIENTID: 0 0 0 0 0 0 0 0 0
        SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0 0
                LOCK: 22512 22512 0 5417628 2521312 1088 17407 18794 2
               LOCKT: 0 0 0 0 0 0 0 0 0
               LOCKU: 21247 21247 0 4589352 2379664 315 8409 9003 0
              ACCESS: 1466 1466 0 298160 246288 22 1394 1492 0
             GETATTR: 52480 52480 0 10015464 12694076 2930 30069 34502 0
              LOOKUP: 11727 11727 0 2518200 2886376 272 16935 17662 3546
         LOOKUP_ROOT: 0 0 0 0 0 0 0 0 0
              REMOVE: 833 833 0 172236 95268 15 4566 4617 68
              RENAME: 11431 11431 0 3150708 1737512 211 52649 53091 0
                LINK: 1 1 0 288 292 0 0 0 0
             SYMLINK: 0 0 0 0 0 0 0 0 0
              CREATE: 77 77 0 18292 23496 0 363 371 11
            PATHCONF: 1 1 0 164 116 0 0 0 0
              STATFS: 7420 7420 0 1394960 1187200 144 4672 4975 0
            READLINK: 4 4 0 704 488 0 1 1 0
             READDIR: 1353 1353 0 304024 2902928 11 4326 4411 0
         SERVER_CAPS: 9 9 0 1548 1476 0 3 3 0
         DELEGRETURN: 232 232 0 48896 37120 811 300 1115 0
              GETACL: 0 0 0 0 0 0 0 0 0
              SETACL: 0 0 0 0 0 0 0 0 0
        FS_LOCATIONS: 0 0 0 0 0 0 0 0 0
        RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0 0
             SECINFO: 0 0 0 0 0 0 0 0 0
        FSID_PRESENT: 0 0 0 0 0 0 0 0 0
         EXCHANGE_ID: 2 2 0 464 200 0 0 0 0
        CREATE_SESSION: 1 1 0 192 124 0 0 0 0
        DESTROY_SESSION: 0 0 0 0 0 0 0 0 0
            SEQUENCE: 0 0 0 0 0 0 0 0 0
        GET_LEASE_TIME: 0 0 0 0 0 0 0 0 0
        RECLAIM_COMPLETE: 1 1 0 124 88 0 81 81 0
           LAYOUTGET: 0 0 0 0 0 0 0 0 0
        GETDEVICEINFO: 0 0 0 0 0 0 0 0 0
        LAYOUTCOMMIT: 0 0 0 0 0 0 0 0 0
        LAYOUTRETURN: 0 0 0 0 0 0 0 0 0
        SECINFO_NO_NAME: 0 0 0 0 0 0 0 0 0
        TEST_STATEID: 0 0 0 0 0 0 0 0 0
        FREE_STATEID: 10413 10413 0 1416168 916344 147 3518 3871 10413
        GETDEVICELIST: 0 0 0 0 0 0 0 0 0
        BIND_CONN_TO_SESSION: 0 0 0 0 0 0 0 0 0
        DESTROY_CLIENTID: 0 0 0 0 0 0 0 0 0
                SEEK: 0 0 0 0 0 0 0 0 0
            ALLOCATE: 0 0 0 0 0 0 0 0 0
          DEALLOCATE: 0 0 0 0 0 0 0 0 0
         LAYOUTSTATS: 0 0 0 0 0 0 0 0 0
               CLONE: 0 0 0 0 0 0 0 0 0
                COPY: 0 0 0 0 0 0 0 0 0
        OFFLOAD_CANCEL: 0 0 0 0 0 0 0 0 0
             LOOKUPP: 0 0 0 0 0 0 0 0 0
         LAYOUTERROR: 0 0 0 0 0 0 0 0 0

Crosspost from prometheus/node_exporter#1642

[pgier]

Looks like there was a new field added in the 5.3 kernel (torvalds/linux@a332518) and we are only handling a specific number of fields. So the first thing to fix is that we should be checking that the number of fields is >= numFields instead of = numFields (

procfs/mountstats.go

Line 511 in 8068798

if len(ss) != numFields {

). And then we probably want to add a new field to the NFSOperationStats struct which is collected only if we find the extra field.

@weikinhuang do you want to submit a PR for this?

[weikinhuang]

@pgier Sure, I can give it a try

[weikinhuang]

Unfortunately, I'm not too familiar with golang, and creating dynamic length arrays

[pgier]

@weikinhuang Ok, no problem, I created PR #276 to address this and added your mountstats example to the tests.

[weikinhuang]

thank you so much, also learned a little bit of go, i thought you can only have static arrays.