-
Notifications
You must be signed in to change notification settings - Fork 37
Utilities for system statistics
sergev edited this page Nov 17, 2014
·
17 revisions
Known bugs:
-
/var/run/dev.db: No such file or directory
-
"vmstat 1" crashes when started right after reboot. Kernel fault at user PC address 00400ed0, badvaddr=00423006/ Looks like division by zero.
-
Same for "vmstat -i".
-
systat crashes. PID -1069483309 (kernel fault at pc=9d00de5c, badvaddr=00000005
-
After "du", see "vmstat -m" -- too much memory allocated to vnodes and FFS nodes.
-
After "ls -lR", see "vmstst -s" -- 57 pages wired down. It's almost half of the physical memory. Need to reduce and limit it somehow.
# ps alx
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT TIME COMMAND
0 0 0 0 -18 0 0 0 schedu DLs ?? 0:00.00 (swapper)
0 1 0 0 10 0 128 0 wait Ss ?? 0:00.00 /sbin/init -s
0 2 0 0 -18 0 0 0 - RL ?? 0:00.00 (pagedaemon)
ps: warning: /var/run/dev.db: No such file or directory
0 3 1 1 10 0 332 0 wait Ss ?? 0:00.05 -sh (sh)
0 5 3 0 28 0 208 0 - R+ ?? 0:00.00 (ps)
# ps aux
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
0 2 2.1 0.0 0 0 ?? RL 10:57PM 0:00.05 (pagedaemon)
ps: warning: /var/run/dev.db: No such file or directory
0 3 2.1 0.0 336 0 ?? Ss 10:57PM 0:00.05 -sh (sh)
0 0 0.0 0.0 0 0 ?? DLs 10:57PM 0:00.00 (swapper)
0 1 0.0 0.0 128 0 ?? Ss 10:57PM 0:00.00 /sbin/init -s
0 7 0.0 0.0 208 0 ?? R+ 10:57PM 0:00.00 (ps)
# uptime
1:28PM up 0 user, load averages: 0.08, 0.02, 0.01
# w
1:15AM up 0 user, load averages: 0.08, 0.02, 0.01
USER TTY FROM LOGIN@ IDLE WHAT
##vmstat # vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr s0 ?1 ?2 ?3 in sy cs us sy id 0 0 0 12956 16 294 38 120 26 0 144 0 0 0 0 259 148 405 2 3 95
# vmstat 1 --- division by zero?
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr s0 ?1 ?2 ?3 in sy cs us sy id
kernel fault at pc=00400ed0, badvaddr=00423006
panic: trap
syncing disks... done
# vmstat 1
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr s0 ?1 ?2 ?3 in sy cs us sy id
0 0 0 12956 16 861 127 363 78 0 460 0 0 0 0 649 347 1348 2 12 86
0 0 0 12960 20 11 1 6 3 0 11 0 0 0 0 102 19 29 0 0 100
0 0 0 12960 20 4 1 1 1 0 1 0 0 0 0 102 18 5 0 0 100
0 0 0 12960 20 3 1 1 1 0 1 0 0 0 0 102 18 5 0 0 100
0 0 0 12960 20 4 1 1 1 0 1 0 0 0 0 102 18 5 0 0 100
0 0 0 12960 20 4 1 1 1 0 1 0 0 0 0 102 18 5 0 0 100
0 0 0 12960 20 5 2 1 1 0 1 0 0 0 0 102 18 5 0 0 100
# vmstat -i
interrupt total rate
kernel fault at pc=00401544, badvaddr=004099b0
# vmstat -s
319 cpu context switches
142 device interrupts
0 software interrupts
393 traps
155 system calls
295 total faults taken
0 swap ins
0 swap outs
0 pages swapped in
0 pages swapped out
110 page ins
20 page outs
110 pages paged in
20 pages paged out
28 pages reactivated
0 intransit blocking page faults
6284 zero fill pages created
59 zero fill page faults
113 pages examined by the clock daemon
75 revolutions of the clock hand
8 VM object cache lookups
8 VM object hits
0 total VM faults taken
31 copy-on-write faults
93 pages freed by daemon
19 pages freed by exiting processes
4 pages free
37 pages wired down
32 pages active
16 pages inactive
4096 bytes per page
16 target inactive pages
5 target free pages
4 minimum free pages
45 total name lookups
cache hits (42% pos + 2% neg) system 0% per-process
deletions 0%, falsehits 0%, toolong 0%
# vmstat -m
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 27 229 209 1280 0
32 36 92 92 640 0
64 147 45 302 320 0
128 249 7 254 160 0
256 256 16 2299 80 0
512 10 6 13 40 0
1024 8 4 2410 20 0
4096 1 1 6 5 0
8192 1 0 1 5 0
Memory usage type by bucket size
Size Type(s)
16 vnodes, namecache, VM objhash, VM pvmap, VM pgdata, proc
32 mbuf, devbuf, pgrp, session, VM pager, VM pgdata, proc, subproc,
LFS segment
64 routetbl, ifaddr, namecache, VM mapent, VM object, VM pgdata, file
128 routetbl, zombie, ifaddr, cred, vnodes, VM pgdata, file desc, ttys
256 devbuf, vnodes, VM map, FFS node
512 mount, namecache, UFS mount, VM pgdata, proc
1024 namei, ttys
4096 UFS mount, exec
8192 UFS mount
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
mbuf 1 1K 1K 96K 1 0 0 32
devbuf 2 1K 1K 96K 2 0 0 32,256
routetbl 3 1K 1K 96K 3 0 0 64,128
zombie 0 0K 1K 96K 2 0 0 128
ifaddr 2 1K 1K 96K 2 0 0 64,128
namei 5 5K 6K 96K 2407 0 0 1024
cred 2 1K 1K 96K 2 0 0 128
pgrp 3 1K 1K 96K 5 0 0 32
session 2 1K 1K 96K 2 0 0 32
mount 1 1K 1K 96K 1 0 0 512
vnodes 255 33K 35K 96K 393 0 0 16,128,256
namecache 69 5K 5K 96K 69 0 0 16,64,512
UFS mount 4 13K 13K 96K 4 0 0 512,4096,8192
VM map 8 2K 3K 96K 20 0 0 256
VM mapent 49 4K 4K 96K 94 0 0 64
VM object 25 2K 2K 96K 58 0 0 64
VM objhash 5 1K 1K 96K 5 0 0 16
VM pvmap 0 0K 1K 96K 41 0 0 16
VM pager 14 1K 1K 96K 20 0 0 32
VM pgdata 23 2K 3K 96K 35 0 0 16,32,64,128,512
file 4 1K 1K 96K 80 0 0 64
file desc 4 1K 1K 96K 6 0 0 128
proc 8 3K 3K 96K 10 0 0 16,32,512
subproc 4 1K 1K 96K 6 0 0 32
LFS segment 0 0K 1K 96K 40 0 0 32
FFS node 237 60K 60K 96K 2268 0 0 256
ttys 5 4K 4K 96K 5 0 0 128,1024
exec 0 0K 4K 96K 5 0 0 4096
Memory Totals: In Use Free Requests
131K 26K 5586
# iostat
tty sd0 dk1 dk2 dk3 cpu
tin tout sps tps msps sps tps msps sps tps msps sps tps msps us ni sy in id
2 142 0 0 0.0 0 0 0.0 0 0 0.0 0 0 0.0 0 0 1 0 98
# systat
PID -1069483309 (kernel fault at pc=9d00de5c, badvaddr=00000005
panic: trap
syncing disks... done
# dmesg
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
4.4BSD-Lite build 19 compiled 2014-11-16
vak@kucha:Tilden/sys/compile/WIFIRE.pic32
cpu: PIC32MZ2048ECG100 rev A4, 200 MHz
oscillator: system PLL div 1:6 mult x50
real mem = 512 kbytes
avail mem = 336 kbytes
using 16 buffers containing 49152 bytes of memory
spi1 at pins sdi=F1/sdo=C1/sck=D1
spi2 at pins sdi=F0/sdo=D11/sck=G6
spi3 at pins sdi=B10/sdo=C4/sck=B14
spi4 at pins sdi=F5/sdo=G0/sck=D10
uart1 at pins rx=D14/tx=D15, interrupts 112/113/114
uart4 at pins rx=F2/tx=F8, interrupts 170/171/172, console
sd0 at port spi3, pin cs=C3
sd0: type I, size 679936 kbytes, speed 12 Mbit/sec
sd0a: partition type b7, sector 2, size 204800 kbytes
sd0b: partition type b8, sector 409602, size 32768 kbytes
sd0c: partition type b7, sector 475138, size 102400 kbytes
bpf: lo0 attached
WARNING: preposterous clock chip time -- CHECK AND RESET THE DATE!
# netstat -i
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
lo0* 1536 <Link> 0 0 0 0 0
# netstat -rn
Routing tables
Internet:
Destination Gateway Flags Refs Use Interface
# netstat -m
0 mbufs in use:
0/4 mapped pages in use
4 Kbytes allocated to network (0% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
# netstat -s
ip:
0 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets reassembled ok
0 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded
0 packets not forwardable
0 redirects sent
0 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
icmp:
0 calls to icmp_error
0 errors not generated 'cuz old message was icmp
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
0 message responses generated
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
tcp:
0 packets sent
0 data packets (0 bytes)
0 data packets (0 bytes) retransmitted
0 ack-only packets (0 delayed)
0 URG only packets
0 window probe packets
0 window update packets
0 control packets
0 packets received
0 acks (for 0 bytes)
0 duplicate acks
0 acks for unsent data
0 packets (0 bytes) received in-sequence
0 completely duplicate packets (0 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
0 out-of-order packets (0 bytes)
0 packets (0 bytes) of data after window
0 window probes
0 window update packets
0 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 connection requests
0 connection accepts
0 bad connection attempts
0 connections established (including accepts)
0 connections closed (including 0 drops)
0 embryonic connections dropped
0 segments updated rtt (of 0 attempts)
0 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections timed out in persist
0 keepalive timeouts
0 keepalive probes sent
0 connections dropped by keepalive
0 correct ACK header predictions
0 correct data packet header predictions
0 PCB cache misses
udp:
0 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
0 dropped due to no socket
0 broadcast/multicast datagrams dropped due to no socket
0 dropped due to full socket buffers
0 delivered
0 datagrams output
# fstat
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W
0 fstat 6 wd / 2 drwxr-xr-x 512 r
fstat: warning: /var/run/dev.db: No such file or directory
0 fstat 6 0 / 12355 crw-rw-r-- 0,0 rw
0 fstat 6 1 / 12355 crw-rw-r-- 0,0 rw
0 fstat 6 2 / 12355 crw-rw-r-- 0,0 rw
0 fstat 6 3 / 12357 crw-r----- 2,0 r
0 fstat 6 4 / 12358 crw-r----- 2,1 r
0 fstat 6 5 / 12361 crw-r----- 3,0 r
0 sh 3 wd / 2 drwxr-xr-x 512 r
0 sh 3 0 / 12355 crw-rw-r-- 0,0 rw
0 sh 3 1 / 12355 crw-rw-r-- 0,0 rw
0 sh 3 2 / 12355 crw-rw-r-- 0,0 rw
0 pagedaemon 2 wd / 2 drwxr-xr-x 512 r
0 init 1 wd / 2 drwxr-xr-x 512 r
0 swapper 0 wd / 2 drwxr-xr-x 512 r
# pstat -T
4/242 files
24 vnodes
0M/32M swap space
# pstat -f
4/242 open files
LOC TYPE FLG CNT MSG DATA OFFSET
c0402600 inode RW 6 0 c0405880 191
c040e1c0 inode R 1 0 c0405000 0
c040e280 inode R 1 0 c0405080 2684355188
c040e200 inode R 1 0 c0405100 0
# pstat -s
Device 1K-blocks Used Avail Capacity Type
pstat: warning: /var/run/dev.db: No such file or directory
/dev/?? 32768 212 32556 1% Interleaved
# pstat -t
5 uart lines
LINE RAW CAN OUT HWT LWT COL STATE SESS PGID DISC
0 0 0 0 0 0 0 - 0 0 term
1 0 0 0 0 0 0 - 0 0 term
2 0 0 0 0 0 0 - 0 0 term
pstat: warning: /var/run/dev.db: No such file or directory
3 0 0 0 2080 256 0 OC c0403e60 8 term
4 0 0 0 0 0 0 - 0 0 term
4 pty lines
LINE RAW CAN OUT HWT LWT COL STATE SESS PGID DISC
0 0 0 0 0 0 0 - 0 0 term
1 0 0 0 0 0 0 - 0 0 term
2 0 0 0 0 0 0 - 0 0 term
3 0 0 0 0 0 0 - 0 0 term
# pstat -v
19 active vnodes
*** MOUNT ufs root_device on / (rdonly,local,rootfs)
ADDR TYP VFLAG USE HOLD FILEID IFLAG RDEV|SZ
c0411c80 dir - 0 1 12635 - 512
c0405300 dir - 0 0 353 - 512
c0405380 dir - 0 1 6179 - 512
pstat: warning: /var/run/dev.db: No such file or directory
c0405000 chr - 1 0 12361 - 3,0
c0405080 chr - 1 0 12358 A 2,1
c0405100 chr - 1 0 12357 - 2,0
c0405280 reg T 1 7 18596 UACM 146046
c0405400 reg T 1 0 6206 AM 93538
c0405480 dir - 0 0 18530 - 1024
c0405500 dir - 0 0 24705 - 512
c0405580 reg - 0 0 72 - 285
c0405600 dir - 0 0 18528 - 1024
c0405680 reg T 1 2 6204 AM 369200
c0405700 dir - 0 0 6176 - 512
c0405880 chr 2 0 12355 UCM 0,0
c0405900 dir - 0 0 12352 - 1536
c0405a00 reg T 1 0 6221 AM 173439
c0405a80 dir - 0 0 6177 - 1024
c0405c80 dir R 6 0 2 - 512
# kgmon
kgmon: profiling not defined in kernel.