Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

qemu memory locking #9

Closed
estibi opened this Issue · 2 comments

2 participants

@estibi

illumos-gate r13632 non debug

illumos-kvm 76be9d4d13f5c5d32d915820bbf956b73fd89aab
illumos-kvm-cmd 4f48eea

prtconf | grep Mem

Memory size: 196599 Megabytes

swap -lh

swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 179,2 4K 190G 190G

swap -sh

total: 141G allocated + 147M reserved = 141G used, 49G available

pgrep kvm | wc -l

  15

[ Mar 5 15:04:44 Executing start method ("/lib/svc/method/kvm_start"). ]
qemu-kvm: -net vnic,vlan=0,name=vm16,ifname=vm16,macaddr=2:8:20:44:4c:1c: vnic dhcp disabled

qemu_mlock: have only locked 8455036928 of 12884901888 bytes; still trying...
qemu_mlock: have only locked 8461008896 of 12884901888 bytes; still trying...
[ Mar 5 15:05:44 Method or service exit timed out. Killing contract 482. ]

This is a log from SMF, because I manage KVM instances by SMF services.
As you can see I'm unable to lock more memory than let say ~150GB.

kstat -n arcstats

module: zfs instance: 0

name: arcstats class: misc
c 6441891072
c_max 6000000000
c_min 6441891072
crtime 206.570697503
data_size 4021757440
deleted 207972
demand_data_hits 786829
demand_data_misses 61181
demand_metadata_hits 562487
demand_metadata_misses 15329
evict_l2_cached 0
evict_l2_eligible 23425769984
evict_l2_ineligible 2239541248
evict_skip 3949640
hash_chain_max 2
hash_chains 799
hash_collisions 10992
hash_elements 79589
hash_elements_max 143756
hdr_size 16153208
hits 1422814
l2_abort_lowmem 0
l2_cksum_bad 0
l2_evict_lock_retry 0
l2_evict_reading 0
l2_feeds 0
l2_free_on_write 0
l2_hdr_size 0
l2_hits 0
l2_io_error 0
l2_misses 0
l2_read_bytes 0
l2_rw_clash 0
l2_size 0
l2_write_bytes 0
l2_writes_done 0
l2_writes_error 0
l2_writes_hdr_miss 0
l2_writes_sent 0
memory_throttle_count 1154
mfu_ghost_hits 28829
mfu_hits 1084501
misses 115318
mru_ghost_hits 17141
mru_hits 269702
mutex_miss 309
other_size 24711328
p 2174031184
prefetch_data_hits 65980
prefetch_data_misses 31926
prefetch_metadata_hits 7518
prefetch_metadata_misses 6882
recycle_miss 192179
size 4062621976
snaptime 7956.28236478

vmstat -p 1 5

 memory           page          executable      anonymous      filesystem 

swap free re mf fr de sr epi epo epf api apo apf fpi fpo fpf
95913840 43182152 477 1390 45 0 3961 0 0 1 0 42 42 0 0 3
51005840 17035264 22 42 0 0 0 0 0 0 0 0 0 0 0 0
51005800 17037432 0 6 0 0 0 0 0 0 0 0 0 0 0 0
51005800 17036860 0 0 0 0 0 0 0 0 0 0 0 0 0 0
51005808 17037192 0 0 0 0 0 0 0 0 0 0 0 0 0 0

::memstat
Page Summary Pages MB %Tot


Kernel 8094484 31619 16%
ZFS File Data 932706 3643 2%
Anon 37021649 144615 74%
Exec and libs 2768 10 0%
Page cache 10367 40 0%
Free (cachelist) 14232 55 0%
Free (freelist) 4251068 16605 8%

Total 50327274 196590
Physical 50327273 196590

::swapinfo
ADDR VNODE PAGES FREE NAME
ffffff34618185d8 ffffff3463d6dc00 49807359 49806844 /dev/zvol/dsk/rpool/swap

::kmastat
cache buf buf buf memory alloc alloc
name size in use total in use succeed fail


kmem_magazine_1 16 2098 5100 208896B 177097 0
kmem_magazine_3 32 24745 25544 1687552B 76846 0
kmem_magazine_7 64 75505 75727 7565312B 102629 0
kmem_magazine_15 128 8731 8760 1794048B 28690 0
kmem_magazine_31 256 784 67080 22896640B 69272 0
kmem_magazine_47 384 2158 90560 46366720B 94118 0
kmem_magazine_63 512 340 201348 137453568B 202149 0
kmem_magazine_95 768 7153 59192 60612608B 65170 0
kmem_magazine_143 1152 110188 110196 150454272B 111201 0
kmem_slab_cache 72 994843 1017087 101609472B 12480881 0
kmem_bufctl_cache 24 55663 77854 3842048B 328480 0
kmem_bufctl_audit_cache 192 42499101 42940728 9771401216B 54365562 0
kmem_va_4096 4096 836069 840544 3442868224B 857995 0
kmem_va_8192 8192 933 944 7733248B 5264 0
kmem_va_12288 12288 202 250 3276800B 2245 0
kmem_va_16384 16384 6713 8528 139722752B 60298 0
kmem_va_20480 20480 40 42 917504B 880 0
kmem_va_24576 24576 17 25 655360B 512 0
kmem_va_28672 28672 38 48 1572864B 629 0
kmem_va_32768 32768 64 128 4194304B 43677 0
kmem_alloc_8 8 18144 38784 1241088B 1987474 0
kmem_alloc_16 16 17299 22236 892928B 2407692 0
kmem_alloc_24 24 23045 29155 1404928B 3288063 0
kmem_alloc_32 32 16373 17374 974848B 1926962 0
kmem_alloc_40 40 73240 74624 4775936B 6254585 0
kmem_alloc_48 48 18494 26824 1961984B 742600 0
kmem_alloc_56 56 16956 17238 1384448B 160010 0
kmem_alloc_64 64 24808 30976 3964928B 762460934 0
kmem_alloc_80 80 12949 25545 2682880B 3676058 0
kmem_alloc_96 96 3063 3468 417792B 2627731 0
kmem_alloc_112 112 10051 12420 1695744B 3319010 0
kmem_alloc_128 128 2115 14973 2920448B 3144445 0
kmem_alloc_160 160 4778 137786 25653248B 2080815 0
kmem_alloc_192 192 9626 9872 2527232B 210946 0
kmem_alloc_224 224 1292 3968 1015808B 705704 0
kmem_alloc_256 256 1200 2496 851968B 8649227 0
kmem_alloc_320 320 7630 8340 3416064B 6873831 0
kmem_alloc_384 384 552 1359 618496B 1232836 0
kmem_alloc_448 448 184 256 131072B 18737 0
kmem_alloc_512 512 21880 24458 14311424B 307523 0
kmem_alloc_640 640 647 715 532480B 34906 0
kmem_alloc_768 768 180 225 204800B 12602 0
kmem_alloc_896 896 1646 1704 1744896B 11240 0
kmem_alloc_1152 1152 360 450 552960B 1426765 0
kmem_alloc_1344 1344 361 400 614400B 3276 0
kmem_alloc_1600 1600 136 182 319488B 151164 0
kmem_alloc_2048 2048 1071 1071 4386816B 45735 0
kmem_alloc_2688 2688 313 313 1282048B 20574 0
kmem_alloc_4096 4096 79438 79438 325378048B 692046 0
kmem_alloc_8192 8192 300 300 2457600B 269832 0
kmem_alloc_12288 12288 16 16 196608B 34125 0
kmem_alloc_16384 16384 55 55 901120B 13173 0
kmem_alloc_24576 24576 63 63 1548288B 11293 0
kmem_alloc_32768 32768 96 96 3145728B 6793 0
kmem_alloc_40960 40960 47 47 1925120B 11809 0
kmem_alloc_49152 49152 15 15 737280B 10797 0
kmem_alloc_57344 57344 0 0 0B 1263 0
kmem_alloc_65536 65536 42 42 2752512B 5648 0
kmem_alloc_73728 73728 6 6 442368B 10795 0
kmem_alloc_81920 81920 2 2 163840B 2685 0
kmem_alloc_90112 90112 6 6 540672B 2824 0
kmem_alloc_98304 98304 5 5 491520B 1194 0
kmem_alloc_106496 106496 0 0 0B 525 0
kmem_alloc_114688 114688 0 0 0B 548 0
kmem_alloc_122880 122880 1 1 122880B 563 0
kmem_alloc_131072 131072 8 8 1048576B 4644 0
streams_mblk 64 4869 8274 806912B 86314389 0
streams_dblk_16 128 235 903 176128B 41746441 0
streams_dblk_80 192 341 3696 946176B 1710434 0
streams_dblk_144 256 17 828 282624B 42121031 0
streams_dblk_208 320 172 460 188416B 107525 0
streams_dblk_272 384 5 63 28672B 129884 0
streams_dblk_336 448 0 80 40960B 8516 0
streams_dblk_528 640 0 88 65536B 90300 0
streams_dblk_1040 1152 0 110 135168B 155542 0
streams_dblk_1488 1600 0 35 61440B 2189 0
streams_dblk_1936 2048 0 0 0B 3145 0
streams_dblk_2576 2688 4 4 16384B 162134 0
streams_dblk_3856 3968 0 0 0B 513 0
streams_dblk_8192 112 0 168 32768B 4425 0
streams_dblk_12048 12160 0 0 0B 22 0
streams_dblk_16384 112 0 0 0B 0 0
streams_dblk_20240 20352 0 0 0B 0 0
streams_dblk_24576 112 0 0 0B 0 0
streams_dblk_28432 28544 0 0 0B 0 0
streams_dblk_32768 112 0 0 0B 0 0
streams_dblk_36624 36736 0 0 0B 0 0
streams_dblk_40960 112 0 21 4096B 1 0
streams_dblk_44816 44928 0 0 0B 0 0
streams_dblk_49152 112 0 0 0B 0 0
streams_dblk_53008 53120 0 0 0B 0 0
streams_dblk_57344 112 0 0 0B 0 0
streams_dblk_61200 61312 0 0 0B 0 0
streams_dblk_65536 112 0 0 0B 0 0
streams_dblk_69392 69504 0 0 0B 0 0
streams_dblk_73728 112 0 0 0B 0 0
streams_dblk_esb 112 4096 4116 802816B 166782 0
streams_fthdr 408 0 0 0B 0 0
streams_ftblk 376 0 0 0B 0 0
multidata 248 0 0 0B 0 0
multidata_pdslab 7112 0 0 0B 0 0
multidata_pattbl 32 0 0 0B 0 0
log_cons_cache 48 23 56 4096B 43 0
taskq_ent_cache 56 1504 1785 143360B 2426 0
taskq_cache 280 197 234 73728B 374 0
kmem_io_4P_128 128 0 0 0B 0 0
kmem_io_4P_256 256 0 0 0B 0 0
kmem_io_4P_512 512 0 0 0B 0 0
kmem_io_4P_1024 1024 0 0 0B 0 0
kmem_io_4P_2048 2048 0 0 0B 0 0
kmem_io_4P_4096 4096 0 0 0B 0 0
kmem_io_4G_128 128 4 288 73728B 4177421 0
kmem_io_4G_256 256 1 32 16384B 251 0
kmem_io_4G_512 512 9 32 32768B 211 0
kmem_io_4G_1024 1024 0 0 0B 0 0
kmem_io_4G_2048 2048 0 0 0B 0 0
kmem_io_4G_4096 4096 0 244 1998848B 2031845 0
kmem_io_2G_128 128 1 16 4096B 1 0
kmem_io_2G_256 256 0 0 0B 0 0
kmem_io_2G_512 512 0 0 0B 0 0
kmem_io_2G_1024 1024 0 0 0B 0 0
kmem_io_2G_2048 2048 0 0 0B 0 0
kmem_io_2G_4096 4096 10 12 98304B 67 0
kmem_io_16M_128 128 0 0 0B 0 0
kmem_io_16M_256 256 0 0 0B 0 0
kmem_io_16M_512 512 0 0 0B 0 0
kmem_io_16M_1024 1024 0 0 0B 0 0
kmem_io_16M_2048 2048 0 0 0B 0 0
kmem_io_16M_4096 4096 0 0 0B 0 0
id32_cache 32 0 0 0B 0 0
bp_map_4096 4096 0 0 0B 1 0
bp_map_8192 8192 0 0 0B 5 0
bp_map_12288 12288 0 0 0B 3 0
bp_map_16384 16384 0 0 0B 3 0
bp_map_20480 20480 0 0 0B 2 0
bp_map_24576 24576 0 0 0B 1 0
bp_map_28672 28672 0 0 0B 0 0
bp_map_32768 32768 0 0 0B 0 0
htable_t 72 26135 27921 2789376B 134121 0
hment_t 64 61061 68572 4530176B 5349672 0
hat_t 176 145 280 57344B 12599 0
HatHash 262144 22 22 5767168B 1717 0
HatVlpHash 4096 123 123 503808B 10754 0
zfs_file_data_4096 4096 196 416 1703936B 26297 0
zfs_file_data_8192 8192 20 64 524288B 82 0
zfs_file_data_12288 12288 59 100 1310720B 133 0
zfs_file_data_16384 16384 27 48 786432B 80 0
zfs_file_data_20480 20480 91 96 2097152B 258 0
zfs_file_data_24576 24576 22 25 655360B 90 0
zfs_file_data_28672 28672 53 76 2490368B 161 0
zfs_file_data_32768 32768 179 180 5898240B 675 0
segkp_4096 4096 139 144 589824B 7282 0
segkp_8192 8192 0 0 0B 0 0
segkp_12288 12288 0 0 0B 0 0
segkp_16384 16384 0 0 0B 0 0
segkp_20480 20480 0 0 0B 0 0
umem_np_4096 4096 15 32 131072B 341 0
umem_np_8192 8192 38 48 393216B 194 0
umem_np_12288 12288 0 0 0B 0 0
umem_np_16384 16384 0 0 0B 170 0
umem_np_20480 20480 0 0 0B 46 0
umem_np_24576 24576 0 0 0B 25 0
umem_np_28672 28672 0 0 0B 0 0
umem_np_32768 32768 0 0 0B 43 0
mod_hash_entries 24 2723 3230 155648B 677041 0
ipp_mod 304 0 0 0B 0 0
ipp_action 368 0 0 0B 0 0
ipp_packet 64 0 0 0B 0 0
seg_cache 96 8151 8330 1003520B 637528 0
seg_pcache 104 0 32 4096B 1 0
fnode_cache 176 20 36 8192B 68 0
pipe_cache 320 102 165 61440B 5303 0
snode_cache 152 769 920 163840B 51488 0
dv_node_cache 176 117 780 159744B 852 0
mac_impl_cache 13568 64 64 1048576B 64 0
mac_ring_cache 192 8 18 4096B 8 0
flow_entry_cache 27112 87 87 2494464B 108 0
flow_tab_cache 216 64 68 16384B 64 0
mac_soft_ring_cache 376 162 171 77824B 168 0
mac_srs_cache 3240 108 108 442368B 112 0
mac_bcast_grp_cache 80 23 39 4096B 41 0
mac_client_impl_cache 2096 64 64 262144B 67 0
mac_promisc_impl_cache 112 15 60 8192B 54 0
dls_link_cache 344 64 77 28672B 67 0
dls_devnet_cache 360 64 70 28672B 67 0
sdev_node_cache 256 1879 1904 557056B 84136 0
dev_info_node_cache 680 215 242 180224B 1174 0
ndi_fm_entry_cache 32 10280 11096 622592B 9987700 0
thread_cache 912 840 960 983040B 34083 0
lwp_cache 1760 1181 1305 2375680B 20200 0
turnstile_cache 64 2109 2300 204800B 34422 0
tslabel_cache 48 2 56 4096B 2 0
cred_cache 184 189 304 65536B 1667 0
rctl_cache 48 2365 2632 192512B 109415 0
rctl_val_cache 64 4473 4922 438272B 234389 0
task_cache 160 114 176 32768B 445 0
kmem_defrag_cache 216 2 16 4096B 2 0
kmem_move_cache 56 0 0 0B 0 0
rootnex_dmahdl 2592 10297 10297 42176512B 9783619 0
timeout_request 128 1 26 4096B 1 0
cyclic_id_cache 72 168 252 24576B 288873 0
callout_cache0 80 808 832 106496B 808 0
callout_lcache0 48 1021 1024 131072B 1021 0
dnlc_space_cache 24 0 0 0B 0 0
vfs_cache 208 125 136 32768B 135 0
vn_cache 208 18535 19312 4943872B 235168 0
vsk_anchor_cache 40 52 64 4096B 66 0
file_cache 56 1163 1683 135168B 230349 0
stream_head_cache 376 494 570 233472B 16846 0
queue_cache 656 929 1014 692224B 23327 0
syncq_cache 160 16 44 8192B 93 0
qband_cache 64 2 46 4096B 2 0
linkinfo_cache 48 52 56 4096B 53 0
ciputctrl_cache 1024 0 0 0B 0 0
serializer_cache 64 58 92 8192B 90 0
as_cache 232 144 272 69632B 12598 0
marker_cache 120 0 28 4096B 8 0
anon_cache 48 37022381 40659954 3265552384B 126336667 0
anonmap_cache 112 5907 6180 843776B 327702 0
segvn_cache 168 8113 8316 1622016B 480975 0
segvn_szc_cache1 4096 0 0 0B 0 0
segvn_szc_cache2 2097152 0 0 0B 0 0
flk_edges 48 0 56 4096B 2 0
fdb_cache 104 0 0 0B 0 0
timer_cache 136 120 225 36864B 458 0
vmu_bound_cache 56 2724 2856 229376B 4368 0
vmu_object_cache 64 1576 1656 147456B 2079 0
physio_buf_cache 248 0 0 0B 0 0
process_cache 3880 152 152 622592B 7339 0
kcf_sreq_cache 56 0 0 0B 0 0
kcf_areq_cache 296 0 0 0B 0 0
kcf_context_cache 112 0 0 0B 0 0
clnt_clts_endpnt_cache 88 0 0 0B 0 0
zio_cache 856 2 6894 6275072B 12929948 0
zio_link_cache 48 0 8568 626688B 12227640 0
zio_buf_512 512 14546 16576 8486912B 366791 0
zio_data_buf_512 512 877 880 450560B 6286 0
zio_buf_1024 1024 3 132 135168B 228377 0
zio_data_buf_1024 1024 70 88 90112B 937 0
zio_buf_1536 1536 2 168 258048B 127676 0
zio_data_buf_1536 1536 21 48 73728B 1510 0
zio_buf_2048 2048 47 94 192512B 53714 0
zio_data_buf_2048 2048 15 46 94208B 685 0
zio_buf_2560 2560 0 56 143360B 15837 0
zio_data_buf_2560 2560 11 56 143360B 751 0
zio_buf_3072 3072 1 56 172032B 33177 0
zio_data_buf_3072 3072 5 56 172032B 277 0
zio_buf_3584 3584 1 32 114688B 21557 0
zio_data_buf_3584 3584 60 72 258048B 903 0
zio_buf_4096 4096 92 127 520192B 78109 0
zio_data_buf_4096 4096 4 41 167936B 69235 0
zio_buf_5120 5120 0 20 102400B 19376 0
zio_data_buf_5120 5120 1 28 143360B 581 0
zio_buf_6144 6144 1 24 147456B 15764 0
zio_data_buf_6144 6144 6 24 147456B 360 0
zio_buf_7168 7168 1 24 172032B 14639 0
zio_data_buf_7168 7168 7 24 172032B 621 0
zio_buf_8192 8192 0 19 155648B 14341 0
zio_data_buf_8192 8192 2 20 163840B 387 0
zio_buf_10240 10240 0 18 184320B 38353 0
zio_data_buf_10240 10240 90 106 1085440B 817 0
zio_buf_12288 12288 0 17 208896B 34204 0
zio_data_buf_12288 12288 5 27 331776B 431 0
zio_buf_14336 14336 0 18 258048B 7040 0
zio_data_buf_14336 14336 2 28 401408B 506 0
zio_buf_16384 16384 6529 6562 107511808B 296743 0
zio_data_buf_16384 16384 4 27 442368B 378 0
zio_buf_20480 20480 0 17 348160B 5643 0
zio_data_buf_20480 20480 3 24 491520B 674 0
zio_buf_24576 24576 0 17 417792B 4761 0
zio_data_buf_24576 24576 2 22 540672B 474 0
zio_buf_28672 28672 0 20 573440B 5925 0
zio_data_buf_28672 28672 2 24 688128B 407 0
zio_buf_32768 32768 0 48 1572864B 444385 0
zio_data_buf_32768 32768 166 179 5865472B 1792 0
zio_buf_36864 36864 6 25 921600B 37694 0
zio_data_buf_36864 36864 0 20 737280B 140 0
zio_buf_40960 40960 1 18 737280B 2123 0
zio_data_buf_40960 40960 2 23 942080B 151 0
zio_buf_45056 45056 0 14 630784B 1204 0
zio_data_buf_45056 45056 1 24 1081344B 66 0
zio_buf_49152 49152 0 17 835584B 1483 0
zio_data_buf_49152 49152 2 20 983040B 101 0
zio_buf_53248 53248 0 14 745472B 738 0
zio_data_buf_53248 53248 0 19 1011712B 71 0
zio_buf_57344 57344 0 15 860160B 791 0
zio_data_buf_57344 57344 2 20 1146880B 64 0
zio_buf_61440 61440 1 19 1167360B 585 0
zio_data_buf_61440 61440 0 19 1167360B 73 0
zio_buf_65536 65536 0 20 1310720B 20430 0
zio_data_buf_65536 65536 2 16 1048576B 99 0
zio_buf_69632 69632 0 15 1044480B 3594 0
zio_data_buf_69632 69632 0 18 1253376B 61 0
zio_buf_73728 73728 0 16 1179648B 1218 0
zio_data_buf_73728 73728 1 17 1253376B 105 0
zio_buf_77824 77824 0 18 1400832B 852 0
zio_data_buf_77824 77824 1 21 1634304B 199 0
zio_buf_81920 81920 0 18 1474560B 1092 0
zio_data_buf_81920 81920 1 17 1392640B 77 0
zio_buf_86016 86016 0 17 1462272B 545 0
zio_data_buf_86016 86016 0 10 860160B 20 0
zio_buf_90112 90112 0 18 1622016B 391 0
zio_data_buf_90112 90112 1 17 1531904B 39 0
zio_buf_94208 94208 0 22 2072576B 344 0
zio_data_buf_94208 94208 0 15 1413120B 59 0
zio_buf_98304 98304 0 18 1769472B 15614 0
zio_data_buf_98304 98304 3 15 1474560B 47 0
zio_buf_102400 102400 0 19 1945600B 29987 0
zio_data_buf_102400 102400 0 6 614400B 6 0
zio_buf_106496 106496 0 21 2236416B 6665 0
zio_data_buf_106496 106496 0 12 1277952B 25 0
zio_buf_110592 110592 0 17 1880064B 1136 0
zio_data_buf_110592 110592 0 15 1658880B 28 0
zio_buf_114688 114688 0 20 2293760B 895 0
zio_data_buf_114688 114688 1 14 1605632B 21 0
zio_buf_118784 118784 0 14 1662976B 343 0
zio_data_buf_118784 118784 1 8 950272B 8 0
zio_buf_122880 122880 0 19 2334720B 269 0
zio_data_buf_122880 122880 1 13 1597440B 23 0
zio_buf_126976 126976 0 19 2412544B 234 0
zio_data_buf_126976 126976 0 11 1396736B 27 0
zio_buf_131072 131072 299 319 41811968B 449213 0
zio_data_buf_131072 131072 28547 28579 3745906688B 295371 0
sa_cache 56 14004 14076 1130496B 14615 0
dnode_t 744 15484 15495 12693504B 23697 0
dmu_buf_impl_t 192 43520 64512 14680064B 575413 0
arc_buf_hdr_t 176 79684 80480 16482304B 432186 0
arc_buf_t 56 36908 60129 4829184B 528704 0
zil_lwb_cache 192 107 1566 356352B 272903 0
zfs_znode_cache 248 14004 14040 3833856B 14615 0
pkt_cache_mpt_0 568 0 0 0B 0 0
mptsas0_cache 592 0 286 180224B 4185336 0
mptsas0_cache_frames 32 0 292 16384B 2020423 0
audit_proc 40 151 448 28672B 7340 0
drv_secobj_cache 296 0 0 0B 0 0
dld_str_cache 304 72 120 40960B 797 0
ip_minor_arena_sa_1 1 47 64 64B 457 0
ip_minor_arena_la_1 1 12 64 64B 4063 0
ip_conn_cache 720 5 20 16384B 41 0
tcp_conn_cache 1808 53 78 159744B 170 0
udp_conn_cache 1256 38 123 167936B 6252 0
rawip_conn_cache 1096 1 7 8192B 2 0
rts_conn_cache 816 17 54 49152B 199 0
ire_cache 352 195 230 94208B 605 0
ncec_cache 200 39 72 16384B 1666 0
nce_cache 88 45 180 20480B 3302 0
rt_entry 152 78 92 16384B 514 0
radix_mask 32 15 73 4096B 22 0
radix_node 120 3 28 4096B 3 0
ipsec_actions 72 0 0 0B 0 0
ipsec_selectors 80 0 0 0B 0 0
ipsec_policy 80 0 0 0B 0 0
tcp_timercache 88 61 108 12288B 195 0
tcp_notsack_blk_cache 24 0 0 0B 0 0
squeue_cache 168 20 21 4096B 20 0
sctp_conn_cache 2528 0 0 0B 0 0
sctp_faddr_cache 176 0 0 0B 0 0
sctp_set_cache 24 0 0 0B 0 0
sctp_ftsn_set_cache 16 0 0 0B 0 0
dce_cache 152 11 23 4096B 12 0
ire_gw_secattr_cache 24 0 0 0B 0 0
socket_cache 640 66 156 106496B 6234 0
socktpi_cache 944 0 4 4096B 4 0
socktpi_unix_cache 944 35 40 40960B 51 0
sock_sod_cache 648 0 0 0B 0 0
exacct_object_cache 40 0 0 0B 0 0
callout_cache1 80 544 544 69632B 544 0
callout_lcache1 48 948 960 122880B 948 0
kssl_cache 1624 0 0 0B 0 0
rds_alloc_cache 88 0 0 0B 0 0
tl_cache 432 89 104 53248B 155 0
keysock_1 1 0 0 0B 0 0
spdsock_1 1 0 0 0B 4 0
fctl_cache 112 0 0 0B 0 0
namefs_inodes_1 1 53 64 64B 54 0
port_cache 80 11 39 4096B 11 0
softmac_cache 568 4 13 8192B 7 0
softmac_upper_cache 232 0 0 0B 0 0
vnic_cache 120 60 84 12288B 60 0
Hex0xffffff3460b2a558_minor_1 1 0 0 0B 0 0
Hex0xffffff3460b2a560_minor_1 1 0 0 0B 0 0
lnode_cache 32 5 73 4096B 67 0
idm_buf_cache 240 0 0 0B 0 0
idm_task_cache 2432 0 0 0B 0 0
idm_tx_pdu_cache 464 0 0 0B 0 0
idm_rx_pdu_cache 513 0 0 0B 0 0
idm_128k_buf_cache 131072 0 0 0B 0 0
dtrace_state_cache 16384 0 0 0B 0 0
ufs_inode_cache 368 0 0 0B 0 0
directio_buf_cache 272 0 0 0B 0 0
lufs_save 24 0 0 0B 0 0
lufs_bufs 256 0 0 0B 0 0
lufs_mapentry_cache 112 0 0 0B 0 0
pty_map 64 2 46 4096B 3 0
sppptun_map 440 0 0 0B 0 0
authkern_cache 72 0 0 0B 0 0
authnone_cache 72 0 0 0B 0 0
authloopback_cache 72 0 0 0B 0 0
authdes_cache_handle 80 0 0 0B 0 0
rnode_cache 656 0 0 0B 0 0
nfs_access_cache 56 0 0 0B 0 0
client_handle_cache 32 0 0 0B 0 0
rnode4_cache 968 0 0 0B 0 0
svnode_cache 40 0 0 0B 0 0
nfs4_access_cache 56 0 0 0B 0 0
client_handle4_cache 32 0 0 0B 0 0
nfs4_ace4vals_cache 48 0 0 0B 0 0
nfs4_ace4_list_cache 264 0 0 0B 0 0
NFS_idmap_cache 48 0 0 0B 0 0
kvm_vcpu 29536 38 38 1245184B 151 0
kvm_pte_chain 56 4007 4029 323584B 69462 0
kvm_rmap_desc 40 1115 4800 307200B 43382 0
kvm_mmu_page_header 264 3737 3752 1097728B 189260 0
crypto_session_cache 104 0 0 0B 0 0
Hex0xffffff3864071558_minor_1 1 0 0 0B 0 0
Hex0xffffff3864071560_minor_1 1 0 0 0B 0 0
Hex0xffffff3897cc1558_minor_1 1 0 0 0B 0 0
Hex0xffffff3897cc1560_minor_1 1 0 0 0B 0 0
Hex0xffffff3a19109558_minor_1 1 0 0 0B 0 0
Hex0xffffff3a19109560_minor_1 1 0 0 0B 0 0


Total [hat_memload] 7319552B 5483793 0
Total [kmem_msb] 10305896448B 68102097 0
Total [kmem_firewall] 402100224B 11116611 0
Total [kmem_va] 3600941056B 971500 0
Total [kmem_default] 3624284160B 1162090769 0
Total [kmem_io_4G] 2121728B 6209728 0
Total [kmem_io_2G] 102400B 68 0
Total [bp_map] 0B 15 0
Total [umem_np] 524288B 819 0
Total [zfs_file_data] 15466496B 27776 0
Total [zfs_file_data_buf] 3785863168B 384893 0
Total [segkp] 589824B 7282 0
Total [ip_minor_arena_sa] 64B 457 0
Total [ip_minor_arena_la] 64B 4063 0
Total [spdsock] 0B 4 0
Total [namefs_inodes] 64B 54 0


vmem memory memory memory alloc alloc
name in use total import succeed fail


heap 27269943296B 882833883136B 0B 11324730 0
vmem_metadata 2545803264B 2545811456B 2545811456B 153771 0
vmem_seg 2517254144B 2517254144B 2517254144B 153641 0
vmem_hash 28253184B 28254208B 28254208B 77 0
vmem_vmem 298480B 331312B 294912B 88 0
static 0B 0B 0B 0 0
static_alloc 0B 0B 0B 0 0
hat_memload 7319552B 7319552B 7319552B 3374 0
kstat 1293840B 1323008B 1257472B 3545 0
kmem_metadata 10586935296B 10587078656B 10587078656B 2516731 0
kmem_msb 10305896448B 10305896448B 10305896448B 2516351 0
kmem_cache 668640B 860160B 860160B 479 0
kmem_hash 280148992B 280178688B 280178688B 596 0
kmem_log 7919637696B 7919640576B 7919640576B 12 0
kmem_firewall_va 2107293696B 2107293696B 2107293696B 11124077 0
kmem_firewall 402104320B 402104320B 402104320B 11119123 0
kmem_oversize 1705145192B 1705189376B 1705189376B 4954 0
mod_sysfile 359B 4096B 4096B 10 0
kmem_va 3676098560B 3676098560B 3676098560B 60991 0
kmem_default 3624677376B 3624677376B 3624677376B 999837 0
kmem_io_4P 0B 0B 0B 0 0
kmem_io_4G 2121728B 2121728B 2121728B 532 0
kmem_io_2G 102400B 102400B 102400B 14 0
kmem_io_16M 0B 0B 0B 0 0
bp_map 0B 0B 0B 5801 0
umem_np 524288B 524288B 524288B 408 0
ksyms 2189346B 2457600B 2457600B 293 0
ctf 1685957B 1998848B 1998848B 293 0
heap_core 3403776B 931115008B 0B 60 0
heaptext 10248192B 67108864B 0B 231 0
module_text 10090752B 10326016B 10248192B 292 0
id32 0B 0B 0B 0 0
module_data 845151B 3710976B 3403776B 425 0
logminor_space 87B 262137B 0B 107 0
taskq_id_arena 108B 2147483647B 0B 241 0
zfs_file_data 3788357632B 206141652992B 0B 119897 0
zfs_file_data_buf 3784814592B 3784814592B 3784814592B 147181 0
device 2437120B 1073741824B 0B 33341 0
segkp 52838400B 2147483648B 0B 11278 0
mac_minor_ids 132B 130070B 0B 855 0
rctl_ids 39B 32767B 0B 39 0
zoneid_space 3B 9998B 0B 3 0
taskid_space 114B 999999B 0B 444 0
pool_ids 0B 999998B 0B 0 0
contracts 132B 2147483646B 0B 485 0
ip_minor_arena_sa 64B 262140B 0B 6 0
ip_minor_arena_la 64B 4294705152B 0B 8 0
lib_va_32 7483392B 2138820608B 0B 20 0
tl_minor_space 89B 262138B 0B 153 0
keysock 0B 4294967295B 0B 0 0
spdsock 0B 4294967295B 0B 4 0
ibcm_local_sid 0B 4294967295B 0B 0 0
ibcm_ip_sid 0B 65535B 0B 0 0
lport-instances 0B 65536B 0B 0 0
rport-instances 0B 65536B 0B 0 0
namefs_inodes 64B 65536B 0B 1 0
lib_va_64 43319296B 137988695961600B 0B 106 0
ds_minors 8B 262140B 0B 8 0
Hex0xffffff3460b2a558_minor 0B 4294967294B 0B 0 0
Hex0xffffff3460b2a560_minor 0B 4294967294B 0B 0 0
syseventd_channel 3B 101B 0B 3 0
syseventd_channel 1B 2B 0B 1 0
syseventconfd_door 0B 101B 0B 0 0
syseventconfd_door 1B 2B 0B 1 0
devfsadm_event_channel 1B 101B 0B 1 0
devfsadm_event_channel 1B 2B 0B 1 0
dtrace 1108B 4294967295B 0B 51191 0
dtrace_minor 0B 4294967293B 0B 0 0
logdmux_minor 0B 256B 0B 0 0
ptms_minor 2B 16B 0B 3 0
sppptun_minor 0B 16B 0B 0 0
lmsysid_space 0B 16383B 0B 1 0
kvm_minor 53B 4294967294B 0B 262 0
ipnet_minor_space 28B 262141B 0B 287 0
crypto 0B 16B 0B 6 0
lofi_minor_id 0B 262143B 0B 0 0
Hex0xffffff3864071558_minor 0B 4294967294B 0B 0 0
Hex0xffffff3864071560_minor 0B 4294967294B 0B 0 0
Hex0xffffff3897cc1558_minor 0B 4294967294B 0B 0 0
Hex0xffffff3897cc1560_minor 0B 4294967294B 0B 0 0
Hex0xffffff3a19109558_minor 0B 4294967294B 0B 0 0
Hex0xffffff3a19109560_minor 0B 4294967294B 0B 0 0


@rmustacc
Owner

How long is the timeout on the SMF service set to? It looks like you were making forward progress evicting memory from the ARC, but that took longer than the timeout allowed.

@estibi

Sorry, it was my fault, kernel eats 30GB of RAM because of kmem_flags=0xf.

@estibi estibi closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.