Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High kernel "SUnreclaimable" memory usage after around 5-7 days of uptime #2637

Open
peat-psuwit opened this issue Nov 16, 2019 · 3 comments

Comments

@peat-psuwit
Copy link

@peat-psuwit peat-psuwit commented Nov 16, 2019

Issue Report

Bug

The kernel's memory usage on a DigitalOcean's 1 GB droplet with CoreOS rises up to an unreasonable level after leaving the machine on for around 5-7 days, making it unresponsive and sometimes trigger OOM.

The machine is not running Kubernetes but is a (single host) Docker swarm manager. It has only 3 containers running.

Container Linux Version

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=2247.6.0
VERSION_ID=2247.6.0
BUILD_ID=2019-11-06-2138
PRETTY_NAME="Container Linux by CoreOS 2247.6.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

DigitalOcean cloud provider, Singapore POP, 1 GB machine.

Expected Behavior

The kernel uses a reasonable amount of memory. SClaimable may grow significantly, but SUnreclaimable shouldn't rise above 100 MB.

Actual Behavior

The kernel's unclaimable memory goes to almost 300 MB, while claimable memory stays around 57 MB. sudo slabtop --sort c shows that the top consumer is cred_jar at around 74 MB.

Reproduction Steps

  1. Start a new DigitalOcean 1GB droplet. Leave it running for about 5-7 days.
  2. Inspect kernel memory usage with cat /proc/meminfo and sudo slabtop --sort c.

Other Information

This is the result of cat /proc/meminfo and sudo slabtop --sort c:

cat /proc/meminfo
MemTotal:        1009980 kB
MemFree:           51916 kB
MemAvailable:      13816 kB
Buffers:             128 kB
Cached:           235944 kB
SwapCached:            0 kB
Active:           400316 kB
Inactive:         157704 kB
Active(anon):     395984 kB
Inactive(anon):   153584 kB
Active(file):       4332 kB
Inactive(file):     4120 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        321956 kB
Mapped:            93824 kB
Shmem:            227616 kB
Slab:             357040 kB
SReclaimable:      57848 kB
SUnreclaim:       299192 kB
KernelStack:        3200 kB
PageTables:         5988 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      504988 kB
Committed_AS:    2279580 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:             3668 kB
HardwareCorrupted:     0 kB
AnonHugePages:    172032 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      417772 kB
DirectMap2M:      630784 kB
DirectMap1G:           0 kB
sudo slabtop --sort c
 Active / Total Objects (% used)    : 3054935 / 3075343 (99.3%)
 Active / Total Slabs (% used)      : 69313 / 69313 (100.0%)
 Active / Total Caches (% used)     : 101 / 130 (77.7%)
 Active / Total Size (% used)       : 345078.60K / 351209.45K (98.3%)
 Minimum / Average / Maximum Object : 0.01K / 0.11K / 23.25K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
392847 392662  99%    0.19K  18707       21     74828K cred_jar
 24855  24855 100%    1.06K   1657       15     26512K mm_struct
329856 329299  99%    0.06K   5154       64     20616K anon_vma_chain
 32727  32389  98%    0.59K   2518       13     20144K inode_cache
 79504  79362  99%    0.25K   4969       16     19876K filp
600064 600064 100%    0.03K   4688      128     18752K kmalloc-32
 39360  39360 100%    0.38K   3936       10     15744K kmem_cache
  7848   7836  99%    2.00K    981        8     15696K kmalloc-2048
159160 159160 100%    0.09K   3460       46     13840K anon_vma
 18821  18811  99%    0.69K   1711       11     13688K sock_inode_cache
 13448  13448 100%    1.00K   1681        8     13448K UNIX
 96720  96720 100%    0.13K   3224       30     12896K kernfs_node_cache
 58254  51782  88%    0.19K   2774       21     11096K dentry
  2336   2334  99%    4.00K    292        8      9344K kmalloc-4096
 15768  15764  99%    0.50K   1971        8      7884K kmalloc-512
 53888  53888 100%    0.12K   1684       32      6736K pid
 26816  26736  99%    0.25K   1676       16      6704K skbuff_head_cache
853504 853504 100%    0.01K   1667      512      6668K kmalloc-8
  9506   7982  83%    0.57K    679       14      5432K radix_tree_node
  5100   5099  99%    1.00K    638        8      5104K kmalloc-1024
  4275   2390  55%    1.05K    285       15      4560K ext4_inode_cache
 47292  47092  99%    0.09K   1126       42      4504K kmalloc-96
   279    254  91%    9.25K     93        3      2976K task_struct
 39488  39488 100%    0.06K    617       64      2468K kmem_cache_node
 36416  36272  99%    0.06K    569       64      2276K kmalloc-64
  7938   6512  82%    0.19K    378       21      1512K kmalloc-192
  1705   1408  82%    0.69K    155       11      1240K shmem_inode_cache
 10335   9796  94%    0.10K    265       39      1060K buffer_head
  1404   1264  90%    0.66K    118       12       944K proc_inode_cache
  3584   3584 100%    0.25K    224       16       896K kmalloc-256
 21522  15903  73%    0.04K    211      102       844K numa_policy
    96     75  78%    8.00K     24        4       768K kmalloc-8192
   300    300 100%    2.06K     20       15       640K sighand_cache
  5024   5024 100%    0.12K    157       32       628K kmalloc-128
 19210  19210 100%    0.02K    113      170       452K avtab_node
   390    390 100%    1.06K     26       15       416K signal_cache
    52     34  65%    8.00K     13        4       416K biovec-max
 26368  26368 100%    0.02K    103      256       412K kmalloc-16
   588    415  70%    0.66K     49       12       392K ovl_inode
   910    853  93%    0.38K     91       10       364K mnt_cache
  4032   4032 100%    0.06K     63       64       252K ebitmap_node
    98     98 100%    2.25K      7       14       224K TCPv6
  1029   1029 100%    0.19K     49       21       196K proc_dir_entry
    48     24  50%    4.00K      6        8       192K names_cache
   242    242 100%    0.69K     22       11       176K files_cache
  3485   3485 100%    0.05K     41       85       164K ftrace_event_field
    75     75 100%    2.12K      5       15       160K TCP
  3570   3162  88%    0.04K     35      102       140K ext4_extent_status
  1380   1380 100%    0.09K     30       46       120K trace_event_file
   870    870 100%    0.13K     29       30       116K dm_bufio_buffer-4
   168    168 100%    0.62K     14       12       112K dio
    98     98 100%    1.12K      7       14       112K RAWv6
  1344   1344 100%    0.07K     24       56        96K Acpi-Operand
   110    110 100%    0.75K     11       10        88K task_group
  1120   1120 100%    0.07K     20       56        80K eventpoll_pwq
    60     60 100%    1.25K      5       12        80K UDPv6
   110    110 100%    0.69K     10       11        80K bio-2
   216    170  78%    0.31K     18       12        72K nf_conntrack
   204    156  76%    0.31K     17       12        68K xfrm_dst_cache
   256    256 100%    0.25K     16       16        64K pool_workqueue
    10     10 100%    5.50K      2        5        64K net_namespace
    26     26 100%    2.40K      2       13        64K request_queue
  3328   3328 100%    0.02K     13      256        52K selinux_file_security
   416    416 100%    0.12K     13       32        52K scsi_sense_cache
   408    408 100%    0.12K     12       34        48K jbd2_journal_head
  1122   1122 100%    0.04K     11      102        44K pde_opener
    88     88 100%    0.50K     11        8        44K skbuff_fclone_cache
   231    231 100%    0.19K     11       21        44K dmaengine-unmap-16
   187    187 100%    0.23K     11       17        44K tw_sock_TCP
   187    187 100%    0.23K     11       17        44K tw_sock_TCPv6
    40     40 100%    0.94K      5        8        40K RAW
   459    459 100%    0.08K      9       51        36K inotify_inode_mark
    38     27  71%    0.81K      2       19        32K bdev_cache
    15     15 100%    2.06K      1       15        32K dmaengine-unmap-256
     8      8 100%    4.00K      1        8        32K sgpool-128
    91     91 100%    0.30K      7       13        28K blkdev_requests
   196    196 100%    0.14K      7       28        28K ext4_groupinfo_4k
    24     24 100%    1.00K      3        8        24K biovec-64
   438    438 100%    0.05K      6       73        24K mbcache
    15     15 100%    1.06K      1       15        16K dmaengine-unmap-128
     8      8 100%    2.00K      1        8        16K biovec-128
    52     52 100%    0.30K      4       13        16K request_sock_TCP
   128    128 100%    0.12K      4       32        16K secpath_cache
   512    512 100%    0.03K      4      128        16K fscrypt_info
    22     22 100%    0.72K      2       11        16K fat_inode_cache
   153    153 100%    0.08K      3       51        12K Acpi-State
   219    219 100%    0.05K      3       73        12K Acpi-Parse
   117    117 100%    0.10K      3       39        12K blkdev_ioc
   192    192 100%    0.06K      3       64        12K ext4_io_end
    24     24 100%    0.32K      2       12         8K taskstats
    40     40 100%    0.20K      2       20         8K file_lock_cache
    13     13 100%    0.61K      1       13         8K hugetlbfs_inode_cache
     8      8 100%    0.94K      1        8         8K mqueue_inode_cache
     9      9 100%    0.43K      1        9         4K uts_namespace
    25     25 100%    0.16K      1       25         4K sigqueue
    16     16 100%    0.25K      1       16         4K dquot
    19     19 100%    0.20K      1       19         4K ip4-frags
    13     13 100%    0.30K      1       13         4K request_sock_TCPv6
    85     85 100%    0.05K      1       85         4K fscrypt_ctx
   256    256 100%    0.02K      1      256         4K jbd2_revoke_table_s
    32     32 100%    0.12K      1       32         4K ext4_allocation_context
     0      0   0%    0.09K      0       42         0K dma-kmalloc-96
     0      0   0%    0.19K      0       21         0K dma-kmalloc-192
     0      0   0%    0.01K      0      512         0K dma-kmalloc-8
     0      0   0%    0.02K      0      256         0K dma-kmalloc-16
     0      0   0%    0.03K      0      128         0K dma-kmalloc-32
     0      0   0%    0.06K      0       64         0K dma-kmalloc-64
     0      0   0%    0.12K      0       32         0K dma-kmalloc-128
     0      0   0%    0.25K      0       16         0K dma-kmalloc-256
     0      0   0%    0.50K      0        8         0K dma-kmalloc-512
     0      0   0%    1.00K      0        8         0K dma-kmalloc-1024
     0      0   0%    2.00K      0        8         0K dma-kmalloc-2048
     0      0   0%    4.00K      0        8         0K dma-kmalloc-4096
     0      0   0%    8.00K      0        4         0K dma-kmalloc-8192
     0      0   0%    0.03K      0      128         0K avc_xperms_data
     0      0   0%    0.12K      0       34         0K iint_cache
     0      0   0%    0.45K      0       17         0K user_namespace
     0      0   0%    0.94K      0        8         0K PING
     0      0   0%    0.23K      0       17         0K posix_timers_cache
     0      0   0%    0.03K      0      128         0K dnotify_struct
     0      0   0%    0.12K      0       34         0K cfq_io_cq
     0      0   0%    0.44K      0        9         0K bfq_queue
     0      0   0%    1.12K      0       14         0K PINGv6
     0      0   0%    2.57K      0       12         0K dm_uevent
     0      0   0%    3.23K      0        9         0K kcopyd_job
     0      0   0%    0.16K      0       24         0K kvm_mmu_page_header
     0      0   0%   23.25K      0        1         0K kvm_vcpu
     0      0   0%    0.13K      0       30         0K kvm_async_pf
     0      0   0%    0.04K      0      102         0K fat_cache
     0      0   0%    0.21K      0       18         0K nf_conntrack_expect

And this is kernel messages, captured about 6 hours later compared to the 2 logs above:

dmesg
[    0.000000] Linux version 4.19.78-coreos (jenkins@ip-10-7-32-103) (gcc version 8.3.0 (Gentoo Hardened 8.3.0-r1 p1.1)) #1 SMP Wed Nov 6 20:58:31 -00 2019
[    0.000000] Command line: BOOT_IMAGE=/coreos/vmlinuz-b mount.usr=/dev/mapper/usr verity.usr=PARTUUID=e03dd35c-7c2d-4a47-b3fe-27f15780a57c rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200n8 console=tty0 coreos.oem.id=digitalocean verity.usrhash=c36835d1bb91ebb00ca7b284c1148235c82498221011cc5f77614b883c441bb7
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[3]:  832, xstate_sizes[3]:   64
[    0.000000] x86/fpu: xstate_offset[4]:  896, xstate_sizes[4]:   64
[    0.000000] x86/fpu: xstate_offset[5]:  960, xstate_sizes[5]:   64
[    0.000000] x86/fpu: xstate_offset[6]: 1024, xstate_sizes[6]:  512
[    0.000000] x86/fpu: xstate_offset[7]: 1536, xstate_sizes[7]: 1024
[    0.000000] x86/fpu: xstate_offset[9]: 2560, xstate_sizes[9]:    8
[    0.000000] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003fffafff] usable
[    0.000000] BIOS-e820: [mem 0x000000003fffb000-0x000000003fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.4 present.
[    0.000000] DMI: DigitalOcean Droplet, BIOS 20171212 12/12/2017
[    0.000000] Hypervisor detected: KVM
[    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[    0.000000] kvm-clock: cpu 0, msr 11e95001, primary cpu clock
[    0.000000] kvm-clock: using sched offset of 9959619446314041 cycles
[    0.000003] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000007] tsc: Detected 2294.608 MHz processor
[    0.001496] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.001499] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.001503] last_pfn = 0x3fffb max_arch_pfn = 0x400000000
[    0.001559] MTRR default type: write-back
[    0.001560] MTRR fixed ranges enabled:
[    0.001561]   00000-9FFFF write-back
[    0.001562]   A0000-BFFFF uncachable
[    0.001562]   C0000-FFFFF write-protect
[    0.001563] MTRR variable ranges enabled:
[    0.001565]   0 base 0080000000 mask FF80000000 uncachable
[    0.001565]   1 disabled
[    0.001566]   2 disabled
[    0.001566]   3 disabled
[    0.001567]   4 disabled
[    0.001567]   5 disabled
[    0.001568]   6 disabled
[    0.001568]   7 disabled
[    0.001583] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.001743] Using GB pages for direct mapping
[    0.001749] BRK [0x12001000, 0x12001fff] PGTABLE
[    0.001750] BRK [0x12002000, 0x12002fff] PGTABLE
[    0.001751] BRK [0x12003000, 0x12003fff] PGTABLE
[    0.001778] BRK [0x12004000, 0x12004fff] PGTABLE
[    0.001835] BRK [0x12005000, 0x12005fff] PGTABLE
[    0.001871] ACPI: Early table checksum verification disabled
[    0.001925] ACPI: RSDP 0x00000000000F1080 000014 (v00 BOCHS )
[    0.001932] ACPI: RSDT 0x000000003FFFE400 000034 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
[    0.001940] ACPI: FACP 0x000000003FFFFF80 000074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
[    0.001946] ACPI: DSDT 0x000000003FFFE440 001189 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
[    0.001950] ACPI: FACS 0x000000003FFFFF40 000040
[    0.001953] ACPI: SSDT 0x000000003FFFF700 00083B (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
[    0.001957] ACPI: APIC 0x000000003FFFF610 000078 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
[    0.001960] ACPI: HPET 0x000000003FFFF5D0 000038 (v01 BOCHS  BXPCHPET 00000001 BXPC 00000001)
[    0.001972] ACPI: Local APIC address 0xfee00000
[    0.002513] No NUMA configuration found
[    0.002515] Faking a node at [mem 0x0000000000000000-0x000000003fffafff]
[    0.002521] NODE_DATA(0) allocated [mem 0x3fff5000-0x3fffafff]
[    0.002556] Zone ranges:
[    0.002557]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.002558]   DMA32    [mem 0x0000000001000000-0x000000003fffafff]
[    0.002559]   Normal   empty
[    0.002560] Movable zone start for each node
[    0.002561] Early memory node ranges
[    0.002562]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.002563]   node   0: [mem 0x0000000000100000-0x000000003fffafff]
[    0.002566] Reserved but unavailable: 98 pages
[    0.002567] Initmem setup node 0 [mem 0x0000000000001000-0x000000003fffafff]
[    0.002568] On node 0 totalpages: 262041
[    0.002569]   DMA zone: 64 pages used for memmap
[    0.002570]   DMA zone: 21 pages reserved
[    0.002570]   DMA zone: 3998 pages, LIFO batch:0
[    0.002655]   DMA32 zone: 4032 pages used for memmap
[    0.002655]   DMA32 zone: 258043 pages, LIFO batch:63
[    0.008884] ACPI: PM-Timer IO Port: 0xb008
[    0.008889] ACPI: Local APIC address 0xfee00000
[    0.008901] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.008945] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
[    0.008948] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.008950] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[    0.008951] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.008960] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[    0.008960] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[    0.008962] ACPI: IRQ0 used by override.
[    0.008963] ACPI: IRQ5 used by override.
[    0.008963] ACPI: IRQ9 used by override.
[    0.008964] ACPI: IRQ10 used by override.
[    0.008964] ACPI: IRQ11 used by override.
[    0.008966] Using ACPI (MADT) for SMP configuration information
[    0.008968] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.008977] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[    0.008997] [mem 0x40000000-0xfeffbfff] available for PCI devices
[    0.008998] Booting paravirtualized kernel on KVM
[    0.009001] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.105550] random: get_random_bytes called from start_kernel+0x93/0x522 with crng_init=0
[    0.105562] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:1 nr_node_ids:1
[    0.105935] percpu: Embedded 44 pages/cpu s143256 r8192 d28776 u2097152
[    0.105943] pcpu-alloc: s143256 r8192 d28776 u2097152 alloc=1*2097152
[    0.105944] pcpu-alloc: [0] 0 
[    0.105965] KVM setup async PF for cpu 0
[    0.105971] kvm-stealtime: cpu 0, msr 3ea161c0
[    0.105978] Built 1 zonelists, mobility grouping on.  Total pages: 257924
[    0.105979] Policy zone: DMA32
[    0.105982] Kernel command line: rootflags=rw mount.usrflags=ro BOOT_IMAGE=/coreos/vmlinuz-b mount.usr=/dev/mapper/usr verity.usr=PARTUUID=e03dd35c-7c2d-4a47-b3fe-27f15780a57c rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200n8 console=tty0 coreos.oem.id=digitalocean verity.usrhash=c36835d1bb91ebb00ca7b284c1148235c82498221011cc5f77614b883c441bb7
[    0.109519] Memory: 965792K/1048164K available (10252K kernel code, 1217K rwdata, 5484K rodata, 41484K init, 1572K bss, 82372K reserved, 0K cma-reserved)
[    0.109641] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.109643] Kernel/User page tables isolation: enabled
[    0.109751] ftrace: allocating 29193 entries in 115 pages
[    0.122904] rcu: Hierarchical RCU implementation.
[    0.122908] rcu: 	RCU event tracing is enabled.
[    0.122909] rcu: 	RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=1.
[    0.122911] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.125667] NR_IRQS: 33024, nr_irqs: 256, preallocated irqs: 16
[    0.140423] Console: colour VGA+ 80x25
[    0.217930] console [tty0] enabled
[    0.498626] console [ttyS0] enabled
[    0.501707] ACPI: Core revision 20180810
[    0.504789] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
[    0.513928] hpet clockevent registered
[    0.514017] APIC: Switch to symmetric I/O mode setup
[    0.520743] x2apic enabled
[    0.524782] Switched APIC routing to physical x2apic.
[    0.532825] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.537414] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x21134f58f0d, max_idle_ns: 440795217993 ns
[    0.545831] Calibrating delay loop (skipped) preset value.. 4589.21 BogoMIPS (lpj=2294608)
[    0.546820] pid_max: default: 32768 minimum: 301
[    0.548858] Security Framework initialized
[    0.549811] SELinux:  Initializing.
[    0.552016] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.552930] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.553841] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes)
[    0.554824] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes)
[    0.557134] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    0.557812] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[    0.558818] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    0.559830] Spectre V2 : Mitigation: Full generic retpoline
[    0.560820] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.561819] Spectre V2 : Enabling Restricted Speculation for firmware calls
[    0.562835] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[    0.563818] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
[    0.564866] MDS: Mitigation: Clear CPU buffers
[    0.575409] Freeing SMP alternatives memory: 28K
[    0.578805] TSC deadline timer enabled
[    0.578805] smpboot: CPU0: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (family: 0x6, model: 0x55, stepping: 0x4)
[    0.579110] Performance Events: Skylake events, Intel PMU driver.
[    0.579832] ... version:                2
[    0.580823] ... bit width:              48
[    0.581818] ... generic registers:      4
[    0.582820] ... value mask:             0000ffffffffffff
[    0.583818] ... max period:             000000007fffffff
[    0.584816] ... fixed-purpose events:   3
[    0.585820] ... event mask:             000000070000000f
[    0.586934] rcu: Hierarchical SRCU implementation.
[    0.588917] smp: Bringing up secondary CPUs ...
[    0.589831] smp: Brought up 1 node, 1 CPU
[    0.590822] smpboot: Max logical packages: 1
[    0.591823] smpboot: Total of 1 processors activated (4589.21 BogoMIPS)
[    0.593178] devtmpfs: initialized
[    0.593916] x86/mm: Memory block size: 128MB
[    0.595230] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.595829] futex hash table entries: 256 (order: 2, 16384 bytes)
[    0.596930] pinctrl core: initialized pinctrl subsystem
[    0.597960] NET: Registered protocol family 16
[    0.598957] audit: initializing netlink subsys (disabled)
[    0.599975] cpuidle: using governor menu
[    0.600945] ACPI: bus type PCI registered
[    0.601827] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.602914] dca service started, version 1.12.1
[    0.603930] PCI: Using configuration type 1 for base access
[    0.604882] audit: type=2000 audit(1573260144.515:1): state=initialized audit_enabled=0 res=1
[    0.606824] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.607818] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.609164] ACPI: Added _OSI(Module Device)
[    0.609827] ACPI: Added _OSI(Processor Device)
[    0.610824] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.611823] ACPI: Added _OSI(Processor Aggregator Device)
[    0.612822] ACPI: Added _OSI(Linux-Dell-Video)
[    0.613819] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    0.616310] ACPI: 2 ACPI AML tables successfully acquired and loaded
[    0.618143] ACPI: Interpreter enabled
[    0.618850] ACPI: (supports S0 S3 S5)
[    0.619822] ACPI: Using IOAPIC for interrupt routing
[    0.620861] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.622245] ACPI: Enabled 16 GPEs in block 00 to 0F
[    0.625325] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.625832] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
[    0.626860] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
[    0.627849] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[    0.629476] acpiphp: Slot [3] registered
[    0.629913] acpiphp: Slot [4] registered
[    0.630892] acpiphp: Slot [5] registered
[    0.631931] acpiphp: Slot [6] registered
[    0.632938] acpiphp: Slot [7] registered
[    0.633989] acpiphp: Slot [8] registered
[    0.634994] acpiphp: Slot [9] registered
[    0.636025] acpiphp: Slot [10] registered
[    0.636948] acpiphp: Slot [11] registered
[    0.637914] acpiphp: Slot [12] registered
[    0.638901] acpiphp: Slot [13] registered
[    0.639912] acpiphp: Slot [14] registered
[    0.640971] acpiphp: Slot [15] registered
[    0.641904] acpiphp: Slot [16] registered
[    0.642886] acpiphp: Slot [17] registered
[    0.643874] acpiphp: Slot [18] registered
[    0.644872] acpiphp: Slot [19] registered
[    0.645991] acpiphp: Slot [20] registered
[    0.646916] acpiphp: Slot [21] registered
[    0.647919] acpiphp: Slot [22] registered
[    0.648912] acpiphp: Slot [23] registered
[    0.649911] acpiphp: Slot [24] registered
[    0.650891] acpiphp: Slot [25] registered
[    0.651888] acpiphp: Slot [26] registered
[    0.652865] acpiphp: Slot [27] registered
[    0.653875] acpiphp: Slot [28] registered
[    0.654903] acpiphp: Slot [29] registered
[    0.655925] acpiphp: Slot [30] registered
[    0.656898] acpiphp: Slot [31] registered
[    0.657851] PCI host bridge to bus 0000:00
[    0.658824] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.659824] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.660824] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.661823] pci_bus 0000:00: root bus resource [mem 0x80000000-0xfebfffff window]
[    0.662824] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.663900] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
[    0.664705] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
[    0.665762] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
[    0.668131] pci 0000:00:01.1: reg 0x20: [io  0xc120-0xc12f]
[    0.669908] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
[    0.670822] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
[    0.671822] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
[    0.672822] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
[    0.674169] pci 0000:00:01.2: [8086:7020] type 00 class 0x0c0300
[    0.677488] pci 0000:00:01.2: reg 0x20: [io  0xc080-0xc09f]
[    0.678472] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
[    0.679183] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 ACPI
[    0.679845] pci 0000:00:01.3: quirk: [io  0xb100-0xb10f] claimed by PIIX4 SMB
[    0.681538] pci 0000:00:02.0: [1b36:0100] type 00 class 0x030000
[    0.682866] pci 0000:00:02.0: reg 0x10: [mem 0xf8000000-0xfbffffff]
[    0.684868] pci 0000:00:02.0: reg 0x14: [mem 0xfc000000-0xfcffffff]
[    0.686899] pci 0000:00:02.0: reg 0x18: [mem 0xfd010000-0xfd011fff]
[    0.688902] pci 0000:00:02.0: reg 0x1c: [io  0xc0a0-0xc0bf]
[    0.694872] pci 0000:00:02.0: reg 0x30: [mem 0xfd000000-0xfd00ffff pref]
[    0.696111] pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000
[    0.696818] pci 0000:00:03.0: reg 0x10: [io  0xc0c0-0xc0df]
[    0.697689] pci 0000:00:03.0: reg 0x14: [mem 0xfd012000-0xfd012fff]
[    0.703060] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000
[    0.703818] pci 0000:00:04.0: reg 0x10: [io  0xc0e0-0xc0ff]
[    0.705366] pci 0000:00:04.0: reg 0x14: [mem 0xfd013000-0xfd013fff]
[    0.711491] pci 0000:00:05.0: [1af4:1004] type 00 class 0x010000
[    0.712309] pci 0000:00:05.0: reg 0x10: [io  0xc000-0xc03f]
[    0.713244] pci 0000:00:05.0: reg 0x14: [mem 0xfd014000-0xfd014fff]
[    0.719516] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000
[    0.720352] pci 0000:00:06.0: reg 0x10: [io  0xc040-0xc07f]
[    0.721350] pci 0000:00:06.0: reg 0x14: [mem 0xfd015000-0xfd015fff]
[    0.728112] pci 0000:00:07.0: [1af4:1002] type 00 class 0x00ff00
[    0.728754] pci 0000:00:07.0: reg 0x10: [io  0xc100-0xc11f]
[    0.732680] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
[    0.733936] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[    0.735062] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[    0.736109] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
[    0.737012] ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
[    0.738402] pci 0000:00:02.0: vgaarb: setting as boot VGA device
[    0.738805] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    0.738825] pci 0000:00:02.0: vgaarb: bridge control possible
[    0.739819] vgaarb: loaded
[    0.740943] pps_core: LinuxPPS API ver. 1 registered
[    0.741818] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti 
[    0.742841] PTP clock support registered
[    0.743917] PCI: Using ACPI for IRQ routing
[    0.744822] PCI: pci_cache_line_size set to 64 bytes
[    0.745112] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff]
[    0.745118] e820: reserve RAM buffer [mem 0x3fffb000-0x3fffffff]
[    0.745693] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[    0.745884] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.746824] hpet0: 3 comparators, 64-bit 100.000000 MHz counter
[    0.751892] clocksource: Switched to clocksource kvm-clock
[    0.768793] VFS: Disk quotas dquot_6.6.0
[    0.771205] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.774890] pnp: PnP ACPI init
[    0.777241] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.777267] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active)
[    0.777283] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active)
[    0.777302] pnp 00:03: [dma 2]
[    0.777310] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active)
[    0.777388] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.777599] pnp: PnP ACPI: found 5 devices
[    0.788030] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.796986] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.796988] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.796989] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    0.796991] pci_bus 0000:00: resource 7 [mem 0x80000000-0xfebfffff window]
[    0.797116] NET: Registered protocol family 2
[    0.800605] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes)
[    0.806250] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[    0.810450] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[    0.814563] TCP: Hash tables configured (established 8192 bind 8192)
[    0.818494] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    0.821941] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    0.827063] NET: Registered protocol family 1
[    0.829882] NET: Registered protocol family 44
[    0.832675] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[    0.836400] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    0.840057] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[    0.843680] PCI Interrupt Link [LNKD] enabled at IRQ 11
[    0.847649] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    0.853296] PCI: CLS 0 bytes, default 64
[    1.496143] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x21134f58f0d, max_idle_ns: 440795217993 ns
[    1.503844] Initialise system trusted keyrings
[    1.508124] workingset: timestamp_bits=39 max_order=18 bucket_order=0
[    1.758294] Key type asymmetric registered
[    1.761449] Asymmetric key parser 'x509' registered
[    1.766939] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[    1.774415] io scheduler noop registered
[    1.778037] io scheduler deadline registered
[    1.781382] io scheduler cfq registered (default)
[    1.784513] io scheduler mq-deadline registered
[    1.787680] io scheduler kyber registered
[    1.792388] io scheduler bfq registered
[    1.795236] intel_idle: Please enable MWAIT in BIOS SETUP
[    1.795476] ioatdma: Intel(R) QuickData Technology Driver 4.00
[    1.799595] PCI Interrupt Link [LNKC] enabled at IRQ 10
[    1.802221] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
[    1.806042] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
[    1.810017] PCI Interrupt Link [LNKA] enabled at IRQ 10
[    1.812152] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
[    1.816254] PCI Interrupt Link [LNKB] enabled at IRQ 11
[    1.822850] virtio-pci 0000:00:06.0: virtio_pci: leaving for legacy driver
[    1.829917] virtio-pci 0000:00:07.0: virtio_pci: leaving for legacy driver
[    1.834834] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    1.863736] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.869110] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[    1.874991] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.877521] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.880972] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
[    1.888544] rtc_cmos 00:00: RTC can wake from S4
[    1.891674] rtc_cmos 00:00: registered as rtc0
[    1.894056] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram, hpet irqs
[    1.897523] NET: Registered protocol family 10
[    1.900680] Segment Routing with IPv6
[    1.903157] NET: Registered protocol family 17
[    1.906120] Key type dns_resolver registered
[    1.908924] mce: Using 10 MCE banks
[    1.911612] sched_clock: Marking stable (1497656423, 413877026)->(2374759917, -463226468)
[    1.919693] registered taskstats version 1
[    1.922651] Loading compiled-in X.509 certificates
[    1.961181] Loaded X.509 cert 'CoreOS, Inc: Module signing key for 4.19.78-coreos: 317d2ce0a4b0e4c89f765793a811eec6a79f49c3'
[    1.968101] ima: No TPM chip found, activating TPM-bypass!
[    1.971482] ima: Allocated hash algorithm: sha1
[    1.974601] rtc_cmos 00:00: setting system clock to 2019-11-09 00:42:25 UTC (1573260145)
[    1.991636] Freeing unused kernel image memory: 41484K
[    1.995869] Write protecting the kernel read-only data: 18432k
[    2.001096] Freeing unused kernel image memory: 2016K
[    2.004846] Freeing unused kernel image memory: 660K
[    2.009673] Run /init as init process
[    2.020731] systemd[1]: systemd 241-30-gf0da8f7+ running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
[    2.037372] systemd[1]: Detected virtualization kvm.
[    2.041044] systemd[1]: Detected architecture x86-64.
[    2.044064] systemd[1]: Running in initial RAM disk.
[    2.055325] systemd[1]: No hostname configured.
[    2.057911] systemd[1]: Set hostname to .
[    2.061098] systemd[1]: Initializing machine ID from KVM UUID.
[    2.065217] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument
[    2.210451] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
[    2.225098] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[    2.517723] device-mapper: uevent: version 1.0.3
[    2.525996] device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: dm-devel@redhat.com
[    3.029192] audit: type=1130 audit(1573260146.554:2): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-journald comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    3.184026] audit: type=1130 audit(1573260146.709:3): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udevd comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    3.339165] audit: type=1130 audit(1573260146.864:4): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-trigger comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    3.505341] audit: type=1130 audit(1573260147.030:5): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udev-trigger comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    3.736327] virtio_blk virtio3: [vda] 52428800 512-byte logical blocks (26.8 GB/25.0 GiB)
[    3.829074] cryptd: max_cpu_qlen set to 1000
[    3.846092] SCSI subsystem initialized
[    3.861043] ACPI: bus type USB registered
[    3.870092] scsi host0: Virtio SCSI HBA
[    3.890726]  vda: vda1 vda2 vda3 vda4 vda6 vda7 vda9
[    3.945008] usbcore: registered new interface driver usbfs
[    4.008157] audit: type=1130 audit(1573260147.533:6): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-networkd comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.025057] random: fast init done
[    4.025703] usbcore: registered new interface driver hub
[    4.025897] usbcore: registered new device driver usb
[    4.059473] AVX2 version of gcm_enc/dec engaged.
[    4.059474] AES CTR mode by8 optimization enabled
[    4.082058] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    4.092444] libata version 3.00 loaded.
[    4.095312] ehci-pci: EHCI PCI platform driver
[    4.119207] ata_piix 0000:00:01.1: version 2.13
[    4.126515] uhci_hcd: USB Universal Host Controller Interface driver
[    4.141319] scsi host1: ata_piix
[    4.170648] scsi host2: ata_piix
[    4.218017] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14
[    4.222520] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15
[    4.242356] uhci_hcd 0000:00:01.2: UHCI Host Controller
[    4.279387] random: veritysetup: uninitialized urandom read (2 bytes read)
[    4.326362] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
[    4.326434] uhci_hcd 0000:00:01.2: detected 2 ports
[    4.326629] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c080
[    4.346338] hub 1-0:1.0: USB hub found
[    4.346348] hub 1-0:1.0: 2 ports detected
[    4.417296] A link change request failed with some changes committed already. Interface eth1 may have been left with an inconsistent configuration, please check.
[    4.426257] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    4.426262] A link change request failed with some changes committed already. Interface eth0 may have been left with an inconsistent configuration, please check.
[    4.712728] audit: type=1130 audit(1573260148.237:7): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-initqueue comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.736898] audit: type=1130 audit(1573260148.255:8): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=verity-setup comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.815049] audit: type=1130 audit(1573260148.340:9): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-mount comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.859849] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    4.888709] audit: type=1130 audit(1573260148.413:10): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-fsck-root comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.979799] EXT4-fs (vda9): mounted filesystem with ordered data mode. Opts: (null)
[    5.036074] random: coreos-metadata: uninitialized urandom read (16 bytes read)
[    5.046906] random: reqwest-interna: uninitialized urandom read (32 bytes read)
[    5.097371] EXT4-fs (dm-0): mounted filesystem without journal. Opts: (null)
[   12.128019] kauditd_printk_skb: 4 callbacks suppressed
[   12.128020] audit: type=1130 audit(1573260155.653:15): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=coreos-digitalocean-network comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.186689] audit: type=1131 audit(1573260155.671:16): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=coreos-digitalocean-network comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.208656] audit: type=1130 audit(1573260155.733:17): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-pivot comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.264469] audit: type=1131 audit(1573260155.787:18): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-pivot comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.297936] audit: type=1131 audit(1573260155.823:19): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-mount comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.311244] audit: type=1131 audit(1573260155.836:20): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-initqueue comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.390163] audit: type=1131 audit(1573260155.915:21): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udev-trigger comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.423312] audit: type=1131 audit(1573260155.948:22): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-trigger comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.466490] audit: type=1130 audit(1573260155.986:23): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-cleanup comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   12.466493] audit: type=1131 audit(1573260155.986:24): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-cleanup comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   13.145934] systemd-journald[143]: Received SIGTERM from PID 1 (n/a).
[   13.193781] systemd: 28 output lines suppressed due to ratelimiting
[   13.363971] SELinux:  Permission validate_trans in class security not defined in policy.
[   13.370003] SELinux:  Permission getrlimit in class process not defined in policy.
[   13.376609] SELinux:  Class process2 not defined in policy.
[   13.381031] SELinux:  Permission module_load in class system not defined in policy.
[   13.386606] SELinux:  Permission map in class file not defined in policy.
[   13.392250] SELinux:  Permission map in class dir not defined in policy.
[   13.396583] SELinux:  Permission map in class lnk_file not defined in policy.
[   13.400174] SELinux:  Permission map in class chr_file not defined in policy.
[   13.403992] SELinux:  Permission map in class blk_file not defined in policy.
[   13.407838] SELinux:  Permission map in class sock_file not defined in policy.
[   13.412735] SELinux:  Permission map in class fifo_file not defined in policy.
[   13.418828] SELinux:  Permission map in class socket not defined in policy.
[   13.426682] SELinux:  Permission map in class tcp_socket not defined in policy.
[   13.433456] SELinux:  Permission map in class udp_socket not defined in policy.
[   13.439723] SELinux:  Permission map in class rawip_socket not defined in policy.
[   13.448026] SELinux:  Permission map in class netlink_socket not defined in policy.
[   13.457157] SELinux:  Permission map in class packet_socket not defined in policy.
[   13.462436] SELinux:  Permission map in class key_socket not defined in policy.
[   13.468121] SELinux:  Permission map in class unix_stream_socket not defined in policy.
[   13.473096] SELinux:  Permission map in class unix_dgram_socket not defined in policy.
[   13.478628] SELinux:  Permission map in class netlink_route_socket not defined in policy.
[   13.485004] SELinux:  Permission map in class netlink_tcpdiag_socket not defined in policy.
[   13.488469] SELinux:  Permission map in class netlink_nflog_socket not defined in policy.
[   13.493001] SELinux:  Permission map in class netlink_xfrm_socket not defined in policy.
[   13.497794] SELinux:  Permission map in class netlink_selinux_socket not defined in policy.
[   13.503468] SELinux:  Permission map in class netlink_iscsi_socket not defined in policy.
[   13.509465] SELinux:  Permission map in class netlink_audit_socket not defined in policy.
[   13.517472] SELinux:  Permission map in class netlink_fib_lookup_socket not defined in policy.
[   13.524126] SELinux:  Permission map in class netlink_connector_socket not defined in policy.
[   13.529969] SELinux:  Permission map in class netlink_netfilter_socket not defined in policy.
[   13.534701] SELinux:  Permission map in class netlink_dnrt_socket not defined in policy.
[   13.539544] SELinux:  Permission map in class netlink_kobject_uevent_socket not defined in policy.
[   13.549639] SELinux:  Permission map in class netlink_generic_socket not defined in policy.
[   13.555971] SELinux:  Permission map in class netlink_scsitransport_socket not defined in policy.
[   13.562031] SELinux:  Permission map in class netlink_rdma_socket not defined in policy.
[   13.567434] SELinux:  Permission map in class netlink_crypto_socket not defined in policy.
[   13.572056] SELinux:  Permission map in class appletalk_socket not defined in policy.
[   13.576993] SELinux:  Permission map in class dccp_socket not defined in policy.
[   13.583345] SELinux:  Permission map in class tun_socket not defined in policy.
[   13.588902] SELinux:  Class cap_userns not defined in policy.
[   13.592825] SELinux:  Class cap2_userns not defined in policy.
[   13.596205] SELinux:  Class sctp_socket not defined in policy.
[   13.600050] SELinux:  Class icmp_socket not defined in policy.
[   13.603712] SELinux:  Class ax25_socket not defined in policy.
[   13.607356] SELinux:  Class ipx_socket not defined in policy.
[   13.613402] SELinux:  Class netrom_socket not defined in policy.
[   13.618219] SELinux:  Class atmpvc_socket not defined in policy.
[   13.622539] SELinux:  Class x25_socket not defined in policy.
[   13.626643] SELinux:  Class rose_socket not defined in policy.
[   13.630095] SELinux:  Class decnet_socket not defined in policy.
[   13.633622] SELinux:  Class atmsvc_socket not defined in policy.
[   13.636927] SELinux:  Class rds_socket not defined in policy.
[   13.642280] SELinux:  Class irda_socket not defined in policy.
[   13.646084] SELinux:  Class pppox_socket not defined in policy.
[   13.649855] SELinux:  Class llc_socket not defined in policy.
[   13.653445] SELinux:  Class can_socket not defined in policy.
[   13.657673] SELinux:  Class tipc_socket not defined in policy.
[   13.663193] SELinux:  Class bluetooth_socket not defined in policy.
[   13.668370] SELinux:  Class iucv_socket not defined in policy.
[   13.673870] SELinux:  Class rxrpc_socket not defined in policy.
[   13.677235] SELinux:  Class isdn_socket not defined in policy.
[   13.680005] SELinux:  Class phonet_socket not defined in policy.
[   13.683483] SELinux:  Class ieee802154_socket not defined in policy.
[   13.686523] SELinux:  Class caif_socket not defined in policy.
[   13.689456] SELinux:  Class alg_socket not defined in policy.
[   13.692905] SELinux:  Class nfc_socket not defined in policy.
[   13.696133] SELinux:  Class vsock_socket not defined in policy.
[   13.700049] SELinux:  Class kcm_socket not defined in policy.
[   13.705354] SELinux:  Class qipcrtr_socket not defined in policy.
[   13.708121] SELinux:  Class smc_socket not defined in policy.
[   13.711840] SELinux:  Class infiniband_pkey not defined in policy.
[   13.716033] SELinux:  Class infiniband_endport not defined in policy.
[   13.719201] SELinux:  Class bpf not defined in policy.
[   13.722055] SELinux:  Class xdp_socket not defined in policy.
[   13.725529] SELinux: the above unknown classes and permissions will be allowed
[   13.730375] SELinux:  policy capability network_peer_controls=1
[   13.735578] SELinux:  policy capability open_perms=1
[   13.738300] SELinux:  policy capability extended_socket_class=0
[   13.741949] SELinux:  policy capability always_check_network=0
[   13.745520] SELinux:  policy capability cgroup_seclabel=0
[   13.748784] SELinux:  policy capability nnp_nosuid_transition=0
[   13.773588] systemd[1]: Successfully loaded SELinux policy in 451.966ms.
[   13.802321] systemd[1]: Relabelled /dev, /dev/shm, /run, /sys/fs/cgroup in 9.597ms.
[   13.809599] systemd[1]: systemd 241-30-gf0da8f7+ running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
[   13.824758] systemd[1]: Detected virtualization kvm.
[   13.828691] systemd[1]: Detected architecture x86-64.
[   13.836896] systemd[1]: Set hostname to .
[   13.845542] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument
[   13.977078] urandom_read: 1 callbacks suppressed
[   13.977080] random: lvm2-activation: uninitialized urandom read (4 bytes read)
[   17.156225] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
[   17.171667] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[   17.344590] kauditd_printk_skb: 10 callbacks suppressed
[   17.344591] audit: type=1131 audit(1573260160.869:35): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   17.374429] systemd[1]: initrd-switch-root.service: Succeeded.
[   17.379784] systemd[1]: Stopped Switch Root.
[   17.387356] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart.
[   17.405850] audit: type=1130 audit(1573260160.911:36): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=initrd-switch-root comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   17.475376] audit: type=1131 audit(1573260160.911:37): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=initrd-switch-root comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   17.475381] audit: type=1130 audit(1573260160.921:38): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   17.475384] audit: type=1131 audit(1573260160.922:39): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   17.664828] audit: type=1131 audit(1573260161.189:40): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-fsck-root comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   17.816025] random: systemd-random-: uninitialized urandom read (512 bytes read)
[   17.854055] audit: type=1131 audit(1573260161.379:41): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=verity-setup comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   17.935020] audit: type=1130 audit(1573260161.460:42): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=coreos-tmpfiles comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   17.974899] audit: type=1131 audit(1573260161.474:43): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=coreos-tmpfiles comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   17.974903] audit: type=1130 audit(1573260161.481:44): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-sysctl comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   18.424338] systemd-journald[510]: Received request to flush runtime journal from PID 1
[   19.399277] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[   19.483828] ACPI: Power Button [PWRF]
[   19.585421] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0
[   19.625712] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
[   19.956102] A link change request failed with some changes committed already. Interface eth1 may have been left with an inconsistent configuration, please check.
[   19.987040] A link change request failed with some changes committed already. Interface eth0 may have been left with an inconsistent configuration, please check.
[   20.188096] mousedev: PS/2 mouse device common for all mice
[   20.368375] random: lvm: uninitialized urandom read (4 bytes read)
[   20.430325] random: lvm: uninitialized urandom read (4 bytes read)
[   20.559995] EXT4-fs (vda6): mounted filesystem with ordered data mode. Opts: (null)
[   21.310707] random: mktemp: uninitialized urandom read (8 bytes read)
[   22.203072] random: dbus-daemon: uninitialized urandom read (12 bytes read)
[   29.844292] random: crng init done
[   33.926309] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   33.935859] Bridge firewalling registered
[   34.051014] Initializing XFRM netlink socket
[   34.177247] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[   34.253042] IPv6: ADDRCONF(NETDEV_UP): docker_gwbridge: link is not ready
[   36.621177] br0: renamed from ov-001000-g9bon
[   36.628624] vxlan0: renamed from vx-001000-g9bon
[   36.632668] br0: port 1(vxlan0) entered blocking state
[   36.635654] br0: port 1(vxlan0) entered disabled state
[   36.640618] device vxlan0 entered promiscuous mode
[   36.650943] br0: port 1(vxlan0) entered blocking state
[   36.654855] br0: port 1(vxlan0) entered forwarding state
[   36.746749] veth0: renamed from vetheb536de
[   36.753988] br0: port 2(veth0) entered blocking state
[   36.756432] br0: port 2(veth0) entered disabled state
[   36.765126] device veth0 entered promiscuous mode
[   36.772951] br0: port 2(veth0) entered blocking state
[   36.778320] br0: port 2(veth0) entered forwarding state
[   37.061016] eth0: renamed from veth0ea39b3
[   37.194076] docker_gwbridge: port 1(veth9c641ee) entered blocking state
[   37.199509] docker_gwbridge: port 1(veth9c641ee) entered disabled state
[   37.218520] device veth9c641ee entered promiscuous mode
[   37.239889] IPv6: ADDRCONF(NETDEV_UP): veth9c641ee: link is not ready
[   37.245582] docker_gwbridge: port 1(veth9c641ee) entered blocking state
[   37.252445] docker_gwbridge: port 1(veth9c641ee) entered forwarding state
[   37.257112] docker_gwbridge: port 1(veth9c641ee) entered disabled state
[   37.391026] eth1: renamed from vethe0c236d
[   37.396135] IPv6: ADDRCONF(NETDEV_CHANGE): veth9c641ee: link becomes ready
[   37.400294] docker_gwbridge: port 1(veth9c641ee) entered blocking state
[   37.403902] docker_gwbridge: port 1(veth9c641ee) entered forwarding state
[   37.417943] IPv6: ADDRCONF(NETDEV_CHANGE): docker_gwbridge: link becomes ready
[   37.522541] br0: renamed from ov-001002-0suyd
[   37.534524] vxlan0: renamed from vx-001002-0suyd
[   37.544607] br0: port 1(vxlan0) entered blocking state
[   37.549367] br0: port 1(vxlan0) entered disabled state
[   37.566940] device vxlan0 entered promiscuous mode
[   37.579990] br0: port 1(vxlan0) entered blocking state
[   37.584287] br0: port 1(vxlan0) entered forwarding state
[   37.617613] br0: renamed from ov-001001-ka7zk
[   37.628040] vxlan0: renamed from vx-001001-ka7zk
[   37.636606] br0: port 1(vxlan0) entered blocking state
[   37.640146] br0: port 1(vxlan0) entered disabled state
[   37.657942] device vxlan0 entered promiscuous mode
[   37.667112] br0: port 1(vxlan0) entered blocking state
[   37.670252] br0: port 1(vxlan0) entered forwarding state
[   37.756629] veth0: renamed from vethbe15515
[   37.766331] br0: port 2(veth0) entered blocking state
[   37.769654] br0: port 2(veth0) entered disabled state
[   37.794938] device veth0 entered promiscuous mode
[   37.827900] br0: port 2(veth0) entered blocking state
[   37.830899] br0: port 2(veth0) entered forwarding state
[   37.872086] veth0: renamed from vetha93d670
[   37.885031] br0: port 2(veth0) entered blocking state
[   37.889036] br0: port 2(veth0) entered disabled state
[   37.913935] device veth0 entered promiscuous mode
[   37.942964] br0: port 2(veth0) entered blocking state
[   37.946090] br0: port 2(veth0) entered forwarding state
[   38.069475] eth0: renamed from vethc42dadf
[   38.074029] br0: port 2(veth0) entered disabled state
[   38.118441] eth0: renamed from veth3dae14a
[   38.125021] br0: port 2(veth0) entered blocking state
[   38.128319] br0: port 2(veth0) entered forwarding state
[   38.807001] br0: port 2(veth0) entered disabled state
[   38.809246] br0: port 1(vxlan0) entered disabled state
[   38.813904] ov-001001-ka7zk: renamed from br0
[   38.816057] device veth0 left promiscuous mode
[   38.818236] ov-001001-ka7zk: port 2(veth0) entered disabled state
[   38.821375] device vxlan0 left promiscuous mode
[   38.823051] ov-001001-ka7zk: port 1(vxlan0) entered disabled state
[   38.829154] vx-001001-ka7zk: renamed from vxlan0
[   38.838113] vetha93d670: renamed from veth0
[   38.863070] veth3dae14a: renamed from eth0
[   38.868010] br0: port 2(veth0) entered disabled state
[   38.871258] br0: port 1(vxlan0) entered disabled state
[   38.884283] ov-001002-0suyd: renamed from br0
[   38.889939] device veth0 left promiscuous mode
[   38.892652] ov-001002-0suyd: port 2(veth0) entered disabled state
[   38.902098] device vxlan0 left promiscuous mode
[   38.905378] ov-001002-0suyd: port 1(vxlan0) entered disabled state
[   38.930911] vx-001002-0suyd: renamed from vxlan0
[   38.951560] vethbe15515: renamed from veth0
[   38.967096] vethc42dadf: renamed from eth0
[   42.062319] br0: renamed from ov-001002-0suyd
[   42.070555] vxlan0: renamed from vx-001002-0suyd
[   42.077711] br0: port 1(vxlan0) entered blocking state
[   42.079978] br0: port 1(vxlan0) entered disabled state
[   42.089902] device vxlan0 entered promiscuous mode
[   42.098943] br0: port 1(vxlan0) entered blocking state
[   42.101684] br0: port 1(vxlan0) entered forwarding state
[   42.134021] br0: renamed from ov-001001-ka7zk
[   42.142550] vxlan0: renamed from vx-001001-ka7zk
[   42.148479] br0: port 1(vxlan0) entered blocking state
[   42.151851] br0: port 1(vxlan0) entered disabled state
[   42.170927] device vxlan0 entered promiscuous mode
[   42.183970] br0: port 1(vxlan0) entered blocking state
[   42.187274] br0: port 1(vxlan0) entered forwarding state
[   42.258008] veth0: renamed from veth518e4fc
[   42.274049] br0: port 2(veth0) entered blocking state
[   42.278369] br0: port 2(veth0) entered disabled state
[   42.301718] device veth0 entered promiscuous mode
[   42.322007] br0: port 2(veth0) entered blocking state
[   42.325441] br0: port 2(veth0) entered forwarding state
[   42.381440] veth0: renamed from vethb441fa7
[   42.390995] br0: port 2(veth0) entered blocking state
[   42.394703] br0: port 2(veth0) entered disabled state
[   42.419949] device veth0 entered promiscuous mode
[   42.455566] br0: port 2(veth0) entered blocking state
[   42.459146] br0: port 2(veth0) entered forwarding state
[   42.553731] eth0: renamed from veth38f4811
[   42.557275] br0: port 2(veth0) entered disabled state
[   42.578364] eth0: renamed from vethf00a2c1
[   42.581351] br0: port 2(veth0) entered blocking state
[   42.585279] br0: port 2(veth0) entered forwarding state
[   42.753503] veth1: renamed from veth53a6a11
[   42.760109] br0: port 3(veth1) entered blocking state
[   42.764140] br0: port 3(veth1) entered disabled state
[   42.778016] device veth1 entered promiscuous mode
[   42.787267] br0: port 3(veth1) entered blocking state
[   42.792486] br0: port 3(veth1) entered forwarding state
[   42.851343] veth1: renamed from veth2e663f5
[   42.858976] br0: port 3(veth1) entered blocking state
[   42.862769] br0: port 3(veth1) entered disabled state
[   42.873106] device veth1 entered promiscuous mode
[   42.879099] br0: port 3(veth1) entered blocking state
[   42.886637] br0: port 3(veth1) entered forwarding state
[   42.919528] veth2: renamed from veth2e0ad97
[   42.951782] br0: port 4(veth2) entered blocking state
[   42.955681] br0: port 4(veth2) entered disabled state
[   42.969222] device veth2 entered promiscuous mode
[   43.026621] br0: port 4(veth2) entered blocking state
[   43.030765] br0: port 4(veth2) entered forwarding state
[   43.052357] docker_gwbridge: port 2(veth97994ff) entered blocking state
[   43.056607] docker_gwbridge: port 2(veth97994ff) entered disabled state
[   43.070184] device veth97994ff entered promiscuous mode
[   43.099902] br0: port 3(veth1) entered disabled state
[   43.105912] br0: port 3(veth1) entered disabled state
[   43.123911] br0: port 4(veth2) entered disabled state
[   43.132867] IPv6: ADDRCONF(NETDEV_UP): veth97994ff: link is not ready
[   43.349185] veth1: renamed from veth533a7aa
[   43.367877] br0: port 3(veth1) entered blocking state
[   43.372752] br0: port 3(veth1) entered disabled state
[   43.385195] device veth1 entered promiscuous mode
[   43.402888] br0: port 3(veth1) entered blocking state
[   43.406180] br0: port 3(veth1) entered forwarding state
[   43.443450] veth2: renamed from veth57d06ae
[   43.452959] br0: port 4(veth2) entered blocking state
[   43.456440] br0: port 4(veth2) entered disabled state
[   43.484927] device veth2 entered promiscuous mode
[   43.493702] docker_gwbridge: port 3(vethf6c8774) entered blocking state
[   43.498602] docker_gwbridge: port 3(vethf6c8774) entered disabled state
[   43.516657] device vethf6c8774 entered promiscuous mode
[   43.529905] br0: port 4(veth2) entered blocking state
[   43.533439] br0: port 4(veth2) entered forwarding state
[   43.552910] IPv6: ADDRCONF(NETDEV_UP): vethf6c8774: link is not ready
[   43.558841] docker_gwbridge: port 3(vethf6c8774) entered blocking state
[   43.563505] docker_gwbridge: port 3(vethf6c8774) entered forwarding state
[   43.811063] eth0: renamed from vethc4601a7
[   43.826995] br0: port 3(veth1) entered disabled state
[   43.836057] br0: port 4(veth2) entered disabled state
[   43.840141] docker_gwbridge: port 3(vethf6c8774) entered disabled state
[   43.854887] br0: port 3(veth1) entered blocking state
[   43.858658] br0: port 3(veth1) entered forwarding state
[   44.307722] eth0: renamed from veth470b051
[   44.319393] br0: port 3(veth1) entered blocking state
[   44.322889] br0: port 3(veth1) entered forwarding state
[   44.604887] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
[   44.625890] IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
[   44.636605] IPVS: ipvs loaded.
[   44.816096] eth0: renamed from veth5f6d95f
[   44.826192] br0: port 3(veth1) entered blocking state
[   44.831842] br0: port 3(veth1) entered forwarding state
[   44.861050] eth1: renamed from veth80c9275
[   44.869304] br0: port 4(veth2) entered blocking state
[   44.873258] br0: port 4(veth2) entered forwarding state
[   44.915722] eth2: renamed from vethc029b67
[   44.918563] IPv6: ADDRCONF(NETDEV_CHANGE): vethf6c8774: link becomes ready
[   44.929778] docker_gwbridge: port 3(vethf6c8774) entered blocking state
[   44.933795] docker_gwbridge: port 3(vethf6c8774) entered forwarding state
[   44.991744] eth1: renamed from vethf5e53d0
[   45.003446] br0: port 4(veth2) entered blocking state
[   45.006588] br0: port 4(veth2) entered forwarding state
[   45.036757] eth2: renamed from vethf31d8f0
[   45.045580] IPv6: ADDRCONF(NETDEV_CHANGE): veth97994ff: link becomes ready
[   45.052366] docker_gwbridge: port 2(veth97994ff) entered blocking state
[   45.056065] docker_gwbridge: port 2(veth97994ff) entered forwarding state
[   45.104525] IPVS: [rr] scheduler registered.
[641371.331352] systemd-cgroups invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
[641371.340376] systemd-cgroups cpuset=/ mems_allowed=0
[641371.342280] CPU: 0 PID: 19787 Comm: systemd-cgroups Not tainted 4.19.78-coreos #1
[641371.344578] Hardware name: DigitalOcean Droplet, BIOS 20171212 12/12/2017
[641371.347030] Call Trace:
[641371.348819]  dump_stack+0x5c/0x80
[641371.350525]  dump_header+0x6e/0x286
[641371.352117]  ? do_try_to_free_pages+0x2ec/0x370
[641371.354210]  oom_kill_process.cold.30+0xb/0x1cf
[641371.356335]  ? oom_badness+0x23/0x140
[641371.358158]  out_of_memory+0x1ba/0x470
[641371.359832]  __alloc_pages_slowpath+0xbd4/0xcb0
[641371.363256]  __alloc_pages_nodemask+0x294/0x2b0
[641371.365922]  filemap_fault+0x398/0x610
[641371.368248]  ? __wake_up_common_lock+0x89/0xc0
[641371.387568]  ? filemap_map_pages+0x1e8/0x350
[641371.391063]  ext4_filemap_fault+0x2c/0xab0 [ext4]
[641371.395100]  __do_fault+0x36/0xf0
[641371.398174]  __handle_mm_fault+0xe25/0x11d0
[641371.401598]  handle_mm_fault+0xd6/0x200
[641371.405524]  __do_page_fault+0x233/0x4c0
[641371.409015]  ? async_page_fault+0x8/0x30
[641371.412853]  async_page_fault+0x1e/0x30
[641371.417158] RIP: 0033:0x7f05bf39f474
[641371.420008] Code: Bad RIP value.
[641371.422174] RSP: 002b:00007fff3cac0f18 EFLAGS: 00010206
[641371.424780] RAX: 00007f05bf39f474 RBX: 00007f05bf850000 RCX: 0000000000000000
[641371.429916] RDX: 00007f05bf5947d8 RSI: 0000000000000000 RDI: 00007f05bf3b4148
[641371.435258] RBP: 00007fff3cac1060 R08: 00007f05bf3b4000 R09: 0000000000000001
[641371.440328] R10: 00007fff3cac0e66 R11: 00007fff3cac0f38 R12: 0000000000000000
[641371.444563] R13: 00007fff3cac0f40 R14: 000000000000001d R15: 00007f05bf8253e0
[641371.450424] Mem-Info:
[641371.452030] active_anon:99727 inactive_anon:38392 isolated_anon:0
                 active_file:164 inactive_file:327 isolated_file:27
                 unevictable:0 dirty:0 writeback:0 unstable:0
                 slab_reclaimable:14549 slab_unreclaimable:75984
                 mapped:22142 shmem:56899 pagetables:1667 bounce:0
                 free:12166 free_pcp:62 free_cma:0
[641371.471312] Node 0 active_anon:398908kB inactive_anon:153568kB active_file:656kB inactive_file:1308kB unevictable:0kB isolated(anon):0kB isolated(file):108kB mapped:88568kB dirty:0kB writeback:0kB shmem:227596kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 159744kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[641371.504210] Node 0 DMA free:4448kB min:740kB low:924kB high:1108kB active_anon:6772kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[641371.516810] lowmem_reserve[]: 0 927 927 927
[641371.519205] Node 0 DMA32 free:44216kB min:44312kB low:55388kB high:66464kB active_anon:392136kB inactive_anon:153568kB active_file:656kB inactive_file:1308kB unevictable:0kB writepending:0kB present:1032172kB managed:994072kB mlocked:0kB kernel_stack:3440kB pagetables:6664kB bounce:0kB free_pcp:248kB local_pcp:248kB free_cma:0kB
[641371.535415] lowmem_reserve[]: 0 0 0 0
[641371.540051] Node 0 DMA: 34*4kB (UME) 23*8kB (ME) 12*16kB (UM) 5*32kB (ME) 9*64kB (ME) 5*128kB (ME) 2*256kB (M) 4*512kB (ME) 0*1024kB 0*2048kB 0*4096kB = 4448kB
[641371.549107] Node 0 DMA32: 188*4kB (UMEH) 53*8kB (UE) 204*16kB (UMEH) 109*32kB (UMEH) 39*64kB (UMEH) 22*128kB (MEH) 11*256kB (ME) 23*512kB (ME) 10*1024kB (ME) 3*2048kB (UM) 0*4096kB = 44216kB
[641371.556585] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[641371.561575] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[641371.567074] 57415 total pagecache pages
[641371.571566] 0 pages in swap cache
[641371.574207] Swap cache stats: add 0, delete 0, find 0/0
[641371.577105] Free swap  = 0kB
[641371.579251] Total swap = 0kB
[641371.581273] 262041 pages RAM
[641371.583510] 0 pages HighMem/MovableOnly
[641371.586097] 9546 pages reserved
[641371.588821] 0 pages hwpoisoned
[641371.591759] Tasks state (memory values in pages):
[641371.595938] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[641371.604823] [    510]     0   510   115368     3925   958464        0             0 systemd-journal
[641371.611820] [    543]     0   543     5534      325    81920        0         -1000 systemd-udevd
[641371.619417] [    546]   244   546     3272      202    73728        0             0 systemd-network
[641371.627312] [    600]   245   600     3336      250    73728        0             0 systemd-resolve
[641371.636212] [    601]   997   601    21560      173    73728        0             0 systemd-timesyn
[641371.641827] [    629]     0   629   120482     1616   139264        0             0 authz-broker
[641371.647695] [    641]     0   641     3347      312    61440        0             0 systemd-logind
[641371.653817] [    642]   201   642     4122      383    57344        0          -900 dbus-daemon
[641371.663041] [    643]     0   643   166082    10011   241664        0          -999 containerd
[641371.676240] [    644]     0   644    43284      705   102400        0             0 update_engine
[641371.683325] [    678]     0   678   120891      972   131072        0             0 locksmithd
[641371.691112] [    698]     0   698     1156       35    49152        0             0 agetty
[641371.696774] [    699]     0   699     1156       36    45056        0             0 agetty
[641371.705766] [    720]     0   720   190506    24769   385024        0          -500 dockerd
[641371.711148] [   1324]     0  1324     1877     1151    73728        0          -999 containerd-shim
[641371.718018] [   1339]   999  1339   365876    24353   544768        0             0 mysqld
[641371.724467] [   1399]     0  1399     1893     1082    73728        0          -999 containerd-shim
[641371.732127] [   1424]     0  1424    40861     3471   208896        0             0 traefik
[641371.738440] [   1435]     0  1435     1877     1125    69632        0          -999 containerd-shim
[641371.743684] [   1460]     0  1460    59605     2722   217088        0             0 apache2
[641371.748171] [   1924]    33  1924    79241     8951   274432        0             0 apache2
[641371.752652] [   1926]    33  1926    79230     8221   274432        0             0 apache2
[641371.759197] [   1927]    33  1927    79229     8650   274432        0             0 apache2
[641371.768629] [   1928]    33  1928    78704     8078   274432        0             0 apache2
[641371.777474] [   2368]    33  2368    79216     8591   274432        0             0 apache2
[641371.785751] [   2373]    33  2373    79251     8656   274432        0             0 apache2
[641371.793755] [  14436]    33 14436    78931     8167   270336        0             0 apache2
[641371.808078] [  11125]    33 11125    79142     8102   270336        0             0 apache2
[641371.813296] [  11126]    33 11126    59966     7128   253952        0             0 apache2
[641371.819783] [  11127]    33 11127    78455     7257   266240        0             0 apache2
[641371.827450] [  18451]   500 18451     3544      296    69632        0             0 systemd
[641371.832928] [  18458]   500 18458    10522      730    86016        0             0 (sd-pam)
[641371.839722] [  19637]     0 19637     3052      136    61440        0             0 systemd-cgroups
[641371.845323] [  19638]     0 19638     3052      136    61440        0             0 systemd-cgroups
[641371.849739] [  19644]     0 19644     3052      136    65536        0             0 systemd-cgroups
[641371.856621] [  19645]     0 19645     3052      135    61440        0             0 systemd-cgroups
[641371.863052] [  19652]     0 19652     3052      137    61440        0             0 systemd-cgroups
[641371.869110] [  19653]     0 19653     3052      136    61440        0             0 systemd-cgroups
[641371.874091] [  19654]     0 19654     3052      137    61440        0             0 systemd-cgroups
[641371.878875] [  19683]     0 19683     3052      137    65536        0             0 systemd-cgroups
[641371.886878] [  19691]     0 19691     3052      136    61440        0             0 systemd-cgroups
[641371.893400] [  19696]     0 19696     3052      136    65536        0             0 systemd-cgroups
[641371.899806] [  19697]     0 19697     3052      135    61440        0             0 systemd-cgroups
[641371.906792] [  19698]     0 19698     3027      129    57344        0             0 systemd-cgroups
[641371.912831] [  19731]     0 19731     3052      136    61440        0             0 systemd-cgroups
[641371.921273] [  19748]     0 19748     3052      137    61440        0             0 systemd-cgroups
[641371.927329] [  19749]     0 19749     3052      136    61440        0             0 systemd-cgroups
[641371.932772] [  19750]     0 19750     3052      137    61440        0             0 systemd-cgroups
[641371.938293] [  19759]     0 19759     3052      135    65536        0             0 systemd-cgroups
[641371.942697] [  19780]     0 19780     3052      136    61440        0             0 systemd-cgroups
[641371.947002] [  19787]     0 19787     3052      137    65536        0             0 systemd-cgroups
[641371.951612] [  19850]     0 19850     1915      138    53248        0             0 sshd
[641371.958532] [  19876]     0 19876     1857      100    45056        0             0 sshd
[641371.964510] [  19887]     0 19887     3052      136    65536        0             0 systemd-cgroups
[641371.970200] [  19902]     0 19902     3052      135    65536        0             0 systemd-cgroups
[641371.976412] [  19903]     0 19903     3052      135    65536        0             0 systemd-cgroups
[641371.981944] [  19952]     0 19952     1857       63    53248        0             0 sshd
[641371.989203] [  20000]     0 20000     1857       64    53248        0             0 sshd
[641371.994926] [  20013]     0 20013     3027       80    53248        0             0 systemd-cgroups
[641372.000221] [  20058]     0 20058     1857       47    53248        0             0 sshd
[641372.006377] [  20091]     0 20091     1855       42    49152        0             0 sshd
[641372.012483] [  20103]     0 20103     2567      993    49152        0          -500 docker-runc
[641372.022187] Out of memory: Kill process 1339 (mysqld) score 96 or sacrifice child
[641372.028193] Killed process 1339 (mysqld) total-vm:1463504kB, anon-rss:97412kB, file-rss:0kB, shmem-rss:0kB
[641372.037535] oom_reaper: reaped process 1339 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[641374.306339] br0: port 3(veth1) entered disabled state
[641374.313350] vethc4601a7: renamed from eth0
[641374.780573] br0: port 3(veth1) entered disabled state
[641374.803014] device veth1 left promiscuous mode
[641374.806182] br0: port 3(veth1) entered disabled state
[641380.979375] veth3: renamed from veth3b3ccb1
[641380.991116] br0: port 3(veth3) entered blocking state
[641380.996282] br0: port 3(veth3) entered disabled state
[641381.010404] device veth3 entered promiscuous mode
[641381.891574] eth0: renamed from veth4a6c895
[641381.910842] br0: port 3(veth3) entered blocking state
[641381.914001] br0: port 3(veth3) entered forwarding state
@bgilbert

This comment has been minimized.

Copy link
Member

@bgilbert bgilbert commented Nov 19, 2019

Thanks for the report. Is there a previous Container Linux version where you weren't seeing this problem?

@peat-psuwit

This comment has been minimized.

Copy link
Author

@peat-psuwit peat-psuwit commented Nov 19, 2019

Is there a previous Container Linux version where you weren't seeing this problem?

I can't remember any. IIRC I've seen this problem since I initially set up the VM since July/August.

@a2yp

This comment has been minimized.

Copy link

@a2yp a2yp commented Nov 20, 2019

We have the same issue with some Container Linux VMs on VMware.
The following data is from a test server running nothing except one monitoring container.

# uptime

16:26:30 up 7 days, 5:50, 1 user, load average: 0.00, 0.00, 0.00

# cat /proc/meminfo

MemTotal:        2017520 kB
MemFree:          213812 kB
MemAvailable:     315436 kB
Buffers:            6128 kB
Cached:           306624 kB
SwapCached:            0 kB
Active:           289840 kB
Inactive:         179560 kB
Active(anon):     215396 kB
Inactive(anon):   145472 kB
Active(file):      74444 kB
Inactive(file):    34088 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               176 kB
Writeback:             0 kB
AnonPages:        156660 kB
Mapped:           176820 kB
Shmem:            204636 kB
Slab:             756624 kB
SReclaimable:     178672 kB
SUnreclaim:       577952 kB
KernelStack:        4748 kB
PageTables:         2240 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1008760 kB
Committed_AS:     804088 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:           557056 kB
HardwareCorrupted:     0 kB
AnonHugePages:     63488 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     1124224 kB
DirectMap2M:      972800 kB
DirectMap1G:           0 kB

# slabtop --sort c -o

 Active / Total Objects (% used)    : 4220674 / 4234894 (99.7%)
 Active / Total Slabs (% used)      : 59399 / 59399 (100.0%)
 Active / Total Caches (% used)     : 101 / 127 (79.5%)
 Active / Total Size (% used)       : 747876.32K / 752120.89K (99.4%)
 Minimum / Average / Maximum Object : 0.01K / 0.18K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
538734 538398  99%    0.19K  12827       42    102616K cred_jar               
132138 131722  99%    0.59K   2447       54     78304K inode_cache            
153856 153815  99%    0.50K   2404       64     76928K kmalloc-512            
 97704  97659  99%    0.69K   2124       46     67968K sock_inode_cache       
 67936  67936 100%    1.00K   2123       32     67936K UNIX                   
 63150  63150 100%    1.06K   2105       30     67360K mm_struct              
139136 138753  99%    0.25K   2174       64     34784K filp                   
135936 135680  99%    0.25K   2124       64     33984K skbuff_head_cache      
 15008  14917  99%    2.00K    938       16     30016K kmalloc-2048           
411840 411338  99%    0.06K   6435       64     25740K anon_vma_chain         
114492 111382  97%    0.19K   2726       42     21808K dentry                 
 47586  47586 100%    0.38K   1133       42     18128K kmem_cache             
198858 198858 100%    0.09K   4323       46     17292K anon_vma               
136128 136128 100%    0.12K   2127       64     17016K pid                    
124740 124740 100%    0.13K   2079       60     16632K kernfs_node_cache      
450048 450048 100%    0.03K   3516      128     14064K kmalloc-32             
  2488   2488 100%    4.00K    311        8      9952K kmalloc-4096           
1086976 1086976 100%    0.01K   2123      512      8492K kmalloc-8              
  5984   5984 100%    1.00K    187       32      5984K kmalloc-1024           
 55524  55484  99%    0.09K   1322       42      5288K kmalloc-96             
 19194  17341  90%    0.19K    457       42      3656K kmalloc-192            
  6384   5493  86%    0.57K    114       56      3648K radix_tree_node        
 14336  14336 100%    0.25K    224       64      3584K kmalloc-256            
   384    363  94%    7.50K     96        4      3072K task_struct            
 47616  47616 100%    0.06K    744       64      2976K kmem_cache_node        
  2610   1924  73%    1.05K     87       30      2784K ext4_inode_cache       
  2880   2391  83%    0.66K     60       48      1920K proc_inode_cache       
 14016  14016 100%    0.12K    219       64      1752K kmalloc-128            
 24064  23619  98%    0.06K    376       64      1504K kmalloc-64             
 14547  14161  97%    0.10K    373       39      1492K buffer_head            
  1978   1426  72%    0.69K     43       46      1376K shmem_inode_cache      
 27948  24742  88%    0.04K    274      102      1096K numa_policy            
   405    405 100%    2.06K     27       15       864K sighand_cache          
   660    660 100%    1.06K     22       30       704K signal_cache           
 32256  32256 100%    0.02K    126      256       504K kmalloc-16             
    60     42  70%    8.00K     15        4       480K kmalloc-8192           
  6384   6384 100%    0.07K    114       56       456K Acpi-Operand           
 19210  19210 100%    0.02K    113      170       452K avtab_node             
    56     42  75%    8.00K     14        4       448K biovec-max             
   598    598 100%    0.69K     13       46       416K files_cache            
   528    528 100%    0.66K     11       48       352K ovl_inode              
   306    306 100%    0.94K      9       34       288K RAW                    
   672    672 100%    0.38K     16       42       256K mnt_cache              
  3520   3520 100%    0.06K     55       64       220K ebitmap_node           
  1440   1440 100%    0.13K     24       60       192K dm_bufio_buffer-4      
   140    140 100%    1.12K      5       28       160K RAWv6                  
  3145   3145 100%    0.05K     37       85       148K ftrace_event_field     
   714    714 100%    0.19K     17       42       136K proc_dir_entry         
    60     60 100%    2.12K      4       15       128K TCP                    
   100    100 100%    1.25K      4       25       128K UDPv6                  
  1288   1288 100%    0.09K     28       46       112K trace_event_file       
   320    320 100%    0.25K      5       64        80K pool_workqueue         
   640    640 100%    0.12K     10       64        80K scsi_sense_cache       
    84     84 100%    0.75K      2       42        64K task_group             
    16     16 100%    4.00K      2        8        64K names_cache            
    78     47  60%    0.81K      2       39        64K bdev_cache             
    28     28 100%    2.25K      2       14        64K TCPv6                  
    56     56 100%    1.12K      2       28        64K PINGv6                 
    92     92 100%    0.69K      2       46        64K bio-2                  
   840    840 100%    0.07K     15       56        60K eventpoll_pwq          
   392    392 100%    0.14K      7       56        56K ext4_groupinfo_4k      
  1224   1224 100%    0.04K     12      102        48K pde_opener             
  1224    816  66%    0.04K     12      102        48K ext4_extent_status     
    64     64 100%    0.50K      1       64        32K dma-kmalloc-512        
    64     64 100%    0.50K      1       64        32K skbuff_fclone_cache    
    30     30 100%    1.06K      1       30        32K dmaengine-unmap-128    
    15     15 100%    2.06K      1       15        32K dmaengine-unmap-256    
    32     32 100%    1.00K      1       32        32K biovec-64              
    16     16 100%    2.00K      1       16        32K biovec-128             
   104    104 100%    0.30K      2       52        32K blkdev_requests        
    13     13 100%    2.40K      1       13        32K request_queue          
   408    408 100%    0.08K      8       51        32K inotify_inode_mark     
    52     52 100%    0.61K      1       52        32K hugetlbfs_inode_cache  
    51     51 100%    0.62K      1       51        32K dio                    
    34     34 100%    0.94K      1       34        32K mqueue_inode_cache     
     8      8 100%    4.00K      1        8        32K sgpool-128             
   272    272 100%    0.12K      4       68        32K jbd2_journal_head      
    44     44 100%    0.72K      1       44        32K fat_inode_cache        
   102    102 100%    0.31K      2       51        32K nf_conntrack           
    37     37 100%    0.43K      1       37        16K uts_namespace          
  1024   1024 100%    0.02K      4      256        16K selinux_file_security  
    49     49 100%    0.32K      1       49        16K taskstats              
    64     64 100%    0.25K      1       64        16K dquot                  
    53     53 100%    0.30K      1       53        16K request_sock_TCP       
    68     68 100%    0.23K      1       68        16K tw_sock_TCP            
   102    102 100%    0.08K      2       51         8K Acpi-State             
   146    146 100%    0.05K      2       73         8K Acpi-Parse             
    51     51 100%    0.16K      1       51         8K sigqueue               
    40     40 100%    0.20K      1       40         8K file_lock_cache        
    42     42 100%    0.19K      1       42         8K dmaengine-unmap-16     
    78     78 100%    0.10K      2       39         8K blkdev_ioc             
    64     64 100%    0.12K      1       64         8K secpath_cache          
    39     39 100%    0.20K      1       39         8K ip4-frags              
   146    146 100%    0.05K      2       73         8K mbcache                
    64     64 100%    0.12K      1       64         8K ext4_allocation_context
   128    128 100%    0.03K      1      128         4K avc_xperms_data        
    85     85 100%    0.05K      1       85         4K fscrypt_ctx            
   128    128 100%    0.03K      1      128         4K fscrypt_info           
   256    256 100%    0.02K      1      256         4K jbd2_revoke_table_s    
    64     64 100%    0.06K      1       64         4K ext4_io_end            
   102    102 100%    0.04K      1      102         4K fat_cache              
     0      0   0%    0.09K      0       42         0K dma-kmalloc-96         
     0      0   0%    0.19K      0       42         0K dma-kmalloc-192        
     0      0   0%    0.01K      0      512         0K dma-kmalloc-8          
     0      0   0%    0.02K      0      256         0K dma-kmalloc-16         
     0      0   0%    0.03K      0      128         0K dma-kmalloc-32         
     0      0   0%    0.06K      0       64         0K dma-kmalloc-64         
     0      0   0%    0.12K      0       64         0K dma-kmalloc-128        
     0      0   0%    0.25K      0       64         0K dma-kmalloc-256        
     0      0   0%    1.00K      0       32         0K dma-kmalloc-1024       
     0      0   0%    2.00K      0       16         0K dma-kmalloc-2048       
     0      0   0%    4.00K      0        8         0K dma-kmalloc-4096       
     0      0   0%    8.00K      0        4         0K dma-kmalloc-8192       
     0      0   0%    0.12K      0       68         0K iint_cache             
     0      0   0%    5.50K      0        5         0K net_namespace          
     0      0   0%    0.45K      0       70         0K user_namespace         
     0      0   0%    0.94K      0       34         0K PING                   
     0      0   0%    0.31K      0       51         0K xfrm_dst_cache         
     0      0   0%    0.23K      0       68         0K posix_timers_cache     
     0      0   0%    0.03K      0      128         0K dnotify_struct         
     0      0   0%    0.12K      0       68         0K cfq_io_cq              
     0      0   0%    0.44K      0       36         0K bfq_queue              
     0      0   0%    0.30K      0       53         0K request_sock_TCPv6     
     0      0   0%    0.23K      0       68         0K tw_sock_TCPv6          
     0      0   0%    2.57K      0       12         0K dm_uevent              
     0      0   0%    3.23K      0        9         0K kcopyd_job             
     0      0   0%    0.21K      0       37         0K nf_conntrack_expect 

Thanks for the report. Is there a previous Container Linux version where you weren't seeing this problem?

@bgilbert: The problems started when we upgraded from 2135.6.0 to 2191.4.1.

Bildschirmfoto 2019-11-20 um 16 46 40

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.