[ 0.000000] Linux version 4.9.27-14.31.amzn1.x86_64 (mockbuild@gobi-build-60006) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Wed May 10 01:58:40 UTC 2017 [ 0.000000] Command line: root=LABEL=/ console=tty1 console=ttyS0 selinux=0 LANG=en_US.UTF-8 KEYTABLE=us [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] x86/fpu: Using 'eager' FPU context switches. [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000efffffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000001efffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] DMI: Xen HVM domU, BIOS 4.2.amazon 11/11/2016 [ 0.000000] Hypervisor detected: Xen [ 0.000000] Xen version 4.2. [ 0.000000] Xen Platform PCI: I/O protocol version 1 [ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs. [ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks. You might have to change the root device from /dev/hd[a-d] to /dev/xvd[a-d] in your root= kernel command line option [ 0.000000] HVMOP_pagetable_dying not supported [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x1f0000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF write-combining [ 0.000000] C0000-FFFFF write-back [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000F0000000 mask 3FFFF8000000 uncachable [ 0.000000] 1 base 0000F8000000 mask 3FFFFC000000 uncachable [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT [ 0.000000] e820: last_pfn = 0xf0000 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000fbc80-0x000fbc8f] mapped at [ffff8800000fbc80] [ 0.000000] Base memory trampoline at [ffff880000098000] 98000 size 24576 [ 0.000000] BRK [0x01f66000, 0x01f66fff] PGTABLE [ 0.000000] BRK [0x01f67000, 0x01f67fff] PGTABLE [ 0.000000] BRK [0x01f68000, 0x01f68fff] PGTABLE [ 0.000000] BRK [0x01f69000, 0x01f69fff] PGTABLE [ 0.000000] BRK [0x01f6a000, 0x01f6afff] PGTABLE [ 0.000000] BRK [0x01f6b000, 0x01f6bfff] PGTABLE [ 0.000000] RAMDISK: [mem 0x3721a000-0x37feffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000EA020 000024 (v02 Xen ) [ 0.000000] ACPI: XSDT 0x00000000FC00DDC0 000054 (v01 Xen HVM 00000000 HVML 00000000) [ 0.000000] ACPI: FACP 0x00000000FC00DA80 0000F4 (v04 Xen HVM 00000000 HVML 00000000) [ 0.000000] ACPI: DSDT 0x00000000FC001CE0 00BD19 (v02 Xen HVM 00000000 INTL 20090123) [ 0.000000] ACPI: FACS 0x00000000FC001CA0 000040 [ 0.000000] ACPI: FACS 0x00000000FC001CA0 000040 [ 0.000000] ACPI: APIC 0x00000000FC00DB80 0000D8 (v02 Xen HVM 00000000 HVML 00000000) [ 0.000000] ACPI: HPET 0x00000000FC00DCD0 000038 (v01 Xen HVM 00000000 HVML 00000000) [ 0.000000] ACPI: WAET 0x00000000FC00DD10 000028 (v01 Xen HVM 00000000 HVML 00000000) [ 0.000000] ACPI: SSDT 0x00000000FC00DD40 000031 (v02 Xen HVM 00000000 INTL 20090123) [ 0.000000] ACPI: SSDT 0x00000000FC00DD80 000031 (v02 Xen HVM 00000000 INTL 20090123) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x00000001efffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1effda000-0x1efffbfff] [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x00000001efffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009dfff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000efffffff] [ 0.000000] node 0: [mem 0x0000000100000000-0x00000001efffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x00000001efffffff] [ 0.000000] On node 0 totalpages: 1965981 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3997 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 15296 pages used for memmap [ 0.000000] DMA32 zone: 978944 pages, LIFO batch:31 [ 0.000000] Normal zone: 15360 pages used for memmap [ 0.000000] Normal zone: 983040 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-47 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 low level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 low level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 low level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 15 CPUs, 13 hotplug CPUs [ 0.000000] e820: [mem 0xf0000000-0xfbffffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on Xen HVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns [ 0.000000] setup_percpu: NR_CPUS:128 nr_cpumask_bits:128 nr_cpu_ids:15 nr_node_ids:1 [ 0.000000] percpu: Embedded 34 pages/cpu @ffff8801e7a00000 s101720 r8192 d29352 u262144 [ 0.000000] pcpu-alloc: s101720 r8192 d29352 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 08 09 10 11 12 13 14 -- [ 0.000000] xen: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1935240 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: root=LABEL=/ console=tty1 console=ttyS0 selinux=0 LANG=en_US.UTF-8 KEYTABLE=us [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] Memory: 7642792K/7863924K available (5344K kernel code, 1108K rwdata, 2440K rodata, 1256K init, 3124K bss, 221132K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=15, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] Build-time adjustment of leaf fanout to 64. [ 0.000000] RCU restricting CPUs from NR_CPUS=128 to nr_cpu_ids=15. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=15 [ 0.000000] NR_IRQS:8448 nr_irqs:952 16 [ 0.000000] xen:events: Using 2-level ABI [ 0.000000] xen:events: Xen HVM callback vector for event delivery is enabled [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty1] enabled [ 0.000000] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22! [ 0.000000] console [ttyS0] enabled [ 0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 30580167144 ns [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 2499.964 MHz processor [57678128.226941] Calibrating delay loop (skipped), value calculated using timer frequency.. 5000.18 BogoMIPS (lpj=10000368) [57678128.486394] pid_max: default: 32768 minimum: 301 [57678128.490504] ACPI: Core revision 20160831 [57678128.499297] ACPI: 3 ACPI AML tables successfully acquired and loaded [57678128.504779] Security Framework initialized [57678128.508900] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes) [57678128.517193] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes) [57678128.523436] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes) [57678128.528391] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes) [57678128.534712] CPU: Physical Processor ID: 0 [57678128.538359] mce: CPU supports 2 MCE banks [57678128.542068] Last level iTLB entries: 4KB 512, 2MB 8, 4MB 8 [57678128.546913] Last level dTLB entries: 4KB 512, 2MB 0, 4MB 0, 1GB 4 [57678128.553103] Freeing SMP alternatives memory: 24K (ffffffff81c51000 - ffffffff81c57000) [57678128.563800] ftrace: allocating 21600 entries in 85 pages [57678128.589573] smpboot: x86_max_cores == zero !?!? [57678128.593834] smpboot: Max logical packages: 15 [57678128.599823] x2apic: IRQ remapping doesn't support X2APIC mode [57678128.606047] Switched APIC routing to physical flat. [57678128.613244] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0 [57678128.658515] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [57678128.666869] Xen: using vcpuop timer interface [57678128.666878] installing Xen timer for CPU 0 [57678128.670342] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz (family: 0x6, model: 0x3e, stepping: 0x4) [57678128.679736] cpu 0 spinlock event irq 53 [57678128.682959] Performance Events: unsupported p6 CPU model 62 no PMU driver, software events only. [57678128.691678] installing Xen timer for CPU 1 [57678128.695353] x86: Booting SMP configuration: [57678128.698967] .... node #0, CPUs: #1 [57678128.785010] cpu 1 spinlock event irq 59 [57678128.790165] x86: Booted up 1 node, 2 CPUs [57678128.794583] smpboot: Total of 2 processors activated (10071.77 BogoMIPS) [57678128.800497] devtmpfs: initialized [57678128.803575] x86/mm: Memory block size: 128MB [57678128.810061] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [57678128.819307] futex hash table entries: 4096 (order: 6, 262144 bytes) [57678128.825727] NET: Registered protocol family 16 [57678128.845729] cpuidle: using governor ladder [57678128.857912] cpuidle: using governor menu [57678128.861809] ACPI: bus type PCI registered [57678128.866197] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [57678128.872628] PCI: Using configuration type 1 for base access [57678128.887348] HugeTLB registered 2 MB page size, pre-allocated 0 pages [57678128.893739] ACPI: Added _OSI(Module Device) [57678128.897746] ACPI: Added _OSI(Processor Device) [57678128.901367] ACPI: Added _OSI(3.0 _SCP Extensions) [57678128.905324] ACPI: Added _OSI(Processor Aggregator Device) [57678128.910133] xen: --> pirq=16 -> irq=9 (gsi=9) [57678128.913297] ACPI: Interpreter enabled [57678128.917174] ACPI: (supports S0 S3 S5) [57678128.921163] ACPI: Using IOAPIC for interrupt routing [57678128.926086] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [57678128.935657] random: fast init done [57678128.991370] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [57678128.996886] acpi PNP0A03:00: _OSC: OS supports [Segments MSI] [57678129.001681] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [57678129.007231] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [57678129.017307] acpiphp: Slot [0] registered [57678129.022120] acpiphp: Slot [3] registered [57678129.041151] acpiphp: Slot [4] registered [57678129.044265] acpiphp: Slot [5] registered [57678129.047256] acpiphp: Slot [6] registered [57678129.050197] acpiphp: Slot [7] registered [57678129.053166] acpiphp: Slot [8] registered [57678129.056169] acpiphp: Slot [9] registered [57678129.059174] acpiphp: Slot [10] registered [57678129.062173] acpiphp: Slot [11] registered [57678129.065220] acpiphp: Slot [12] registered [57678129.068222] acpiphp: Slot [13] registered [57678129.071223] acpiphp: Slot [14] registered [57678129.074253] acpiphp: Slot [15] registered [57678129.077280] acpiphp: Slot [16] registered [57678129.080350] acpiphp: Slot [17] registered [57678129.083408] acpiphp: Slot [18] registered [57678129.086423] acpiphp: Slot [19] registered [57678129.089431] acpiphp: Slot [20] registered [57678129.092431] acpiphp: Slot [21] registered [57678129.095492] acpiphp: Slot [22] registered [57678129.098491] acpiphp: Slot [23] registered [57678129.101474] acpiphp: Slot [24] registered [57678129.104489] acpiphp: Slot [25] registered [57678129.107520] acpiphp: Slot [26] registered [57678129.110506] acpiphp: Slot [27] registered [57678129.113576] acpiphp: Slot [28] registered [57678129.116577] acpiphp: Slot [29] registered [57678129.119597] acpiphp: Slot [30] registered [57678129.122580] acpiphp: Slot [31] registered [57678129.125918] PCI host bridge to bus 0000:00 [57678129.128775] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [57678129.132837] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [57678129.136971] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [57678129.141725] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfbffffff window] [57678129.146690] pci_bus 0000:00: root bus resource [bus 00-ff] [57678129.150252] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [57678129.151731] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [57678129.153763] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [57678129.154712] pci 0000:00:01.1: reg 0x20: [io 0xc100-0xc10f] [57678129.155171] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [57678129.159827] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [57678129.163757] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [57678129.168324] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [57678129.172954] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [57678129.172984] * Found PM-Timer Bug on the chipset. Due to workarounds for a bug, * this clock source is slow. Consider trying other clock sources [57678129.183054] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [57678129.189081] pci 0000:00:02.0: [1013:00b8] type 00 class 0x030000 [57678129.189444] pci 0000:00:02.0: reg 0x10: [mem 0xf0000000-0xf1ffffff pref] [57678129.189762] pci 0000:00:02.0: reg 0x14: [mem 0xf3000000-0xf3000fff] [57678129.192078] pci 0000:00:03.0: [5853:0001] type 00 class 0xff8000 [57678129.192382] pci 0000:00:03.0: reg 0x10: [io 0xc000-0xc0ff] [57678129.192564] pci 0000:00:03.0: reg 0x14: [mem 0xf2000000-0xf2ffffff pref] [57678129.195141] ACPI: PCI Interrupt Link [LNKA] (IRQs *5 10 11) [57678129.202807] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [57678129.208502] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [57678129.214001] ACPI: PCI Interrupt Link [LNKD] (IRQs *5 10 11) [57678129.233833] ACPI: Enabled 2 GPEs in block 00 to 0F [57678129.238032] xen:balloon: Initialising balloon driver [57678129.247080] vgaarb: setting as boot device: PCI:0000:00:02.0 [57678129.250607] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [57678129.255791] vgaarb: loaded [57678129.257921] vgaarb: bridge control possible 0000:00:02.0 [57678129.261475] PCI: Using ACPI for IRQ routing [57678129.264390] PCI: pci_cache_line_size set to 64 bytes [57678129.264715] e820: reserve RAM buffer [mem 0x0009e000-0x0009ffff] [57678129.264863] NetLabel: Initializing [57678129.267320] NetLabel: domain hash size = 128 [57678129.270241] NetLabel: protocols = UNLABELED CIPSOv4 [57678129.273490] NetLabel: unlabeled traffic allowed by default [57678129.276965] HPET: 3 timers in total, 0 timers will be used for per-cpu timer [57678129.281511] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [57678129.286161] hpet0: 3 comparators, 64-bit 62.500000 MHz counter [57678129.293968] clocksource: Switched to clocksource xen [57678129.304072] VFS: Disk quotas dquot_6.6.0 [57678129.306836] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [57678129.311034] pnp: PnP ACPI init [57678129.313419] system 00:00: [mem 0x00000000-0x0009ffff] could not be reserved [57678129.318149] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active) [57678129.318243] system 00:01: [io 0x08a0-0x08a3] has been reserved [57678129.322087] system 00:01: [io 0x0cc0-0x0ccf] has been reserved [57678129.325939] system 00:01: [io 0x04d0-0x04d1] has been reserved [57678129.329609] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active) [57678129.329648] xen: --> pirq=17 -> irq=8 (gsi=8) [57678129.329688] pnp 00:02: Plug and Play ACPI device, IDs PNP0b00 (active) [57678129.329727] xen: --> pirq=18 -> irq=12 (gsi=12) [57678129.329762] pnp 00:03: Plug and Play ACPI device, IDs PNP0f13 (active) [57678129.329793] xen: --> pirq=19 -> irq=1 (gsi=1) [57678129.329820] pnp 00:04: Plug and Play ACPI device, IDs PNP0303 PNP030b (active) [57678129.329850] xen: --> pirq=20 -> irq=6 (gsi=6) [57678129.329854] pnp 00:05: [dma 2] [57678129.329878] pnp 00:05: Plug and Play ACPI device, IDs PNP0700 (active) [57678129.329920] xen: --> pirq=21 -> irq=4 (gsi=4) [57678129.329944] pnp 00:06: Plug and Play ACPI device, IDs PNP0501 (active) [57678129.330033] system 00:07: [io 0x10c0-0x1141] has been reserved [57678129.333670] system 00:07: [io 0xb044-0xb047] has been reserved [57678129.337376] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active) [57678129.351364] pnp: PnP ACPI: found 8 devices [57678129.362139] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [57678129.367763] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [57678129.367765] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [57678129.367767] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [57678129.367769] pci_bus 0000:00: resource 7 [mem 0xf0000000-0xfbffffff window] [57678129.367946] NET: Registered protocol family 2 [57678129.371084] TCP established hash table entries: 65536 (order: 7, 524288 bytes) [57678129.375923] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [57678129.380103] TCP: Hash tables configured (established 65536 bind 65536) [57678129.384274] UDP hash table entries: 4096 (order: 5, 131072 bytes) [57678129.388115] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes) [57678129.392149] NET: Registered protocol family 1 [57678129.395111] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [57678129.398794] pci 0000:00:01.0: PIIX3: Enabling Passive Release [57678129.402518] pci 0000:00:01.0: Activating ISA DMA hang workarounds [57678129.406625] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff] [57678129.411812] PCI: CLS 0 bytes, default 64 [57678129.411889] Unpacking initramfs... [57678129.621190] Freeing initrd memory: 14168K (ffff88003721a000 - ffff880037ff0000) [57678129.626362] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [57678129.631591] software IO TLB [mem 0xec000000-0xf0000000] (64MB) mapped at [ffff8800ec000000-ffff8800efffffff] [57678129.639032] audit: initializing netlink subsys (disabled) [57678129.642540] audit: type=2000 audit(1495215151.683:1): initialized [57678129.646674] Initialise system trusted keyrings [57678129.650283] workingset: timestamp_bits=36 max_order=21 bucket_order=0 [57678129.693578] Key type asymmetric registered [57678129.696428] Asymmetric key parser 'x509' registered [57678129.699691] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254) [57678129.704819] io scheduler noop registered (default) [57678129.709416] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [57678129.714198] intel_idle: does not run on family 6 model 62 [57678129.714708] xen: --> pirq=22 -> irq=28 (gsi=28) [57678129.714802] xen:grant_table: Grant tables using version 1 layout [57678129.719835] Grant table initialized [57678129.723115] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22! [57678129.728660] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [57678129.763449] 00:06: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [57678129.769473] xen_netfront: Initialising Xen virtual ethernet driver [57678129.774324] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12 [57678129.792975] serio: i8042 KBD port at 0x60,0x64 irq 1 [57678129.797623] serio: i8042 AUX port at 0x60,0x64 irq 12 [57678129.801615] hidraw: raw HID events driver (C) Jiri Kosina [57678129.805059] Invalid max_queues (4), will use default max: 2. [57678129.810003] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [57678129.810157] NET: Registered protocol family 17 [57678129.822055] registered taskstats version 1 [57678129.826232] Loading compiled-in X.509 certificates [57678129.832552] blkfront: xvda: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled; [57678129.833216] alg: No test for pkcs1pad(rsa,sha256) (pkcs1pad(rsa-generic,sha256)) [57678129.833949] Loaded X.509 cert 'Build time autogenerated kernel key: e5ce9fea87770e101779898e2a7fd312d140b79f' [57678129.834574] zswap: default zpool zbud not available [57678129.834575] zswap: pool creation failed [57678129.864717] xvda: xvda1 [57678129.935529] Freeing unused kernel memory: 1256K (ffffffff81b17000 - ffffffff81c51000) [57678129.940583] Write protecting the kernel read-only data: 10240k [57678129.944699] Freeing unused kernel memory: 784K (ffff88000153c000 - ffff880001600000) [57678129.952882] Freeing unused kernel memory: 1656K (ffff880001862000 - ffff880001a00000) [57678129.973254] dm_mod: module verification failed: signature and/or required key missing - tainting kernel [57678129.980880] device-mapper: uevent: version 1.0.3 [57678129.984386] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com [57678129.996195] udevd[825]: starting version 173 [57678130.042112] SCSI subsystem initialized [57678130.073809] libata version 3.00 loaded. [57678130.074599] ata_piix 0000:00:01.1: version 2.13 [57678130.091054] scsi host0: ata_piix [57678130.096031] scsi host1: ata_piix [57678130.100671] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc100 irq 14 [57678130.107292] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc108 irq 15 [57678130.325241] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null) [57678130.358683] dracut: Remounting /dev/disk/by-label/\x2f with -o noatime,ro [57678130.368778] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null) [57678130.378861] dracut: Mounted root filesystem /dev/xvda1 [57678130.406204] dracut: Switching root [57678130.666072] tsc: Refined TSC clocksource calibration: 2500.094 MHz [57678130.670830] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x240992b804d, max_idle_ns: 440795300532 ns [57678130.694272] udevd[1603]: starting version 173 [57678130.756934] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3 [57678130.764331] ACPI: Power Button [PWRF] [57678130.768133] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input4 [57678130.775061] ACPI: Sleep Button [SLPF] [57678130.826379] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input5 [57678130.901338] mousedev: PS/2 mouse device common for all mice [57678132.013324] EXT4-fs (xvda1): re-mounted. Opts: (null) [57678132.294449] NET: Registered protocol family 10 [57678133.202042] random: crng init done [57678134.203528] audit: type=1305 audit(1495215156.243:2): audit_pid=2309 old=0 auid=4294967295 ses=4294967295 res=1 [57678170.499850] spl: loading out-of-tree module taints kernel. [57678170.504678] SPL: Loaded module v0.7.0-rc4_1_g8f87971 (DEBUG mode) [57678170.543891] SPLAT: Loaded module v0.7.0-rc4_1_g8f87971 (DEBUG mode) [57678170.552407] zavl: module license 'CDDL' taints kernel. [57678170.555490] Disabling lock debugging due to kernel taint [57678172.330292] ZFS: Loaded module v0.7.0-rc4_17_ga32df59 (DEBUG mode), ZFS pool version 5000, ZFS filesystem version 5 [57678180.750186] loop: module loaded [57678181.198357] loop0: p1 [57678181.655033] loop0: p1 p2 [57678182.073012] SPL: using hostid 0x00000000 [57678186.562442] loop0: p1 [57678187.054543] loop0: p1 p2 [57678191.154686] loop0: p1 [57678191.588971] loop0: p1 p2 [57678326.850517] loop0: p1 [57678327.366809] loop0: p1 p2 [57678328.034975] zd0: p1 p9 [57678379.146025] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 50s! [57678379.152244] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=-20 stuck for 46s! [57678379.158687] Showing busy workqueues and worker pools: [57678379.162067] workqueue events_power_efficient: flags=0x80 [57678379.165414] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 [57678379.170463] pending: neigh_periodic_work, neigh_periodic_work [57678379.175595] workqueue kblockd: flags=0x18 [57678379.178440] pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=1/256 [57678379.183585] pending: blk_mq_timeout_work [57678379.187130] workqueue vmstat: flags=0xc [57678379.189764] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 [57678379.194828] pending: vmstat_update [57678387.182016] INFO: rcu_sched self-detected stall on CPU [57678387.185430] 0-...: (14743 ticks this GP) idle=d77/140000000000001/0 softirq=12474/12474 fqs=7210 [57678387.191196] (t=14750 jiffies g=2180 c=2179 q=433) [57678387.194708] Task dump for CPU 0: [57678387.197078] blkid R running task 0 7776 4485 0x00000008 [57678387.202339] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57678387.208389] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57678387.214501] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57678387.220570] Call Trace: [57678387.222581] [57678387.223989] [] sched_show_task+0xd7/0x140 [57678387.228182] [] dump_cpu_task+0x39/0x40 [57678387.231596] [] rcu_dump_cpu_stacks+0x80/0xbc [57678387.235214] [] rcu_check_callbacks+0x6ef/0x850 [57678387.238939] [] ? account_system_time+0x81/0x110 [57678387.242789] [] ? account_process_tick+0x60/0x170 [57678387.246614] [] ? tick_sched_do_timer+0x30/0x30 [57678387.250325] [] update_process_times+0x2f/0x60 [57678387.254090] [] tick_sched_handle.isra.13+0x36/0x50 [57678387.258047] [] tick_sched_timer+0x3d/0x70 [57678387.261553] [] __hrtimer_run_queues+0xd6/0x230 [57678387.265264] [] hrtimer_interrupt+0xa8/0x1a0 [57678387.268891] [] xen_timer_interrupt+0x1f/0x30 [57678387.272571] [] __handle_irq_event_percpu+0x3c/0x1a0 [57678387.276499] [] handle_irq_event_percpu+0x23/0x60 [57678387.280299] [] handle_percpu_irq+0x3a/0x50 [57678387.284019] [] generic_handle_irq+0x22/0x30 [57678387.287663] [] evtchn_2l_handle_events+0x238/0x240 [57678387.291542] [] __xen_evtchn_do_upcall+0x43/0x80 [57678387.295310] [] xen_evtchn_do_upcall+0x30/0x50 [57678387.299005] [] xen_hvm_callback_vector+0x82/0x90 [57678387.302943] [57678387.304313] [] ? __raw_callee_save___pv_queued_spin_unlock+0x10/0x17 [57678387.309710] [] zvol_open+0x112/0x3a0 [zfs] [57678387.313378] [] __blkdev_get+0xc4/0x420 [57678387.316774] [] ? bdget+0x3e/0x130 [57678387.319971] [] __blkdev_get+0x296/0x420 [57678387.323488] [] blkdev_get+0x1af/0x300 [57678387.326838] [] blkdev_open+0x5b/0x70 [57678387.330165] [] do_dentry_open+0x213/0x310 [57678387.333709] [] ? blkdev_get_by_dev+0x50/0x50 [57678387.337346] [] vfs_open+0x4f/0x70 [57678387.340538] [] ? may_open+0x9b/0x100 [57678387.343880] [] path_openat+0x529/0x1300 [57678387.347308] [] ? page_add_file_rmap+0x52/0x150 [57678387.351079] [] ? filemap_map_pages+0x366/0x3a0 [57678387.354835] [] do_filp_open+0x7e/0xd0 [57678387.358217] [] ? vma_link+0xb9/0xc0 [57678387.361504] [] ? kmem_cache_alloc+0x181/0x190 [57678387.365238] [] ? __alloc_fd+0xb0/0x170 [57678387.368667] [] do_sys_open+0x115/0x1f0 [57678387.372097] [] SyS_open+0x1e/0x20 [57678387.375282] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57678491.786084] INFO: task lt-zpool:7624 blocked for more than 120 seconds. [57678491.790665] Tainted: P OE 4.9.27-14.31.amzn1.x86_64 #1 [57678491.795168] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [57678491.800709] lt-zpool D 0 7624 6784 0x00000000 [57678491.805401] ffff8801e2cf7bc0 0000000000000000 ffff8801e662bb00 ffff8801d3281d80 [57678491.812342] ffff8801e7a57bc0 ffffc900066bb9f8 ffffffff8152ed4c ffffc900066bb9c8 [57678491.819356] ffffffff8152f45e ffffc900066bba20 ffff8801d3281d80 ffff8801e59ea71c [57678491.826660] Call Trace: [57678491.828921] [] ? __schedule+0x23c/0x680 [57678491.832798] [] ? schedule_preempt_disabled+0xe/0x10 [57678491.837209] [] schedule+0x36/0x80 [57678491.840765] [] schedule_preempt_disabled+0xe/0x10 [57678491.845098] [] __mutex_lock_slowpath+0x95/0x110 [57678491.849305] [] mutex_lock+0x17/0x27 [57678491.852987] [] __blkdev_get+0x5f/0x420 [57678491.856851] [] blkdev_get+0x1ff/0x300 [57678491.860619] [] ? mntput+0x24/0x40 [57678491.864252] [] ? path_put+0x1e/0x30 [57678491.867918] [] blkdev_get_by_path+0x53/0x90 [57678491.871987] [] vdev_disk_open+0x3b4/0x420 [zfs] [57678491.876194] [] ? iput+0x8a/0x200 [57678491.879817] [] vdev_open+0x12e/0x730 [zfs] [57678491.883830] [] vdev_open_children+0x55/0x170 [zfs] [57678491.888249] [] vdev_root_open+0x50/0x110 [zfs] [57678491.892621] [] vdev_open+0x12e/0x730 [zfs] [57678491.896685] [] vdev_create+0x22/0xa0 [zfs] [57678491.900681] [] ? zfs_allocatable_devs+0x5c/0x80 [zcommon] [57678491.905977] [] spa_create+0x435/0xaa0 [zfs] [57678491.910032] [] ? nvlist_add_uint64+0x35/0x40 [znvpair] [57678491.914608] [] ? zfs_fill_zplprops_impl+0x231/0x410 [zfs] [57678491.919807] [] zfs_ioc_pool_create+0x12e/0x230 [zfs] [57678491.924250] [] ? strdup+0x3f/0x60 [spl] [57678491.928161] [] zfsdev_ioctl+0x4d9/0x600 [zfs] [57678491.932276] [] do_vfs_ioctl+0x96/0x5b0 [57678491.936107] [] ? __do_page_fault+0x24a/0x4a0 [57678491.940168] [] SyS_ioctl+0x79/0x90 [57678491.943792] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57678564.194015] INFO: rcu_sched self-detected stall on CPU [57678564.198873] 0-...: (58948 ticks this GP) idle=d77/140000000000001/0 softirq=12474/12474 fqs=28798 [57678564.206323] (t=59003 jiffies g=2180 c=2179 q=1498) [57678564.211004] Task dump for CPU 0: [57678564.214099] blkid R running task 0 7776 4485 0x0000000c [57678564.221047] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57678564.229089] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57678564.236887] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57678564.244393] Call Trace: [57678564.247405] [57678564.249338] [] sched_show_task+0xd7/0x140 [57678564.255001] [] dump_cpu_task+0x39/0x40 [57678564.259595] [] rcu_dump_cpu_stacks+0x80/0xbc [57678564.264497] [] rcu_check_callbacks+0x6ef/0x850 [57678564.269537] [] ? account_system_time+0x81/0x110 [57678564.274611] [] ? account_process_tick+0x60/0x170 [57678564.279748] [] ? tick_sched_do_timer+0x30/0x30 [57678564.284779] [] update_process_times+0x2f/0x60 [57678564.289879] [] tick_sched_handle.isra.13+0x36/0x50 [57678564.295114] [] tick_sched_timer+0x3d/0x70 [57678564.300005] [] __hrtimer_run_queues+0xd6/0x230 [57678564.305147] [] hrtimer_interrupt+0xa8/0x1a0 [57678564.309988] [] xen_timer_interrupt+0x1f/0x30 [57678564.314906] [] __handle_irq_event_percpu+0x3c/0x1a0 [57678564.320268] [] handle_irq_event_percpu+0x23/0x60 [57678564.325427] [] handle_percpu_irq+0x3a/0x50 [57678564.329907] [] generic_handle_irq+0x22/0x30 [57678564.334596] [] evtchn_2l_handle_events+0x238/0x240 [57678564.339611] [] __xen_evtchn_do_upcall+0x43/0x80 [57678564.344954] [] xen_evtchn_do_upcall+0x30/0x50 [57678564.349937] [] xen_hvm_callback_vector+0x82/0x90 [57678564.355200] [57678564.357177] [] ? __blkdev_get+0x52/0x420 [57678564.362638] [] ? bdget+0x3e/0x130 [57678564.366935] [] __blkdev_get+0x296/0x420 [57678564.371578] [] blkdev_get+0x1af/0x300 [57678564.376047] [] blkdev_open+0x5b/0x70 [57678564.380475] [] do_dentry_open+0x213/0x310 [57678564.385194] [] ? blkdev_get_by_dev+0x50/0x50 [57678564.390160] [] vfs_open+0x4f/0x70 [57678564.394416] [] ? may_open+0x9b/0x100 [57678564.398883] [] path_openat+0x529/0x1300 [57678564.403457] [] ? page_add_file_rmap+0x52/0x150 [57678564.408477] [] ? filemap_map_pages+0x366/0x3a0 [57678564.413474] [] do_filp_open+0x7e/0xd0 [57678564.417774] [] ? vma_link+0xb9/0xc0 [57678564.421851] [] ? kmem_cache_alloc+0x181/0x190 [57678564.426503] [] ? __alloc_fd+0xb0/0x170 [57678564.431662] [] do_sys_open+0x115/0x1f0 [57678564.436270] [] SyS_open+0x1e/0x20 [57678564.440502] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57678612.622095] INFO: task lt-zpool:7624 blocked for more than 120 seconds. [57678612.626705] Tainted: P OE 4.9.27-14.31.amzn1.x86_64 #1 [57678612.631187] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [57678612.636922] lt-zpool D 0 7624 6784 0x00000000 [57678612.641235] ffff8801e2cf7bc0 0000000000000000 ffff8801e662bb00 ffff8801d3281d80 [57678612.648235] ffff8801e7a57bc0 ffffc900066bb9f8 ffffffff8152ed4c ffffc900066bb9c8 [57678612.678626] ffffffff8152f45e ffffc900066bba20 ffff8801d3281d80 ffff8801e59ea71c [57678612.687754] Call Trace: [57678612.690231] [] ? __schedule+0x23c/0x680 [57678612.694255] [] ? schedule_preempt_disabled+0xe/0x10 [57678612.699245] [] schedule+0x36/0x80 [57678612.702817] [] schedule_preempt_disabled+0xe/0x10 [57678612.707164] [] __mutex_lock_slowpath+0x95/0x110 [57678612.711562] [] mutex_lock+0x17/0x27 [57678612.716373] [] __blkdev_get+0x5f/0x420 [57678612.720570] [] blkdev_get+0x1ff/0x300 [57678612.724336] [] ? mntput+0x24/0x40 [57678612.728025] [] ? path_put+0x1e/0x30 [57678612.732019] [] blkdev_get_by_path+0x53/0x90 [57678612.736432] [] vdev_disk_open+0x3b4/0x420 [zfs] [57678612.740831] [] ? iput+0x8a/0x200 [57678612.744481] [] vdev_open+0x12e/0x730 [zfs] [57678612.748661] [] vdev_open_children+0x55/0x170 [zfs] [57678612.753263] [] vdev_root_open+0x50/0x110 [zfs] [57678612.757713] [] vdev_open+0x12e/0x730 [zfs] [57678612.763262] [] vdev_create+0x22/0xa0 [zfs] [57678612.768726] [] ? zfs_allocatable_devs+0x5c/0x80 [zcommon] [57678612.774509] [] spa_create+0x435/0xaa0 [zfs] [57678612.778782] [] ? nvlist_add_uint64+0x35/0x40 [znvpair] [57678612.784961] [] ? zfs_fill_zplprops_impl+0x231/0x410 [zfs] [57678612.791577] [] zfs_ioc_pool_create+0x12e/0x230 [zfs] [57678612.796175] [] ? strdup+0x3f/0x60 [spl] [57678612.800418] [] zfsdev_ioctl+0x4d9/0x600 [zfs] [57678612.804509] [] do_vfs_ioctl+0x96/0x5b0 [57678612.809224] [] ? __do_page_fault+0x24a/0x4a0 [57678612.813419] [] SyS_ioctl+0x79/0x90 [57678612.817259] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57678733.450080] INFO: task lt-zpool:7624 blocked for more than 120 seconds. [57678733.454568] Tainted: P OE 4.9.27-14.31.amzn1.x86_64 #1 [57678733.459038] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [57678733.464625] lt-zpool D 0 7624 1 0x00000004 [57678733.468864] ffff8801e2cf7bc0 0000000000000000 ffff8801e662bb00 ffff8801d3281d80 [57678733.475940] ffff8801e7a57bc0 ffffc900066bb9f8 ffffffff8152ed4c ffffc900066bb9c8 [57678733.482938] ffffffff8152f45e ffffc900066bba20 ffff8801d3281d80 ffff8801e59ea71c [57678733.490128] Call Trace: [57678733.492417] [] ? __schedule+0x23c/0x680 [57678733.496265] [] ? schedule_preempt_disabled+0xe/0x10 [57678733.500628] [] schedule+0x36/0x80 [57678733.504231] [] schedule_preempt_disabled+0xe/0x10 [57678733.508542] [] __mutex_lock_slowpath+0x95/0x110 [57678733.512813] [] mutex_lock+0x17/0x27 [57678733.516516] [] __blkdev_get+0x5f/0x420 [57678733.520373] [] blkdev_get+0x1ff/0x300 [57678733.524154] [] ? mntput+0x24/0x40 [57678733.527725] [] ? path_put+0x1e/0x30 [57678733.531399] [] blkdev_get_by_path+0x53/0x90 [57678733.535500] [] vdev_disk_open+0x3b4/0x420 [zfs] [57678733.539719] [] ? iput+0x8a/0x200 [57678733.543511] [] vdev_open+0x12e/0x730 [zfs] [57678733.547501] [] vdev_open_children+0x55/0x170 [zfs] [57678733.552006] [] vdev_root_open+0x50/0x110 [zfs] [57678733.556318] [] vdev_open+0x12e/0x730 [zfs] [57678733.560362] [] vdev_create+0x22/0xa0 [zfs] [57678733.564353] [] ? zfs_allocatable_devs+0x5c/0x80 [zcommon] [57678733.569540] [] spa_create+0x435/0xaa0 [zfs] [57678733.573587] [] ? nvlist_add_uint64+0x35/0x40 [znvpair] [57678733.578176] [] ? zfs_fill_zplprops_impl+0x231/0x410 [zfs] [57678733.583496] [] zfs_ioc_pool_create+0x12e/0x230 [zfs] [57678733.587922] [] ? strdup+0x3f/0x60 [spl] [57678733.592008] [] zfsdev_ioctl+0x4d9/0x600 [zfs] [57678733.596147] [] do_vfs_ioctl+0x96/0x5b0 [57678733.599951] [] ? __do_page_fault+0x24a/0x4a0 [57678733.604018] [] SyS_ioctl+0x79/0x90 [57678733.607612] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57678739.594034] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 32s! [57678739.600664] Showing busy workqueues and worker pools: [57678739.603958] workqueue events: flags=0x0 [57678739.606652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 [57678739.611702] pending: console_callback [57678739.615120] workqueue events_power_efficient: flags=0x80 [57678739.618551] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 [57678739.623633] pending: neigh_periodic_work, neigh_periodic_work [57678739.628574] workqueue vmstat: flags=0xc [57678739.631262] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 [57678739.636311] pending: vmstat_update [57678758.862015] INFO: rcu_sched self-detected stall on CPU [57678758.866789] 0-...: (14746 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=7199 [57678758.874190] (t=14750 jiffies g=2184 c=2183 q=781) [57678758.878880] Task dump for CPU 0: [57678758.882059] blkid R running task 0 7776 4485 0x0000000c [57678758.889064] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57678758.897203] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57678758.904845] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57678758.912552] Call Trace: [57678758.915525] [57678758.917445] [] sched_show_task+0xd7/0x140 [57678758.922997] [] dump_cpu_task+0x39/0x40 [57678758.927598] [] rcu_dump_cpu_stacks+0x80/0xbc [57678758.932510] [] rcu_check_callbacks+0x6ef/0x850 [57678758.937578] [] ? account_system_time+0x81/0x110 [57678758.942690] [] ? account_process_tick+0x60/0x170 [57678758.947861] [] ? tick_sched_do_timer+0x30/0x30 [57678758.952972] [] update_process_times+0x2f/0x60 [57678758.957926] [] tick_sched_handle.isra.13+0x36/0x50 [57678758.963215] [] tick_sched_timer+0x3d/0x70 [57678758.968032] [] __hrtimer_run_queues+0xd6/0x230 [57678758.973122] [] hrtimer_interrupt+0xa8/0x1a0 [57678758.977981] [] xen_timer_interrupt+0x1f/0x30 [57678758.982896] [] __handle_irq_event_percpu+0x3c/0x1a0 [57678758.988254] [] handle_irq_event_percpu+0x23/0x60 [57678758.993685] [] handle_percpu_irq+0x3a/0x50 [57678758.998369] [] generic_handle_irq+0x22/0x30 [57678759.002873] [] evtchn_2l_handle_events+0x238/0x240 [57678759.007801] [] __xen_evtchn_do_upcall+0x43/0x80 [57678759.013243] [] xen_evtchn_do_upcall+0x30/0x50 [57678759.018255] [] xen_hvm_callback_vector+0x82/0x90 [57678759.023698] [57678759.025649] [] ? down_read+0x5/0x30 [57678759.030982] [] ? zvol_open+0x1e1/0x3a0 [zfs] [57678759.035905] [] __blkdev_get+0xc4/0x420 [57678759.040503] [] ? bdget+0x3e/0x130 [57678759.044799] [] __blkdev_get+0x296/0x420 [57678759.049411] [] blkdev_get+0x1af/0x300 [57678759.053926] [] blkdev_open+0x5b/0x70 [57678759.058341] [] do_dentry_open+0x213/0x310 [57678759.063182] [] ? blkdev_get_by_dev+0x50/0x50 [57678759.068172] [] vfs_open+0x4f/0x70 [57678759.072486] [] ? may_open+0x9b/0x100 [57678759.076954] [] path_openat+0x529/0x1300 [57678759.081601] [] ? page_add_file_rmap+0x52/0x150 [57678759.086629] [] ? filemap_map_pages+0x366/0x3a0 [57678759.091738] [] do_filp_open+0x7e/0xd0 [57678759.095921] [] ? vma_link+0xb9/0xc0 [57678759.099960] [] ? kmem_cache_alloc+0x181/0x190 [57678759.104567] [] ? __alloc_fd+0xb0/0x170 [57678759.109555] [] do_sys_open+0x115/0x1f0 [57678759.114295] [] SyS_open+0x1e/0x20 [57678759.118638] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57678854.282084] INFO: task lt-zpool:7624 blocked for more than 120 seconds. [57678854.286701] Tainted: P OE 4.9.27-14.31.amzn1.x86_64 #1 [57678854.291369] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [57678854.297057] lt-zpool D 0 7624 1 0x00000004 [57678854.301319] ffff8801e2cf7bc0 0000000000000000 ffff8801e662bb00 ffff8801d3281d80 [57678854.308344] ffff8801e7a57bc0 ffffc900066bb9f8 ffffffff8152ed4c ffffc900066bb9c8 [57678854.315418] ffffffff8152f45e ffffc900066bba20 ffff8801d3281d80 ffff8801e59ea71c [57678854.322504] Call Trace: [57678854.324772] [] ? __schedule+0x23c/0x680 [57678854.328718] [] ? schedule_preempt_disabled+0xe/0x10 [57678854.333118] [] schedule+0x36/0x80 [57678854.336703] [] schedule_preempt_disabled+0xe/0x10 [57678854.340980] [] __mutex_lock_slowpath+0x95/0x110 [57678854.345217] [] mutex_lock+0x17/0x27 [57678854.348894] [] __blkdev_get+0x5f/0x420 [57678854.352699] [] blkdev_get+0x1ff/0x300 [57678854.356505] [] ? mntput+0x24/0x40 [57678854.360113] [] ? path_put+0x1e/0x30 [57678854.363831] [] blkdev_get_by_path+0x53/0x90 [57678854.367994] [] vdev_disk_open+0x3b4/0x420 [zfs] [57678854.372268] [] ? iput+0x8a/0x200 [57678854.376069] [] vdev_open+0x12e/0x730 [zfs] [57678854.380218] [] vdev_open_children+0x55/0x170 [zfs] [57678854.384637] [] vdev_root_open+0x50/0x110 [zfs] [57678854.388891] [] vdev_open+0x12e/0x730 [zfs] [57678854.392918] [] vdev_create+0x22/0xa0 [zfs] [57678854.396894] [] ? zfs_allocatable_devs+0x5c/0x80 [zcommon] [57678854.402187] [] spa_create+0x435/0xaa0 [zfs] [57678854.429298] [] ? nvlist_add_uint64+0x35/0x40 [znvpair] [57678854.433871] [] ? zfs_fill_zplprops_impl+0x231/0x410 [zfs] [57678854.439061] [] zfs_ioc_pool_create+0x12e/0x230 [zfs] [57678854.443474] [] ? strdup+0x3f/0x60 [spl] [57678854.447344] [] zfsdev_ioctl+0x4d9/0x600 [zfs] [57678854.451514] [] do_vfs_ioctl+0x96/0x5b0 [57678854.455381] [] ? __do_page_fault+0x24a/0x4a0 [57678854.459475] [] SyS_ioctl+0x79/0x90 [57678854.463181] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57678854.467589] INFO: task losetup:7859 blocked for more than 120 seconds. [57678854.472065] Tainted: P OE 4.9.27-14.31.amzn1.x86_64 #1 [57678854.476615] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [57678854.482170] losetup D 0 7859 7858 0x00000000 [57678854.486501] ffff8801e2cf6540 0000000000000000 ffff8801e662bb00 ffff8801d0943b00 [57678854.493471] ffff8801e7a57bc0 ffffc90006bffd30 ffffffff8152ed4c 0000000000000000 [57678854.500417] ffffc90006bffd08 0000000000000296 ffff8801d0943b00 ffff8801e2de19b8 [57678854.507896] Call Trace: [57678854.510216] [] ? __schedule+0x23c/0x680 [57678854.514076] [] schedule+0x36/0x80 [57678854.517652] [] blk_mq_freeze_queue_wait+0x37/0x90 [57678854.522099] [] ? prepare_to_wait_event+0x110/0x110 [57678854.526499] [] blk_mq_freeze_queue+0x1a/0x20 [57678854.530565] [] loop_clr_fd+0x64/0x2b0 [loop] [57678854.534629] [] lo_ioctl+0x290/0x710 [loop] [57678854.538626] [] blkdev_ioctl+0x24e/0x8d0 [57678854.542489] [] block_ioctl+0x41/0x50 [57678854.546277] [] do_vfs_ioctl+0x96/0x5b0 [57678854.550060] [] ? putname+0x53/0x60 [57678854.553685] [] SyS_ioctl+0x79/0x90 [57678854.557319] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57678935.874015] INFO: rcu_sched self-detected stall on CPU [57678935.877749] 0-...: (58935 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=28810 [57678935.883230] (t=59003 jiffies g=2184 c=2183 q=1769) [57678935.886766] Task dump for CPU 0: [57678935.889136] blkid R running task 0 7776 4485 0x0000000c [57678935.894481] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57678935.900665] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57678935.906796] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57678935.912836] Call Trace: [57678935.914814] [57678935.916251] [] sched_show_task+0xd7/0x140 [57678935.920530] [] dump_cpu_task+0x39/0x40 [57678935.923995] [] rcu_dump_cpu_stacks+0x80/0xbc [57678935.927733] [] rcu_check_callbacks+0x6ef/0x850 [57678935.931478] [] ? account_system_time+0x81/0x110 [57678935.935260] [] ? account_process_tick+0x60/0x170 [57678935.939081] [] ? tick_sched_do_timer+0x30/0x30 [57678935.942794] [] update_process_times+0x2f/0x60 [57678935.946671] [] tick_sched_handle.isra.13+0x36/0x50 [57678935.950677] [] tick_sched_timer+0x3d/0x70 [57678935.954197] [] __hrtimer_run_queues+0xd6/0x230 [57678935.957967] [] hrtimer_interrupt+0xa8/0x1a0 [57678935.961581] [] xen_timer_interrupt+0x1f/0x30 [57678935.965234] [] __handle_irq_event_percpu+0x3c/0x1a0 [57678935.969183] [] handle_irq_event_percpu+0x23/0x60 [57678935.972974] [] handle_percpu_irq+0x3a/0x50 [57678935.976573] [] generic_handle_irq+0x22/0x30 [57678935.980452] [] evtchn_2l_handle_events+0x238/0x240 [57678935.984308] [] __xen_evtchn_do_upcall+0x43/0x80 [57678935.988100] [] xen_evtchn_do_upcall+0x30/0x50 [57678935.991763] [] xen_hvm_callback_vector+0x82/0x90 [57678935.995551] [57678935.996952] [] ? mutex_unlock+0x1c/0x20 [57678936.001098] [] ? zvol_open+0x108/0x3a0 [zfs] [57678936.004717] [] __blkdev_get+0xc4/0x420 [57678936.008196] [] ? bdget+0x3e/0x130 [57678936.011366] [] __blkdev_get+0x296/0x420 [57678936.014783] [] blkdev_get+0x1af/0x300 [57678936.018169] [] blkdev_open+0x5b/0x70 [57678936.021468] [] do_dentry_open+0x213/0x310 [57678936.025238] [] ? blkdev_get_by_dev+0x50/0x50 [57678936.028932] [] vfs_open+0x4f/0x70 [57678936.032219] [] ? may_open+0x9b/0x100 [57678936.035506] [] path_openat+0x529/0x1300 [57678936.038942] [] ? page_add_file_rmap+0x52/0x150 [57678936.042685] [] ? filemap_map_pages+0x366/0x3a0 [57678936.046439] [] do_filp_open+0x7e/0xd0 [57678936.049791] [] ? vma_link+0xb9/0xc0 [57678936.053057] [] ? kmem_cache_alloc+0x181/0x190 [57678936.056750] [] ? __alloc_fd+0xb0/0x170 [57678936.060103] [] do_sys_open+0x115/0x1f0 [57678936.063574] [] SyS_open+0x1e/0x20 [57678936.066805] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57678975.114090] INFO: task lt-zpool:7624 blocked for more than 120 seconds. [57678975.118643] Tainted: P OE 4.9.27-14.31.amzn1.x86_64 #1 [57678975.123839] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [57678975.131684] lt-zpool D 0 7624 1 0x00000004 [57678975.136328] ffff8801e2cf7bc0 0000000000000000 ffff8801e662bb00 ffff8801d3281d80 [57678975.143719] ffff8801e7a57bc0 ffffc900066bb9f8 ffffffff8152ed4c ffffc900066bb9c8 [57678975.150855] ffffffff8152f45e ffffc900066bba20 ffff8801d3281d80 ffff8801e59ea71c [57678975.157931] Call Trace: [57678975.160375] [] ? __schedule+0x23c/0x680 [57678975.164299] [] ? schedule_preempt_disabled+0xe/0x10 [57678975.168811] [] schedule+0x36/0x80 [57678975.172389] [] schedule_preempt_disabled+0xe/0x10 [57678975.176725] [] __mutex_lock_slowpath+0x95/0x110 [57678975.180909] [] mutex_lock+0x17/0x27 [57678975.184626] [] __blkdev_get+0x5f/0x420 [57678975.188506] [] blkdev_get+0x1ff/0x300 [57678975.192297] [] ? mntput+0x24/0x40 [57678975.195866] [] ? path_put+0x1e/0x30 [57678975.199558] [] blkdev_get_by_path+0x53/0x90 [57678975.203658] [] vdev_disk_open+0x3b4/0x420 [zfs] [57678975.207869] [] ? iput+0x8a/0x200 [57678975.212111] [] vdev_open+0x12e/0x730 [zfs] [57678975.216140] [] vdev_open_children+0x55/0x170 [zfs] [57678975.220660] [] vdev_root_open+0x50/0x110 [zfs] [57678975.224862] [] vdev_open+0x12e/0x730 [zfs] [57678975.229014] [] vdev_create+0x22/0xa0 [zfs] [57678975.233055] [] ? zfs_allocatable_devs+0x5c/0x80 [zcommon] [57678975.238373] [] spa_create+0x435/0xaa0 [zfs] [57678975.242385] [] ? nvlist_add_uint64+0x35/0x40 [znvpair] [57678975.246943] [] ? zfs_fill_zplprops_impl+0x231/0x410 [zfs] [57678975.252322] [] zfs_ioc_pool_create+0x12e/0x230 [zfs] [57678975.256751] [] ? strdup+0x3f/0x60 [spl] [57678975.260728] [] zfsdev_ioctl+0x4d9/0x600 [zfs] [57678975.264938] [] do_vfs_ioctl+0x96/0x5b0 [57678975.268774] [] ? __do_page_fault+0x24a/0x4a0 [57678975.272956] [] SyS_ioctl+0x79/0x90 [57678975.276707] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57678975.281029] INFO: task losetup:7859 blocked for more than 120 seconds. [57678975.285698] Tainted: P OE 4.9.27-14.31.amzn1.x86_64 #1 [57678975.290164] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [57678975.295726] losetup D 0 7859 7858 0x00000000 [57678975.300003] ffff8801e2cf6540 0000000000000000 ffff8801e662bb00 ffff8801d0943b00 [57678975.306976] ffff8801e7a57bc0 ffffc90006bffd30 ffffffff8152ed4c 0000000000000000 [57678975.313906] ffffc90006bffd08 0000000000000296 ffff8801d0943b00 ffff8801e2de19b8 [57678975.320989] Call Trace: [57678975.323276] [] ? __schedule+0x23c/0x680 [57678975.327260] [] schedule+0x36/0x80 [57678975.330984] [] blk_mq_freeze_queue_wait+0x37/0x90 [57678975.335310] [] ? prepare_to_wait_event+0x110/0x110 [57678975.339766] [] blk_mq_freeze_queue+0x1a/0x20 [57678975.343938] [] loop_clr_fd+0x64/0x2b0 [loop] [57678975.348203] [] lo_ioctl+0x290/0x710 [loop] [57678975.352256] [] blkdev_ioctl+0x24e/0x8d0 [57678975.356140] [] block_ioctl+0x41/0x50 [57678975.359999] [] do_vfs_ioctl+0x96/0x5b0 [57678975.363944] [] ? putname+0x53/0x60 [57678975.367658] [] SyS_ioctl+0x79/0x90 [57678975.371428] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57679095.946185] INFO: task lt-zpool:7624 blocked for more than 120 seconds. [57679095.951823] Tainted: P OE 4.9.27-14.31.amzn1.x86_64 #1 [57679095.957495] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [57679095.964465] lt-zpool D 0 7624 1 0x00000004 [57679095.969745] ffff8801e2cf7bc0 0000000000000000 ffff8801e662bb00 ffff8801d3281d80 [57679095.977687] ffff8801e7a57bc0 ffffc900066bb9f8 ffffffff8152ed4c ffffc900066bb9c8 [57679095.986281] ffffffff8152f45e ffffc900066bba20 ffff8801d3281d80 ffff8801e59ea71c [57679095.995023] Call Trace: [57679095.997796] [] ? __schedule+0x23c/0x680 [57679096.002660] [] ? schedule_preempt_disabled+0xe/0x10 [57679096.008198] [] schedule+0x36/0x80 [57679096.012711] [] schedule_preempt_disabled+0xe/0x10 [57679096.018112] [] __mutex_lock_slowpath+0x95/0x110 [57679096.023984] [] mutex_lock+0x17/0x27 [57679096.028648] [] __blkdev_get+0x5f/0x420 [57679096.056406] [] blkdev_get+0x1ff/0x300 [57679096.061177] [] ? mntput+0x24/0x40 [57679096.065653] [] ? path_put+0x1e/0x30 [57679096.070290] [] blkdev_get_by_path+0x53/0x90 [57679096.075451] [] vdev_disk_open+0x3b4/0x420 [zfs] [57679096.080789] [] ? iput+0x8a/0x200 [57679096.085249] [] vdev_open+0x12e/0x730 [zfs] [57679096.090358] [] vdev_open_children+0x55/0x170 [zfs] [57679096.095924] [] vdev_root_open+0x50/0x110 [zfs] [57679096.101055] [] vdev_open+0x12e/0x730 [zfs] [57679096.105900] [] vdev_create+0x22/0xa0 [zfs] [57679096.111303] [] ? zfs_allocatable_devs+0x5c/0x80 [zcommon] [57679096.117764] [] spa_create+0x435/0xaa0 [zfs] [57679096.122855] [] ? nvlist_add_uint64+0x35/0x40 [znvpair] [57679096.128630] [] ? zfs_fill_zplprops_impl+0x231/0x410 [zfs] [57679096.135073] [] zfs_ioc_pool_create+0x12e/0x230 [zfs] [57679096.140731] [] ? strdup+0x3f/0x60 [spl] [57679096.145615] [] zfsdev_ioctl+0x4d9/0x600 [zfs] [57679096.150832] [] do_vfs_ioctl+0x96/0x5b0 [57679096.155673] [] ? __do_page_fault+0x24a/0x4a0 [57679096.160833] [] SyS_ioctl+0x79/0x90 [57679096.165350] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57679096.170875] INFO: task losetup:7859 blocked for more than 120 seconds. [57679096.176543] Tainted: P OE 4.9.27-14.31.amzn1.x86_64 #1 [57679096.182259] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [57679096.189302] losetup D 0 7859 7858 0x00000000 [57679096.194538] ffff8801e2cf6540 0000000000000000 ffff8801e662bb00 ffff8801d0943b00 [57679096.202676] ffff8801e7a57bc0 ffffc90006bffd30 ffffffff8152ed4c 0000000000000000 [57679096.211697] ffffc90006bffd08 0000000000000296 ffff8801d0943b00 ffff8801e2de19b8 [57679096.220198] Call Trace: [57679096.223001] [] ? __schedule+0x23c/0x680 [57679096.227973] [] schedule+0x36/0x80 [57679096.232760] [] blk_mq_freeze_queue_wait+0x37/0x90 [57679096.238202] [] ? prepare_to_wait_event+0x110/0x110 [57679096.243741] [] blk_mq_freeze_queue+0x1a/0x20 [57679096.248948] [] loop_clr_fd+0x64/0x2b0 [loop] [57679096.254133] [] lo_ioctl+0x290/0x710 [loop] [57679096.259158] [] blkdev_ioctl+0x24e/0x8d0 [57679096.264196] [] block_ioctl+0x41/0x50 [57679096.268848] [] do_vfs_ioctl+0x96/0x5b0 [57679096.273682] [] ? putname+0x53/0x60 [57679096.278222] [] SyS_ioctl+0x79/0x90 [57679096.282757] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57679112.886529] INFO: rcu_sched self-detected stall on CPU [57679112.890243] 0-...: (103140 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=50351 [57679112.895750] (t=103256 jiffies g=2184 c=2183 q=2725) [57679112.899314] Task dump for CPU 0: [57679112.901790] blkid R running task 0 7776 4485 0x0000000c [57679112.907032] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57679112.913119] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57679112.919124] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57679112.925171] Call Trace: [57679112.927129] [57679112.928502] [] sched_show_task+0xd7/0x140 [57679112.932671] [] dump_cpu_task+0x39/0x40 [57679112.936053] [] rcu_dump_cpu_stacks+0x80/0xbc [57679112.939722] [] rcu_check_callbacks+0x6ef/0x850 [57679112.943417] [] ? account_system_time+0x81/0x110 [57679112.947176] [] ? account_process_tick+0x60/0x170 [57679112.951003] [] ? tick_sched_do_timer+0x30/0x30 [57679112.954916] [] update_process_times+0x2f/0x60 [57679112.958601] [] tick_sched_handle.isra.13+0x36/0x50 [57679112.962550] [] tick_sched_timer+0x3d/0x70 [57679112.966054] [] __hrtimer_run_queues+0xd6/0x230 [57679112.969757] [] hrtimer_interrupt+0xa8/0x1a0 [57679112.973342] [] xen_timer_interrupt+0x1f/0x30 [57679112.976966] [] __handle_irq_event_percpu+0x3c/0x1a0 [57679112.980953] [] handle_irq_event_percpu+0x23/0x60 [57679112.984772] [] handle_percpu_irq+0x3a/0x50 [57679112.988313] [] generic_handle_irq+0x22/0x30 [57679112.991960] [] evtchn_2l_handle_events+0x238/0x240 [57679112.995816] [] __xen_evtchn_do_upcall+0x43/0x80 [57679112.999576] [] xen_evtchn_do_upcall+0x30/0x50 [57679113.003282] [] xen_hvm_callback_vector+0x82/0x90 [57679113.007061] [57679113.008430] [] ? module_put+0x2f/0xa0 [57679113.012611] [] __blkdev_get+0x122/0x420 [57679113.016010] [] ? bdget+0x3e/0x130 [57679113.019241] [] __blkdev_get+0x296/0x420 [57679113.022889] [] blkdev_get+0x1af/0x300 [57679113.026212] [] blkdev_open+0x5b/0x70 [57679113.029501] [] do_dentry_open+0x213/0x310 [57679113.033100] [] ? blkdev_get_by_dev+0x50/0x50 [57679113.036710] [] vfs_open+0x4f/0x70 [57679113.039866] [] ? may_open+0x9b/0x100 [57679113.043183] [] path_openat+0x529/0x1300 [57679113.046599] [] ? page_add_file_rmap+0x52/0x150 [57679113.050420] [] ? filemap_map_pages+0x366/0x3a0 [57679113.054183] [] do_filp_open+0x7e/0xd0 [57679113.057501] [] ? vma_link+0xb9/0xc0 [57679113.060755] [] ? kmem_cache_alloc+0x181/0x190 [57679113.064403] [] ? __alloc_fd+0xb0/0x170 [57679113.067796] [] do_sys_open+0x115/0x1f0 [57679113.071210] [] SyS_open+0x1e/0x20 [57679113.074567] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57679216.781158] INFO: task lt-zpool:7624 blocked for more than 120 seconds. [57679216.785326] Tainted: P OE 4.9.27-14.31.amzn1.x86_64 #1 [57679216.789752] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [57679216.795299] lt-zpool D 0 7624 1 0x00000004 [57679216.799564] ffff8801e2cf7bc0 0000000000000000 ffff8801e662bb00 ffff8801d3281d80 [57679216.806735] ffff8801e7a57bc0 ffffc900066bb9f8 ffffffff8152ed4c ffffc900066bb9c8 [57679216.813664] ffffffff8152f45e ffffc900066bba20 ffff8801d3281d80 ffff8801e59ea71c [57679216.820558] Call Trace: [57679216.822872] [] ? __schedule+0x23c/0x680 [57679216.827236] [] ? schedule_preempt_disabled+0xe/0x10 [57679216.832539] [] schedule+0x36/0x80 [57679216.842909] [] schedule_preempt_disabled+0xe/0x10 [57679216.847173] [] __mutex_lock_slowpath+0x95/0x110 [57679216.851341] [] mutex_lock+0x17/0x27 [57679216.855011] [] __blkdev_get+0x5f/0x420 [57679216.858796] [] blkdev_get+0x1ff/0x300 [57679216.862580] [] ? mntput+0x24/0x40 [57679216.866127] [] ? path_put+0x1e/0x30 [57679216.869813] [] blkdev_get_by_path+0x53/0x90 [57679216.873910] [] vdev_disk_open+0x3b4/0x420 [zfs] [57679216.878097] [] ? iput+0x8a/0x200 [57679216.881655] [] vdev_open+0x12e/0x730 [zfs] [57679216.885724] [] vdev_open_children+0x55/0x170 [zfs] [57679216.890096] [] vdev_root_open+0x50/0x110 [zfs] [57679216.894390] [] vdev_open+0x12e/0x730 [zfs] [57679216.898383] [] vdev_create+0x22/0xa0 [zfs] [57679216.902343] [] ? zfs_allocatable_devs+0x5c/0x80 [zcommon] [57679216.907517] [] spa_create+0x435/0xaa0 [zfs] [57679216.911572] [] ? nvlist_add_uint64+0x35/0x40 [znvpair] [57679216.916219] [] ? zfs_fill_zplprops_impl+0x231/0x410 [zfs] [57679216.921382] [] zfs_ioc_pool_create+0x12e/0x230 [zfs] [57679216.925854] [] ? strdup+0x3f/0x60 [spl] [57679216.929764] [] zfsdev_ioctl+0x4d9/0x600 [zfs] [57679216.933914] [] do_vfs_ioctl+0x96/0x5b0 [57679216.937765] [] ? __do_page_fault+0x24a/0x4a0 [57679216.941873] [] SyS_ioctl+0x79/0x90 [57679216.945498] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57679289.902906] INFO: rcu_sched self-detected stall on CPU [57679289.906646] 0-...: (147347 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=71732 [57679289.912210] (t=147509 jiffies g=2184 c=2183 q=4025) [57679289.915798] Task dump for CPU 0: [57679289.918119] blkid R running task 0 7776 4485 0x0000000c [57679289.923313] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57679289.929371] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57679289.935407] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57679289.941406] Call Trace: [57679289.943377] [57679289.944779] [] sched_show_task+0xd7/0x140 [57679289.948979] [] dump_cpu_task+0x39/0x40 [57679289.952363] [] rcu_dump_cpu_stacks+0x80/0xbc [57679289.956009] [] rcu_check_callbacks+0x6ef/0x850 [57679289.959748] [] ? account_system_time+0x81/0x110 [57679289.963536] [] ? account_process_tick+0x60/0x170 [57679289.967404] [] ? tick_sched_do_timer+0x30/0x30 [57679289.971122] [] update_process_times+0x2f/0x60 [57679289.974787] [] tick_sched_handle.isra.13+0x36/0x50 [57679289.978695] [] tick_sched_timer+0x3d/0x70 [57679289.982215] [] __hrtimer_run_queues+0xd6/0x230 [57679289.985923] [] hrtimer_interrupt+0xa8/0x1a0 [57679289.989515] [] xen_timer_interrupt+0x1f/0x30 [57679289.993150] [] __handle_irq_event_percpu+0x3c/0x1a0 [57679289.997112] [] handle_irq_event_percpu+0x23/0x60 [57679290.000893] [] handle_percpu_irq+0x3a/0x50 [57679290.019802] [] generic_handle_irq+0x22/0x30 [57679290.023417] [] evtchn_2l_handle_events+0x238/0x240 [57679290.027399] [] __xen_evtchn_do_upcall+0x43/0x80 [57679290.031172] [] xen_evtchn_do_upcall+0x30/0x50 [57679290.034836] [] xen_hvm_callback_vector+0x82/0x90 [57679290.038673] [57679290.040063] [] ? kobject_put+0x18/0x50 [57679290.044128] [] put_device+0x17/0x20 [57679290.047447] [] __blkdev_get+0xe4/0x420 [57679290.050838] [] ? bdget+0x3e/0x130 [57679290.054042] [] __blkdev_get+0x296/0x420 [57679290.057510] [] blkdev_get+0x1af/0x300 [57679290.060886] [] blkdev_open+0x5b/0x70 [57679290.064223] [] do_dentry_open+0x213/0x310 [57679290.067723] [] ? blkdev_get_by_dev+0x50/0x50 [57679290.071437] [] vfs_open+0x4f/0x70 [57679290.074620] [] ? may_open+0x9b/0x100 [57679290.077949] [] path_openat+0x529/0x1300 [57679290.081367] [] ? page_add_file_rmap+0x52/0x150 [57679290.085098] [] ? filemap_map_pages+0x366/0x3a0 [57679290.088808] [] do_filp_open+0x7e/0xd0 [57679290.092134] [] ? vma_link+0xb9/0xc0 [57679290.095497] [] ? kmem_cache_alloc+0x181/0x190 [57679290.099195] [] ? __alloc_fd+0xb0/0x170 [57679290.102581] [] do_sys_open+0x115/0x1f0 [57679290.106006] [] SyS_open+0x1e/0x20 [57679290.109199] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57679466.931857] INFO: rcu_sched self-detected stall on CPU [57679466.935663] 0-...: (191549 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=92979 [57679466.941360] (t=191762 jiffies g=2184 c=2183 q=5776) [57679466.944926] Task dump for CPU 0: [57679466.947390] blkid R running task 0 7776 4485 0x0000000c [57679466.952659] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57679466.958702] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57679466.964912] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57679466.971110] Call Trace: [57679466.973100] [57679466.974490] [] sched_show_task+0xd7/0x140 [57679466.978908] [] dump_cpu_task+0x39/0x40 [57679466.982290] [] rcu_dump_cpu_stacks+0x80/0xbc [57679466.985946] [] rcu_check_callbacks+0x6ef/0x850 [57679466.989754] [] ? account_system_time+0x81/0x110 [57679466.993533] [] ? account_process_tick+0x60/0x170 [57679466.997352] [] ? tick_sched_do_timer+0x30/0x30 [57679467.001585] [] update_process_times+0x2f/0x60 [57679467.006225] [] tick_sched_handle.isra.13+0x36/0x50 [57679467.010448] [] tick_sched_timer+0x3d/0x70 [57679467.013953] [] __hrtimer_run_queues+0xd6/0x230 [57679467.017730] [] hrtimer_interrupt+0xa8/0x1a0 [57679467.021395] [] xen_timer_interrupt+0x1f/0x30 [57679467.025651] [] __handle_irq_event_percpu+0x3c/0x1a0 [57679467.029750] [] handle_irq_event_percpu+0x23/0x60 [57679467.033539] [] handle_percpu_irq+0x3a/0x50 [57679467.037158] [] generic_handle_irq+0x22/0x30 [57679467.041032] [] evtchn_2l_handle_events+0x238/0x240 [57679467.045014] [] __xen_evtchn_do_upcall+0x43/0x80 [57679467.048807] [] xen_evtchn_do_upcall+0x30/0x50 [57679467.052592] [] xen_hvm_callback_vector+0x82/0x90 [57679467.056437] [57679467.057832] [] ? _raw_spin_lock+0x10/0x30 [57679467.062231] [] ? zvol_open+0x100/0x3a0 [zfs] [57679467.065877] [] __blkdev_get+0xc4/0x420 [57679467.069318] [] ? bdget+0x3e/0x130 [57679467.072557] [] __blkdev_get+0x296/0x420 [57679467.076141] [] blkdev_get+0x1af/0x300 [57679467.079507] [] blkdev_open+0x5b/0x70 [57679467.082837] [] do_dentry_open+0x213/0x310 [57679467.086365] [] ? blkdev_get_by_dev+0x50/0x50 [57679467.090064] [] vfs_open+0x4f/0x70 [57679467.093306] [] ? may_open+0x9b/0x100 [57679467.096616] [] path_openat+0x529/0x1300 [57679467.100059] [] ? page_add_file_rmap+0x52/0x150 [57679467.103785] [] ? filemap_map_pages+0x366/0x3a0 [57679467.107710] [] do_filp_open+0x7e/0xd0 [57679467.111138] [] ? vma_link+0xb9/0xc0 [57679467.114415] [] ? kmem_cache_alloc+0x181/0x190 [57679467.118129] [] ? __alloc_fd+0xb0/0x170 [57679467.121612] [] do_sys_open+0x115/0x1f0 [57679467.125150] [] SyS_open+0x1e/0x20 [57679467.128423] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57679643.955298] INFO: rcu_sched self-detected stall on CPU [57679643.960716] 0-...: (235753 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=114235 [57679643.967966] (t=236015 jiffies g=2184 c=2183 q=7923) [57679643.972815] Task dump for CPU 0: [57679643.976673] blkid R running task 0 7776 4485 0x0000000c [57679643.984098] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57679643.992338] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57679644.000338] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57679644.008788] Call Trace: [57679644.011347] [57679644.013174] [] sched_show_task+0xd7/0x140 [57679644.018637] [] dump_cpu_task+0x39/0x40 [57679644.023361] [] rcu_dump_cpu_stacks+0x80/0xbc [57679644.028808] [] rcu_check_callbacks+0x6ef/0x850 [57679644.033942] [] ? account_system_time+0x81/0x110 [57679644.039092] [] ? account_process_tick+0x60/0x170 [57679644.044903] [] ? tick_sched_do_timer+0x30/0x30 [57679644.050419] [] update_process_times+0x2f/0x60 [57679644.055258] [] tick_sched_handle.isra.13+0x36/0x50 [57679644.060990] [] tick_sched_timer+0x3d/0x70 [57679644.066135] [] __hrtimer_run_queues+0xd6/0x230 [57679644.071305] [] hrtimer_interrupt+0xa8/0x1a0 [57679644.076067] [] xen_timer_interrupt+0x1f/0x30 [57679644.081587] [] __handle_irq_event_percpu+0x3c/0x1a0 [57679644.087410] [] handle_irq_event_percpu+0x23/0x60 [57679644.092555] [] handle_percpu_irq+0x3a/0x50 [57679644.097689] [] generic_handle_irq+0x22/0x30 [57679644.103143] [] evtchn_2l_handle_events+0x238/0x240 [57679644.108478] [] __xen_evtchn_do_upcall+0x43/0x80 [57679644.113529] [] xen_evtchn_do_upcall+0x30/0x50 [57679644.119256] [] xen_hvm_callback_vector+0x82/0x90 [57679644.124868] [57679644.126725] [] ? _raw_spin_lock+0xc/0x30 [57679644.132129] [] ? zvol_open+0xa6/0x3a0 [zfs] [57679644.137425] [] __blkdev_get+0xc4/0x420 [57679644.142398] [] ? bdget+0x3e/0x130 [57679644.146843] [] __blkdev_get+0x296/0x420 [57679644.151318] [] blkdev_get+0x1af/0x300 [57679644.156255] [] blkdev_open+0x5b/0x70 [57679644.161151] [] do_dentry_open+0x213/0x310 [57679644.165999] [] ? blkdev_get_by_dev+0x50/0x50 [57679644.170771] [] vfs_open+0x4f/0x70 [57679644.175551] [] ? may_open+0x9b/0x100 [57679644.180356] [] path_openat+0x529/0x1300 [57679644.185291] [] ? page_add_file_rmap+0x52/0x150 [57679644.190381] [] ? filemap_map_pages+0x366/0x3a0 [57679644.195572] [] do_filp_open+0x7e/0xd0 [57679644.198968] [] ? vma_link+0xb9/0xc0 [57679644.202230] [] ? kmem_cache_alloc+0x181/0x190 [57679644.205906] [] ? __alloc_fd+0xb0/0x170 [57679644.209266] [] do_sys_open+0x115/0x1f0 [57679644.212705] [] SyS_open+0x1e/0x20 [57679644.216021] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57679820.976532] INFO: rcu_sched self-detected stall on CPU [57679820.980535] 0-...: (279941 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=135435 [57679820.986159] (t=280268 jiffies g=2184 c=2183 q=9170) [57679820.989744] Task dump for CPU 0: [57679820.992114] blkid R running task 0 7776 4485 0x0000000c [57679820.997291] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57679821.003482] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57679821.009758] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57679821.015888] Call Trace: [57679821.017858] [57679821.019324] [] sched_show_task+0xd7/0x140 [57679821.023684] [] dump_cpu_task+0x39/0x40 [57679821.027170] [] rcu_dump_cpu_stacks+0x80/0xbc [57679821.030832] [] rcu_check_callbacks+0x6ef/0x850 [57679821.034545] [] ? account_system_time+0x81/0x110 [57679821.038532] [] ? account_process_tick+0x60/0x170 [57679821.042475] [] ? tick_sched_do_timer+0x30/0x30 [57679821.046192] [] update_process_times+0x2f/0x60 [57679821.049978] [] tick_sched_handle.isra.13+0x36/0x50 [57679821.054002] [] tick_sched_timer+0x3d/0x70 [57679821.057689] [] __hrtimer_run_queues+0xd6/0x230 [57679821.062024] [] hrtimer_interrupt+0xa8/0x1a0 [57679821.065598] [] xen_timer_interrupt+0x1f/0x30 [57679821.069338] [] __handle_irq_event_percpu+0x3c/0x1a0 [57679821.073453] [] handle_irq_event_percpu+0x23/0x60 [57679821.077353] [] handle_percpu_irq+0x3a/0x50 [57679821.081016] [] generic_handle_irq+0x22/0x30 [57679821.084649] [] evtchn_2l_handle_events+0x238/0x240 [57679821.088544] [] __xen_evtchn_do_upcall+0x43/0x80 [57679821.092595] [] xen_evtchn_do_upcall+0x30/0x50 [57679821.096355] [] xen_hvm_callback_vector+0x82/0x90 [57679821.100253] [57679821.101634] [] ? mutex_unlock+0x10/0x20 [57679821.121153] [] ? __blkdev_get+0x10a/0x420 [57679821.125149] [] ? bdget+0x3e/0x130 [57679821.128490] [] __blkdev_get+0x296/0x420 [57679821.131949] [] blkdev_get+0x1af/0x300 [57679821.135287] [] blkdev_open+0x5b/0x70 [57679821.138575] [] do_dentry_open+0x213/0x310 [57679821.142103] [] ? blkdev_get_by_dev+0x50/0x50 [57679821.145965] [] vfs_open+0x4f/0x70 [57679821.149270] [] ? may_open+0x9b/0x100 [57679821.152564] [] path_openat+0x529/0x1300 [57679821.155983] [] ? page_add_file_rmap+0x52/0x150 [57679821.159711] [] ? filemap_map_pages+0x366/0x3a0 [57679821.163609] [] do_filp_open+0x7e/0xd0 [57679821.167052] [] ? vma_link+0xb9/0xc0 [57679821.170445] [] ? kmem_cache_alloc+0x181/0x190 [57679821.174127] [] ? __alloc_fd+0xb0/0x170 [57679821.177503] [] do_sys_open+0x115/0x1f0 [57679821.181193] [] SyS_open+0x1e/0x20 [57679821.184533] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57679997.999609] INFO: rcu_sched self-detected stall on CPU [57679998.003485] 0-...: (324143 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=156675 [57679998.009067] (t=324521 jiffies g=2184 c=2183 q=10448) [57679998.012672] Task dump for CPU 0: [57679998.015030] blkid R running task 0 7776 4485 0x0000000c [57679998.020250] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57679998.026434] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57679998.032606] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57679998.038632] Call Trace: [57679998.040588] [57679998.041980] [] sched_show_task+0xd7/0x140 [57679998.047137] [] dump_cpu_task+0x39/0x40 [57679998.050707] [] rcu_dump_cpu_stacks+0x80/0xbc [57679998.054362] [] rcu_check_callbacks+0x6ef/0x850 [57679998.058063] [] ? account_system_time+0x81/0x110 [57679998.061789] [] ? account_process_tick+0x60/0x170 [57679998.065828] [] ? tick_sched_do_timer+0x30/0x30 [57679998.069690] [] update_process_times+0x2f/0x60 [57679998.073335] [] tick_sched_handle.isra.13+0x36/0x50 [57679998.077240] [] tick_sched_timer+0x3d/0x70 [57679998.080743] [] __hrtimer_run_queues+0xd6/0x230 [57679998.084667] [] hrtimer_interrupt+0xa8/0x1a0 [57679998.088374] [] xen_timer_interrupt+0x1f/0x30 [57679998.092009] [] __handle_irq_event_percpu+0x3c/0x1a0 [57679998.095985] [] handle_irq_event_percpu+0x23/0x60 [57679998.099768] [] handle_percpu_irq+0x3a/0x50 [57679998.103515] [] generic_handle_irq+0x22/0x30 [57679998.107284] [] evtchn_2l_handle_events+0x238/0x240 [57679998.111158] [] __xen_evtchn_do_upcall+0x43/0x80 [57679998.114907] [] xen_evtchn_do_upcall+0x30/0x50 [57679998.118564] [] xen_hvm_callback_vector+0x82/0x90 [57679998.122557] [57679998.124023] [] ? mutex_lock+0x17/0x27 [57679998.128095] [] __blkdev_get+0x5f/0x420 [57679998.131467] [] ? bdget+0x3e/0x130 [57679998.134683] [] __blkdev_get+0x296/0x420 [57679998.138113] [] blkdev_get+0x1af/0x300 [57679998.141602] [] blkdev_open+0x5b/0x70 [57679998.145069] [] do_dentry_open+0x213/0x310 [57679998.148547] [] ? blkdev_get_by_dev+0x50/0x50 [57679998.152184] [] vfs_open+0x4f/0x70 [57679998.155391] [] ? may_open+0x9b/0x100 [57679998.158794] [] path_openat+0x529/0x1300 [57679998.162377] [] ? page_add_file_rmap+0x52/0x150 [57679998.166113] [] ? filemap_map_pages+0x366/0x3a0 [57679998.169798] [] do_filp_open+0x7e/0xd0 [57679998.173131] [] ? vma_link+0xb9/0xc0 [57679998.176448] [] ? kmem_cache_alloc+0x181/0x190 [57679998.180316] [] ? __alloc_fd+0xb0/0x170 [57679998.183747] [] do_sys_open+0x115/0x1f0 [57679998.187189] [] SyS_open+0x1e/0x20 [57679998.190342] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57680175.020218] INFO: rcu_sched self-detected stall on CPU [57680175.025546] 0-...: (368349 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=177886 [57680175.032473] (t=368774 jiffies g=2184 c=2183 q=11434) [57680175.036952] Task dump for CPU 0: [57680175.040483] blkid R running task 0 7776 4485 0x0000000c [57680175.047698] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57680175.055932] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57680175.064118] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57680175.072073] Call Trace: [57680175.074853] [57680175.076869] [] sched_show_task+0xd7/0x140 [57680175.082949] [] dump_cpu_task+0x39/0x40 [57680175.087408] [] rcu_dump_cpu_stacks+0x80/0xbc [57680175.092445] [] rcu_check_callbacks+0x6ef/0x850 [57680175.098101] [] ? account_system_time+0x81/0x110 [57680175.103496] [] ? account_process_tick+0x60/0x170 [57680175.108552] [] ? tick_sched_do_timer+0x30/0x30 [57680175.114331] [] update_process_times+0x2f/0x60 [57680175.119900] [] tick_sched_handle.isra.13+0x36/0x50 [57680175.125045] [] tick_sched_timer+0x3d/0x70 [57680175.129936] [] __hrtimer_run_queues+0xd6/0x230 [57680175.135589] [] hrtimer_interrupt+0xa8/0x1a0 [57680175.140638] [] xen_timer_interrupt+0x1f/0x30 [57680175.145390] [] __handle_irq_event_percpu+0x3c/0x1a0 [57680175.151305] [] handle_irq_event_percpu+0x23/0x60 [57680175.157009] [] handle_percpu_irq+0x3a/0x50 [57680175.161747] [] generic_handle_irq+0x22/0x30 [57680175.166737] [] evtchn_2l_handle_events+0x238/0x240 [57680175.172556] [] __xen_evtchn_do_upcall+0x43/0x80 [57680175.177847] [] xen_evtchn_do_upcall+0x30/0x50 [57680175.182770] [] xen_hvm_callback_vector+0x82/0x90 [57680175.188503] [57680175.190525] [] ? try_module_get+0x45/0xe0 [57680175.196316] [] get_disk+0x2c/0x60 [57680175.200530] [] exact_lock+0x11/0x20 [57680175.205146] [] kobj_lookup+0xe1/0x160 [57680175.210138] [] ? disk_map_sector_rcu+0x70/0x70 [57680175.215604] [] get_gendisk+0x34/0x110 [57680175.220036] [] __blkdev_get+0x12d/0x420 [57680175.225136] [] ? bdget+0x3e/0x130 [57680175.229964] [] __blkdev_get+0x296/0x420 [57680175.234709] [] blkdev_get+0x1af/0x300 [57680175.239075] [] blkdev_open+0x5b/0x70 [57680175.243916] [] do_dentry_open+0x213/0x310 [57680175.249126] [] ? blkdev_get_by_dev+0x50/0x50 [57680175.254252] [] vfs_open+0x4f/0x70 [57680175.258438] [] ? may_open+0x9b/0x100 [57680175.263135] [] path_openat+0x529/0x1300 [57680175.268254] [] ? page_add_file_rmap+0x52/0x150 [57680175.273618] [] ? filemap_map_pages+0x366/0x3a0 [57680175.278522] [] do_filp_open+0x7e/0xd0 [57680175.284164] [] ? vma_link+0xb9/0xc0 [57680175.291048] [] ? kmem_cache_alloc+0x181/0x190 [57680175.296802] [] ? __alloc_fd+0xb0/0x170 [57680175.301559] [] do_sys_open+0x115/0x1f0 [57680175.306850] [] SyS_open+0x1e/0x20 [57680175.311786] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57680352.041421] INFO: rcu_sched self-detected stall on CPU [57680352.045338] 0-...: (412529 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=199103 [57680352.050983] (t=413027 jiffies g=2184 c=2183 q=130631) [57680352.054655] Task dump for CPU 0: [57680352.056997] blkid R running task 0 7776 4485 0x0000000c [57680352.062491] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57680352.068631] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57680352.075044] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57680352.081124] Call Trace: [57680352.083080] [57680352.084471] [] sched_show_task+0xd7/0x140 [57680352.088756] [] dump_cpu_task+0x39/0x40 [57680352.092201] [] rcu_dump_cpu_stacks+0x80/0xbc [57680352.095933] [] rcu_check_callbacks+0x6ef/0x850 [57680352.099804] [] ? account_system_time+0x81/0x110 [57680352.103565] [] ? account_process_tick+0x60/0x170 [57680352.107360] [] ? tick_sched_do_timer+0x30/0x30 [57680352.111143] [] update_process_times+0x2f/0x60 [57680352.114936] [] tick_sched_handle.isra.13+0x36/0x50 [57680352.118817] [] tick_sched_timer+0x3d/0x70 [57680352.122312] [] __hrtimer_run_queues+0xd6/0x230 [57680352.126206] [] hrtimer_interrupt+0xa8/0x1a0 [57680352.129903] [] xen_timer_interrupt+0x1f/0x30 [57680352.133540] [] __handle_irq_event_percpu+0x3c/0x1a0 [57680352.137508] [] handle_irq_event_percpu+0x23/0x60 [57680352.141327] [] handle_percpu_irq+0x3a/0x50 [57680352.145185] [] generic_handle_irq+0x22/0x30 [57680352.148768] [] evtchn_2l_handle_events+0x238/0x240 [57680352.152743] [] __xen_evtchn_do_upcall+0x43/0x80 [57680352.156674] [] ? get_disk+0x2c/0x60 [57680352.159957] [] xen_evtchn_do_upcall+0x30/0x50 [57680352.163676] [] xen_hvm_callback_vector+0x82/0x90 [57680352.167466] [57680352.168858] [] ? get_disk+0x2c/0x60 [57680352.172829] [] ? try_module_get+0x59/0xe0 [57680352.176388] [] get_disk+0x2c/0x60 [57680352.195016] [] exact_lock+0x11/0x20 [57680352.198420] [] kobj_lookup+0xe1/0x160 [57680352.201872] [] ? disk_map_sector_rcu+0x70/0x70 [57680352.205596] [] get_gendisk+0x34/0x110 [57680352.208938] [] __blkdev_get+0x12d/0x420 [57680352.212389] [] ? bdget+0x3e/0x130 [57680352.215701] [] __blkdev_get+0x296/0x420 [57680352.219247] [] blkdev_get+0x1af/0x300 [57680352.222636] [] blkdev_open+0x5b/0x70 [57680352.226124] [] do_dentry_open+0x213/0x310 [57680352.229707] [] ? blkdev_get_by_dev+0x50/0x50 [57680352.233532] [] vfs_open+0x4f/0x70 [57680352.236856] [] ? may_open+0x9b/0x100 [57680352.240146] [] path_openat+0x529/0x1300 [57680352.243682] [] ? page_add_file_rmap+0x52/0x150 [57680352.247488] [] ? filemap_map_pages+0x366/0x3a0 [57680352.251197] [] do_filp_open+0x7e/0xd0 [57680352.254814] [] ? vma_link+0xb9/0xc0 [57680352.258247] [] ? kmem_cache_alloc+0x181/0x190 [57680352.261976] [] ? __alloc_fd+0xb0/0x170 [57680352.265398] [] do_sys_open+0x115/0x1f0 [57680352.268869] [] SyS_open+0x1e/0x20 [57680352.272072] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57680529.057998] INFO: rcu_sched self-detected stall on CPU [57680529.062799] 0-...: (456725 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=220313 [57680529.069777] (t=457280 jiffies g=2184 c=2183 q=131933) [57680529.074342] Task dump for CPU 0: [57680529.077736] blkid R running task 0 7776 4485 0x0000000c [57680529.084982] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57680529.093334] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57680529.101386] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57680529.109306] Call Trace: [57680529.112117] [57680529.114259] [] sched_show_task+0xd7/0x140 [57680529.120190] [] dump_cpu_task+0x39/0x40 [57680529.124709] [] rcu_dump_cpu_stacks+0x80/0xbc [57680529.129490] [] rcu_check_callbacks+0x6ef/0x850 [57680529.135251] [] ? account_system_time+0x81/0x110 [57680529.141149] [] ? account_process_tick+0x60/0x170 [57680529.146468] [] ? tick_sched_do_timer+0x30/0x30 [57680529.151383] [] update_process_times+0x2f/0x60 [57680529.156954] [] tick_sched_handle.isra.13+0x36/0x50 [57680529.162987] [] tick_sched_timer+0x3d/0x70 [57680529.167607] [] __hrtimer_run_queues+0xd6/0x230 [57680529.172747] [] hrtimer_interrupt+0xa8/0x1a0 [57680529.178158] [] xen_timer_interrupt+0x1f/0x30 [57680529.183280] [] __handle_irq_event_percpu+0x3c/0x1a0 [57680529.188439] [] handle_irq_event_percpu+0x23/0x60 [57680529.194012] [] handle_percpu_irq+0x3a/0x50 [57680529.199364] [] generic_handle_irq+0x22/0x30 [57680529.204555] [] evtchn_2l_handle_events+0x238/0x240 [57680529.209684] [] __xen_evtchn_do_upcall+0x43/0x80 [57680529.215419] [] xen_evtchn_do_upcall+0x30/0x50 [57680529.220960] [] xen_hvm_callback_vector+0x82/0x90 [57680529.226118] [57680529.228001] [] ? mutex_lock+0x10/0x27 [57680529.233511] [] kobj_lookup+0x59/0x160 [57680529.238534] [] ? disk_map_sector_rcu+0x70/0x70 [57680529.243928] [] get_gendisk+0x34/0x110 [57680529.248307] [] __blkdev_get+0x12d/0x420 [57680529.253209] [] ? bdget+0x3e/0x130 [57680529.257842] [] __blkdev_get+0x296/0x420 [57680529.262884] [] blkdev_get+0x1af/0x300 [57680529.267403] [] blkdev_open+0x5b/0x70 [57680529.272068] [] do_dentry_open+0x213/0x310 [57680529.276709] [] ? blkdev_get_by_dev+0x50/0x50 [57680529.282342] [] vfs_open+0x4f/0x70 [57680529.286843] [] ? may_open+0x9b/0x100 [57680529.291371] [] path_openat+0x529/0x1300 [57680529.296260] [] ? page_add_file_rmap+0x52/0x150 [57680529.301901] [] ? filemap_map_pages+0x366/0x3a0 [57680529.307299] [] do_filp_open+0x7e/0xd0 [57680529.311803] [] ? vma_link+0xb9/0xc0 [57680529.316377] [] ? kmem_cache_alloc+0x181/0x190 [57680529.321865] [] ? __alloc_fd+0xb0/0x170 [57680529.326746] [] do_sys_open+0x115/0x1f0 [57680529.331275] [] SyS_open+0x1e/0x20 [57680529.335796] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57680706.075005] INFO: rcu_sched self-detected stall on CPU [57680706.078741] 0-...: (500909 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=241513 [57680706.084354] (t=501533 jiffies g=2184 c=2183 q=132936) [57680706.087973] Task dump for CPU 0: [57680706.090311] blkid R running task 0 7776 4485 0x0000000c [57680706.095506] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57680706.101601] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57680706.107691] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57680706.113785] Call Trace: [57680706.115792] [57680706.117193] [] sched_show_task+0xd7/0x140 [57680706.121411] [] dump_cpu_task+0x39/0x40 [57680706.124861] [] rcu_dump_cpu_stacks+0x80/0xbc [57680706.128493] [] rcu_check_callbacks+0x6ef/0x850 [57680706.132210] [] ? account_system_time+0x81/0x110 [57680706.136077] [] ? account_process_tick+0x60/0x170 [57680706.139894] [] ? tick_sched_do_timer+0x30/0x30 [57680706.143629] [] update_process_times+0x2f/0x60 [57680706.147296] [] tick_sched_handle.isra.13+0x36/0x50 [57680706.151250] [] tick_sched_timer+0x3d/0x70 [57680706.154794] [] __hrtimer_run_queues+0xd6/0x230 [57680706.158479] [] hrtimer_interrupt+0xa8/0x1a0 [57680706.162068] [] xen_timer_interrupt+0x1f/0x30 [57680706.165783] [] __handle_irq_event_percpu+0x3c/0x1a0 [57680706.169736] [] handle_irq_event_percpu+0x23/0x60 [57680706.173528] [] handle_percpu_irq+0x3a/0x50 [57680706.177054] [] generic_handle_irq+0x22/0x30 [57680706.180735] [] evtchn_2l_handle_events+0x238/0x240 [57680706.184646] [] __xen_evtchn_do_upcall+0x43/0x80 [57680706.188441] [] xen_evtchn_do_upcall+0x30/0x50 [57680706.192109] [] xen_hvm_callback_vector+0x82/0x90 [57680706.196047] [57680706.197446] [] ? up_read+0x18/0x30 [57680706.201391] [] zvol_open+0x1cd/0x3a0 [zfs] [57680706.205101] [] __blkdev_get+0xc4/0x420 [57680706.208458] [] ? bdget+0x3e/0x130 [57680706.211695] [] __blkdev_get+0x296/0x420 [57680706.215279] [] blkdev_get+0x1af/0x300 [57680706.218594] [] blkdev_open+0x5b/0x70 [57680706.221872] [] do_dentry_open+0x213/0x310 [57680706.225457] [] ? blkdev_get_by_dev+0x50/0x50 [57680706.229123] [] vfs_open+0x4f/0x70 [57680706.232311] [] ? may_open+0x9b/0x100 [57680706.235654] [] path_openat+0x529/0x1300 [57680706.239088] [] ? page_add_file_rmap+0x52/0x150 [57680706.242892] [] ? filemap_map_pages+0x366/0x3a0 [57680706.246599] [] do_filp_open+0x7e/0xd0 [57680706.249927] [] ? vma_link+0xb9/0xc0 [57680706.253414] [] ? kmem_cache_alloc+0x181/0x190 [57680706.257130] [] ? __alloc_fd+0xb0/0x170 [57680706.260543] [] do_sys_open+0x115/0x1f0 [57680706.263955] [] SyS_open+0x1e/0x20 [57680706.267121] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57680883.092861] INFO: rcu_sched self-detected stall on CPU [57680883.096971] 0-...: (545114 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=262724 [57680883.102688] (t=545786 jiffies g=2184 c=2183 q=133889) [57680883.106411] Task dump for CPU 0: [57680883.108858] blkid R running task 0 7776 4485 0x0000000c [57680883.114071] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57680883.120363] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57680883.126457] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57680883.132488] Call Trace: [57680883.134441] [57680883.135810] [] sched_show_task+0xd7/0x140 [57680883.140142] [] dump_cpu_task+0x39/0x40 [57680883.143738] [] rcu_dump_cpu_stacks+0x80/0xbc [57680883.147400] [] rcu_check_callbacks+0x6ef/0x850 [57680883.151195] [] ? account_system_time+0x81/0x110 [57680883.154962] [] ? account_process_tick+0x60/0x170 [57680883.158983] [] ? tick_sched_do_timer+0x30/0x30 [57680883.162959] [] update_process_times+0x2f/0x60 [57680883.166620] [] tick_sched_handle.isra.13+0x36/0x50 [57680883.170615] [] tick_sched_timer+0x3d/0x70 [57680883.174224] [] __hrtimer_run_queues+0xd6/0x230 [57680883.178165] [] hrtimer_interrupt+0xa8/0x1a0 [57680883.181821] [] xen_timer_interrupt+0x1f/0x30 [57680883.185516] [] __handle_irq_event_percpu+0x3c/0x1a0 [57680883.189491] [] handle_irq_event_percpu+0x23/0x60 [57680883.193307] [] handle_percpu_irq+0x3a/0x50 [57680883.197119] [] generic_handle_irq+0x22/0x30 [57680883.200783] [] evtchn_2l_handle_events+0x238/0x240 [57680883.204674] [] __xen_evtchn_do_upcall+0x43/0x80 [57680883.208478] [] xen_evtchn_do_upcall+0x30/0x50 [57680883.212159] [] xen_hvm_callback_vector+0x82/0x90 [57680883.216153] [57680883.217652] [] ? mutex_lock+0x10/0x27 [57680883.221755] [] kobj_lookup+0x59/0x160 [57680883.240480] [] ? disk_map_sector_rcu+0x70/0x70 [57680883.244437] [] get_gendisk+0x34/0x110 [57680883.247930] [] __blkdev_get+0x12d/0x420 [57680883.251404] [] ? bdget+0x3e/0x130 [57680883.254600] [] __blkdev_get+0x296/0x420 [57680883.258105] [] blkdev_get+0x1af/0x300 [57680883.261473] [] blkdev_open+0x5b/0x70 [57680883.264933] [] do_dentry_open+0x213/0x310 [57680883.268622] [] ? blkdev_get_by_dev+0x50/0x50 [57680883.272256] [] vfs_open+0x4f/0x70 [57680883.275440] [] ? may_open+0x9b/0x100 [57680883.278755] [] path_openat+0x529/0x1300 [57680883.282316] [] ? page_add_file_rmap+0x52/0x150 [57680883.286192] [] ? filemap_map_pages+0x366/0x3a0 [57680883.289991] [] do_filp_open+0x7e/0xd0 [57680883.293356] [] ? vma_link+0xb9/0xc0 [57680883.296632] [] ? kmem_cache_alloc+0x181/0x190 [57680883.300384] [] ? __alloc_fd+0xb0/0x170 [57680883.303943] [] do_sys_open+0x115/0x1f0 [57680883.307464] [] SyS_open+0x1e/0x20 [57680883.310633] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57681060.111142] INFO: rcu_sched self-detected stall on CPU [57681060.115012] 0-...: (589313 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=283940 [57681060.120599] (t=590039 jiffies g=2184 c=2183 q=135190) [57681060.124310] Task dump for CPU 0: [57681060.126638] blkid R running task 0 7776 4485 0x0000000c [57681060.131947] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57681060.138108] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57681060.144391] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57681060.150581] Call Trace: [57681060.152565] [57681060.153949] [] sched_show_task+0xd7/0x140 [57681060.158168] [] dump_cpu_task+0x39/0x40 [57681060.161779] [] rcu_dump_cpu_stacks+0x80/0xbc [57681060.165505] [] rcu_check_callbacks+0x6ef/0x850 [57681060.169205] [] ? account_system_time+0x81/0x110 [57681060.172960] [] ? account_process_tick+0x60/0x170 [57681060.176744] [] ? tick_sched_do_timer+0x30/0x30 [57681060.180612] [] update_process_times+0x2f/0x60 [57681060.184434] [] tick_sched_handle.isra.13+0x36/0x50 [57681060.188309] [] tick_sched_timer+0x3d/0x70 [57681060.191815] [] __hrtimer_run_queues+0xd6/0x230 [57681060.195563] [] hrtimer_interrupt+0xa8/0x1a0 [57681060.199409] [] xen_timer_interrupt+0x1f/0x30 [57681060.203222] [] __handle_irq_event_percpu+0x3c/0x1a0 [57681060.207138] [] handle_irq_event_percpu+0x23/0x60 [57681060.210943] [] handle_percpu_irq+0x3a/0x50 [57681060.214529] [] generic_handle_irq+0x22/0x30 [57681060.218289] [] evtchn_2l_handle_events+0x238/0x240 [57681060.222330] [] __xen_evtchn_do_upcall+0x43/0x80 [57681060.226110] [] xen_evtchn_do_upcall+0x30/0x50 [57681060.229778] [] xen_hvm_callback_vector+0x82/0x90 [57681060.233582] [57681060.234980] [] ? disk_unblock_events+0x1b/0x20 [57681060.239497] [] ? __blkdev_get+0x112/0x420 [57681060.243325] [] ? bdget+0x3e/0x130 [57681060.246640] [] __blkdev_get+0x296/0x420 [57681060.250037] [] blkdev_get+0x1af/0x300 [57681060.253400] [] blkdev_open+0x5b/0x70 [57681060.256698] [] do_dentry_open+0x213/0x310 [57681060.260328] [] ? blkdev_get_by_dev+0x50/0x50 [57681060.264158] [] vfs_open+0x4f/0x70 [57681060.267361] [] ? may_open+0x9b/0x100 [57681060.270647] [] path_openat+0x529/0x1300 [57681060.274186] [] ? page_add_file_rmap+0x52/0x150 [57681060.277985] [] ? filemap_map_pages+0x366/0x3a0 [57681060.282161] [] do_filp_open+0x7e/0xd0 [57681060.285543] [] ? vma_link+0xb9/0xc0 [57681060.288788] [] ? kmem_cache_alloc+0x181/0x190 [57681060.292480] [] ? __alloc_fd+0xb0/0x170 [57681060.295869] [] do_sys_open+0x115/0x1f0 [57681060.299446] [] SyS_open+0x1e/0x20 [57681060.302865] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57681237.129633] INFO: rcu_sched self-detected stall on CPU [57681237.133573] 0-...: (633519 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=305163 [57681237.139177] (t=634292 jiffies g=2184 c=2183 q=136218) [57681237.142856] Task dump for CPU 0: [57681237.145197] blkid R running task 0 7776 4485 0x0000000c [57681237.150405] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57681237.156574] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57681237.162777] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57681237.168785] Call Trace: [57681237.170736] [57681237.172117] [] sched_show_task+0xd7/0x140 [57681237.176343] [] dump_cpu_task+0x39/0x40 [57681237.179998] [] rcu_dump_cpu_stacks+0x80/0xbc [57681237.183710] [] rcu_check_callbacks+0x6ef/0x850 [57681237.187468] [] ? account_system_time+0x81/0x110 [57681237.191232] [] ? account_process_tick+0x60/0x170 [57681237.195048] [] ? tick_sched_do_timer+0x30/0x30 [57681237.198932] [] update_process_times+0x2f/0x60 [57681237.202814] [] tick_sched_handle.isra.13+0x36/0x50 [57681237.206718] [] tick_sched_timer+0x3d/0x70 [57681237.210323] [] __hrtimer_run_queues+0xd6/0x230 [57681237.214033] [] hrtimer_interrupt+0xa8/0x1a0 [57681237.217745] [] xen_timer_interrupt+0x1f/0x30 [57681237.221700] [] __handle_irq_event_percpu+0x3c/0x1a0 [57681237.225681] [] handle_irq_event_percpu+0x23/0x60 [57681237.229643] [] handle_percpu_irq+0x3a/0x50 [57681237.233187] [] generic_handle_irq+0x22/0x30 [57681237.236993] [] evtchn_2l_handle_events+0x238/0x240 [57681237.240987] [] __xen_evtchn_do_upcall+0x43/0x80 [57681237.244910] [] xen_evtchn_do_upcall+0x30/0x50 [57681237.248673] [] xen_hvm_callback_vector+0x82/0x90 [57681237.252658] [57681237.254108] [] ? __raw_callee_save___pv_queued_spin_unlock+0x10/0x17 [57681237.259497] [] zvol_open+0xc0/0x3a0 [zfs] [57681237.262976] [] __blkdev_get+0xc4/0x420 [57681237.266420] [] ? bdget+0x3e/0x130 [57681237.269737] [] __blkdev_get+0x296/0x420 [57681237.273253] [] blkdev_get+0x1af/0x300 [57681237.276595] [] blkdev_open+0x5b/0x70 [57681237.279873] [] do_dentry_open+0x213/0x310 [57681237.283447] [] ? blkdev_get_by_dev+0x50/0x50 [57681237.287273] [] vfs_open+0x4f/0x70 [57681237.290588] [] ? may_open+0x9b/0x100 [57681237.293880] [] path_openat+0x529/0x1300 [57681237.297354] [] ? page_add_file_rmap+0x52/0x150 [57681237.301071] [] ? filemap_map_pages+0x366/0x3a0 [57681237.305005] [] do_filp_open+0x7e/0xd0 [57681237.308443] [] ? vma_link+0xb9/0xc0 [57681237.311706] [] ? kmem_cache_alloc+0x181/0x190 [57681237.315358] [] ? __alloc_fd+0xb0/0x170 [57681237.318771] [] do_sys_open+0x115/0x1f0 [57681237.322180] [] SyS_open+0x1e/0x20 [57681237.325510] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57681414.147965] INFO: rcu_sched self-detected stall on CPU [57681414.152123] 0-...: (677724 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=326412 [57681414.160433] (t=678545 jiffies g=2184 c=2183 q=139761) [57681414.165281] Task dump for CPU 0: [57681414.167689] blkid R running task 0 7776 4485 0x0000000c [57681414.173194] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57681414.179223] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57681414.185595] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57681414.192357] Call Trace: [57681414.194327] [57681414.195697] [] sched_show_task+0xd7/0x140 [57681414.200145] [] dump_cpu_task+0x39/0x40 [57681414.203882] [] rcu_dump_cpu_stacks+0x80/0xbc [57681414.207717] [] rcu_check_callbacks+0x6ef/0x850 [57681414.213102] [] ? account_system_time+0x81/0x110 [57681414.218553] [] ? account_process_tick+0x60/0x170 [57681414.223739] [] ? tick_sched_do_timer+0x30/0x30 [57681414.227562] [] update_process_times+0x2f/0x60 [57681414.231589] [] tick_sched_handle.isra.13+0x36/0x50 [57681414.237221] [] tick_sched_timer+0x3d/0x70 [57681414.242386] [] __hrtimer_run_queues+0xd6/0x230 [57681414.247147] [] hrtimer_interrupt+0xa8/0x1a0 [57681414.250836] [] xen_timer_interrupt+0x1f/0x30 [57681414.254960] [] __handle_irq_event_percpu+0x3c/0x1a0 [57681414.258964] [] handle_irq_event_percpu+0x23/0x60 [57681414.262901] [] handle_percpu_irq+0x3a/0x50 [57681414.266521] [] generic_handle_irq+0x22/0x30 [57681414.270197] [] evtchn_2l_handle_events+0x238/0x240 [57681414.274438] [] __xen_evtchn_do_upcall+0x43/0x80 [57681414.280249] [] xen_evtchn_do_upcall+0x30/0x50 [57681414.285930] [] xen_hvm_callback_vector+0x82/0x90 [57681414.290859] [57681414.292250] [] ? _raw_spin_lock+0x14/0x30 [57681414.296747] [] ? zvol_open+0x100/0x3a0 [zfs] [57681414.301069] [] __blkdev_get+0xc4/0x420 [57681414.304476] [] ? bdget+0x3e/0x130 [57681414.307728] [] __blkdev_get+0x296/0x420 [57681414.311370] [] blkdev_get+0x1af/0x300 [57681414.314868] [] blkdev_open+0x5b/0x70 [57681414.333826] [] do_dentry_open+0x213/0x310 [57681414.337530] [] ? blkdev_get_by_dev+0x50/0x50 [57681414.341159] [] vfs_open+0x4f/0x70 [57681414.344354] [] ? may_open+0x9b/0x100 [57681414.347802] [] path_openat+0x529/0x1300 [57681414.351497] [] ? page_add_file_rmap+0x52/0x150 [57681414.355664] [] ? filemap_map_pages+0x366/0x3a0 [57681414.359395] [] do_filp_open+0x7e/0xd0 [57681414.362871] [] ? vma_link+0xb9/0xc0 [57681414.366271] [] ? kmem_cache_alloc+0x181/0x190 [57681414.370181] [] ? __alloc_fd+0xb0/0x170 [57681414.373642] [] do_sys_open+0x115/0x1f0 [57681414.377027] [] SyS_open+0x1e/0x20 [57681414.380209] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57681591.163876] INFO: rcu_sched self-detected stall on CPU [57681591.169217] 0-...: (721919 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=347662 [57681591.176054] (t=722798 jiffies g=2184 c=2183 q=143279) [57681591.180623] Task dump for CPU 0: [57681591.183508] blkid R running task 0 7776 4485 0x0000000c [57681591.190856] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57681591.199342] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57681591.207557] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57681591.215344] Call Trace: [57681591.218095] [57681591.220171] [] sched_show_task+0xd7/0x140 [57681591.226068] [] dump_cpu_task+0x39/0x40 [57681591.230786] [] rcu_dump_cpu_stacks+0x80/0xbc [57681591.235600] [] rcu_check_callbacks+0x6ef/0x850 [57681591.240828] [] ? account_system_time+0x81/0x110 [57681591.246407] [] ? account_process_tick+0x60/0x170 [57681591.251759] [] ? tick_sched_do_timer+0x30/0x30 [57681591.256684] [] update_process_times+0x2f/0x60 [57681591.262077] [] tick_sched_handle.isra.13+0x36/0x50 [57681591.267927] [] tick_sched_timer+0x3d/0x70 [57681591.272759] [] __hrtimer_run_queues+0xd6/0x230 [57681591.277730] [] hrtimer_interrupt+0xa8/0x1a0 [57681591.283034] [] xen_timer_interrupt+0x1f/0x30 [57681591.288473] [] __handle_irq_event_percpu+0x3c/0x1a0 [57681591.293910] [] handle_irq_event_percpu+0x23/0x60 [57681591.299110] [] handle_percpu_irq+0x3a/0x50 [57681591.304351] [] generic_handle_irq+0x22/0x30 [57681591.309470] [] evtchn_2l_handle_events+0x238/0x240 [57681591.314925] [] __xen_evtchn_do_upcall+0x43/0x80 [57681591.320409] [] xen_evtchn_do_upcall+0x30/0x50 [57681591.325268] [] xen_hvm_callback_vector+0x82/0x90 [57681591.330754] [57681591.332798] [] ? __raw_callee_save___pv_queued_spin_unlock+0x14/0x17 [57681591.340453] [] zvol_open+0x112/0x3a0 [zfs] [57681591.345145] [] __blkdev_get+0xc4/0x420 [57681591.350003] [] ? bdget+0x3e/0x130 [57681591.354639] [] __blkdev_get+0x296/0x420 [57681591.359713] [] blkdev_get+0x1af/0x300 [57681591.364374] [] blkdev_open+0x5b/0x70 [57681591.368857] [] do_dentry_open+0x213/0x310 [57681591.374084] [] ? blkdev_get_by_dev+0x50/0x50 [57681591.379400] [] vfs_open+0x4f/0x70 [57681591.383610] [] ? may_open+0x9b/0x100 [57681591.387972] [] path_openat+0x529/0x1300 [57681591.393080] [] ? page_add_file_rmap+0x52/0x150 [57681591.398795] [] ? filemap_map_pages+0x366/0x3a0 [57681591.403729] [] do_filp_open+0x7e/0xd0 [57681591.408155] [] ? vma_link+0xb9/0xc0 [57681591.412970] [] ? kmem_cache_alloc+0x181/0x190 [57681591.418412] [] ? __alloc_fd+0xb0/0x170 [57681591.422878] [] do_sys_open+0x115/0x1f0 [57681591.427322] [] SyS_open+0x1e/0x20 [57681591.431999] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57681768.180500] INFO: rcu_sched self-detected stall on CPU [57681768.184382] 0-...: (766105 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=368887 [57681768.189954] (t=767051 jiffies g=2184 c=2183 q=178692) [57681768.193687] Task dump for CPU 0: [57681768.196027] blkid R running task 0 7776 4485 0x0000000c [57681768.201327] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57681768.207505] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57681768.213836] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57681768.219843] Call Trace: [57681768.221850] [57681768.223236] [] sched_show_task+0xd7/0x140 [57681768.227422] [] dump_cpu_task+0x39/0x40 [57681768.230963] [] rcu_dump_cpu_stacks+0x80/0xbc [57681768.234704] [] rcu_check_callbacks+0x6ef/0x850 [57681768.238414] [] ? account_system_time+0x81/0x110 [57681768.242183] [] ? account_process_tick+0x60/0x170 [57681768.246029] [] ? tick_sched_do_timer+0x30/0x30 [57681768.249939] [] update_process_times+0x2f/0x60 [57681768.253712] [] tick_sched_handle.isra.13+0x36/0x50 [57681768.257586] [] tick_sched_timer+0x3d/0x70 [57681768.261263] [] __hrtimer_run_queues+0xd6/0x230 [57681768.265043] [] hrtimer_interrupt+0xa8/0x1a0 [57681768.268841] [] xen_timer_interrupt+0x1f/0x30 [57681768.272617] [] __handle_irq_event_percpu+0x3c/0x1a0 [57681768.276621] [] handle_irq_event_percpu+0x23/0x60 [57681768.280541] [] handle_percpu_irq+0x3a/0x50 [57681768.284149] [] generic_handle_irq+0x22/0x30 [57681768.287866] [] evtchn_2l_handle_events+0x238/0x240 [57681768.291919] [] __xen_evtchn_do_upcall+0x43/0x80 [57681768.295676] [] xen_evtchn_do_upcall+0x30/0x50 [57681768.299434] [] xen_hvm_callback_vector+0x82/0x90 [57681768.303508] [57681768.304916] [] ? mutex_lock+0x10/0x27 [57681768.309281] [] zvol_open+0x5a/0x3a0 [zfs] [57681768.313265] [] __blkdev_get+0xc4/0x420 [57681768.316710] [] ? bdget+0x3e/0x130 [57681768.319871] [] __blkdev_get+0x296/0x420 [57681768.323321] [] blkdev_get+0x1af/0x300 [57681768.326678] [] blkdev_open+0x5b/0x70 [57681768.330075] [] do_dentry_open+0x213/0x310 [57681768.333788] [] ? blkdev_get_by_dev+0x50/0x50 [57681768.337478] [] vfs_open+0x4f/0x70 [57681768.340639] [] ? may_open+0x9b/0x100 [57681768.343936] [] path_openat+0x529/0x1300 [57681768.347354] [] ? page_add_file_rmap+0x52/0x150 [57681768.351112] [] ? filemap_map_pages+0x366/0x3a0 [57681768.354976] [] do_filp_open+0x7e/0xd0 [57681768.358400] [] ? vma_link+0xb9/0xc0 [57681768.361893] [] ? kmem_cache_alloc+0x181/0x190 [57681768.365546] [] ? __alloc_fd+0xb0/0x170 [57681768.368945] [] do_sys_open+0x115/0x1f0 [57681768.372355] [] SyS_open+0x1e/0x20 [57681768.375523] [] entry_SYSCALL_64_fastpath+0x1a/0xa9 [57681945.196014] INFO: rcu_sched self-detected stall on CPU [57681945.200890] 0-...: (810310 ticks this GP) idle=d79/140000000000001/0 softirq=12492/12492 fqs=390191 [57681945.208037] (t=811304 jiffies g=2184 c=2183 q=184757) [57681945.212598] Task dump for CPU 0: [57681945.216041] blkid R running task 0 7776 4485 0x0000000c [57681945.223149] ffff8801e7a03ce8 ffffffff8109d6a7 0000000000000000 0000000000000000 [57681945.231547] ffff8801e7a03d00 ffffffff8109fff9 ffffffff81a54100 ffff8801e7a03d30 [57681945.239814] ffffffff81173a40 ffff8801e7a18940 ffffffff81a54100 0000000000000000 [57681945.247500] Call Trace: [57681945.250000] [57681945.252138] [] sched_show_task+0xd7/0x140 [57681945.258019] [] dump_cpu_task+0x39/0x40 [57681945.263046] [] rcu_dump_cpu_stacks+0x80/0xbc [57681945.267945] [] rcu_check_callbacks+0x6ef/0x850 [57681945.273128] [] ? account_system_time+0x81/0x110 [57681945.278875] [] ? account_process_tick+0x60/0x170 [57681945.284512] [] ? tick_sched_do_timer+0x30/0x30 [57681945.289404] [] update_process_times+0x2f/0x60 [57681945.294347] [] tick_sched_handle.isra.13+0x36/0x50 [57681945.300363] [] tick_sched_timer+0x3d/0x70 [57681945.305537] [] __hrtimer_run_queues+0xd6/0x230 [57681945.310434] [] hrtimer_interrupt+0xa8/0x1a0 [57681945.315109] [] xen_timer_interrupt+0x1f/0x30 [57681945.320682] [] __handle_irq_event_percpu+0x3c/0x1a0 [57681945.326659] [] handle_irq_event_percpu+0x23/0x60 [57681945.331683] [] handle_percpu_irq+0x3a/0x50 [57681945.336714] [] generic_handle_irq+0x22/0x30 [57681945.342032] [] evtchn_2l_handle_events+0x238/0x240 [57681945.347877] [] __xen_evtchn_do_upcall+0x43/0x80 [57681945.352839] [] xen_evtchn_do_upcall+0x30/0x50 [57681945.358359] [] xen_hvm_callback_vector+0x82/0x90 [57681945.364230] [57681945.366342] [] ? mutex_lock+0x17/0x27 [57681945.371566] [] zvol_open+0x9a/0x3a0 [zfs] [57681945.376186] [] __blkdev_get+0xc4/0x420 [57681945.380984] [] ? bdget+0x3e/0x130 [57681945.385747] [] __blkdev_get+0x296/0x420 [57681945.390642] [] blkdev_get+0x1af/0x300 [57681945.395104] [] blkdev_open+0x5b/0x70 [57681945.399411] [] do_dentry_open+0x213/0x310 [57681945.404733] [] ? blkdev_get_by_dev+0x50/0x50 [57681945.410186] [] vfs_open+0x4f/0x70 [57681945.414362] [] ? may_open+0x9b/0x100 [57681945.418859] [] path_openat+0x529/0x1300 [57681945.423928] [] ? page_add_file_rmap+0x52/0x150 [57681945.444843] [] ? filemap_map_pages+0x366/0x3a0 [57681945.449948] [] do_filp_open+0x7e/0xd0 [57681945.454288] [] ? vma_link+0xb9/0xc0 [57681945.458762] [] ? kmem_cache_alloc+0x181/0x190 [57681945.464261] [] ? __alloc_fd+0xb0/0x170 [57681945.469317] [] do_sys_open+0x115/0x1f0 [57681945.473925] [] SyS_open+0x1e/0x20 [57681945.478280] [] entry_SYSCALL_64_fastpath+0x1a/0xa9