page allocation failure

[nzt4567]

Hi,

not sure if this can be related to performance issues reported in #93 but we are also experiencing the following crash in dmesg output

[Wed Oct 31 10:41:44 2018] QAT: Stopping all acceleration devices.
[Wed Oct 31 10:41:44 2018] c6xx 0000:06:00.0: qat_dev0 stopped 10 acceleration engines
[Wed Oct 31 10:41:44 2018] c6xx 0000:06:00.0: Resetting device qat_dev0
[Wed Oct 31 10:41:44 2018] c6xx 0000:06:00.0: Function level reset
[Wed Oct 31 10:41:44 2018] c6xx 0000:07:00.0: qat_dev1 stopped 10 acceleration engines
[Wed Oct 31 10:41:44 2018] c6xx 0000:07:00.0: Resetting device qat_dev1
[Wed Oct 31 10:41:44 2018] c6xx 0000:07:00.0: Function level reset
[Wed Oct 31 10:41:44 2018] c6xx 0000:08:00.0: qat_dev2 stopped 10 acceleration engines
[Wed Oct 31 10:41:44 2018] c6xx 0000:08:00.0: Resetting device qat_dev2
[Wed Oct 31 10:41:44 2018] c6xx 0000:08:00.0: Function level reset
[Wed Oct 31 10:41:44 2018] c6xx 0000:06:00.0: Starting acceleration device qat_dev0.
[Wed Oct 31 10:41:44 2018] c6xx 0000:06:00.0: firmware: direct-loading firmware qat_c62x_mmp.bin
[Wed Oct 31 10:41:44 2018] c6xx 0000:06:00.0: firmware: direct-loading firmware qat_c62x.bin
[Wed Oct 31 10:41:45 2018] c6xx 0000:06:00.0: qat_dev0 started 10 acceleration engines
[Wed Oct 31 10:41:45 2018] c6xx 0000:07:00.0: Starting acceleration device qat_dev1.
[Wed Oct 31 10:41:45 2018] c6xx 0000:07:00.0: qat_dev1 started 10 acceleration engines
[Wed Oct 31 10:41:45 2018] c6xx 0000:08:00.0: Starting acceleration device qat_dev2.
[Wed Oct 31 10:41:46 2018] c6xx 0000:08:00.0: qat_dev2 started 10 acceleration engines
[Wed Oct 31 10:55:30 2018] perf: interrupt took too long (4997 > 4992), lowering kernel.perf_event_max_sample_rate to 40000
[Wed Oct 31 10:58:19 2018] nginx: page allocation failure: order:9, mode:0x26040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK)
[Wed Oct 31 10:58:19 2018] CPU: 26 PID: 29088 Comm: nginx Tainted: G           O    4.9.0-8-amd64 #1 Debian 4.9.110-3+deb9u6
[Wed Oct 31 10:58:19 2018] Hardware name: Supermicro Super Server/X10DRW-N, BIOS 3.1 06/07/2018
[Wed Oct 31 10:58:19 2018]  0000000000000000 ffffffff95531e54 ffffffff95c016e8 ffffb2620cce7b00
[Wed Oct 31 10:58:19 2018]  ffffffff9538a84a 026040c0026040c0 ffffffff95c016e8 ffffb2620cce7aa0
[Wed Oct 31 10:58:19 2018]  ffffa07000000010 ffffb2620cce7b10 ffffb2620cce7ac0 e706296b54befc67
[Wed Oct 31 10:58:19 2018] Call Trace:
[Wed Oct 31 10:58:19 2018]  [<ffffffff95531e54>] ? dump_stack+0x5c/0x78
[Wed Oct 31 10:58:19 2018]  [<ffffffff9538a84a>] ? warn_alloc+0x13a/0x160
[Wed Oct 31 10:58:19 2018]  [<ffffffff9538a58a>] ? __alloc_pages_direct_compact+0x4a/0xf0
[Wed Oct 31 10:58:19 2018]  [<ffffffff9538ab74>] ? __alloc_pages_slowpath+0x294/0xbf0
[Wed Oct 31 10:58:19 2018]  [<ffffffff9538ab74>] ? __alloc_pages_slowpath+0x294/0xbf0
[Wed Oct 31 10:58:19 2018]  [<ffffffff9538b6d1>] ? __alloc_pages_nodemask+0x201/0x260
[Wed Oct 31 10:58:19 2018]  [<ffffffff953e4d0a>] ? cache_grow_begin+0x9a/0x560
[Wed Oct 31 10:58:19 2018]  [<ffffffff953e4d0a>] ? cache_grow_begin+0x9a/0x560
[Wed Oct 31 10:58:19 2018]  [<ffffffff953e5481>] ? fallback_alloc+0x161/0x200
[Wed Oct 31 10:58:19 2018]  [<ffffffff953e5bc1>] ? kmem_cache_alloc_node_trace+0xb1/0x5a0
[Wed Oct 31 10:58:19 2018]  [<ffffffffc0825a27>] ? mem_ioctl+0x447/0x800 [usdm_drv]
[Wed Oct 31 10:58:19 2018]  [<ffffffff9541d5d2>] ? do_vfs_ioctl+0xa2/0x620
[Wed Oct 31 10:58:19 2018]  [<ffffffff9541dbc4>] ? SyS_ioctl+0x74/0x80
[Wed Oct 31 10:58:19 2018]  [<ffffffff95203b7d>] ? do_syscall_64+0x8d/0xf0
[Wed Oct 31 10:58:19 2018]  [<ffffffff95815c4e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
[Wed Oct 31 10:58:19 2018] Mem-Info:
[Wed Oct 31 10:58:19 2018] active_anon:1128871 inactive_anon:129222 isolated_anon:0
                            active_file:29125668 inactive_file:882274 isolated_file:0
                            unevictable:8 dirty:328 writeback:0 unstable:0
                            slab_reclaimable:488035 slab_unreclaimable:237499
                            mapped:125914 shmem:130561 pagetables:14790 bounce:0
                            free:335526 free_pcp:361 free_cma:0
[Wed Oct 31 10:58:19 2018] Node 0 active_anon:4515484kB inactive_anon:516888kB active_file:116502672kB inactive_file:3529096kB unevictable:32kB isolated(anon):0kB isolated(file):0kB mapped:503656kB dirty:1312kB writeback:0kB shmem:522244kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 376832kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
[Wed Oct 31 10:58:19 2018] Node 0 DMA free:15884kB min:60kB low:72kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15968kB managed:15884kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Wed Oct 31 10:58:19 2018] lowmem_reserve[]: 0 1833 128787 128787 128787
[Wed Oct 31 10:58:19 2018] Node 0 DMA32 free:515980kB min:7368kB low:9264kB high:11160kB active_anon:70048kB inactive_anon:600kB active_file:1151904kB inactive_file:34736kB unevictable:0kB writepending:0kB present:1964600kB managed:1899032kB mlocked:0kB slab_reclaimable:9960kB slab_unreclaimable:109276kB kernel_stack:0kB pagetables:80kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Wed Oct 31 10:58:19 2018] lowmem_reserve[]: 0 0 126954 126954 126954
[Wed Oct 31 10:58:19 2018] Node 0 Normal free:810208kB min:504564kB low:634568kB high:764572kB active_anon:4445436kB inactive_anon:516288kB active_file:115350768kB inactive_file:3494372kB unevictable:32kB writepending:1312kB present:132120576kB managed:130005724kB mlocked:32kB slab_reclaimable:1942180kB slab_unreclaimable:840720kB kernel_stack:11856kB pagetables:59080kB bounce:0kB free_pcp:1432kB local_pcp:0kB free_cma:0kB
[Wed Oct 31 10:58:19 2018] lowmem_reserve[]: 0 0 0 0 0
[Wed Oct 31 10:58:19 2018] Node 0 DMA: 1*4kB (U) 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15884kB
[Wed Oct 31 10:58:19 2018] Node 0 DMA32: 1898*4kB (UME) 1852*8kB (UME) 1404*16kB (UME) 2597*32kB (UME) 831*64kB (UME) 218*128kB (UM) 179*256kB (UM) 70*512kB (UM) 36*1024kB (UME) 2*2048kB (UM) 45*4096kB (UM) = 516008kB
[Wed Oct 31 10:58:19 2018] Node 0 Normal: 145343*4kB (UME) 14635*8kB (UME) 5327*16kB (UME) 826*32kB (UM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 810116kB
[Wed Oct 31 10:58:19 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Wed Oct 31 10:58:19 2018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Wed Oct 31 10:58:19 2018] 30139229 total pagecache pages
[Wed Oct 31 10:58:19 2018] 0 pages in swap cache
[Wed Oct 31 10:58:19 2018] Swap cache stats: add 0, delete 0, find 0/0
[Wed Oct 31 10:58:19 2018] Free swap  = 0kB
[Wed Oct 31 10:58:19 2018] Total swap = 0kB
[Wed Oct 31 10:58:19 2018] 33525286 pages RAM
[Wed Oct 31 10:58:19 2018] 0 pages HighMem/MovableOnly
[Wed Oct 31 10:58:19 2018] 545126 pages reserved
[Wed Oct 31 10:58:19 2018] 0 pages hwpoisoned
[Wed Oct 31 10:58:19 2018] usdm_drv: userMemAlloc:394 Unable to allocate memory slab or wrong alignment:           (null)
[Wed Oct 31 10:58:19 2018] usdm_drv: dev_mem_alloc:599 userMemAlloc failed

Setup, configuration and everything else is the same as described in #93. I can provide you with any other info you might need to debug this :)

Thanks,
Tomas

[nzt4567]

Hi @stevelinsell,

I just wanted to ask if there is anything else I can do to help debug this. I feel like this might be an issue of the driver itself, but I have not found a way how to report bugs to its creators.

Thanks,
Tomas