async io for blk_posix module

[fengggli]

I added the following test function in the "src/components/block/posix/unit_test/test1.cpp"(before the "BasicAsync" test).
It uses one 4k page(mem) as a buffer and first writes 0xcc into each byte of the first block,and 0xff into each byte of the second block.
It's more like a stress test where the values are frequently read in and written back between one memory page and two logical blocks. (it also represents one extreme case in Pager_Simple: there is only once physical page and two blocks for swapping)
But the test cannot pass in my qemu vm.
Could you help to confirm it in your side?

Thanks

TEST_F(Block_posix_test, AsyncSwap)                                                                                                                                                                                                           
{   
    uint64_t nr_iterations = 10000;
    io_buffer_t mem = _block->allocate_io_buffer(4096,4096,Component::NUMA_NODE_ANY);
    void * ptr = _block->virt_addr(mem);
    char *p = (char*) ptr;
    
    uint64_t tag;
    
    memset(ptr, 0x0, 4096);
    
    /* two blocks, 1 filled with 0xcc and the other with 0xff*/
    for(unsigned i=0;i<4096;i++) p[i] = 0xcc;
    
    tag = _block->async_write(mem, 0, 1, 1);
    
    for(unsigned i=0;i<4096;i++) p[i] = 0xff;
    
    for(uint64_t i = 0; i < nr_iterations; i++){
        /*
         * write second blk and read first blk
         */
        while(!_block->check_completion(tag)) {
            cpu_relax();
        }
        tag = _block->async_write(mem, 0, 2, 1);
        
        _block->read(mem, 0, 1, 1);
        
        if(p[0] !=0xcc){
            PWRN("eeek on blk 1!! p[0]=0x%x%x",0xf&(p[0]>>4), 0xf &p[0]);
            ASSERT_TRUE(0);
        }
        
        /*
         * write first blk and read second blk
         */
        while(!_block->check_completion(tag)) {
            cpu_relax();
        }
    
        tag = _block->async_write(mem, 0, 1, 1);
        _block->read(mem, 0, 2, 1);
    
        if(p[0] !=0xff){
            PWRN("eeek on blk 2!! p[0]=0x%x%x",0xf&(p[0]>>4), 0xf &p[0]);
            ASSERT_TRUE(0);
        }
    
    }
    _block->free_io_buffer(mem);
    PMAJOR("> async swap test OK");

}

The first byte of block 1 should be read as 0xcc, but 0xff is read instead:

root@node0:/home/lifeng/comanche/src/components/block/posix/unit_test# ./blockposix-test1 /home/lifeng/comanche/blockfile.dat 
[==========] Running 9 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 9 tests from Block_posix_test
[ RUN      ] Block_posix_test.InstantiateBlockDevice
[LOG]:Load path: libcomanche-blkposix.so
[LOG]:Creating 'Block_posix' factory.
[LOG]:Block_device factory loaded OK.
[LOG]:config: {"path":"/home/lifeng/comanche/blockfile.dat","size_in_blocks" : 100 }
[LOG]:Block_posix: path=(/home/lifeng/comanche/blockfile.dat) size=(100)
[LOG]:Block_posix: XMS available!
[LOG]:Using API defined memory limit 256 MB
[LOG]:CPU count: 4
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
# DPDK EAL set maps & initialized OK.
# DPDK EAL initialized ok (--proc-type=primary).
[LOG]:unloading component (0x1f26f50)
POSIX-based block-layer component loaded OK.
[       OK ] Block_posix_test.InstantiateBlockDevice (825 ms)
[ RUN      ] Block_posix_test.AsyncSwap
[WRN]: eeek on blk 1!! p[0]=0xff
/home/lifeng/comanche/src/components/block/posix/unit_test/test1.cpp:121: Failure
Value of: 0
  Actual: false
Expected: true

[dwaddington]

Yes, this fails on my machine. blk 1!! p[0]=0xff

[dwaddington]

Did it pass on the NVMe block device?

[fengggli]

@dwaddington cannot pass in block_nvme(using uio-generic in qemu)

[fengggli]

The log of block_nvme(using uio):
BTW, there are some warnings/nvme for nvme init and "512" block size also seems incorrect in my emulator.
I might not have configured it correctly(previously I only use the posix_block).

root@node0:/home/lifeng/comanche/src/components/block/nvme/unit_test# ./blocknvme-test1 00:09.0
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from Block_nvme_test
[ RUN      ] Block_nvme_test.InstantiateBlockDevice
[LOG]:Load path: libcomanche-blknvme.so
[LOG]:Using API defined memory limit 48 MB
[LOG]:CPU count: 4
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
# DPDK EAL set maps & initialized OK.
# DPDK EAL initialized ok (--proc-type=primary).
Creating 'Block_device_factory' component.
[LOG]:Block_device factory loaded OK.
[LOG]:Ring_buffer_base [rbuf-0]: allocated at 0x7f20be4ebd00 size=131072
[LOG]:Nvme_device (IO threads):0x71bfe0
[LOG]:Nvme_device: descriptor ring size 131072
[LOG]:Looking for NVMe Controller (00:09.0)...
[LOG]:Using device: 0000:00:09.0
nvme_qpair.c: 112:nvme_admin_qpair_print_command: *NOTICE*: SET FEATURES (09) sqid:0 cid:63 nsid:0 cdw10:0000000b cdw11:0000001f
nvme_qpair.c: 283:nvme_qpair_print_completion: *NOTICE*: INVALID FIELD (00/02) sqid:0 cid:63 cdw0:0 sqhd:0005 p:1 m:0 dnr:1
nvme_ctrlr.c:1030:nvme_ctrlr_configure_aer: *ERROR*: nvme_ctrlr_cmd_set_async_event_config failed!
nvme_qpair.c: 112:nvme_admin_qpair_print_command: *NOTICE*: GET LOG PAGE (02) sqid:0 cid:63 nsid:ffffffff cdw10:007f00c0 cdw11:00000000
nvme_qpair.c: 283:nvme_qpair_print_completion: *NOTICE*: INVALID OPCODE (00/01) sqid:0 cid:63 cdw0:0 sqhd:0006 p:1 m:0 dnr:1
nvme_ctrlr.c: 360:nvme_ctrlr_set_intel_support_log_pages: *ERROR*: nvme_ctrlr_cmd_get_log_page failed!
Attaching to NVMe device 0000:00:09.0::
Using controller QEMU NVMe Ctrl       (1236                ) with 1 namespaces
[LOG]:Probe complete (0x7f20be4e90c0,0x7f20be4dce40)
[ctlr-info] sqes: min(64) max(64)
[ctlr-info] cqes: min(16) max(16)
[ctlr-info] awun: 1
[ctlr-info] awupf: 1
[ctlr-info] acwu: 0
[ctlr-info] fused op: N
[ctlr-info] metadata size: 0
[ctlr-info] max IO size: 2072576
[ns-info] extended LBA support: N
[ns-info] metadata transfer as extended LBA: N
[ns-info] metadata transfer as separate pointer: N
[ns-info] nsze: 4194304
[ns-info] ncap: 4194304
[LOG]:Launching IO thread: 1
[LOG]:allocating queue [0x7f20be4dcd90] (0) on namespace:1 block size=512
[LOG]:created new IO queue: namespace=1 max_lba=4194304
NVMe queue IO thread (core 1) entered.
[LOG]:Launching IO thread: 2
[LOG]:registering queue message ring (core=1) : 0x7f20be18fe00
[LOG]:allocating queue [0x7f20be393f50] (1) on namespace:1 block size=512
[LOG]:created new IO queue: namespace=1 max_lba=4194304
NVMe queue IO thread (core 2) entered.
[LOG]:registering queue message ring (core=2) : 0x7f20be084e00
[LOG]:unloading component (0x711510)
nvme-based block-layer component loaded OK.
[       OK ] Block_nvme_test.InstantiateBlockDevice (570 ms)
[ RUN      ] Block_nvme_test.AsyncSwap
[WRN]: eeek on blk 1!! p[0]=0xff
/home/lifeng/comanche/src/components/block/nvme/unit_test/test1.cpp:115: Failure
Value of: 0
  Actual: false
Expected: true
[  FAILED  ] Block_nvme_test.AsyncSwap (1 ms)

[fengggli]

explained and fixed in this pull request: (#6)
I will close this first. You can comment on the pull request above