Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nvdla_runtime inference issue #19

Closed
yirs2001 opened this issue Feb 1, 2018 · 24 comments
Closed

nvdla_runtime inference issue #19

yirs2001 opened this issue Feb 1, 2018 · 24 comments

Comments

@yirs2001
Copy link

yirs2001 commented Feb 1, 2018

Issue 1 : no desc get due to index==-1
Issue 2: dla_enable_operation without actual enable due to processor hasn't been programmed
any clues?
Thanks for helping!!

Steps:

  1. using nvdla_compiler to generate default.nvdla from MNIST model (caffe)
  2. In simuation environment:
    ./nvdla_runtime --softmax --loadable default.nvdla --image 2.pgm
  3. the following is log:
    ./nvdla_runtime --softmax --loadable default.nvdla --image 2.pgm
    creating new runtime context...
    Emulator starting
    ppgminfo 1 28 28
    pgm2dimg 1 28 28 1 32 896 896
    submitting tasks...
    [ 3688.898671] Enter:dla_read_network_config
    [ 3688.899068] Exit:dla_read_network_config status=0
    [ 3688.899277] Enter: dla_initiate_processors
    [ 3688.899565] Enter: dla_submit_operation
    [ 3688.899769] Prepare Convolution operation index 0 ROI 0 dep_count 1
    [ 3688.900058] Enter: dla_prepare_operation
    [ 3688.900320] processor:Convolution group:0, rdma_group:0 available
    [ 3688.902057] Enter: dla_read_config
    [ 3688.903844] Exit: dla_read_config
    [ 3688.904017] Exit: dla_prepare_operation status=0
    [ 3688.904226] Enter: dla_program_operation
    [ 3688.904409] Program Convolution operation index 0 ROI 0 Group[0]
    [ 3688.906926] no desc get due to index==-1
    [ 3688.907343] no desc get due to index==-1
    [ 3688.907535] no desc get due to index==-1
    [ 3688.907713] no desc get due to index==-1
    [ 3688.907890] no desc get due to index==-1
    [ 3688.908064] Enter: dla_op_programmed
    [ 3688.908241] Update dependency operation index 3 ROI 0 DEP_COUNT=3
    [ 3688.908486] Update dependency operation index 1 ROI 0 DEP_COUNT=1
    [ 3688.909963] enable SDP in dla_update_dependency as depdency are resolved
    [ 3688.910266] Enter: dla_enable_operation
    [ 3688.910437] exit dla_enable_operation without actual enable due to processor hasn't been programmed
    [ 3688.910797] Exit: dla_enable_operation status=0
    [ 3688.910986] Exit: dla_op_programmed
    [ 3688.911145] Exit: dla_program_operation status=0
    [ 3688.911348] Exit: dla_submit_operation
    [ 3688.911520] Enter: dla_dequeue_operation
    [ 3688.911694] Dequeue op from Convolution processor, index=3 ROI=0
    [ 3688.911944] Enter: dla_submit_operation
    [ 3688.912117] Prepare Convolution operation index 3 ROI 0 dep_count 2
    [ 3688.912376] Enter: dla_prepare_operation
    [ 3688.914273] processor:Convolution group:1, rdma_group:0 available
    [ 3688.914552] Enter: dla_read_config
    [ 3688.916315] Exit: dla_read_config
    [ 3688.916474] Exit: dla_prepare_operation status=0
    [ 3688.917528] Enter: dla_program_operation
    [ 3688.917737] Program Convolution operation index 3 ROI 0 Group[1]
    [ 3688.919080] no desc get due to index==-1
    [ 3688.919434] no desc get due to index==-1
    [ 3688.919613] no desc get due to index==-1
    [ 3688.919782] no desc get due to index==-1
    [ 3688.919949] no desc get due to index==-1
    [ 3688.920118] Enter: dla_op_programmed
    [ 3688.920282] Update dependency operation index 6 ROI 0 DEP_COUNT=3
    [ 3688.921794] Update dependency operation index 4 ROI 0 DEP_COUNT=2
    [ 3688.922071] Exit: dla_op_programmed
    [ 3688.922224] Exit: dla_program_operation status=0
    [ 3688.922418] Exit: dla_submit_operation
    [ 3688.922580] Exit: dla_dequeue_operation
    [ 3688.922745] Enter: dla_submit_operation
    [ 3688.922917] Prepare SDP operation index 1 ROI 0 dep_count 0
    [ 3688.923136] Enter: dla_prepare_operation
    [ 3688.923378] processor:SDP group:1, rdma_group:1 available
    [ 3688.923598] Enter: dla_read_config
    [ 3688.925410] Exit: dla_read_config
    [ 3688.926896] Exit: dla_prepare_operation status=0
    [ 3688.927091] Enter: dla_program_operation
    [ 3688.927263] Program SDP operation index 1 ROI 0 Group[1]
    [ 3688.928020] no desc get due to index==-1
    [ 3688.928194] no desc get due to index==-1
    [ 3688.928468] no desc get due to index==-1
    [ 3688.929676] no desc get due to index==-1
    [ 3688.929875] Enter: dla_op_programmed
    [ 3688.930038] Update dependency operation index 4 ROI 0 DEP_COUNT=1
    [ 3688.930278] enable SDP in dla_update_dependency as depdency are resolved
    [ 3688.930538] Enter: dla_enable_operation
    [ 3688.930709] exit dla_enable_operation without actual enable due to processor hasn't been programmed
    [ 3688.931060] Exit: dla_enable_operation status=0
    [ 3688.931252] Exit: dla_op_programmed
    [ 3688.931403] Exit: dla_program_operation status=0
    [ 3688.931588] Enter: dla_enable_operation
    [ 3688.931753] Enable SDP operation index 1 ROI 0
    [ 3688.932025] Enter: dla_op_enabled
    [ 3688.932180] Update dependency operation index 0 ROI 0 DEP_COUNT=1
    [ 3688.932417] enable Convolution in dla_update_dependency as depdency are resolved
    [ 3688.934580] Enter: dla_enable_operation
    [ 3688.934785] Enable Convolution operation index 0 ROI 0
    [ 3688.935446] Enter: dla_op_enabled
    [ 3688.935603] Exit: dla_op_enabled
    [ 3688.935746] Exit: dla_enable_operation status=0
    [ 3688.935959] Exit: dla_op_enabled
    [ 3688.936117] Exit: dla_enable_operation status=0
    [ 3688.936305] Exit: dla_submit_operation
    [ 3688.936481] Enter: dla_dequeue_operation
    [ 3688.937952] Dequeue op from SDP processor, index=4 ROI=0
    [ 3688.938204] Enter: dla_submit_operation
    [ 3688.938386] Prepare SDP operation index 4 ROI 0 dep_count 0
    [ 3688.938615] Enter: dla_prepare_operation
    [ 3688.943351] processor:SDP group:0, rdma_group:0 available
    [ 3688.943624] Enter: dla_read_config
    [ 3688.969032] Exit: dla_read_config
    [ 3688.969228] Exit: dla_prepare_operation status=0
    [ 3688.969436] Enter: dla_program_operation
    [ 3688.969608] Program SDP operation index 4 ROI 0 Group[0]
    [ 3688.974613] no desc get due to index==-1
    [ 3688.974824] no desc get due to index==-1
    [ 3688.976313] no desc get due to index==-1
    [ 3688.976487] no desc get due to index==-1
    [ 3688.977577] Enter: dla_op_programmed
    [ 3688.977785] Update dependency operation index 7 ROI 0 DEP_COUNT=2
    [ 3688.978038] Exit: dla_op_programmed
    [ 3688.978205] Exit: dla_program_operation status=0
    [ 3688.978412] Enter: dla_enable_operation
    [ 3688.978586] Enable SDP operation index 4 ROI 0
    [ 3688.980151] Enter: dla_op_enabled
    [ 3688.980318] Update dependency operation index 3 ROI 0 DEP_COUNT=2
    [ 3688.981631] Exit: dla_op_enabled
    [ 3688.981830] Exit: dla_enable_operation status=0
    [ 3688.982030] Exit: dla_submit_operation
    [ 3688.982197] Exit: dla_dequeue_operation
    [ 3688.982366] Enter: dla_submit_operation
    [ 3688.982539] Prepare PDP operation index 2 ROI 0 dep_count 1
    [ 3688.982765] Enter: dla_prepare_operation
    [ 3688.983333] processor:PDP group:0, rdma_group:0 available
    [ 3688.983559] Enter: dla_read_config
    [ 3689.003402] Exit: dla_read_config
    [ 3689.003624] Exit: dla_prepare_operation status=0
    [ 3689.003818] Enter: dla_program_operation
    [ 3689.003982] Program PDP operation index 2 ROI 0 Group[0]
    [ 3689.004176] group id 0 rdma id 0
    [ 3689.013963] no desc get due to index==-1
    [ 3689.014182] no desc get due to index==-1
    [ 3689.014345] no desc get due to index==-1
    [ 3689.014502] no desc get due to index==-1
    [ 3689.014670] no desc get due to index==-1
    [ 3689.014835] Enter: dla_op_programmed
    [ 3689.014989] Update dependency operation index 5 ROI 0 DEP_COUNT=2
    [ 3689.015220] Exit: dla_op_programmed
    [ 3689.015379] Exit: dla_program_operation status=0
    [ 3689.015571] Exit: dla_submit_operation
    [ 3689.015740] Enter: dla_dequeue_operation
    [ 3689.015914] Dequeue op from PDP processor, index=5 ROI=0
    [ 3689.016125] Enter: dla_submit_operation
    [ 3689.016284] Prepare PDP operation index 5 ROI 0 dep_count 1
    [ 3689.016485] Enter: dla_prepare_operation
    [ 3689.018908] processor:PDP group:1, rdma_group:1 available
    [ 3689.019175] Enter: dla_read_config
    [ 3714.082055] random: crng init done
    [ 3714.083460] Exit: dla_read_config
    [ 3714.083642] Exit: dla_prepare_operation status=0
    [ 3714.083841] Enter: dla_program_operation
    [ 3714.084017] Program PDP operation index 5 ROI 0 Group[1]
    [ 3714.084229] group id 1 rdma id 1
    [ 3714.086364] no desc get due to index==-1
    [ 3714.086588] no desc get due to index==-1
    [ 3714.086758] no desc get due to index==-1
    [ 3714.086923] no desc get due to index==-1
    [ 3714.087109] no desc get due to index==-1
    [ 3714.087276] no desc get due to index==-1
    [ 3714.087458] Enter: dla_op_programmed
    [ 3714.087625] Exit: dla_op_programmed
    [ 3714.087790] Exit: dla_program_operation status=0
    [ 3714.087998] Exit: dla_submit_operation
    [ 3714.088160] Exit: dla_dequeue_operation
    [ 3714.088326] Exit: dla_initiate_processors status=0
    [ 3714.090184] Enter:dla_handle_events, processor:BDMA
    [ 3714.090470] Exit:dla_handle_events, ret:0
    [ 3714.090640] Enter:dla_handle_events, processor:Convolution
    [ 3714.090865] Handle cdma weight done event, processor Convolution group 0
    [ 3714.091132] Handle cdma data done event, processor Convolution group 0
    [ 3714.091388] Handle op complete event, processor Convolution group 0
    [ 3714.091641] Enter:dla_op_completion processor Convolution group0
    [ 3714.091884] Completed Convolution operation index 0 ROI 0
    [ 3714.092109] 1 HWLs done, totally 11 layers
    [ 3714.092288] Enter: dla_dequeue_operation
    [ 3714.092464] Dequeue op from Convolution processor, index=6 ROI=0
    [ 3714.092749] Enter: dla_submit_operation
    [ 3714.092923] Prepare Convolution operation index 6 ROI 0 dep_count 2
    [ 3714.093171] Enter: dla_prepare_operation
    [ 3714.093393] processor:Convolution group:0, rdma_group:0 available
    [ 3714.093641] Enter: dla_read_config
    [ 3714.095333] Exit: dla_read_config
    [ 3714.095490] Exit: dla_prepare_operation status=0
    [ 3714.095674] Enter: dla_program_operation
    [ 3714.095841] Program Convolution operation index 6 ROI 0 Group[0]
    [ 3714.097160] no desc get due to index==-1
    [ 3714.097465] no desc get due to index==-1
    [ 3714.097648] no desc get due to index==-1
    [ 3714.097815] no desc get due to index==-1
    [ 3714.097984] no desc get due to index==-1
    [ 3714.098153] Enter: dla_op_programmed
    [ 3714.098312] Update dependency operation index 9 ROI 0 DEP_COUNT=3
    [ 3714.098557] Update dependency operation index 7 ROI 0 DEP_COUNT=1
    [ 3714.098807] enable SDP in dla_update_dependency as depdency are resolved
    [ 3714.099066] Enter: dla_enable_operation
    [ 3714.099234] exit dla_enable_operation without actual enable due to processor hasn't been programmed
    [ 3714.099591] Exit: dla_enable_operation status=0
    [ 3714.099789] Exit: dla_op_programmed
    [ 3714.099942] Exit: dla_program_operation status=0
    [ 3714.100135] Exit: dla_submit_operation
    [ 3714.100298] Exit: dla_dequeue_operation
    [ 3714.100472] Exit:dla_op_completion processor Convolution group0 status=0
    [ 3714.100741] Exit:dla_handle_events, ret:0
    [ 3714.100911] Enter:dla_handle_events, processor:SDP
    [ 3714.101111] Handle op complete event, processor SDP group 1
    [ 3714.101338] Enter:dla_op_completion processor SDP group1
    [ 3714.101558] Completed SDP operation index 1 ROI 0
    [ 3714.101758] Update dependency operation index 2 ROI 0 DEP_COUNT=1
    [ 3714.101999] enable PDP in dla_update_dependency as depdency are resolved
    [ 3714.102260] Enter: dla_enable_operation
    [ 3714.102429] Enable PDP operation index 2 ROI 0
    [ 3714.102613] group id 0 rdma id 0
    [ 3714.102812] rdma needed 1
    [ 3714.102980] Enter: dla_op_enabled
    [ 3714.103136] Exit: dla_op_enabled
    [ 3714.103286] Exit: dla_enable_operation status=0
    [ 3714.103475] 2 HWLs done, totally 11 layers
    [ 3714.103654] Enter: dla_free_op_desc op desc index 0 ROI 0
    [ 3714.108590] Exit: dla_free_op_desc
    [ 3714.108755] Enter: dla_dequeue_operation
    [ 3714.108927] Dequeue op from SDP processor, index=7 ROI=0
    [ 3714.109149] Enter: dla_submit_operation
    [ 3714.109322] Prepare SDP operation index 7 ROI 0 dep_count 0
    [ 3714.109545] Enter: dla_prepare_operation
    [ 3714.110792] processor:SDP group:1, rdma_group:1 available
    [ 3714.111028] Enter: dla_read_config
    [ 3714.164393] Exit: dla_read_config
    [ 3714.164573] Exit: dla_prepare_operation status=0
    [ 3714.164767] Enter: dla_program_operation
    [ 3714.164943] Program SDP operation index 7 ROI 0 Group[1]
    [ 3714.176621] no desc get due to index==-1
    [ 3714.176809] no desc get due to index==-1
    [ 3714.179585] no desc get due to index==-1
    [ 3714.179761] no desc get due to index==-1
    [ 3714.179929] no desc get due to index==-1
    [ 3714.180099] Enter: dla_op_programmed
    [ 3714.180259] Exit: dla_op_programmed
    [ 3714.180414] Exit: dla_program_operation status=0
    [ 3714.180609] Enter: dla_enable_operation
    [ 3714.180780] Enable SDP operation index 7 ROI 0
    [ 3714.182902] Enter: dla_op_enabled
    [ 3714.183066] Update dependency operation index 6 ROI 0 DEP_COUNT=2
    [ 3714.183314] Exit: dla_op_enabled
    [ 3714.183460] Exit: dla_enable_operation status=0
    [ 3714.183646] Exit: dla_submit_operation
    [ 3714.183804] Exit: dla_dequeue_operation
    [ 3714.183975] Enter: dla_free_op_desc op desc index 1 ROI 0
    [ 3714.195310] Exit: dla_free_op_desc
    [ 3714.195490] Exit:dla_op_completion processor SDP group1 status=0
    [ 3714.195713] Exit:dla_handle_events, ret:0
    [ 3714.195889] Enter:dla_handle_events, processor:PDP
    [ 3714.196085] Exit:dla_handle_events, ret:0
    [ 3714.196251] Enter:dla_handle_events, processor:CDP
    [ 3714.196442] Exit:dla_handle_events, ret:0
    [ 3714.196608] Enter:dla_handle_events, processor:RUBIK
    [ 3714.196799] Exit:dla_handle_events, ret:0
    [ 3714.204137] Enter:dla_handle_events, processor:BDMA
    [ 3714.204389] Exit:dla_handle_events, ret:0
    [ 3714.204626] Enter:dla_handle_events, processor:Convolution
    [ 3714.204896] Exit:dla_handle_events, ret:0
    [ 3714.205074] Enter:dla_handle_events, processor:SDP
    [ 3714.205282] Exit:dla_handle_events, ret:0
    [ 3714.205446] Enter:dla_handle_events, processor:PDP
    [ 3714.205634] Exit:dla_handle_events, ret:0
    [ 3714.205792] Enter:dla_handle_events, processor:CDP
    [ 3714.205971] Exit:dla_handle_events, ret:0
    [ 3714.206130] Enter:dla_handle_events, processor:RUBIK
    [ 3714.206315] Exit:dla_handle_events, ret:0
    [ 3714.206599] Enter:dla_handle_events, processor:BDMA
    [ 3714.206794] Exit:dla_handle_events, ret:0
    [ 3714.206950] Enter:dla_handle_events, processor:Convolution
    [ 3714.207149] Exit:dla_handle_events, ret:0
    [ 3714.207304] Enter:dla_handle_events, processor:SDP
    [ 3714.207503] Exit:dla_handle_events, ret:0
    [ 3714.207673] Enter:dla_handle_events, processor:PDP
    [ 3714.207868] Exit:dla_handle_events, ret:0
    [ 3714.208032] Enter:dla_handle_events, processor:CDP
    [ 3714.208221] Exit:dla_handle_events, ret:0
    [ 3714.208383] Enter:dla_handle_events, processor:RUBIK
    [ 3714.208611] Exit:dla_handle_events, ret:0
    [ 3714.416359] Enter:dla_handle_events, processor:BDMA
    [ 3714.416665] Exit:dla_handle_events, ret:0
    [ 3714.416884] Enter:dla_handle_events, processor:Convolution
    [ 3714.417126] Exit:dla_handle_events, ret:0
    [ 3714.417323] Enter:dla_handle_events, processor:SDP
    [ 3714.417525] Exit:dla_handle_events, ret:0
    [ 3714.417716] Enter:dla_handle_events, processor:PDP
    [ 3714.417916] Handle op complete event, processor PDP group 0
    [ 3714.418140] Enter:dla_op_completion processor PDP group0
    [ 3714.418354] Completed PDP operation index 2 ROI 0
    [ 3714.418552] Update dependency operation index 3 ROI 0 DEP_COUNT=1
    [ 3714.418799] enable Convolution in dla_update_dependency as depdency are resolved
    [ 3714.419093] Enter: dla_enable_operation
    [ 3714.419254] Enable Convolution operation index 3 ROI 0
    [ 3714.419741] Enter: dla_op_enabled
    [ 3714.419908] Exit: dla_op_enabled
    [ 3714.420057] Exit: dla_enable_operation status=0
    [ 3714.420243] 3 HWLs done, totally 11 layers
    [ 3714.420422] Enter: dla_dequeue_operation
    [ 3714.420592] exit PDP as there's no further operation
    [ 3714.420797] Exit: dla_dequeue_operation
    [ 3714.420969] Enter: dla_free_op_desc op desc index 2 ROI 0
    [ 3714.427803] Exit: dla_free_op_desc
    [ 3714.427974] Exit:dla_op_completion processor PDP group0 status=0
    [ 3714.428216] Exit:dla_handle_events, ret:0
    [ 3714.428412] Enter:dla_handle_events, processor:CDP
    [ 3714.428613] Exit:dla_handle_events, ret:0
    [ 3714.428787] Enter:dla_handle_events, processor:RUBIK
    [ 3714.428997] Exit:dla_handle_events, ret:0
    [ 3714.567790] Enter:dla_handle_events, processor:BDMA
    [ 3714.568091] Exit:dla_handle_events, ret:0
    [ 3714.568255] Enter:dla_handle_events, processor:Convolution
    [ 3714.568474] Handle cdma data done event, processor Convolution group 1
    [ 3714.568736] Exit:dla_handle_events, ret:0
    [ 3714.568901] Enter:dla_handle_events, processor:SDP
    [ 3714.569083] Exit:dla_handle_events, ret:0
    [ 3714.569255] Enter:dla_handle_events, processor:PDP
    [ 3714.569444] Exit:dla_handle_events, ret:0
    [ 3714.569611] Enter:dla_handle_events, processor:CDP
    [ 3714.569805] Exit:dla_handle_events, ret:0
    [ 3714.569976] Enter:dla_handle_events, processor:RUBIK
    [ 3714.570174] Exit:dla_handle_events, ret:0
    [ 3721.334091] Enter:dla_handle_events, processor:BDMA
    [ 3721.334396] Exit:dla_handle_events, ret:0
    [ 3721.334575] Enter:dla_handle_events, processor:Convolution
    [ 3721.334812] Handle cdma weight done event, processor Convolution group 1
    [ 3721.335062] Handle op complete event, processor Convolution group 1
    [ 3721.335305] Enter:dla_op_completion processor Convolution group1
    [ 3721.335551] Completed Convolution operation index 3 ROI 0
    [ 3721.335860] 4 HWLs done, totally 11 layers
    [ 3721.336060] Enter: dla_dequeue_operation
    [ 3721.336231] Dequeue op from Convolution processor, index=9 ROI=0
    [ 3721.336468] Enter: dla_submit_operation
    [ 3721.336664] Prepare Convolution operation index 9 ROI 0 dep_count 2
    [ 3721.336908] Enter: dla_prepare_operation
    [ 3721.337159] processor:Convolution group:1, rdma_group:0 available
    [ 3721.337416] Enter: dla_read_config
    [ 3721.339158] Exit: dla_read_config
    [ 3721.339323] Exit: dla_prepare_operation status=0
    [ 3721.339518] Enter: dla_program_operation
    [ 3721.339702] Program Convolution operation index 9 ROI 0 Group[1]
    [ 3721.341082] no desc get due to index==-1
    [ 3721.341277] no desc get due to index==-1
    [ 3721.341554] no desc get due to index==-1
    [ 3721.341736] no desc get due to index==-1
    [ 3721.341916] no desc get due to index==-1
    [ 3721.342092] no desc get due to index==-1
    [ 3721.342274] Enter: dla_op_programmed
    [ 3721.342443] Update dependency operation index 10 ROI 0 DEP_COUNT=2
    [ 3721.342713] Exit: dla_op_programmed
    [ 3721.342869] Exit: dla_program_operation status=0
    [ 3721.343055] Exit: dla_submit_operation
    [ 3721.343212] Exit: dla_dequeue_operation
    [ 3721.343381] Exit:dla_op_completion processor Convolution group1 status=0
    [ 3721.343646] Exit:dla_handle_events, ret:0
    [ 3721.343822] Enter:dla_handle_events, processor:SDP
    [ 3721.344019] Handle op complete event, processor SDP group 0
    [ 3721.344241] Enter:dla_op_completion processor SDP group0
    [ 3721.344457] Completed SDP operation index 4 ROI 0
    [ 3721.344651] Update dependency operation index 5 ROI 0 DEP_COUNT=1
    [ 3721.344892] enable PDP in dla_update_dependency as depdency are resolved
    [ 3721.345150] Enter: dla_enable_operation
    [ 3721.345322] Enable PDP operation index 5 ROI 0
    [ 3721.345510] group id 1 rdma id 1
    [ 3721.345744] rdma needed 1
    [ 3721.345945] Enter: dla_op_enabled
    [ 3721.346108] Exit: dla_op_enabled
    [ 3721.346260] Exit: dla_enable_operation status=0
    [ 3721.346451] 5 HWLs done, totally 11 layers
    [ 3721.346625] Enter: dla_free_op_desc op desc index 3 ROI 0
    [ 3721.349402] Exit: dla_free_op_desc
    [ 3721.349577] Enter: dla_dequeue_operation
    [ 3721.349750] Dequeue op from SDP processor, index=8 ROI=0
    [ 3721.349988] Enter: dla_submit_operation
    [ 3721.350169] Prepare SDP operation index 8 ROI 0 dep_count 1
    [ 3721.350406] Enter: dla_prepare_operation
    [ 3721.351781] processor:SDP group:0, rdma_group:0 available
    [ 3721.352025] Enter: dla_read_config
    [ 3721.405789] Exit: dla_read_config
    [ 3721.405983] Exit: dla_prepare_operation status=0
    [ 3721.406172] Enter: dla_program_operation
    [ 3721.406339] Program SDP operation index 8 ROI 0 Group[0]
    [ 3721.418160] no desc get due to index==-1
    [ 3721.418353] no desc get due to index==-1
    [ 3721.418516] no desc get due to index==-1
    [ 3721.418688] no desc get due to index==-1
    [ 3721.418859] no desc get due to index==-1
    [ 3721.419042] Enter: dla_op_programmed
    [ 3721.419207] Update dependency operation index 10 ROI 0 DEP_COUNT=1
    [ 3721.419466] enable SDP in dla_update_dependency as depdency are resolved
    [ 3721.419737] Enter: dla_enable_operation
    [ 3721.419912] exit dla_enable_operation without actual enable due to processor hasn't been programmed
    [ 3721.420264] Exit: dla_enable_operation status=0
    [ 3721.420451] Exit: dla_op_programmed
    [ 3721.420601] Exit: dla_program_operation status=0
    [ 3721.420788] Exit: dla_submit_operation
    [ 3721.420948] Exit: dla_dequeue_operation
    [ 3721.421119] Enter: dla_free_op_desc op desc index 4 ROI 0
    [ 3721.426588] Exit: dla_free_op_desc
    [ 3721.426757] Exit:dla_op_completion processor SDP group0 status=0
    [ 3721.426995] Exit:dla_handle_events, ret:0
    [ 3721.427185] Enter:dla_handle_events, processor:PDP
    [ 3721.427393] Exit:dla_handle_events, ret:0
    [ 3721.427567] Enter:dla_handle_events, processor:CDP
    [ 3721.427783] Exit:dla_handle_events, ret:0
    [ 3721.427970] Enter:dla_handle_events, processor:RUBIK
    [ 3721.428179] Exit:dla_handle_events, ret:0
    [ 3721.435900] Enter:dla_handle_events, processor:BDMA
    [ 3721.436172] Exit:dla_handle_events, ret:0
    [ 3721.436345] Enter:dla_handle_events, processor:Convolution
    [ 3721.436561] Exit:dla_handle_events, ret:0
    [ 3721.436727] Enter:dla_handle_events, processor:SDP
    [ 3721.436925] Exit:dla_handle_events, ret:0
    [ 3721.437095] Enter:dla_handle_events, processor:PDP
    [ 3721.437293] Handle op complete event, processor PDP group 1
    [ 3721.437521] Enter:dla_op_completion processor PDP group1
    [ 3721.437741] Completed PDP operation index 5 ROI 0
    [ 3721.437937] Update dependency operation index 6 ROI 0 DEP_COUNT=1
    [ 3721.438183] enable Convolution in dla_update_dependency as depdency are resolved
    [ 3721.438478] Enter: dla_enable_operation
    [ 3721.438648] Enable Convolution operation index 6 ROI 0
    [ 3721.439064] Enter: dla_op_enabled
    [ 3721.439212] Exit: dla_op_enabled
    [ 3721.439357] Exit: dla_enable_operation status=0
    [ 3721.439565] 6 HWLs done, totally 11 layers
    [ 3721.439740] Enter: dla_dequeue_operation
    [ 3721.439910] exit PDP as there's no further operation
    [ 3721.440107] Exit: dla_dequeue_operation
    [ 3721.440271] Enter: dla_free_op_desc op desc index 5 ROI 0
    [ 3721.447906] Exit: dla_free_op_desc
    [ 3721.448076] Exit:dla_op_completion processor PDP group1 status=0
    [ 3721.448314] Exit:dla_handle_events, ret:0
    [ 3721.448492] Enter:dla_handle_events, processor:CDP
    [ 3721.448694] Exit:dla_handle_events, ret:0
    [ 3721.448868] Enter:dla_handle_events, processor:RUBIK
    [ 3721.449080] Exit:dla_handle_events, ret:0
    [ 3721.545029] Enter:dla_handle_events, processor:BDMA
    [ 3721.545338] Exit:dla_handle_events, ret:0
    [ 3721.545506] Enter:dla_handle_events, processor:Convolution
    [ 3721.545717] Handle cdma data done event, processor Convolution group 0
    [ 3721.545971] Exit:dla_handle_events, ret:0
    [ 3721.546131] Enter:dla_handle_events, processor:SDP
    [ 3721.546328] Exit:dla_handle_events, ret:0
    [ 3721.546489] Enter:dla_handle_events, processor:PDP
    [ 3721.546667] Exit:dla_handle_events, ret:0
    [ 3721.546831] Enter:dla_handle_events, processor:CDP
    [ 3721.547015] Exit:dla_handle_events, ret:0
    [ 3721.547168] Enter:dla_handle_events, processor:RUBIK
    [ 3721.547355] Exit:dla_handle_events, ret:0
    [ 3726.151448] Enter:dla_handle_events, processor:BDMA
    [ 3726.151776] Exit:dla_handle_events, ret:0
    [ 3726.151965] Enter:dla_handle_events, processor:Convolution
    [ 3726.152207] Handle cdma weight done event, processor Convolution group 0
    [ 3726.152476] Handle op complete event, processor Convolution group 0
    [ 3726.152776] Enter:dla_op_completion processor Convolution group0
    [ 3726.153016] Completed Convolution operation index 6 ROI 0
    [ 3726.153247] 7 HWLs done, totally 11 layers
    [ 3726.153428] Enter: dla_dequeue_operation
    [ 3726.153595] exit Convolution as there's no further operation
    [ 3726.153810] Exit: dla_dequeue_operation
    [ 3726.153978] Exit:dla_op_completion processor Convolution group0 status=0
    [ 3726.154235] Exit:dla_handle_events, ret:0
    [ 3726.154406] Enter:dla_handle_events, processor:SDP
    [ 3726.154603] Handle op complete event, processor SDP group 1
    [ 3726.154829] Enter:dla_op_completion processor SDP group1
    [ 3726.155035] Completed SDP operation index 7 ROI 0
    [ 3726.155231] Update dependency operation index 8 ROI 0 DEP_COUNT=1
    [ 3726.155474] enable SDP in dla_update_dependency as depdency are resolved
    [ 3726.155732] Enter: dla_enable_operation
    [ 3726.155902] Enable SDP operation index 8 ROI 0
    [ 3726.156298] Enter: dla_op_enabled
    [ 3726.156463] Exit: dla_op_enabled
    [ 3726.156614] Exit: dla_enable_operation status=0
    [ 3726.156807] 8 HWLs done, totally 11 layers
    [ 3726.156981] Enter: dla_free_op_desc op desc index 6 ROI 0
    [ 3726.159328] Exit: dla_free_op_desc
    [ 3726.159499] Enter: dla_dequeue_operation
    [ 3726.159672] Dequeue op from SDP processor, index=10 ROI=0
    [ 3726.159887] Enter: dla_submit_operation
    [ 3726.160057] Prepare SDP operation index 10 ROI 0 dep_count 0
    [ 3726.160284] Enter: dla_prepare_operation
    [ 3726.161682] processor:SDP group:1, rdma_group:1 available
    [ 3726.161928] Enter: dla_read_config
    [ 3726.170226] Exit: dla_read_config
    [ 3726.170399] Exit: dla_prepare_operation status=0
    [ 3726.170601] Enter: dla_program_operation
    [ 3726.170777] Program SDP operation index 10 ROI 0 Group[1]
    [ 3726.171565] no desc get due to index==-1
    [ 3726.171750] no desc get due to index==-1
    [ 3726.171923] no desc get due to index==-1
    [ 3726.172097] no desc get due to index==-1
    [ 3726.172274] no desc get due to index==-1
    [ 3726.172449] no desc get due to index==-1
    [ 3726.172630] Enter: dla_op_programmed
    [ 3726.172790] Exit: dla_op_programmed
    [ 3726.172951] Exit: dla_program_operation status=0
    [ 3726.173143] Enter: dla_enable_operation
    [ 3726.173315] Enable SDP operation index 10 ROI 0
    [ 3726.173578] Enter: dla_op_enabled
    [ 3726.173734] Update dependency operation index 9 ROI 0 DEP_COUNT=2
    [ 3726.173973] Exit: dla_op_enabled
    [ 3726.174121] Exit: dla_enable_operation status=0
    [ 3726.174311] Exit: dla_submit_operation
    [ 3726.174471] Exit: dla_dequeue_operation
    [ 3726.174642] Enter: dla_free_op_desc op desc index 7 ROI 0
    [ 3726.175276] Exit: dla_free_op_desc
    [ 3726.175432] Exit:dla_op_completion processor SDP group1 status=0
    [ 3726.175685] Exit:dla_handle_events, ret:0
    [ 3726.175871] Enter:dla_handle_events, processor:PDP
    [ 3726.176081] Exit:dla_handle_events, ret:0
    [ 3726.176265] Enter:dla_handle_events, processor:CDP
    [ 3726.176473] Exit:dla_handle_events, ret:0
    [ 3726.176661] Enter:dla_handle_events, processor:RUBIK
    [ 3726.176865] Exit:dla_handle_events, ret:0
    [ 3726.251011] Enter:dla_handle_events, processor:BDMA
    [ 3726.251300] Exit:dla_handle_events, ret:0
    [ 3726.251484] Enter:dla_handle_events, processor:Convolution
    [ 3726.251718] Exit:dla_handle_events, ret:0
    [ 3726.251902] Enter:dla_handle_events, processor:SDP
    [ 3726.252106] Handle op complete event, processor SDP group 0
    [ 3726.252334] Enter:dla_op_completion processor SDP group0
    [ 3726.252567] Completed SDP operation index 8 ROI 0
    [ 3726.252770] Update dependency operation index 9 ROI 0 DEP_COUNT=1
    [ 3726.253009] enable Convolution in dla_update_dependency as depdency are resolved
    [ 3726.253316] Enter: dla_enable_operation
    [ 3726.253486] Enable Convolution operation index 9 ROI 0
    [ 3726.253916] Enter: dla_op_enabled
    [ 3726.254073] Exit: dla_op_enabled
    [ 3726.254224] Exit: dla_enable_operation status=0
    [ 3726.254415] 9 HWLs done, totally 11 layers
    [ 3726.254595] Enter: dla_dequeue_operation
    [ 3726.254764] exit SDP as there's no further operation
    [ 3726.254959] Exit: dla_dequeue_operation
    [ 3726.255123] Enter: dla_free_op_desc op desc index 8 ROI 0
    [ 3726.260282] Exit: dla_free_op_desc
    [ 3726.260492] Exit:dla_op_completion processor SDP group0 status=0
    [ 3726.260733] Exit:dla_handle_events, ret:0
    [ 3726.260900] Enter:dla_handle_events, processor:PDP
    [ 3726.261097] Exit:dla_handle_events, ret:0
    [ 3726.261400] Enter:dla_handle_events, processor:CDP
    [ 3726.261741] Exit:dla_handle_events, ret:0
    [ 3726.262037] Enter:dla_handle_events, processor:RUBIK
    [ 3726.262386] Exit:dla_handle_events, ret:0
    [ 3726.356118] Enter:dla_handle_events, processor:BDMA
    [ 3726.356425] Exit:dla_handle_events, ret:0
    [ 3726.356664] Enter:dla_handle_events, processor:Convolution
    [ 3726.356942] Handle cdma data done event, processor Convolution group 1
    [ 3726.357479] Exit:dla_handle_events, ret:0
    [ 3726.358014] Enter:dla_handle_events, processor:SDP
    [ 3726.358669] Exit:dla_handle_events, ret:0
    [ 3726.359191] Enter:dla_handle_events, processor:PDP
    [ 3726.359793] Exit:dla_handle_events, ret:0
    [ 3726.360313] Enter:dla_handle_events, processor:CDP
    [ 3726.360907] Exit:dla_handle_events, ret:0
    [ 3726.361434] Enter:dla_handle_events, processor:RUBIK
    [ 3726.362039] Exit:dla_handle_events, ret:0
    [ 3726.417167] Enter:dla_handle_events, processor:BDMA
    [ 3726.417434] Exit:dla_handle_events, ret:0
    [ 3726.417616] Enter:dla_handle_events, processor:Convolution
    [ 3726.417837] Handle cdma weight done event, processor Convolution group 1
    [ 3726.418104] Exit:dla_handle_events, ret:0
    [ 3726.418274] Enter:dla_handle_events, processor:SDP
    [ 3726.418473] Exit:dla_handle_events, ret:0
    [ 3726.418650] Enter:dla_handle_events, processor:PDP
    [ 3726.418846] Exit:dla_handle_events, ret:0
    [ 3726.419019] Enter:dla_handle_events, processor:CDP
    [ 3726.419225] Exit:dla_handle_events, ret:0
    [ 3726.419392] Enter:dla_handle_events, processor:RUBIK
    [ 3726.419586] Exit:dla_handle_events, ret:0
    [ 3726.422740] Enter:dla_handle_events, processor:BDMA
    [ 3726.423007] Exit:dla_handle_events, ret:0
    [ 3726.423210] Enter:dla_handle_events, processor:Convolution
    [ 3726.423458] Handle op complete event, processor Convolution group 1
    [ 3726.423728] Enter:dla_op_completion processor Convolution group1
    [ 3726.423991] Completed Convolution operation index 9 ROI 0
    [ 3726.424239] 10 HWLs done, totally 11 layers
    [ 3726.424426] Enter: dla_dequeue_operation
    [ 3726.424662] exit Convolution as there's no further operation
    [ 3726.424918] Exit: dla_dequeue_operation
    [ 3726.425117] Exit:dla_op_completion processor Convolution group1 status=0
    [ 3726.425416] Exit:dla_handle_events, ret:0
    [ 3726.425607] Enter:dla_handle_events, processor:SDP
    [ 3726.425834] Handle op complete event, processor SDP group 1
    [ 3726.426084] Enter:dla_op_completion processor SDP group1
    [ 3726.426331] Completed SDP operation index 10 ROI 0
    [ 3726.426556] 11 HWLs done, totally 11 layers
    [ 3726.426744] Enter: dla_free_op_desc op desc index 9 ROI 0
    [ 3726.427128] Exit: dla_free_op_desc
    [ 3726.427311] Enter: dla_free_op_desc op desc index 10 ROI 0
    [ 3726.427658] Exit: dla_free_op_desc
    [ 3726.427844] Exit:dla_op_completion processor SDP group1 status=0
    [ 3726.428110] Exit:dla_handle_events, ret:0
    [ 3726.428299] Enter:dla_handle_events, processor:PDP
    [ 3726.428511] Exit:dla_handle_events, ret:0
    [ 3726.428713] Enter:dla_handle_events, processor:CDP
    [ 3726.428935] Exit:dla_handle_events, ret:0
    [ 3726.429124] Enter:dla_handle_events, processor:RUBIK
    [ 3726.429367] Exit:dla_handle_events, ret:0
    [ 3726.433004] reset engine done
    Work Found!
    Work Done
    ========performSoftwareSoftmax======
    Result:
    0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
    Test pass
@prasshantg
Copy link
Collaborator

prasshantg commented Feb 1, 2018

[ 3726.426556] 11 HWLs done, totally 11 layers

above log indicates that network execution completed successfully, you can ignore messages you mentioned as issue #1 and #2

Do not use --softmax option, now it is integrated in network execution i.e. it will be part of default.nvdla.

Currently it will not output the softmax result. We will soon push change to print the output.

@yirs2001
Copy link
Author

yirs2001 commented Feb 2, 2018

Dear Prasshantg:
Thanks for your response.

@yirs2001 yirs2001 closed this as completed Feb 2, 2018
@JunningWu
Copy link

@yirs2001 index=-1, which because the sw will checkout all 6 Processors, when there are not actual OPs for one Processor, its index will be -1. the check process is insided in a for() loop.

as for issue 2, it is because you shall program one processor, then you can initiate it or enable.

BTW, hoping for the NEXT BIG update.

@JunningWu
Copy link

@yirs2001
Copy link
Author

yirs2001 commented Feb 2, 2018

@JunningWu appreciated!! I will check it out

@yirs2001
Copy link
Author

@JunningWu for issue2, Did you program any processor successfully? thanks.

@yirs2001 yirs2001 reopened this Mar 7, 2018
@yirs2001
Copy link
Author

yirs2001 commented Mar 7, 2018

@prasshantg , Does 0bf766e commit(6 Mar 2018) support "output the softmax result"? thanks.
I set NvDlaDebugPrintf in Emulator::executeSoftmax function, but the result seems the same with previous commit

@prasshantg
Copy link
Collaborator

@yirs2001 0bf766e should not make any difference to softmax.

What is issue here? All values 0?

@yirs2001
Copy link
Author

yirs2001 commented Mar 7, 2018

@prasshantg
I copy the image from ~/nvdla/sw/regression/images/digits
And I use the following command to test all number pgm.
The results are not I expect.
Modified code segment:
I replace debugOps() as true in Emulator::executeSoftmax function
and add
NvF32 sumexp = 0.0f;
for (NvU32 ii=0; ii<*src.channel(); ii++)
{
sumexp += expf(float(pSrc[ii])-maxval);
NvDlaDebugPrintf("[%1f] ", float(pSrc[ii]-maxval));// Will add
}
NvDlaDebugPrintf("\n Destination\n"); // Will add
for (NvU32 ii=0; ii<*src.channel(); ii++)
{
pDst[ii] = expf(float(pSrc[ii])-maxval) / sumexp;
NvDlaDebugPrintf("[%1f] ", float(pDst[ii])); //Will add
}

The following are two logs( 9 & 5):
Command:

./nvdla_runtime --loadable mnist.nvdla --image digits/nine.pgm

src format 2
address[9] 0xffffa19e8000 (1x1x10) 32B
line_stride 32B surface_stride 32B
dst format 2
address[11] 0xffffa11b5000 (1x1x10) 32B
line_stride 32B surface_stride 32B
[-856.500000] [-2283.500000] [-847.500000] [-1116.250000] [-956.500000] [-1698.000000] [-2392.500000] [-2034.500000] [0.000000] [-2161.000000]
Destination
[0.000000] [0.000000] [0.000000] [0.000000] [0.000000] [0.000000] [0.000000] [0.000000] [1.000000] [0.000000] Work Done
Test pass

./nvdla_runtime --loadable mnist.nvdla --image digits/five.pgm

src format 2
address[9] 0xffff9717a000 (1x1x10) 32B
line_stride 32B surface_stride 32B
dst format 2
address[11] 0xffff96947000 (1x1x10) 32B
line_stride 32B surface_stride 32B
[-1757.750000] [-4326.000000] [-1697.250000] [-658.000000] [-2902.500000] [-2399.125000] [-1246.000000] [-4390.000000] [0.000000] [-2747.000000]
Destination
[0.000000] [0.000000] [0.000000] [0.000000] [0.000000] [0.000000] [0.000000] [0.000000] [1.000000] [0.000000] Work Done
Test pass

@prasshantg
Copy link
Collaborator

@yirs2001

We have fixed issue for incorrect output in latest release. Please update all components compiler, runtime and kmd to check it.

You will have to recompile model.

@JunningWu
Copy link

@prasshantg can you share your prototxt and caffemodel, when I try the BVLC lenet model, the output is still incorrect.

@NienfengYao
Copy link

@prasshantg
The vp/sw is what we need to update only?
I check out vp/sw and compile lenet in host, generate basic.nvdla and I run it on simulator. Got the result but it seems not correct.

# ./nvdla_runtime --loadable basic.nvdla --image digits/one.pgm
Work Found!
Processing softmax [axis=1]
src format 2
	address[8] 0xffffbcf11000 (1x1x10) 32B
	line_stride 32B surface_stride 32B
dst format 2
	address[11] 0xffffbc6de000 (1x1x10) 32B
	line_stride 32B surface_stride 32B
pSrc[0]=0.000000, pSrc[1]=0.000000, pSrc[2]=0.000000, pSrc[3]=0.000000, pSrc[4]=0.000000, pSrc[5]=0.000000, pSrc[6]=0.000000, pSrc[7]=0.000000, pSrc[8]=0.000000, pSrc[9]=0.000000, 
pDst[0]=0.000000, pDst[1]=0.000000, pDst[2]=0.000000, pDst[3]=0.000000, pDst[4]=0.000000, pDst[5]=0.000000, pDst[6]=0.000000, pDst[7]=0.000000, pDst[8]=0.000000, pDst[9]=0.000000, 
Work Done

Test pass

@prasshantg
Copy link
Collaborator

@JunningWu @NienfengYao

did you update opendla.ko too?

I am getting below result for one.pgm, please confirm if you also get same.

2.21729e-05 0.96875 0.000106931 0.00115108 0.0072403 0.00423813 5.74589e-05 0.0171356 1.49012e-06 0.00135899

@prasshantg
Copy link
Collaborator

@JunningWu @NienfengYao can you try with LeNet loadable regression/flatbufs/kmd/NN/NN_L0_0_fbuf to make sure all other components are updated correctly? You should get below result for one.pgm with this loadable.

2.21729e-05 0.96875 0.000106931 0.00115108 0.0072403 0.00423813 5.74589e-05 0.0171356 1.49012e-06 0.00135899

@prasshantg
Copy link
Collaborator

prasshantg commented Apr 26, 2018

@wyxsky I tried using some other pre-trained mnist model. You can download it from https://github.com/mravendi/caffe-test-mnist-jpg/tree/master/model

and test images (testSample.zip) from https://www.kaggle.com/scolianni/mnistasjpg/data

below are the results for it

Image Label 0 1 2 3 4 5 6 7 8 9
img_1.jpg 2 0 0 1 5.96E-08 0 0 0 0 0 0
img_2.jpg 0 1 2.62E-06 9.30E-06 0 2.98E-07 2.19E-05 4.95E-06 2.50E-06 0 5.96E-08
img_3.jpg 9 5.96E-08 5.96E-07 1.73E-06 1.09E-05 3.58E-05 1.07E-06 0 1.19E-05 0.000208616 0.999512
img_4.jpg 0 0.766113 1.01E-05 0.00599289 0.00321579 0.00187874 0.000323772 0.000316858 0.00191498 0.0137939 0.206299
img_5.jpg 3 0 0 5.96E-08 1 0 0 0 0 1.07E-06 0
img_6.jpg 7 8.94E-07 1.19E-07 1.40E-05 7.24E-05 0.000295877 4.97E-05 1.19E-07 0.688965 2.55E-05 0.310547

@yirs2001
Copy link
Author

yirs2001 commented Apr 27, 2018

@prasshantg I clone all and setup everything to do the test, the same result with JuningWu & NienfengYao.
I also download the model in above link, all result are 0

@NienfengYao
Copy link

@prasshantg
I tried the command, the result is the same with yours. But still can't get the correct inference for lenet.

./nvdla_runtime --loadable regression/flatbufs/kmd/NN/NN_L0_0_fbuf  --image digits/one.pgm   --rawdump
# cat output.dimg
2.21729e-05 0.96875 0.000106931 0.00115108 0.0072403 0.00423813 5.74589e-05 0.0171356 1.49012e-06 0.00135899

@wyxsky
Copy link

wyxsky commented Apr 27, 2018

@NienfengYao
the index of the label begin with 0, so the result is correct.

@wyxsky
Copy link

wyxsky commented Apr 27, 2018

@prasshantg
Follow your instructions, I use jpg file to do the test and pass the test, while the pgm file in regression folder is still not correct. Thanks.

@NienfengYao
Copy link

@wyxsky
Let me explain more in detail.

  1. Test test is requested by @prasshantg , we have the same result.
/nvdla_runtime --loadable regression/flatbufs/kmd/NN/NN_L0_0_fbuf  --image digits/one.pgm   --rawdump
# cat output.dimg
2.21729e-05 0.96875 0.000106931 0.00115108 0.0072403 0.00423813 5.74589e-05 0.0171356 1.49012e-06 0.00135899
  1. Base the 1's result, I do the inference of MNIST. Get the result, it seems not correct. (The lenet.prototxt and lenet_iter_10000.caffemodel comes from caffe/examples/mnist/, it works correctly in PC. I have synced with vp/sw and use this to compile and build umd/kmd and use them in simulator)
one.pgm: 0.999512 0.000422239 0 1.43051e-06 7.51019e-06 5.96046e-07 4.31538e-05 1.3113e-06 0.000175238 1.85966e-05
two.pgm: 0.0113754 1.24574e-05 0.000127316 0.0245056 1.13249e-06 0.000159383 3.09944e-06 2.14577e-06 0.219238 0.744629
three.pgm: 0.0761108 2.73585e-05 1.93715e-05 0.370361 1.71661e-05 0.000618458 6.12736e-05 2.38419e-07 0.266602 0.286133
four.pgm: 0.464355 0.0440979 4.11272e-06 0.0344238 0.0402527 0.00463104 0.000963688 3.51667e-05 0.401855 0.00931549

@wyxsky
Copy link

wyxsky commented Apr 27, 2018

@NienfengYao You can use jpeg format file to do the test not pgm, test images (testSample.zip) from https://www.kaggle.com/scolianni/mnistasjpg/data

@prasshantg
Copy link
Collaborator

Please use jpeg format files to test model other than regression/flatbufs/kmd/NN/NN_L0_0_fbuf

Also, compiler does not support multi-batch currently so to be on safe side you may want to change batch number to 1 in lenet.prototxt

@NienfengYao
Copy link

@wyxsky , @prasshantg
Thanks. I use the images (testSample.zip) from https://www.kaggle.com/scolianni/mnistasjpg/data and get the correct inference with MNIST.

@ghost
Copy link

ghost commented Feb 25, 2019

Issue 1 : no desc get due to index==-1
Issue 2: dla_enable_operation without actual enable due to processor hasn't been programmed
any clues?
Thanks for helping!!

Steps:

  1. using nvdla_compiler to generate default.nvdla from MNIST model (caffe)
  2. In simuation environment:
    ./nvdla_runtime --softmax --loadable default.nvdla --image 2.pgm
  3. the following is log:
    ./nvdla_runtime --softmax --loadable default.nvdla --image 2.pgm
    creating new runtime context...
    Emulator starting
    ppgminfo 1 28 28
    pgm2dimg 1 28 28 1 32 896 896
    submitting tasks...
    [ 3688.898671] Enter:dla_read_network_config
    [ 3688.899068] Exit:dla_read_network_config status=0
    [ 3688.899277] Enter: dla_initiate_processors
    [ 3688.899565] Enter: dla_submit_operation
    [ 3688.899769] Prepare Convolution operation index 0 ROI 0 dep_count 1
    [ 3688.900058] Enter: dla_prepare_operation
    [ 3688.900320] processor:Convolution group:0, rdma_group:0 available
    [ 3688.902057] Enter: dla_read_config
    [ 3688.903844] Exit: dla_read_config
    [ 3688.904017] Exit: dla_prepare_operation status=0
    [ 3688.904226] Enter: dla_program_operation
    [ 3688.904409] Program Convolution operation index 0 ROI 0 Group[0]
    [ 3688.906926] no desc get due to index==-1
    [ 3688.907343] no desc get due to index==-1
    [ 3688.907535] no desc get due to index==-1
    [ 3688.907713] no desc get due to index==-1
    [ 3688.907890] no desc get due to index==-1
    [ 3688.908064] Enter: dla_op_programmed
    [ 3688.908241] Update dependency operation index 3 ROI 0 DEP_COUNT=3
    [ 3688.908486] Update dependency operation index 1 ROI 0 DEP_COUNT=1
    [ 3688.909963] enable SDP in dla_update_dependency as depdency are resolved
    [ 3688.910266] Enter: dla_enable_operation
    [ 3688.910437] exit dla_enable_operation without actual enable due to processor hasn't been programmed
    [ 3688.910797] Exit: dla_enable_operation status=0
    [ 3688.910986] Exit: dla_op_programmed
    [ 3688.911145] Exit: dla_program_operation status=0
    [ 3688.911348] Exit: dla_submit_operation
    [ 3688.911520] Enter: dla_dequeue_operation
    [ 3688.911694] Dequeue op from Convolution processor, index=3 ROI=0
    [ 3688.911944] Enter: dla_submit_operation
    [ 3688.912117] Prepare Convolution operation index 3 ROI 0 dep_count 2
    [ 3688.912376] Enter: dla_prepare_operation
    [ 3688.914273] processor:Convolution group:1, rdma_group:0 available
    [ 3688.914552] Enter: dla_read_config
    [ 3688.916315] Exit: dla_read_config
    [ 3688.916474] Exit: dla_prepare_operation status=0
    [ 3688.917528] Enter: dla_program_operation
    [ 3688.917737] Program Convolution operation index 3 ROI 0 Group[1]
    [ 3688.919080] no desc get due to index==-1
    [ 3688.919434] no desc get due to index==-1
    [ 3688.919613] no desc get due to index==-1
    [ 3688.919782] no desc get due to index==-1
    [ 3688.919949] no desc get due to index==-1
    [ 3688.920118] Enter: dla_op_programmed
    [ 3688.920282] Update dependency operation index 6 ROI 0 DEP_COUNT=3
    [ 3688.921794] Update dependency operation index 4 ROI 0 DEP_COUNT=2
    [ 3688.922071] Exit: dla_op_programmed
    [ 3688.922224] Exit: dla_program_operation status=0
    [ 3688.922418] Exit: dla_submit_operation
    [ 3688.922580] Exit: dla_dequeue_operation
    [ 3688.922745] Enter: dla_submit_operation
    [ 3688.922917] Prepare SDP operation index 1 ROI 0 dep_count 0
    [ 3688.923136] Enter: dla_prepare_operation
    [ 3688.923378] processor:SDP group:1, rdma_group:1 available
    [ 3688.923598] Enter: dla_read_config
    [ 3688.925410] Exit: dla_read_config
    [ 3688.926896] Exit: dla_prepare_operation status=0
    [ 3688.927091] Enter: dla_program_operation
    [ 3688.927263] Program SDP operation index 1 ROI 0 Group[1]
    [ 3688.928020] no desc get due to index==-1
    [ 3688.928194] no desc get due to index==-1
    [ 3688.928468] no desc get due to index==-1
    [ 3688.929676] no desc get due to index==-1
    [ 3688.929875] Enter: dla_op_programmed
    [ 3688.930038] Update dependency operation index 4 ROI 0 DEP_COUNT=1
    [ 3688.930278] enable SDP in dla_update_dependency as depdency are resolved
    [ 3688.930538] Enter: dla_enable_operation
    [ 3688.930709] exit dla_enable_operation without actual enable due to processor hasn't been programmed
    [ 3688.931060] Exit: dla_enable_operation status=0
    [ 3688.931252] Exit: dla_op_programmed
    [ 3688.931403] Exit: dla_program_operation status=0
    [ 3688.931588] Enter: dla_enable_operation
    [ 3688.931753] Enable SDP operation index 1 ROI 0
    [ 3688.932025] Enter: dla_op_enabled
    [ 3688.932180] Update dependency operation index 0 ROI 0 DEP_COUNT=1
    [ 3688.932417] enable Convolution in dla_update_dependency as depdency are resolved
    [ 3688.934580] Enter: dla_enable_operation
    [ 3688.934785] Enable Convolution operation index 0 ROI 0
    [ 3688.935446] Enter: dla_op_enabled
    [ 3688.935603] Exit: dla_op_enabled
    [ 3688.935746] Exit: dla_enable_operation status=0
    [ 3688.935959] Exit: dla_op_enabled
    [ 3688.936117] Exit: dla_enable_operation status=0
    [ 3688.936305] Exit: dla_submit_operation
    [ 3688.936481] Enter: dla_dequeue_operation
    [ 3688.937952] Dequeue op from SDP processor, index=4 ROI=0
    [ 3688.938204] Enter: dla_submit_operation
    [ 3688.938386] Prepare SDP operation index 4 ROI 0 dep_count 0
    [ 3688.938615] Enter: dla_prepare_operation
    [ 3688.943351] processor:SDP group:0, rdma_group:0 available
    [ 3688.943624] Enter: dla_read_config
    [ 3688.969032] Exit: dla_read_config
    [ 3688.969228] Exit: dla_prepare_operation status=0
    [ 3688.969436] Enter: dla_program_operation
    [ 3688.969608] Program SDP operation index 4 ROI 0 Group[0]
    [ 3688.974613] no desc get due to index==-1
    [ 3688.974824] no desc get due to index==-1
    [ 3688.976313] no desc get due to index==-1
    [ 3688.976487] no desc get due to index==-1
    [ 3688.977577] Enter: dla_op_programmed
    [ 3688.977785] Update dependency operation index 7 ROI 0 DEP_COUNT=2
    [ 3688.978038] Exit: dla_op_programmed
    [ 3688.978205] Exit: dla_program_operation status=0
    [ 3688.978412] Enter: dla_enable_operation
    [ 3688.978586] Enable SDP operation index 4 ROI 0
    [ 3688.980151] Enter: dla_op_enabled
    [ 3688.980318] Update dependency operation index 3 ROI 0 DEP_COUNT=2
    [ 3688.981631] Exit: dla_op_enabled
    [ 3688.981830] Exit: dla_enable_operation status=0
    [ 3688.982030] Exit: dla_submit_operation
    [ 3688.982197] Exit: dla_dequeue_operation
    [ 3688.982366] Enter: dla_submit_operation
    [ 3688.982539] Prepare PDP operation index 2 ROI 0 dep_count 1
    [ 3688.982765] Enter: dla_prepare_operation
    [ 3688.983333] processor:PDP group:0, rdma_group:0 available
    [ 3688.983559] Enter: dla_read_config
    [ 3689.003402] Exit: dla_read_config
    [ 3689.003624] Exit: dla_prepare_operation status=0
    [ 3689.003818] Enter: dla_program_operation
    [ 3689.003982] Program PDP operation index 2 ROI 0 Group[0]
    [ 3689.004176] group id 0 rdma id 0
    [ 3689.013963] no desc get due to index==-1
    [ 3689.014182] no desc get due to index==-1
    [ 3689.014345] no desc get due to index==-1
    [ 3689.014502] no desc get due to index==-1
    [ 3689.014670] no desc get due to index==-1
    [ 3689.014835] Enter: dla_op_programmed
    [ 3689.014989] Update dependency operation index 5 ROI 0 DEP_COUNT=2
    [ 3689.015220] Exit: dla_op_programmed
    [ 3689.015379] Exit: dla_program_operation status=0
    [ 3689.015571] Exit: dla_submit_operation
    [ 3689.015740] Enter: dla_dequeue_operation
    [ 3689.015914] Dequeue op from PDP processor, index=5 ROI=0
    [ 3689.016125] Enter: dla_submit_operation
    [ 3689.016284] Prepare PDP operation index 5 ROI 0 dep_count 1
    [ 3689.016485] Enter: dla_prepare_operation
    [ 3689.018908] processor:PDP group:1, rdma_group:1 available
    [ 3689.019175] Enter: dla_read_config
    [ 3714.082055] random: crng init done
    [ 3714.083460] Exit: dla_read_config
    [ 3714.083642] Exit: dla_prepare_operation status=0
    [ 3714.083841] Enter: dla_program_operation
    [ 3714.084017] Program PDP operation index 5 ROI 0 Group[1]
    [ 3714.084229] group id 1 rdma id 1
    [ 3714.086364] no desc get due to index==-1
    [ 3714.086588] no desc get due to index==-1
    [ 3714.086758] no desc get due to index==-1
    [ 3714.086923] no desc get due to index==-1
    [ 3714.087109] no desc get due to index==-1
    [ 3714.087276] no desc get due to index==-1
    [ 3714.087458] Enter: dla_op_programmed
    [ 3714.087625] Exit: dla_op_programmed
    [ 3714.087790] Exit: dla_program_operation status=0
    [ 3714.087998] Exit: dla_submit_operation
    [ 3714.088160] Exit: dla_dequeue_operation
    [ 3714.088326] Exit: dla_initiate_processors status=0
    [ 3714.090184] Enter:dla_handle_events, processor:BDMA
    [ 3714.090470] Exit:dla_handle_events, ret:0
    [ 3714.090640] Enter:dla_handle_events, processor:Convolution
    [ 3714.090865] Handle cdma weight done event, processor Convolution group 0
    [ 3714.091132] Handle cdma data done event, processor Convolution group 0
    [ 3714.091388] Handle op complete event, processor Convolution group 0
    [ 3714.091641] Enter:dla_op_completion processor Convolution group0
    [ 3714.091884] Completed Convolution operation index 0 ROI 0
    [ 3714.092109] 1 HWLs done, totally 11 layers
    [ 3714.092288] Enter: dla_dequeue_operation
    [ 3714.092464] Dequeue op from Convolution processor, index=6 ROI=0
    [ 3714.092749] Enter: dla_submit_operation
    [ 3714.092923] Prepare Convolution operation index 6 ROI 0 dep_count 2
    [ 3714.093171] Enter: dla_prepare_operation
    [ 3714.093393] processor:Convolution group:0, rdma_group:0 available
    [ 3714.093641] Enter: dla_read_config
    [ 3714.095333] Exit: dla_read_config
    [ 3714.095490] Exit: dla_prepare_operation status=0
    [ 3714.095674] Enter: dla_program_operation
    [ 3714.095841] Program Convolution operation index 6 ROI 0 Group[0]
    [ 3714.097160] no desc get due to index==-1
    [ 3714.097465] no desc get due to index==-1
    [ 3714.097648] no desc get due to index==-1
    [ 3714.097815] no desc get due to index==-1
    [ 3714.097984] no desc get due to index==-1
    [ 3714.098153] Enter: dla_op_programmed
    [ 3714.098312] Update dependency operation index 9 ROI 0 DEP_COUNT=3
    [ 3714.098557] Update dependency operation index 7 ROI 0 DEP_COUNT=1
    [ 3714.098807] enable SDP in dla_update_dependency as depdency are resolved
    [ 3714.099066] Enter: dla_enable_operation
    [ 3714.099234] exit dla_enable_operation without actual enable due to processor hasn't been programmed
    [ 3714.099591] Exit: dla_enable_operation status=0
    [ 3714.099789] Exit: dla_op_programmed
    [ 3714.099942] Exit: dla_program_operation status=0
    [ 3714.100135] Exit: dla_submit_operation
    [ 3714.100298] Exit: dla_dequeue_operation
    [ 3714.100472] Exit:dla_op_completion processor Convolution group0 status=0
    [ 3714.100741] Exit:dla_handle_events, ret:0
    [ 3714.100911] Enter:dla_handle_events, processor:SDP
    [ 3714.101111] Handle op complete event, processor SDP group 1
    [ 3714.101338] Enter:dla_op_completion processor SDP group1
    [ 3714.101558] Completed SDP operation index 1 ROI 0
    [ 3714.101758] Update dependency operation index 2 ROI 0 DEP_COUNT=1
    [ 3714.101999] enable PDP in dla_update_dependency as depdency are resolved
    [ 3714.102260] Enter: dla_enable_operation
    [ 3714.102429] Enable PDP operation index 2 ROI 0
    [ 3714.102613] group id 0 rdma id 0
    [ 3714.102812] rdma needed 1
    [ 3714.102980] Enter: dla_op_enabled
    [ 3714.103136] Exit: dla_op_enabled
    [ 3714.103286] Exit: dla_enable_operation status=0
    [ 3714.103475] 2 HWLs done, totally 11 layers
    [ 3714.103654] Enter: dla_free_op_desc op desc index 0 ROI 0
    [ 3714.108590] Exit: dla_free_op_desc
    [ 3714.108755] Enter: dla_dequeue_operation
    [ 3714.108927] Dequeue op from SDP processor, index=7 ROI=0
    [ 3714.109149] Enter: dla_submit_operation
    [ 3714.109322] Prepare SDP operation index 7 ROI 0 dep_count 0
    [ 3714.109545] Enter: dla_prepare_operation
    [ 3714.110792] processor:SDP group:1, rdma_group:1 available
    [ 3714.111028] Enter: dla_read_config
    [ 3714.164393] Exit: dla_read_config
    [ 3714.164573] Exit: dla_prepare_operation status=0
    [ 3714.164767] Enter: dla_program_operation
    [ 3714.164943] Program SDP operation index 7 ROI 0 Group[1]
    [ 3714.176621] no desc get due to index==-1
    [ 3714.176809] no desc get due to index==-1
    [ 3714.179585] no desc get due to index==-1
    [ 3714.179761] no desc get due to index==-1
    [ 3714.179929] no desc get due to index==-1
    [ 3714.180099] Enter: dla_op_programmed
    [ 3714.180259] Exit: dla_op_programmed
    [ 3714.180414] Exit: dla_program_operation status=0
    [ 3714.180609] Enter: dla_enable_operation
    [ 3714.180780] Enable SDP operation index 7 ROI 0
    [ 3714.182902] Enter: dla_op_enabled
    [ 3714.183066] Update dependency operation index 6 ROI 0 DEP_COUNT=2
    [ 3714.183314] Exit: dla_op_enabled
    [ 3714.183460] Exit: dla_enable_operation status=0
    [ 3714.183646] Exit: dla_submit_operation
    [ 3714.183804] Exit: dla_dequeue_operation
    [ 3714.183975] Enter: dla_free_op_desc op desc index 1 ROI 0
    [ 3714.195310] Exit: dla_free_op_desc
    [ 3714.195490] Exit:dla_op_completion processor SDP group1 status=0
    [ 3714.195713] Exit:dla_handle_events, ret:0
    [ 3714.195889] Enter:dla_handle_events, processor:PDP
    [ 3714.196085] Exit:dla_handle_events, ret:0
    [ 3714.196251] Enter:dla_handle_events, processor:CDP
    [ 3714.196442] Exit:dla_handle_events, ret:0
    [ 3714.196608] Enter:dla_handle_events, processor:RUBIK
    [ 3714.196799] Exit:dla_handle_events, ret:0
    [ 3714.204137] Enter:dla_handle_events, processor:BDMA
    [ 3714.204389] Exit:dla_handle_events, ret:0
    [ 3714.204626] Enter:dla_handle_events, processor:Convolution
    [ 3714.204896] Exit:dla_handle_events, ret:0
    [ 3714.205074] Enter:dla_handle_events, processor:SDP
    [ 3714.205282] Exit:dla_handle_events, ret:0
    [ 3714.205446] Enter:dla_handle_events, processor:PDP
    [ 3714.205634] Exit:dla_handle_events, ret:0
    [ 3714.205792] Enter:dla_handle_events, processor:CDP
    [ 3714.205971] Exit:dla_handle_events, ret:0
    [ 3714.206130] Enter:dla_handle_events, processor:RUBIK
    [ 3714.206315] Exit:dla_handle_events, ret:0
    [ 3714.206599] Enter:dla_handle_events, processor:BDMA
    [ 3714.206794] Exit:dla_handle_events, ret:0
    [ 3714.206950] Enter:dla_handle_events, processor:Convolution
    [ 3714.207149] Exit:dla_handle_events, ret:0
    [ 3714.207304] Enter:dla_handle_events, processor:SDP
    [ 3714.207503] Exit:dla_handle_events, ret:0
    [ 3714.207673] Enter:dla_handle_events, processor:PDP
    [ 3714.207868] Exit:dla_handle_events, ret:0
    [ 3714.208032] Enter:dla_handle_events, processor:CDP
    [ 3714.208221] Exit:dla_handle_events, ret:0
    [ 3714.208383] Enter:dla_handle_events, processor:RUBIK
    [ 3714.208611] Exit:dla_handle_events, ret:0
    [ 3714.416359] Enter:dla_handle_events, processor:BDMA
    [ 3714.416665] Exit:dla_handle_events, ret:0
    [ 3714.416884] Enter:dla_handle_events, processor:Convolution
    [ 3714.417126] Exit:dla_handle_events, ret:0
    [ 3714.417323] Enter:dla_handle_events, processor:SDP
    [ 3714.417525] Exit:dla_handle_events, ret:0
    [ 3714.417716] Enter:dla_handle_events, processor:PDP
    [ 3714.417916] Handle op complete event, processor PDP group 0
    [ 3714.418140] Enter:dla_op_completion processor PDP group0
    [ 3714.418354] Completed PDP operation index 2 ROI 0
    [ 3714.418552] Update dependency operation index 3 ROI 0 DEP_COUNT=1
    [ 3714.418799] enable Convolution in dla_update_dependency as depdency are resolved
    [ 3714.419093] Enter: dla_enable_operation
    [ 3714.419254] Enable Convolution operation index 3 ROI 0
    [ 3714.419741] Enter: dla_op_enabled
    [ 3714.419908] Exit: dla_op_enabled
    [ 3714.420057] Exit: dla_enable_operation status=0
    [ 3714.420243] 3 HWLs done, totally 11 layers
    [ 3714.420422] Enter: dla_dequeue_operation
    [ 3714.420592] exit PDP as there's no further operation
    [ 3714.420797] Exit: dla_dequeue_operation
    [ 3714.420969] Enter: dla_free_op_desc op desc index 2 ROI 0
    [ 3714.427803] Exit: dla_free_op_desc
    [ 3714.427974] Exit:dla_op_completion processor PDP group0 status=0
    [ 3714.428216] Exit:dla_handle_events, ret:0
    [ 3714.428412] Enter:dla_handle_events, processor:CDP
    [ 3714.428613] Exit:dla_handle_events, ret:0
    [ 3714.428787] Enter:dla_handle_events, processor:RUBIK
    [ 3714.428997] Exit:dla_handle_events, ret:0
    [ 3714.567790] Enter:dla_handle_events, processor:BDMA
    [ 3714.568091] Exit:dla_handle_events, ret:0
    [ 3714.568255] Enter:dla_handle_events, processor:Convolution
    [ 3714.568474] Handle cdma data done event, processor Convolution group 1
    [ 3714.568736] Exit:dla_handle_events, ret:0
    [ 3714.568901] Enter:dla_handle_events, processor:SDP
    [ 3714.569083] Exit:dla_handle_events, ret:0
    [ 3714.569255] Enter:dla_handle_events, processor:PDP
    [ 3714.569444] Exit:dla_handle_events, ret:0
    [ 3714.569611] Enter:dla_handle_events, processor:CDP
    [ 3714.569805] Exit:dla_handle_events, ret:0
    [ 3714.569976] Enter:dla_handle_events, processor:RUBIK
    [ 3714.570174] Exit:dla_handle_events, ret:0
    [ 3721.334091] Enter:dla_handle_events, processor:BDMA
    [ 3721.334396] Exit:dla_handle_events, ret:0
    [ 3721.334575] Enter:dla_handle_events, processor:Convolution
    [ 3721.334812] Handle cdma weight done event, processor Convolution group 1
    [ 3721.335062] Handle op complete event, processor Convolution group 1
    [ 3721.335305] Enter:dla_op_completion processor Convolution group1
    [ 3721.335551] Completed Convolution operation index 3 ROI 0
    [ 3721.335860] 4 HWLs done, totally 11 layers
    [ 3721.336060] Enter: dla_dequeue_operation
    [ 3721.336231] Dequeue op from Convolution processor, index=9 ROI=0
    [ 3721.336468] Enter: dla_submit_operation
    [ 3721.336664] Prepare Convolution operation index 9 ROI 0 dep_count 2
    [ 3721.336908] Enter: dla_prepare_operation
    [ 3721.337159] processor:Convolution group:1, rdma_group:0 available
    [ 3721.337416] Enter: dla_read_config
    [ 3721.339158] Exit: dla_read_config
    [ 3721.339323] Exit: dla_prepare_operation status=0
    [ 3721.339518] Enter: dla_program_operation
    [ 3721.339702] Program Convolution operation index 9 ROI 0 Group[1]
    [ 3721.341082] no desc get due to index==-1
    [ 3721.341277] no desc get due to index==-1
    [ 3721.341554] no desc get due to index==-1
    [ 3721.341736] no desc get due to index==-1
    [ 3721.341916] no desc get due to index==-1
    [ 3721.342092] no desc get due to index==-1
    [ 3721.342274] Enter: dla_op_programmed
    [ 3721.342443] Update dependency operation index 10 ROI 0 DEP_COUNT=2
    [ 3721.342713] Exit: dla_op_programmed
    [ 3721.342869] Exit: dla_program_operation status=0
    [ 3721.343055] Exit: dla_submit_operation
    [ 3721.343212] Exit: dla_dequeue_operation
    [ 3721.343381] Exit:dla_op_completion processor Convolution group1 status=0
    [ 3721.343646] Exit:dla_handle_events, ret:0
    [ 3721.343822] Enter:dla_handle_events, processor:SDP
    [ 3721.344019] Handle op complete event, processor SDP group 0
    [ 3721.344241] Enter:dla_op_completion processor SDP group0
    [ 3721.344457] Completed SDP operation index 4 ROI 0
    [ 3721.344651] Update dependency operation index 5 ROI 0 DEP_COUNT=1
    [ 3721.344892] enable PDP in dla_update_dependency as depdency are resolved
    [ 3721.345150] Enter: dla_enable_operation
    [ 3721.345322] Enable PDP operation index 5 ROI 0
    [ 3721.345510] group id 1 rdma id 1
    [ 3721.345744] rdma needed 1
    [ 3721.345945] Enter: dla_op_enabled
    [ 3721.346108] Exit: dla_op_enabled
    [ 3721.346260] Exit: dla_enable_operation status=0
    [ 3721.346451] 5 HWLs done, totally 11 layers
    [ 3721.346625] Enter: dla_free_op_desc op desc index 3 ROI 0
    [ 3721.349402] Exit: dla_free_op_desc
    [ 3721.349577] Enter: dla_dequeue_operation
    [ 3721.349750] Dequeue op from SDP processor, index=8 ROI=0
    [ 3721.349988] Enter: dla_submit_operation
    [ 3721.350169] Prepare SDP operation index 8 ROI 0 dep_count 1
    [ 3721.350406] Enter: dla_prepare_operation
    [ 3721.351781] processor:SDP group:0, rdma_group:0 available
    [ 3721.352025] Enter: dla_read_config
    [ 3721.405789] Exit: dla_read_config
    [ 3721.405983] Exit: dla_prepare_operation status=0
    [ 3721.406172] Enter: dla_program_operation
    [ 3721.406339] Program SDP operation index 8 ROI 0 Group[0]
    [ 3721.418160] no desc get due to index==-1
    [ 3721.418353] no desc get due to index==-1
    [ 3721.418516] no desc get due to index==-1
    [ 3721.418688] no desc get due to index==-1
    [ 3721.418859] no desc get due to index==-1
    [ 3721.419042] Enter: dla_op_programmed
    [ 3721.419207] Update dependency operation index 10 ROI 0 DEP_COUNT=1
    [ 3721.419466] enable SDP in dla_update_dependency as depdency are resolved
    [ 3721.419737] Enter: dla_enable_operation
    [ 3721.419912] exit dla_enable_operation without actual enable due to processor hasn't been programmed
    [ 3721.420264] Exit: dla_enable_operation status=0
    [ 3721.420451] Exit: dla_op_programmed
    [ 3721.420601] Exit: dla_program_operation status=0
    [ 3721.420788] Exit: dla_submit_operation
    [ 3721.420948] Exit: dla_dequeue_operation
    [ 3721.421119] Enter: dla_free_op_desc op desc index 4 ROI 0
    [ 3721.426588] Exit: dla_free_op_desc
    [ 3721.426757] Exit:dla_op_completion processor SDP group0 status=0
    [ 3721.426995] Exit:dla_handle_events, ret:0
    [ 3721.427185] Enter:dla_handle_events, processor:PDP
    [ 3721.427393] Exit:dla_handle_events, ret:0
    [ 3721.427567] Enter:dla_handle_events, processor:CDP
    [ 3721.427783] Exit:dla_handle_events, ret:0
    [ 3721.427970] Enter:dla_handle_events, processor:RUBIK
    [ 3721.428179] Exit:dla_handle_events, ret:0
    [ 3721.435900] Enter:dla_handle_events, processor:BDMA
    [ 3721.436172] Exit:dla_handle_events, ret:0
    [ 3721.436345] Enter:dla_handle_events, processor:Convolution
    [ 3721.436561] Exit:dla_handle_events, ret:0
    [ 3721.436727] Enter:dla_handle_events, processor:SDP
    [ 3721.436925] Exit:dla_handle_events, ret:0
    [ 3721.437095] Enter:dla_handle_events, processor:PDP
    [ 3721.437293] Handle op complete event, processor PDP group 1
    [ 3721.437521] Enter:dla_op_completion processor PDP group1
    [ 3721.437741] Completed PDP operation index 5 ROI 0
    [ 3721.437937] Update dependency operation index 6 ROI 0 DEP_COUNT=1
    [ 3721.438183] enable Convolution in dla_update_dependency as depdency are resolved
    [ 3721.438478] Enter: dla_enable_operation
    [ 3721.438648] Enable Convolution operation index 6 ROI 0
    [ 3721.439064] Enter: dla_op_enabled
    [ 3721.439212] Exit: dla_op_enabled
    [ 3721.439357] Exit: dla_enable_operation status=0
    [ 3721.439565] 6 HWLs done, totally 11 layers
    [ 3721.439740] Enter: dla_dequeue_operation
    [ 3721.439910] exit PDP as there's no further operation
    [ 3721.440107] Exit: dla_dequeue_operation
    [ 3721.440271] Enter: dla_free_op_desc op desc index 5 ROI 0
    [ 3721.447906] Exit: dla_free_op_desc
    [ 3721.448076] Exit:dla_op_completion processor PDP group1 status=0
    [ 3721.448314] Exit:dla_handle_events, ret:0
    [ 3721.448492] Enter:dla_handle_events, processor:CDP
    [ 3721.448694] Exit:dla_handle_events, ret:0
    [ 3721.448868] Enter:dla_handle_events, processor:RUBIK
    [ 3721.449080] Exit:dla_handle_events, ret:0
    [ 3721.545029] Enter:dla_handle_events, processor:BDMA
    [ 3721.545338] Exit:dla_handle_events, ret:0
    [ 3721.545506] Enter:dla_handle_events, processor:Convolution
    [ 3721.545717] Handle cdma data done event, processor Convolution group 0
    [ 3721.545971] Exit:dla_handle_events, ret:0
    [ 3721.546131] Enter:dla_handle_events, processor:SDP
    [ 3721.546328] Exit:dla_handle_events, ret:0
    [ 3721.546489] Enter:dla_handle_events, processor:PDP
    [ 3721.546667] Exit:dla_handle_events, ret:0
    [ 3721.546831] Enter:dla_handle_events, processor:CDP
    [ 3721.547015] Exit:dla_handle_events, ret:0
    [ 3721.547168] Enter:dla_handle_events, processor:RUBIK
    [ 3721.547355] Exit:dla_handle_events, ret:0
    [ 3726.151448] Enter:dla_handle_events, processor:BDMA
    [ 3726.151776] Exit:dla_handle_events, ret:0
    [ 3726.151965] Enter:dla_handle_events, processor:Convolution
    [ 3726.152207] Handle cdma weight done event, processor Convolution group 0
    [ 3726.152476] Handle op complete event, processor Convolution group 0
    [ 3726.152776] Enter:dla_op_completion processor Convolution group0
    [ 3726.153016] Completed Convolution operation index 6 ROI 0
    [ 3726.153247] 7 HWLs done, totally 11 layers
    [ 3726.153428] Enter: dla_dequeue_operation
    [ 3726.153595] exit Convolution as there's no further operation
    [ 3726.153810] Exit: dla_dequeue_operation
    [ 3726.153978] Exit:dla_op_completion processor Convolution group0 status=0
    [ 3726.154235] Exit:dla_handle_events, ret:0
    [ 3726.154406] Enter:dla_handle_events, processor:SDP
    [ 3726.154603] Handle op complete event, processor SDP group 1
    [ 3726.154829] Enter:dla_op_completion processor SDP group1
    [ 3726.155035] Completed SDP operation index 7 ROI 0
    [ 3726.155231] Update dependency operation index 8 ROI 0 DEP_COUNT=1
    [ 3726.155474] enable SDP in dla_update_dependency as depdency are resolved
    [ 3726.155732] Enter: dla_enable_operation
    [ 3726.155902] Enable SDP operation index 8 ROI 0
    [ 3726.156298] Enter: dla_op_enabled
    [ 3726.156463] Exit: dla_op_enabled
    [ 3726.156614] Exit: dla_enable_operation status=0
    [ 3726.156807] 8 HWLs done, totally 11 layers
    [ 3726.156981] Enter: dla_free_op_desc op desc index 6 ROI 0
    [ 3726.159328] Exit: dla_free_op_desc
    [ 3726.159499] Enter: dla_dequeue_operation
    [ 3726.159672] Dequeue op from SDP processor, index=10 ROI=0
    [ 3726.159887] Enter: dla_submit_operation
    [ 3726.160057] Prepare SDP operation index 10 ROI 0 dep_count 0
    [ 3726.160284] Enter: dla_prepare_operation
    [ 3726.161682] processor:SDP group:1, rdma_group:1 available
    [ 3726.161928] Enter: dla_read_config
    [ 3726.170226] Exit: dla_read_config
    [ 3726.170399] Exit: dla_prepare_operation status=0
    [ 3726.170601] Enter: dla_program_operation
    [ 3726.170777] Program SDP operation index 10 ROI 0 Group[1]
    [ 3726.171565] no desc get due to index==-1
    [ 3726.171750] no desc get due to index==-1
    [ 3726.171923] no desc get due to index==-1
    [ 3726.172097] no desc get due to index==-1
    [ 3726.172274] no desc get due to index==-1
    [ 3726.172449] no desc get due to index==-1
    [ 3726.172630] Enter: dla_op_programmed
    [ 3726.172790] Exit: dla_op_programmed
    [ 3726.172951] Exit: dla_program_operation status=0
    [ 3726.173143] Enter: dla_enable_operation
    [ 3726.173315] Enable SDP operation index 10 ROI 0
    [ 3726.173578] Enter: dla_op_enabled
    [ 3726.173734] Update dependency operation index 9 ROI 0 DEP_COUNT=2
    [ 3726.173973] Exit: dla_op_enabled
    [ 3726.174121] Exit: dla_enable_operation status=0
    [ 3726.174311] Exit: dla_submit_operation
    [ 3726.174471] Exit: dla_dequeue_operation
    [ 3726.174642] Enter: dla_free_op_desc op desc index 7 ROI 0
    [ 3726.175276] Exit: dla_free_op_desc
    [ 3726.175432] Exit:dla_op_completion processor SDP group1 status=0
    [ 3726.175685] Exit:dla_handle_events, ret:0
    [ 3726.175871] Enter:dla_handle_events, processor:PDP
    [ 3726.176081] Exit:dla_handle_events, ret:0
    [ 3726.176265] Enter:dla_handle_events, processor:CDP
    [ 3726.176473] Exit:dla_handle_events, ret:0
    [ 3726.176661] Enter:dla_handle_events, processor:RUBIK
    [ 3726.176865] Exit:dla_handle_events, ret:0
    [ 3726.251011] Enter:dla_handle_events, processor:BDMA
    [ 3726.251300] Exit:dla_handle_events, ret:0
    [ 3726.251484] Enter:dla_handle_events, processor:Convolution
    [ 3726.251718] Exit:dla_handle_events, ret:0
    [ 3726.251902] Enter:dla_handle_events, processor:SDP
    [ 3726.252106] Handle op complete event, processor SDP group 0
    [ 3726.252334] Enter:dla_op_completion processor SDP group0
    [ 3726.252567] Completed SDP operation index 8 ROI 0
    [ 3726.252770] Update dependency operation index 9 ROI 0 DEP_COUNT=1
    [ 3726.253009] enable Convolution in dla_update_dependency as depdency are resolved
    [ 3726.253316] Enter: dla_enable_operation
    [ 3726.253486] Enable Convolution operation index 9 ROI 0
    [ 3726.253916] Enter: dla_op_enabled
    [ 3726.254073] Exit: dla_op_enabled
    [ 3726.254224] Exit: dla_enable_operation status=0
    [ 3726.254415] 9 HWLs done, totally 11 layers
    [ 3726.254595] Enter: dla_dequeue_operation
    [ 3726.254764] exit SDP as there's no further operation
    [ 3726.254959] Exit: dla_dequeue_operation
    [ 3726.255123] Enter: dla_free_op_desc op desc index 8 ROI 0
    [ 3726.260282] Exit: dla_free_op_desc
    [ 3726.260492] Exit:dla_op_completion processor SDP group0 status=0
    [ 3726.260733] Exit:dla_handle_events, ret:0
    [ 3726.260900] Enter:dla_handle_events, processor:PDP
    [ 3726.261097] Exit:dla_handle_events, ret:0
    [ 3726.261400] Enter:dla_handle_events, processor:CDP
    [ 3726.261741] Exit:dla_handle_events, ret:0
    [ 3726.262037] Enter:dla_handle_events, processor:RUBIK
    [ 3726.262386] Exit:dla_handle_events, ret:0
    [ 3726.356118] Enter:dla_handle_events, processor:BDMA
    [ 3726.356425] Exit:dla_handle_events, ret:0
    [ 3726.356664] Enter:dla_handle_events, processor:Convolution
    [ 3726.356942] Handle cdma data done event, processor Convolution group 1
    [ 3726.357479] Exit:dla_handle_events, ret:0
    [ 3726.358014] Enter:dla_handle_events, processor:SDP
    [ 3726.358669] Exit:dla_handle_events, ret:0
    [ 3726.359191] Enter:dla_handle_events, processor:PDP
    [ 3726.359793] Exit:dla_handle_events, ret:0
    [ 3726.360313] Enter:dla_handle_events, processor:CDP
    [ 3726.360907] Exit:dla_handle_events, ret:0
    [ 3726.361434] Enter:dla_handle_events, processor:RUBIK
    [ 3726.362039] Exit:dla_handle_events, ret:0
    [ 3726.417167] Enter:dla_handle_events, processor:BDMA
    [ 3726.417434] Exit:dla_handle_events, ret:0
    [ 3726.417616] Enter:dla_handle_events, processor:Convolution
    [ 3726.417837] Handle cdma weight done event, processor Convolution group 1
    [ 3726.418104] Exit:dla_handle_events, ret:0
    [ 3726.418274] Enter:dla_handle_events, processor:SDP
    [ 3726.418473] Exit:dla_handle_events, ret:0
    [ 3726.418650] Enter:dla_handle_events, processor:PDP
    [ 3726.418846] Exit:dla_handle_events, ret:0
    [ 3726.419019] Enter:dla_handle_events, processor:CDP
    [ 3726.419225] Exit:dla_handle_events, ret:0
    [ 3726.419392] Enter:dla_handle_events, processor:RUBIK
    [ 3726.419586] Exit:dla_handle_events, ret:0
    [ 3726.422740] Enter:dla_handle_events, processor:BDMA
    [ 3726.423007] Exit:dla_handle_events, ret:0
    [ 3726.423210] Enter:dla_handle_events, processor:Convolution
    [ 3726.423458] Handle op complete event, processor Convolution group 1
    [ 3726.423728] Enter:dla_op_completion processor Convolution group1
    [ 3726.423991] Completed Convolution operation index 9 ROI 0
    [ 3726.424239] 10 HWLs done, totally 11 layers
    [ 3726.424426] Enter: dla_dequeue_operation
    [ 3726.424662] exit Convolution as there's no further operation
    [ 3726.424918] Exit: dla_dequeue_operation
    [ 3726.425117] Exit:dla_op_completion processor Convolution group1 status=0
    [ 3726.425416] Exit:dla_handle_events, ret:0
    [ 3726.425607] Enter:dla_handle_events, processor:SDP
    [ 3726.425834] Handle op complete event, processor SDP group 1
    [ 3726.426084] Enter:dla_op_completion processor SDP group1
    [ 3726.426331] Completed SDP operation index 10 ROI 0
    [ 3726.426556] 11 HWLs done, totally 11 layers
    [ 3726.426744] Enter: dla_free_op_desc op desc index 9 ROI 0
    [ 3726.427128] Exit: dla_free_op_desc
    [ 3726.427311] Enter: dla_free_op_desc op desc index 10 ROI 0
    [ 3726.427658] Exit: dla_free_op_desc
    [ 3726.427844] Exit:dla_op_completion processor SDP group1 status=0
    [ 3726.428110] Exit:dla_handle_events, ret:0
    [ 3726.428299] Enter:dla_handle_events, processor:PDP
    [ 3726.428511] Exit:dla_handle_events, ret:0
    [ 3726.428713] Enter:dla_handle_events, processor:CDP
    [ 3726.428935] Exit:dla_handle_events, ret:0
    [ 3726.429124] Enter:dla_handle_events, processor:RUBIK
    [ 3726.429367] Exit:dla_handle_events, ret:0
    [ 3726.433004] reset engine done
    Work Found!
    Work Done
    ========performSoftwareSoftmax======
    Result:
    0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
    Test pass

I have the same problem.

Run NN/NN_L0_1_small_fbuf
creating new runtime context...
Emulator starting
submitting tasks...
[ 231.546798] Enter:dla_read_network_config
[ 231.552459] Exit:dla_read_network_config status=0
[ 231.557136] Enter: dla_initiate_processors
[ 231.561215] Enter: dla_submit_operation
[ 231.565033] Prepare Convolution operation index 0 ROI 0 dep_count 1
[ 231.571281] Enter: dla_prepare_operation
[ 231.575192] processor:Convolution group:1, rdma_group:0 available
[ 231.581264] Enter: dla_read_config
[ 231.584657] Exit: dla_read_config
[ 231.587941] Exit: dla_prepare_operation status=0
[ 231.592549] Enter: dla_program_operation
[ 231.596455] Program Convolution operation index 0 ROI 0 Group[1]
[ 231.602471] no desc get due to index==-1
[ 231.606353] no desc get due to index==-1
[ 231.610265] no desc get due to index==-1
[ 231.614165] no desc get due to index==-1
[ 231.618070] no desc get due to index==-1
[ 231.621976] Enter: dla_op_programmed
[ 231.625537] Update dependency operation index 2 ROI 0 DEP_COUNT=2
[ 231.631613] Update dependency operation index 1 ROI 0 DEP_COUNT=1
[ 231.637691] enable SDP in dla_update_dependency as depdency are resolved
[ 231.644374] Enter: dla_enable_operation
[ 231.648195] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[ 231.657225] Exit: dla_enable_operation status=0
[ 231.661736] Exit: dla_op_programmed
[ 231.665207] Exit: dla_program_operation status=0
[ 231.669809] Exit: dla_submit_operation
[ 231.673541] Enter: dla_dequeue_operation
[ 231.677449] Dequeue op from Convolution processor, index=2 ROI=0
[ 231.683437] Enter: dla_submit_operation
[ 231.687258] Prepare Convolution operation index 2 ROI 0 dep_count 1
[ 231.693507] Enter: dla_prepare_operation
[ 231.697418] processor:Convolution group:0, rdma_group:0 available
[ 231.703490] Enter: dla_read_config
[ 231.706883] Exit: dla_read_config
[ 231.710174] Exit: dla_prepare_operation status=0
[ 231.714775] Enter: dla_program_operation
[ 231.718681] Program Convolution operation index 2 ROI 0 Group[0]
[ 231.724694] no desc get due to index==-1
[ 231.728573] no desc get due to index==-1
[ 231.732485] no desc get due to index==-1
[ 231.736390] no desc get due to index==-1
[ 231.740297] no desc get due to index==-1
[ 231.744202] Enter: dla_op_programmed
[ 231.747763] Update dependency operation index 6 ROI 0 DEP_COUNT=3
[ 231.753840] Update dependency operation index 3 ROI 0 DEP_COUNT=2
[ 231.759916] Exit: dla_op_programmed
[ 231.763391] Exit: dla_program_operation status=0
[ 231.767990] Exit: dla_submit_operation
[ 231.771721] Exit: dla_dequeue_operation
[ 231.775542] Enter: dla_submit_operation
[ 231.779361] Prepare SDP operation index 1 ROI 0 dep_count 0
[ 231.784917] Enter: dla_prepare_operation
[ 231.788824] processor:SDP group:0, rdma_group:1 available
[ 231.794206] Enter: dla_read_config
[ 231.797597] Exit: dla_read_config
[ 231.800890] Exit: dla_prepare_operation status=0
[ 231.805490] Enter: dla_program_operation
[ 231.809398] Program SDP operation index 1 ROI 0 Group[0]
[ 231.814705] no desc get due to index==-1
[ 231.818598] no desc get due to index==-1
[ 231.822503] no desc get due to index==-1
[ 231.826415] no desc get due to index==-1
[ 231.830317] Enter: dla_op_programmed
[ 231.833877] Update dependency operation index 3 ROI 0 DEP_COUNT=1
[ 231.839954] enable SDP in dla_update_dependency as depdency are resolved
[ 231.846638] Enter: dla_enable_operation
[ 231.850457] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[ 231.859490] Exit: dla_enable_operation status=0
[ 231.864000] Exit: dla_op_programmed
[ 231.867472] Exit: dla_program_operation status=0
[ 231.872072] Enter: dla_enable_operation
[ 231.875893] Enable SDP operation index 1 ROI 0
[ 231.880320] Enter: dla_op_enabled
[ 231.883618] Update dependency operation index 0 ROI 0 DEP_COUNT=1
[ 231.889696] enable Convolution in dla_update_dependency as depdency are resolved
[ 231.897076] Enter: dla_enable_operation
[ 231.900893] Enable Convolution operation index 0 ROI 0
[ 231.906019] Enter: dla_op_enabled
[ 231.909313] Exit: dla_op_enabled
[ 231.912519] Exit: dla_enable_operation status=0
[ 231.917039] Exit: dla_op_enabled
[ 231.920244] Exit: dla_enable_operation status=0
[ 231.924765] Exit: dla_submit_operation
[ 231.928491] Enter: dla_dequeue_operation
[ 231.932405] Dequeue op from SDP processor, index=3 ROI=0
[ 231.937700] Enter: dla_submit_operation
[ 231.941520] Prepare SDP operation index 3 ROI 0 dep_count 0
[ 231.947075] Enter: dla_prepare_operation
[ 231.950986] processor:SDP group:1, rdma_group:0 available
[ 231.956364] Enter: dla_read_config
[ 231.959756] Exit: dla_read_config
[ 231.963047] Exit: dla_prepare_operation status=0
[ 231.967649] Enter: dla_program_operation
[ 231.971555] Program SDP operation index 3 ROI 0 Group[1]
[ 231.976860] no desc get due to index==-1
[ 231.980756] no desc get due to index==-1
[ 231.984667] no desc get due to index==-1
[ 231.988563] no desc get due to index==-1
[ 231.992476] Enter: dla_op_programmed
[ 231.996035] Update dependency operation index 7 ROI 0 DEP_COUNT=2
[ 232.002113] Exit: dla_op_programmed
[ 232.005584] Exit: dla_program_operation status=0
[ 232.010186] Enter: dla_enable_operation
[ 232.014005] Enable SDP operation index 3 ROI 0
[ 232.018437] Enter: dla_op_enabled
[ 232.021731] Update dependency operation index 2 ROI 0 DEP_COUNT=1
[ 232.027808] enable Convolution in dla_update_dependency as depdency are resolved
[ 232.035188] Enter: dla_enable_operation
[ 232.039006] Enable Convolution operation index 2 ROI 0
[ 232.044130] Enter: dla_op_enabled
[ 232.047426] Exit: dla_op_enabled
[ 232.050637] Exit: dla_enable_operation status=0
[ 232.055153] Exit: dla_op_enabled
[ 232.058363] Exit: dla_enable_operation status=0
[ 232.062878] Exit: dla_submit_operation
[ 232.066610] Exit: dla_dequeue_operation
[ 232.070434] Enter: dla_submit_operation
[ 232.074250] Prepare PDP operation index 5 ROI 0 dep_count 1
[ 232.079807] Enter: dla_prepare_operation
[ 232.083713] processor:PDP group:1, rdma_group:1 available
[ 232.089095] Enter: dla_read_config
[ 232.092480] Exit: dla_read_config
[ 232.095779] Exit: dla_prepare_operation status=0
[ 232.100379] Enter: dla_program_operation
[ 232.104287] Program PDP operation index 5 ROI 0 Group[1]
[ 232.109582] group id 1 rdma id 1
[ 232.112808] no desc get due to index==-1
[ 232.116698] no desc get due to index==-1
[ 232.120607] no desc get due to index==-1
[ 232.124505] no desc get due to index==-1
[ 232.128419] no desc get due to index==-1
[ 232.132323] Enter: dla_op_programmed
[ 232.135884] Update dependency operation index 11 ROI 0 DEP_COUNT=2
[ 232.142048] Exit: dla_op_programmed
[ 232.145520] Exit: dla_program_operation status=0
[ 232.150120] Exit: dla_submit_operation
[ 232.153853] Enter: dla_dequeue_operation
[ 232.157760] Dequeue op from PDP processor, index=11 ROI=0
[ 232.163143] Enter: dla_submit_operation
[ 232.166962] Prepare PDP operation index 11 ROI 0 dep_count 1
[ 232.172608] Enter: dla_prepare_operation
[ 232.176505] processor:PDP group:0, rdma_group:0 available
[ 232.181894] Enter: dla_read_config
[ 232.185284] Exit: dla_read_config
[ 232.188570] Exit: dla_prepare_operation status=0
[ 232.193178] Enter: dla_program_operation
[ 232.197085] Program PDP operation index 11 ROI 0 Group[0]
[ 232.202467] group id 0 rdma id 0
[ 232.205691] no desc get due to index==-1
[ 232.209588] no desc get due to index==-1
[ 232.213491] no desc get due to index==-1
[ 232.217398] no desc get due to index==-1
[ 232.221302] no desc get due to index==-1
[ 232.225210] Enter: dla_op_programmed
[ 232.228769] Update dependency operation index 22 ROI 0 DEP_COUNT=2
[ 232.234933] Exit: dla_op_programmed
[ 232.238403] Exit: dla_program_operation status=0
[ 232.243006] Exit: dla_submit_operation
[ 232.246738] Exit: dla_dequeue_operation
[ 232.250559] Enter: dla_submit_operation
[ 232.254378] Prepare CDP operation index 4 ROI 0 dep_count 2
[ 232.259934] Enter: dla_prepare_operation
[ 232.263841] processor:CDP group:1, rdma_group:1 available
[ 232.269223] Enter: dla_read_config
[ 232.272616] Exit: dla_read_config
[ 232.275900] Exit: dla_prepare_operation status=0
[ 232.280508] Enter: dla_program_operation
[ 232.284414] Program CDP operation index 4 ROI 0 Group[1]
[ 232.289710] Enter: dla_cdp_program
[ 232.289716] Enter: processor_cdp_program
[ 232.297105] Exit: processor_cdp_program
[ 232.297107] Exit: dla_cdp_program
[ 232.300865] no desc get due to index==-1
[ 232.308068] no desc get due to index==-1
[ 232.311975] no desc get due to index==-1
[ 232.315882] no desc get due to index==-1
[ 232.319788] no desc get due to index==-1
[ 232.323693] Enter: dla_op_programmed
[ 232.327255] Update dependency operation index 10 ROI 0 DEP_COUNT=3
[ 232.333418] Exit: dla_op_programmed
[ 232.336889] Exit: dla_program_operation status=0
[ 232.341489] Exit: dla_submit_operation
[ 232.345223] Enter: dla_dequeue_operation
[ 232.349129] Dequeue op from CDP processor, index=10 ROI=0
[ 232.354513] Enter: dla_submit_operation
[ 232.358331] Prepare CDP operation index 10 ROI 0 dep_count 2
[ 232.363975] Enter: dla_prepare_operation
[ 232.367881] processor:CDP group:0, rdma_group:0 available
[ 232.373263] Enter: dla_read_config
[ 232.376653] Exit: dla_read_config
[ 232.379940] Exit: dla_prepare_operation status=0
[ 232.384548] Exit: dla_submit_operation
[ 232.388279] Exit: dla_dequeue_operation
[ 232.392100] Exit: dla_initiate_processors status=0
[ 232.396875] Enter:dla_handle_events, processor:BDMA
[ 232.401730] Exit:dla_handle_events, ret:0
[ 232.405722] Enter:dla_handle_events, processor:Convolution
[ 232.411192] Exit:dla_handle_events, ret:0
[ 232.415184] Enter:dla_handle_events, processor:SDP
[ 232.419959] Exit:dla_handle_events, ret:0
[ 232.423952] Enter:dla_handle_events, processor:PDP
[ 232.428727] Exit:dla_handle_events, ret:0
[ 232.432719] Enter:dla_handle_events, processor:CDP
[ 232.437494] Exit:dla_handle_events, ret:0
[ 232.441487] Enter:dla_handle_events, processor:RUBIK
[ 232.446436] Exit:dla_handle_events, ret:0
[ 232.450440] Enter:dla_handle_events, processor:BDMA
[ 232.455291] Exit:dla_handle_events, ret:0
[ 232.459283] Enter:dla_handle_events, processor:Convolution
[ 232.464753] Exit:dla_handle_events, ret:0
[ 232.468745] Enter:dla_handle_events, processor:SDP
[ 232.473520] Exit:dla_handle_events, ret:0
[ 232.477513] Enter:dla_handle_events, processor:PDP
[ 232.482288] Exit:dla_handle_events, ret:0
[ 232.486280] Enter:dla_handle_events, processor:CDP
[ 232.491055] Exit:dla_handle_events, ret:0
[ 232.495048] Enter:dla_handle_events, processor:RUBIK
[ 232.499997] Exit:dla_handle_events, ret:0
Then, the process stalls and the FPGA is not responding anymore. Do you find an idea of which could be the problem? I wil really appreciate your help, thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants