Bug in service shell crashing the node

[papeg]

The current version of the service shell is crashing the node. We have first found this while testing the 06_dcmac example, but the following output is from the 01_aximm example synthesized with shell=service:

diff --git a/examples/01_aximm/config.cfg b/examples/01_aximm/config.cfg
index 78a1c9ac..577d23a8 100644
--- a/examples/01_aximm/config.cfg
+++ b/examples/01_aximm/config.cfg
@@ -19,6 +19,7 @@
 # ##################################################################################################

 [connectivity]
+shell=service
 nk=dma:1:dma_0
 nk=offset:1:offset_0
 stream_connect=offset_0.axis_out:dma_0.axis_in

The test itself passes, but the node crashes. The device is then gone after the automatic warm reboot.

./01_aximm 91:00 aximm_hw.vbin
[2026-08-27 10:01:22.131] [INFO ] void vrt::impl::Device::programDevice()                                         : Programming PDI via vrtd design writer /pc2/users/d/deffel/.cache/SLASH/vrt/vrtbin_91_00/images/top_i_slash_slash_aximm_hw_inst_0_partial.pdi
[2026-08-27 10:01:23.277] [INFO ] vrt::impl::Device::Device(const string&, const string&, bool, vrt::ProgramType) : Programmed user clock to 4000000000 Hz (target 200000000 Hz)
Test passed

click for dmesg output

[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_ioctl: slash_hotplug: ioctl: received cmd=0x40247731
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_ioctl: slash_hotplug: ioctl: dispatching REMOVE
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_ioctl: slash_hotplug: remove: BDF 0000:91:00.2
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_remove: slash_hotplug: remove: starting for BDF 0000:91:00.2
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_get_pci_dev: slash_hotplug: get_pci_dev: looking up 0000:91:00.2 (domain=0000 bus=91 slot=00 func=2)
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_get_pci_dev: slash_hotplug: get_pci_dev: found 0000:91:00.2
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_remove: slash_hotplug: remove: 0000:91:00.2 upstream bridge=0000:90:01.1 bridge_ctrl=0x0012 before remove
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_remove: slash_hotplug: remove: acquiring pci_lock_rescan_remove
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_remove: slash_hotplug: remove: clearing bus master for 0000:91:00.2
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_remove: slash_hotplug: remove: calling pci_stop_and_remove_bus_device for 0000:91:00.2
[Thu Aug 27 10:01:21 2026] slash_ctl 0000:91:00.2: slash: remove start for 0000:91:00.2
[Thu Aug 27 10:01:21 2026] slash_ctl 0000:91:00.2: ctldev: destroying control device
[Thu Aug 27 10:01:21 2026] slash_ctl 0000:91:00.2: slash: remove complete
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_remove: slash_hotplug: remove: released pci_lock_rescan_remove
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_remove: slash_hotplug: remove: 0000:91:00.2 complete
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_ioctl: slash_hotplug: ioctl: REMOVE succeeded
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_ioctl: slash_hotplug: ioctl: received cmd=0x7730
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_ioctl: slash_hotplug: ioctl: dispatching RESCAN
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: acquiring pci_lock_rescan_remove
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:50
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:30
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.1: bridge window [io  0x1000-0x0fff] to [bus 31] add_size 1000
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.2: bridge window [io  0x1000-0x0fff] to [bus 32] add_size 1000
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.3: bridge window [io  0x1000-0x0fff] to [bus 33] add_size 1000
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.1: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.1: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.2: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.2: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.3: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.3: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.3: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.3: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.2: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.2: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.1: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:30:01.1: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:70
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:00
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:20
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:40
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:60
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:10
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:f0
[Thu Aug 27 10:01:21 2026] pcieport 0000:f0:01.1: bridge window [io  0x1000-0x0fff] to [bus f1] add_size 1000
[Thu Aug 27 10:01:21 2026] pcieport 0000:f0:01.2: bridge window [io  0x1000-0x0fff] to [bus f2] add_size 1000
[Thu Aug 27 10:01:21 2026] pcieport 0000:f0:01.1: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:f0:01.1: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] pcieport 0000:f0:01.2: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:f0:01.2: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] pcieport 0000:f0:01.2: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:f0:01.2: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] pcieport 0000:f0:01.1: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:f0:01.1: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:d0
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:80
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:a0
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:c0
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.1: bridge window [io  0x1000-0x0fff] to [bus c1] add_size 1000
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.2: bridge window [io  0x1000-0x0fff] to [bus c2] add_size 1000
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.3: bridge window [io  0x1000-0x0fff] to [bus c3] add_size 1000
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.1: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.1: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.2: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.2: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.3: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.3: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.3: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.3: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.2: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.2: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.1: bridge window [io  size 0x1000]: can't assign; no space
[Thu Aug 27 10:01:21 2026] pcieport 0000:c0:01.1: bridge window [io  size 0x1000]: failed to assign
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:b0
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:90
[Thu Aug 27 10:01:21 2026] pci 0000:91:00.2: [10ee:50c2] type 00 class 0x128000 PCIe Endpoint
[Thu Aug 27 10:01:21 2026] pci 0000:91:00.2: BAR 0 [mem 0xb0000000-0xb7ffffff 64bit]
[Thu Aug 27 10:01:21 2026] pci 0000:91:00.2: BAR 2 [mem 0xb8000000-0xbfffffff 64bit]
[Thu Aug 27 10:01:21 2026] pci 0000:91:00.2: BAR 4 [mem 0xf0280000000-0xf0287ffffff 64bit pref]
[Thu Aug 27 10:01:21 2026] pci 0000:91:00.2: supports D1
[Thu Aug 27 10:01:21 2026] pci 0000:91:00.2: PME# supported from D0 D1 D3hot D3cold
[Thu Aug 27 10:01:21 2026] pci 0000:91:00.2: BAR 0 [mem 0xb0000000-0xb7ffffff 64bit]: assigned
[Thu Aug 27 10:01:21 2026] pci 0000:91:00.2: BAR 2 [mem 0xb8000000-0xbfffffff 64bit]: assigned
[Thu Aug 27 10:01:21 2026] pci 0000:91:00.2: BAR 4 [mem 0xf0280000000-0xf0287ffffff 64bit pref]: assigned
[Thu Aug 27 10:01:21 2026] ami 0000:91:00.2: probe with driver ami failed with error -22
[Thu Aug 27 10:01:21 2026] slash_ctl 0000:91:00.2: slash: probe start for 0000:91:00.2
[Thu Aug 27 10:01:21 2026] slash_ctl 0000:91:00.2: ctldev: creating control device
[Thu Aug 27 10:01:21 2026] slash_ctl 0000:91:00.2: Found BAR0: 0x0x00000000b0000000 - 0x0x00000000b7ffffff (size: 0x0000000008000000) MMIO
[Thu Aug 27 10:01:21 2026] slash_ctl 0000:91:00.2: Found BAR2: 0x0x00000000b8000000 - 0x0x00000000bfffffff (size: 0x0000000008000000) MMIO
[Thu Aug 27 10:01:21 2026] slash_ctl 0000:91:00.2: Found BAR4: 0x0x00000f0280000000 - 0x0x00000f0287ffffff (size: 0x0000000008000000) MMIO
[Thu Aug 27 10:01:21 2026] slash_ctl 0000:91:00.2: slash: BAR0 exported as dma-buf (size=0x0000000008000000)
[Thu Aug 27 10:01:21 2026] slash_ctl 0000:91:00.2: slash: BAR2 exported as dma-buf (size=0x0000000008000000)
[Thu Aug 27 10:01:21 2026] slash_ctl 0000:91:00.2: slash: BAR4 exported as dma-buf (size=0x0000000008000000)
[Thu Aug 27 10:01:21 2026] slash_ctl 0000:91:00.2: ctldev: device created successfully
[Thu Aug 27 10:01:21 2026] slash_ctl 0000:91:00.2: slash: probe successful
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: scanning root bus 0000:e0
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_handle_rescan: slash_hotplug: rescan: complete (16 root bus(es) scanned)
[Thu Aug 27 10:01:21 2026] slash:slash_hotplug_ioctl: slash_hotplug: ioctl: RESCAN succeeded
[Thu Aug 27 10:01:22 2026] ami 0000:91:00.0: ERROR           : Logging thread error - invalid PCI data read!
[Thu Aug 27 10:01:23 2026] eqdma_cpm5_hw_error_process: Global Err Reg(0x248) = 0xffffffff
[Thu Aug 27 10:01:23 2026] addr = 0x00000254 val = 0xffffffff
[Thu Aug 27 10:01:23 2026] ami: CRITICAL WARNING: cmd id: 40 timed out(timeout), hot reset is required
[Thu Aug 27 10:01:23 2026] ami 0000:91:00.0: ERROR           : Submitted command timed out
[Thu Aug 27 10:01:23 2026] ami: ERROR           : Failed to get the heartbeat msg!
[Thu Aug 27 10:01:23 2026] ami: ERROR           : AMC Heartbeat expired event received
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS                         0x254     0xffffffff 4294967295
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_RSVD_1                  [31,26]   0x3f      
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_PORT_ID                 [   25]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_SBE                     [   24]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_DBE                     [   23]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_RQ_CANCEL               [   22]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_DSC                     [   21]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_DMA                     [   20]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_FLR_CANCEL              [   19]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_RSVD_2                  [18,17]   0x3       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_DAT_POISON              [   16]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_TIMEOUT                 [    9]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_FLR                     [    8]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_TAG                     [    6]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_ADDR                    [    5]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_PARAM                   [    4]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_BCNT                    [    3]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_UR_CA                   [    2]   0x1       
[Thu Aug 27 10:01:23 2026] GLBL_DSC_ERR_STS_POISON                  [    1]   0x1       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_ERR_LOG0                        0x25c     0xffffffff 4294967295
[Thu Aug 27 10:01:24 2026] GLBL_DSC_ERR_LOG0_VALID                  [   31]   0x1       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_ERR_LOG0_SEL                    [   30]   0x1       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_ERR_LOG0_RSVD_1                 [29,13]   0x1ffff   
[Thu Aug 27 10:01:24 2026] GLBL_DSC_ERR_LOG0_QID                    [12, 0]   0x1fff    
[Thu Aug 27 10:01:24 2026] ami: ERROR           : FW_IF write request failed; 28
[Thu Aug 27 10:01:24 2026] ami 0000:91:00.0: ERROR           : Submitted command killed, abort.
[Thu Aug 27 10:01:24 2026] ami: ERROR           : Failed to get the heartbeat msg!
[Thu Aug 27 10:01:24 2026] ami: ERROR           : AMC Heartbeat expired event received
[Thu Aug 27 10:01:24 2026] GLBL_DSC_ERR_LOG1                        0x260     0xffffffff 4294967295
[Thu Aug 27 10:01:24 2026] GLBL_DSC_ERR_LOG1_RSVD_1                 [31,28]   0xf       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_ERR_LOG1_CIDX                   [27,12]   0xffff    
[Thu Aug 27 10:01:24 2026] GLBL_DSC_ERR_LOG1_RSVD_2                 [11, 9]   0x7       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_ERR_LOG1_SUB_TYPE               [ 8, 5]   0xf       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_ERR_LOG1_ERR_TYPE               [ 4, 0]   0x1f      
[Thu Aug 27 10:01:24 2026] GLBL_DSC_DBG_DAT0                        0x270     0xffffffff 4294967295
[Thu Aug 27 10:01:24 2026] GLBL_DSC_DAT0_RSVD_1                     [31,30]   0x3       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_DAT0_CTXT_ARB_DIR               [   29]   0x1       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_DAT0_CTXT_ARB_QID               [28,17]   0xfff     
[Thu Aug 27 10:01:24 2026] GLBL_DSC_DAT0_CTXT_ARB_REQ               [16,12]   0x1f      
[Thu Aug 27 10:01:24 2026] GLBL_DSC_DAT0_IRQ_FIFO_FL                [   11]   0x1       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_DAT0_TMSTALL                    [   10]   0x1       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_DAT0_RRQ_STALL                  [ 9, 8]   0x3       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_DAT0_RCP_FIFO_SPC_STALL         [ 7, 6]   0x3       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_DAT0_RRQ_FIFO_SPC_STALL         [ 5, 4]   0x3       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_DAT0_FAB_MRKR_RSP_STALL         [ 3, 2]   0x3       
[Thu Aug 27 10:01:24 2026] GLBL_DSC_DAT0_DSC_OUT_STALL              [ 1, 0]   0x3       
[Thu Aug 27 10:01:25 2026] ami: ERROR           : FW_IF write request failed; 28
[Thu Aug 27 10:01:25 2026] ami 0000:91:00.0: ERROR           : Submitted command killed, abort.
[Thu Aug 27 10:01:25 2026] ami: ERROR           : Failed to get the heartbeat msg!
[Thu Aug 27 10:01:25 2026] ami: ERROR           : AMC Heartbeat expired event received
[Thu Aug 27 10:01:25 2026] GLBL_DSC_DBG_DAT1                        0x274     0xffffffff 4294967295
[Thu Aug 27 10:01:25 2026] GLBL_DSC_DAT1_RSVD_1                     [31,28]   0xf       
[Thu Aug 27 10:01:25 2026] GLBL_DSC_DAT1_EVT_SPC_C2H                [27,22]   0x3f      
[Thu Aug 27 10:01:25 2026] GLBL_DSC_DAT1_EVT_SP_H2C                 [21,16]   0x3f      
[Thu Aug 27 10:01:25 2026] GLBL_DSC_DAT1_DSC_SPC_C2H                [15, 8]   0xff      
[Thu Aug 27 10:01:25 2026] GLBL_DSC_DAT1_DSC_SPC_H2C                [ 7, 0]   0xff      
[Thu Aug 27 10:01:25 2026] GLBL_DSC_ERR_LOG2                        0x27c     0xffffffff 4294967295
[Thu Aug 27 10:01:25 2026] GLBL_DSC_ERR_LOG2_OLD_PIDX               [31,16]   0xffff    
[Thu Aug 27 10:01:25 2026] GLBL_DSC_ERR_LOG2_NEW_PIDX               [15, 0]   0xffff    
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected Poison error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected Unsupported request or completer aborted error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected Unexpected Byte count in completion error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected Parameter mismatch error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected Address mismatch error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected Unexpected tag error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected FLR error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected Timed out error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected Poison data error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected Descriptor fetch cancelled due to FLR error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected DMA engine error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected Invalid PIDX update error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected Descriptor fetch cancelled due to disable register status error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected UNC_ERR_RAM_DBE error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected UNC_ERR_RAM_SBE error
[Thu Aug 27 10:01:25 2026] eqdma_cpm5_hw_error_process detected Port ID Error
[Thu Aug 27 10:01:25 2026] addr = 0x00000264 val = 0xffffffff
[Thu Aug 27 10:01:25 2026] ami: ERROR           : Heartbeat fail count above threshold! Raising fatal event...
[Thu Aug 27 10:01:25 2026] ami: ERROR           : AMC Heartbeat fatal event received, stopping GCQ...
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Service is halted
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Submit command failed
[Thu Aug 27 10:01:26 2026] ami 0000:91:00.0: ERROR           : Failed to fetch sensors

click for vrtd output

Aug 27 10:01:22 fpga1608 vrtd[388560]: Design write submitted dev=0 uid=46476 conn_id=8
Aug 27 10:01:22 fpga1608 vrtd[388560]: Design writer transfer starting
Aug 27 10:01:22 fpga1608 vrtd[388560]: Transferring design writer payload to device offset 0x102100000 (bytes=8314016 buffer_bytes=8314880 keyhole_bytes=4096)
Aug 27 10:01:22 fpga1608 vrtd[388560]: Design write completed successfully for uid=46476 conn_id=8
Aug 27 10:01:23 fpga1608 vrtd[388560]: clock_driver: request_hz=200000000 trying candidate=1/50 m=200 d=5 o=20 est_hz=200000000 diff_hz=0 predicted_divo=20 predicted_rate_hz=200000000
Aug 27 10:01:23 fpga1608 vrtd[388560]: clock_driver[before_program]: wiz=0x00000000 clk=0 prim_in_hz=100000000 fvco_hz=3000000000 rate_hz=200000000 status=0x00000303 reconfig=0x00000000 reg1=0x00001600 reg2=0x00000f0f reg12=0x00000000 reg13=0x00000000 leaf0=0x0000bb00 leaf1=0x00000303
Aug 27 10:01:23 fpga1608 vrtd[388560]: clock_driver[after_program]: wiz=0x00000000 clk=0 prim_in_hz=100000000 fvco_hz=4000000000 rate_hz=200000000 status=0x00000505 reconfig=0x00000000 reg1=0x00000000 reg2=0x00006464 reg12=0x00000400 reg13=0x00000202 leaf0=0x00001a00 leaf1=0x00000505
Aug 27 10:01:23 fpga1608 vrtd[388560]: clock_driver: set_rate request_hz=200000000 reported_hz=4000000000 m=200 d=5 o=20 candidate=1/50
Aug 27 10:01:23 fpga1608 vrtd[388560]: clock_op: op=2 region=1 rate_hz=4000000000 dev=0 uid=46476 conn_id=8
Aug 27 10:01:23 fpga1608 vrtd[388560]: Buffer opened size=67108864 phys_addr=0x60000000000 dev=0 uid=46476 conn_id=8
Aug 27 10:01:23 fpga1608 vrtd[388560]: Buffer opened size=67108864 phys_addr=0x4000000000 dev=0 uid=46476 conn_id=8
Aug 27 10:01:23 fpga1608 vrtd[388560]: Buffer closed addr=0x4000000000 size=67108864 dev=0 uid=46476 conn_id=8

If you need more details or tests, let me know.
Thanks a lot