AXI Stream Slave issue? - ModelSim Microsemi Pro 2020.3 Crash

[mkuklewski]

Hello,
In my project I want to receive large amount of AXI4 Stream transactions (to validate my IP Cores).
In my test bench I am receiving data by using axi_stream_slave entity and pop_axi_stream() function.

After more than 65 536 transactions - but less than 131 072 (I think), ModelSim completely crashes (without any error) and inside output.txt file I find following message:

querycommand error: cmd=namespace inscope ::FlatMemlist {::.main_pane.memory.interior.cs.body.tree GenerateFlatMemList /ccsds_tb/axi_stream_slave_inst/bus_process/rnd 0 150}
results=expected integer but got ""
Stack Trace:
expected integer but got ""
    while executing
"incr i -1"
    (object "::.main_pane.memory.interior.cs.body.tree" method "::FlatMemlist::GetChildrenMem" body line 23)
    invoked from within
"GetChildrenMem $memName $idx $count"
    (object "::.main_pane.memory.interior.cs.body.tree" method "::FlatMemlist::GenerateFlatMemList" body line 21)
    invoked from within
"::.main_pane.memory.interior.cs.body.tree GenerateFlatMemList /ccsds_tb/axi_stream_slave_inst/bus_process/rnd 0 150"
    (in namespace inscope "::FlatMemlist" script line 1)
    invoked from within
"namespace inscope ::FlatMemlist {::.main_pane.memory.interior.cs.body.tree GenerateFlatMemList /ccsds_tb/axi_stream_slave_inst/bus_process/rnd 0 15..."
querycommand error: cmd=namespace inscope ::FlatMemlist {::.main_pane.memory.interior.cs.body.tree GenerateFlatMemList /ccsds_tb/axi_stream_slave_inst/bus_process/rnd 0 150}
results=expected integer but got ""
Stack Trace:
expected integer but got ""
    while executing
"incr i -1"
    (object "::.main_pane.memory.interior.cs.body.tree" method "::FlatMemlist::GetChildrenMem" body line 23)
    invoked from within
"GetChildrenMem $memName $idx $count"
    (object "::.main_pane.memory.interior.cs.body.tree" method "::FlatMemlist::GenerateFlatMemList" body line 21)
    invoked from within
"::.main_pane.memory.interior.cs.body.tree GenerateFlatMemList /ccsds_tb/axi_stream_slave_inst/bus_process/rnd 0 150"
    (in namespace inscope "::FlatMemlist" script line 1)
    invoked from within
"namespace inscope ::FlatMemlist {::.main_pane.memory.interior.cs.body.tree GenerateFlatMemList /ccsds_tb/axi_stream_slave_inst/bus_process/rnd 0 15..."

Also, when it happens ModelSim takes over 4GB of memory.

[LarsAsplund]

@mkuklewski Sounds like you're running out of memory. Can you print the output when calling the get_messenger_state_string function and share the result?

[mkuklewski]

Here it is:

#    1333587500 ps - default                      -    INFO - Active actors:
#                                                               Name: slave
#                                                                 Is deferred: no
#                                                                 Mailbox: inbox
#                                                                   Size: 2147483647
#                                                                   Messages:
#                                                                 Mailbox: outbox
#                                                                   Size: 2147483647
#                                                                   Messages:
#                                                                 Subscriptions:
#                                                                 Subscribers:
#                                                             
#                                                               Name: receiver
#                                                                 Is deferred: no
#                                                                 Mailbox: inbox
#                                                                   Size: 2147483647
#                                                                   Messages:
#                                                                 Mailbox: outbox
#                                                                   Size: 2147483647
#                                                                   Messages:
#                                                                 Subscriptions:
#                                                                 Subscribers:
#                                                             
#                                                               Name: sender
#                                                                 Is deferred: no
#                                                                 Mailbox: inbox
#                                                                   Size: 2147483647
#                                                                   Messages:
#                                                                 Mailbox: outbox
#                                                                   Size: 2147483647
#                                                                   Messages:
#                                                                 Subscriptions:
#                                                                 Subscribers:
#                                                             
#                                                               Name: 
#                                                                 Is deferred: no
#                                                                 Mailbox: inbox
#                                                                   Size: 2147483647
#                                                                   Messages:
#                                                                 Mailbox: outbox
#                                                                   Size: 2147483647
#                                                                   Messages:
#                                                                 Subscriptions:
#                                                                 Subscribers:
#                                                             
#                                                               Name: main
#                                                                 Is deferred: no
#                                                                 Mailbox: inbox
#                                                                   Size: 2147483647
#                                                                   Messages:
#                                                                 Mailbox: outbox
#                                                                   Size: 2147483647
#                                                                   Messages:
#                                                                 Subscriptions:
#                                                                 Subscribers:
#                                                             
#                                                             Deferred actors:

Hm... All inboxes are empty.

Should I try to deallocate memory used by AXI4Stream Slave (for example after receiving a 100 messages)? If so how I can do it?

EDIT:
After some digging I have found that a table inside vec_ptr_storage is getting bigger and bigger

The issue seems to be too many transaction.
Each time pop_axi_stream is used, a new pointer is created, which then increases size inside vec_ptr_storage and this causes memory issue.

[LarsAsplund]

pop will do a delete of messages which is actually a recycling of memory in the sense that we keep it for future use since constantly allocating and deallocating memory affects performance a lot. I will have to do some tests on this but it seems reasonable to have a limit on how much we should recycle before starting to deallocate.

[mkuklewski]

Is there any workaround to manually deallocate memory?
In my case it seems recycling memory do not work, as the only messages which are sent are pop type and process assign new record for each new one.

[LarsAsplund]

@mkuklewski I thought about this a bit more and I was wrong in my previous statement. That doesn't explain why there is a memory leak. If I for example have a a stream master and a slave and let them send data between each other then RivieraPro shows constant memory usage if I look in the task manager. In my test I have 1 million transactions, each one byte. There is no fundamental leak in the code. If I use ModelSim I see that the memory usage is constantly increasing during the simulation, same thing with GHDL. Hard to say if this is the cause of your problem or if there is something else in your testbench, Also hard to say what's causing this. Does it have to do with how these simulators handles VUnit or is it something else. I think I would need a minimal working example from you to understand what is happening.

[mkuklewski]

@LarsAsplund
Here is example tb which shows the issue, I think this is also correlated with size of the transaction.
memory_leak_test_tb.zip

With this testbench it takes quite long time to actually fail. Probably it would fail faster if more com transactions would be used.

# Start time: 14:21:29 on Dec 14,2022
# //  ModelSim Microsemi Pro 2020.3 Jul 13 2020
# //
# //  Copyright 1991-2020 Mentor Graphics Corporation
# //  All Rights Reserved.
# //
# //  ModelSim Microsemi Pro and its associated documentation contain trade
# //  secrets and commercial or financial information that are the property of
# //  Mentor Graphics Corporation and are privileged, confidential,
# //  and exempt from disclosure under the Freedom of Information Act,
# //  5 U.S.C. Section 552. Furthermore, this information
# //  is prohibited from disclosure under the Trade Secrets Act,
# //  18 U.S.C. Section 1905.
# //
# ============================================================
# ** Fatal: (vsim-4) ****** Memory allocation failure. *****
# Attempting to allocate 131072 bytes
# Please check your system for available memory and swap space.
# ** Fatal: (vsim-4) ****** Memory allocation failure. *****
# Attempting to allocate 131072 bytes
# Please check your system for available memory and swap space.
# End time: 15:09:32 on Dec 14,2022, Elapsed time: 0:48:03
# Errors: 1, Warnings: 0

[LarsAsplund]

@mkuklewski When running your example I see the same behavior as before. Riviera-PRO use constant memory while the other simulators constantly require more memory. There is a difference between the example and the original code though. In the example I see that the number of allocated vectors in the data structure settles at 703 (st.idx)

This means that all but the 703 first indices in st.ptrs will have the value NULL. The number of indices is by default 65k but if the application needs more than that, the structure can be resized. That is what seems to be happening in your original example where the number of entries is larger.

It looks like there are two issues here.

1. Some simulators leak although the size of the data structure remains the same
2. Your original testbench has something else consuming memory

What is causing your testbench to run out of memory is hard to tell but I would start investigating what that other thing is. I suggest that you run your testbench for a short while so that those allocating constant memory have done that. Then set a breakpoint here:

vunit/vunit/vhdl/data_types/src/integer_vector_ptr_pkg-body-2002p.vhd

Line 159 in f1eb251

st.idx := st.idx + 1;

That line should be executed continuously by the code allocating more memory. Step out of the function to see where the calls come from.

[mkuklewski]

@LarsAsplund
Ok, so here is a screenshot with call stack and locals values:

As you can see idx value constantly increase and I can confirm, that this call stack is consistent -> no other function calls it.
Here are additional stacks which might help you: