Memory leak when driving simulations with a VPI plugin.

[bellaz89]

Description
In simulations where events are scheduled by a VPI plugin using vpi_register_cb, even if no wave files are specified, the memory usage increases every time a new callback is registered. If to many calls are done, the system ram and cache is completely filled and the process is killed. In my machine such amount is around to ~5GB of ram.

Expected behavior
I would have expected that a fixed amount of memory is needed in a simulation if the wave traces have not to be saved

How to reproduce?

To reproduce the issue the following vpi plugin is given:

#include<vpi_user.h>
#include<inttypes.h>
#include<stdio.h>

#define STOP_ITERATION 1000000000

uint32_t iteration = 0;


PLI_INT32 start_cb(p_cb_data);
PLI_INT32 end_cb(p_cb_data);
PLI_INT32 rw_cb(p_cb_data);
PLI_INT32 ro_cb(p_cb_data);
PLI_INT32 delay_rw_cb(p_cb_data);
PLI_INT32 delay_ro_cb(p_cb_data);

void register_cb(PLI_INT32(*f)(p_cb_data),
                 PLI_INT32 reason,
                 int64_t cycles){

    s_cb_data cbData;
    s_vpi_time simuTime;
    if (cycles < 0){
        cbData.time = NULL;
    } else {
        cbData.time = &simuTime;
        simuTime.type = vpiSimTime;
        simuTime.high = (PLI_INT32) (cycles >> 32);
        simuTime.low = (PLI_INT32) (cycles & 0xFFFFFFFF);
    }

    cbData.reason = reason;
    cbData.cb_rtn = f;
    cbData.user_data = 0;
    cbData.value = 0;

    vpi_register_cb(&cbData);
}

void entry_point_cb() {
    register_cb(start_cb, cbStartOfSimulation, -1);
    register_cb(end_cb, cbEndOfSimulation, -1);
    register_cb(delay_ro_cb, cbAfterDelay, 0);
}

PLI_INT32 start_cb(p_cb_data data){
    (void) data;
    printf("Start of simulation \n");
    return 0;
}

PLI_INT32 end_cb(p_cb_data data){
    (void) data;
    printf("End of simulation %u \n", iteration);
    return 0;
}


PLI_INT32 rw_cb(p_cb_data data){
    (void) data;
    if(iteration < STOP_ITERATION) {
        register_cb(delay_ro_cb, cbAfterDelay, 1);
    } else {
        vpi_control(vpiFinish, 0);
    }

    iteration++;
    return 0;
}

PLI_INT32 ro_cb(p_cb_data data){
    (void) data;
    register_cb(delay_rw_cb, cbAfterDelay, 0);
    return 0;
}

PLI_INT32 delay_rw_cb(p_cb_data data){
    (void) data;
    register_cb(rw_cb, cbReadWriteSynch, 0);
    return 0;
}

PLI_INT32 delay_ro_cb(p_cb_data data){
    (void) data;
    register_cb(ro_cb, cbReadOnlySynch, 0);
    return 0;
}

void (*vlog_startup_routines[]) () = {
    entry_point_cb,
    0
};

The VPI plugin generate a chain of callbacks that repeats
cbAfterDelay -> cbReadWriteSynch -> cbAfterDelay -> cbReadOnlySynch one billion times

A 'dummy' vhdl source(any other vhdl file can be used):

library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;

entity adder is
   port
   (
      nibble1, nibble2 : in unsigned(3 downto 0); 
      sum       : out unsigned(3 downto 0); 
      carry_out : out std_logic
   );
end entity adder;
 
architecture behavioral of adder is
   signal temp : unsigned(4 downto 0); 
begin 
   temp <= ("0" & nibble1) + nibble2; 
   sum       <= temp(3 downto 0); 
   carry_out <= temp(4);
end architecture behavioral;

A script file to execute the simulation

$GHDL -a adder.vhd
$GHDL --vpi-compile gcc -c vpi_plugin.c
$GHDL --vpi-link gcc -o vpi_plugin.vpi vpi_plugin.o -lc
$GHDL --elab-run adder --vpi=./vpi_plugin.vpi

to run the simulation use

GHDL=ghdl-mcode sh execute.sh
GHDL=ghdl-gcc execute.sh
GHDL=ghdl-llvm sh execute.sh

Context

• OS: Debian 11 (Bullseye)
• Origin:
  • Package manager: GHDL 0.37 (Debian 0.37+dfsg-1) [Dunoon edition]
  • Released binaries:
  • Built from sources: commit 471f916

Memory usage report example

GHDL=ghdl-mcode /usr/bin/time -v sh execute.sh 
loading VPI module './vpi_plugin.vpi'
VPI module loaded!
Start of simulation 
../../src/openieee/v93/numeric_std-body.vhdl:398:9:@0ms:(assertion warning): NUMERIC_STD."+": non logical value detected
Killed
Command exited with non-zero status 137
	Command being timed: "sh execute.sh"
	User time (seconds): 12.14
	System time (seconds): 2.31
	Percent of CPU this job got: 97%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:14.77
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 4959252
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 496
	Minor (reclaiming a frame) page faults: 2029823
	Voluntary context switches: 863
	Involuntary context switches: 1858
	Swaps: 0
	File system inputs: 118752
	File system outputs: 72
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 137

Notes

This was discovered working on SpinalHDL/SpinalHDL#146

[tgingold]

I will investigate.

The memory handling in VPI is not clear. In all examples you can find in books or on the internet, vpi_free is rarely called.

[tgingold]

And thank you for your standalone reproducer. That's very useful.

[bellaz89]

@tgingold

Thank you :)!

[bellaz89]

@tgingold
Maybe this might help.
From Sutherland's book:

PLI_INT32 vpi_free_object(vpiHandle handle)/* handle for an object */

Frees memory allocated by the PLI for the specified handle. Returns 1 if successful, and
0 if an error occurred. Should be called to free an iterator handle whenever vpi_scan() is
terminated before it returns NULL. Should also be called to free a handle returned from
vpi_register_cb() or vpi_put_value() when the handle is no longer needed.

I tried to use it on the handle of vpi_register_cb but it crashes the ghdl execution :P ..

[tgingold]

Can you post the code which use vpi_free_object ? That could be useful too!

[bellaz89]

From Sutherland's

When a simulation callback is registered, the handle for the callback will persist the
remainder of simulation, unless the callback is removed. The only usage of the call-
back handle is for removing the callback using vpi_remove_cb (). If the PLI application has no need to remove the callback, then the handle for the callback should be
freed immediately after the callback is registered. The recommended method of registering most simulation callbacks is:

callback_handle = vpi_register_cb(&cb_data_s);
vpi_free_object(callback_handle); /* free callback handle */

Freeing the callback handle immediately after registering the callback is registered
prevents a memory leak in simulation. For example, if a new simulation callback
were registered every positive edge of a design clock, and the design were simulated
millions of clock cycles, then millions of handles would be created. If the handles
were not released, a serious memory usage problem would occur.

Do not call vpi_free_object () on a callback handle if the callback has been
removed using vpi_remove_cb (). The latter routine automatically releases the
callback handle, making the handle invalid.

So I tried to wrap vpi_register_cb(&cbData) of the reproducer I posted inside vpi_free_object, without success due to crashes.

There are some examples in The Verilog PLI Handbook that use vpi_free_object () to delete callback handles, but the way they do that is not anything more than the two lines of code above