vcomponents tests are failling with ModelSim

[NanooooK]

When running vcomponents tests, several are failling:
tox -v -e py36-vcomponents-modelsim

fail vunit_lib.tb_axi_stream.id_l=0 dest_l=0 user_l=0.test back-to-back failing check                                                                         (1.0 seconds)
fail vunit_lib.tb_axi_stream.id_l=0 dest_l=0 user_l=8.test back-to-back failing check                                                                         (1.0 seconds)
fail vunit_lib.tb_axi_stream.id_l=0 dest_l=8 user_l=0.test back-to-back failing check                                                                         (1.0 seconds)
fail vunit_lib.tb_axi_stream.id_l=0 dest_l=8 user_l=8.test back-to-back failing check                                                                         (1.0 seconds)
fail vunit_lib.tb_axi_stream.id_l=8 dest_l=0 user_l=0.test back-to-back failing check                                                                         (1.0 seconds)
fail vunit_lib.tb_axi_stream.id_l=8 dest_l=0 user_l=8.test back-to-back failing check                                                                         (1.0 seconds)
fail vunit_lib.tb_axi_stream.id_l=8 dest_l=8 user_l=0.test back-to-back failing check                                                                         (1.0 seconds)
fail vunit_lib.tb_axi_stream.id_l=8 dest_l=8 user_l=8.test back-to-back failing check                                                                         (1.0 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.0,write_prob:1.0,read_prob:1.0,transfers:64.wait until idle                         (1.0 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.0,write_prob:1.0,read_prob:0.3,transfers:64.wait until idle                         (1.1 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.0,write_prob:0.3,read_prob:1.0,transfers:64.wait until idle                         (1.0 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.0,write_prob:0.3,read_prob:0.3,transfers:64.wait until idle                         (1.0 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.7,write_prob:1.0,read_prob:1.0,transfers:64.wait until idle                         (1.0 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.7,write_prob:1.0,read_prob:0.3,transfers:64.wait until idle                         (1.1 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.7,write_prob:0.3,read_prob:1.0,transfers:64.wait until idle                         (1.1 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.7,write_prob:0.3,read_prob:0.3,transfers:64.wait until idle                         (1.0 seconds)
=================================================================================================================================================================================
pass 473 of 489
fail 16 of 489
=================================================================================================================================================================================
Total time was 493.0 seconds
Elapsed time was 493.9 seconds

 # ** Note: Got mocked log item 
#    time = 45000 ps
#    logger = check
#    log_level = error
#    msg = TLAST mismatch, check non-blocking - Got 1. Expected 0.
#    file_name:line_num = :0
# 
#    Time: 45 ns  Iteration: 0  Instance: /tb_axi_stream/axi_stream_slave_inst
# ** Failure: log item mismatch:
# 
# Got:
#    logger = check
#    log_level = pass
#    msg = Invalid strb not X
#    file_name:line_num = :0
# 
# expected:
#    logger = check
#    log_level = error
#    msg = TDATA mismatch, check non-blocking - Got 0000_0011 (3). Expected 0000_0110 (6).
#    file_name:line_num = :0

fail (P=0 S=0 F=1 T=1) vunit_lib.tb_axi_stream.id_l=0 dest_l=0 user_l=0.test back-to-back failing check (2.4 seconds)

Details for the failures can be found here and there.
Environment is VUnit 4.4.0 and ModelSim 10.6c.

[eine]

@NanooooK is this still failing with v4.5.0 or v4.6.0?

[dalex78]

I also have fails using:

• python 3.9.2
• ModelSim - Interl FPGA Starter Edition 2020.1
• Vunit: v4.6.0
• tox -v -e py39-vcomponents-modelsim to launch the tests

fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.0,write_prob:1.0,read_prob:1.0,transfers:64.wait until idle                         (3.9 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.0,write_prob:1.0,read_prob:0.3,transfers:64.wait until idle                         (4.1 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.0,write_prob:0.3,read_prob:1.0,transfers:64.wait until idle                         (3.5 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.0,write_prob:0.3,read_prob:0.3,transfers:64.wait until idle                         (3.6 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.7,write_prob:1.0,read_prob:1.0,transfers:64.wait until idle                         (3.3 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.7,write_prob:1.0,read_prob:0.3,transfers:64.wait until idle                         (3.1 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.7,write_prob:0.3,read_prob:1.0,transfers:64.wait until idle                         (3.0 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.7,write_prob:0.3,read_prob:0.3,transfers:64.wait until idle                         (3.3 seconds)
fail vunit_lib.tb_axi_stream.id_l=0 dest_l=0 user_l=0.test back-to-back failing check                                                                         (2.4 seconds)
fail vunit_lib.tb_axi_stream.id_l=0 dest_l=0 user_l=8.test back-to-back failing check                                                                         (2.4 seconds)
fail vunit_lib.tb_axi_stream.id_l=0 dest_l=8 user_l=0.test back-to-back failing check                                                                         (2.3 seconds)
fail vunit_lib.tb_axi_stream.id_l=0 dest_l=8 user_l=8.test back-to-back failing check                                                                         (2.3 seconds)
fail vunit_lib.tb_axi_stream.id_l=8 dest_l=0 user_l=0.test back-to-back failing check                                                                         (2.3 seconds)
fail vunit_lib.tb_axi_stream.id_l=8 dest_l=0 user_l=8.test back-to-back failing check                                                                         (2.3 seconds)
fail vunit_lib.tb_axi_stream.id_l=8 dest_l=8 user_l=0.test back-to-back failing check                                                                         (2.3 seconds)
fail vunit_lib.tb_axi_stream.id_l=8 dest_l=8 user_l=8.test back-to-back failing check                                                                         (2.3 seconds)
=================================================================================================================================================================================
pass 484 of 500
fail 16 of 500
=================================================================================================================================================================================
Total time was 1288.9 seconds
Elapsed time was 1290.6 seconds
=================================================================================================================================================================================
Some failed!

vunit_test.log

Tell me if you need more inputs.

[dalex78]

I also have fails using:

• python 3.9.2
• GHDL 2.0.0-dev (1.0.0.r838.ge14d7e67) [Dunoon edition] (Compiled with GNAT Version: 10.2.1 20210110 and GCC back-end code generator)
• Vunit: v4.6.0
• tox -v -e py39-vcomponents-ghdl to launch the tests

fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.0,write_prob:1.0,read_prob:1.0,transfers:64.wait until idle                         (2.6 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.0,write_prob:1.0,read_prob:0.3,transfers:64.wait until idle                         (2.7 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.0,write_prob:0.3,read_prob:1.0,transfers:64.wait until idle                         (2.8 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.0,write_prob:0.3,read_prob:0.3,transfers:64.wait until idle                         (2.6 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.7,write_prob:1.0,read_prob:1.0,transfers:64.wait until idle                         (2.6 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.7,write_prob:1.0,read_prob:0.3,transfers:64.wait until idle                         (2.7 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.7,write_prob:0.3,read_prob:1.0,transfers:64.wait until idle                         (2.7 seconds)
fail vunit_lib.tb_avalon_master.readdatavalid_prob:1.0,waitrequest_prob:0.7,write_prob:0.3,read_prob:0.3,transfers:64.wait until idle                         (2.5 seconds)
fail vunit_lib.tb_axi_stream.id_l=0 dest_l=0 user_l=0.test back-to-back failing check                                                                         (2.4 seconds)
fail vunit_lib.tb_axi_stream.id_l=0 dest_l=0 user_l=8.test back-to-back failing check                                                                         (2.4 seconds)
fail vunit_lib.tb_axi_stream.id_l=0 dest_l=8 user_l=0.test back-to-back failing check                                                                         (2.4 seconds)
fail vunit_lib.tb_axi_stream.id_l=0 dest_l=8 user_l=8.test back-to-back failing check                                                                         (2.5 seconds)
fail vunit_lib.tb_axi_stream.id_l=8 dest_l=0 user_l=0.test back-to-back failing check                                                                         (2.4 seconds)
fail vunit_lib.tb_axi_stream.id_l=8 dest_l=0 user_l=8.test back-to-back failing check                                                                         (2.5 seconds)
fail vunit_lib.tb_axi_stream.id_l=8 dest_l=8 user_l=0.test back-to-back failing check                                                                         (2.4 seconds)
fail vunit_lib.tb_axi_stream.id_l=8 dest_l=8 user_l=8.test back-to-back failing check                                                                         (2.4 seconds)
=================================================================================================================================================================================
pass 484 of 500
fail 16 of 500
=================================================================================================================================================================================
Total time was 1308.8 seconds
Elapsed time was 1310.4 seconds
=================================================================================================================================================================================
Some failed!

vunit_test_ghdl.log

[LarsAsplund]

I didn't manage to recreate this problem in GHDL when using

$ ghdl --version                                                                   
GHDL 2.0.0-dev (1.0.0.r89.g7722e309) [Dunoon edition]                              
 Compiled with GNAT Version: 10.2.0                                                
 mcode code generator                                                              
Written by Tristan Gingold.                                                        
                                                                                   
Copyright (C) 2003 - 2021 Tristan Gingold.                                         
GHDL is free software, covered by the GNU General Public License.  There is NO     
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.        
                                                                                                           
$ python --version                                                                 
Python 3.9.5

It was reproduceable in ModelSim and I fixed the root problem (delta cycle race conditions). Please verify in your GHDL setup.

[dalex78]

I launched a new test by setting VUNIT_SIMULATOR=ghdl and using python run.py vunit_lib.tb_avalon_master.readdatavalid_prob* but it seems to launch the test with modelsim as I the directory created under vunit_out is named modelsim. This seems weird ? I might be doing something wrong.
Is there a command in vunit to list the simulators it detects and can use ?
Anyway, I have launched the tests with modelsim (using the issue-642 branch and tox -v -e py39-vcomponents-modelsim), it worked for me (no more errors).

[LarsAsplund]

Yes, that is weird. The commit only changes VHDL files so I can't see that I introduced anything that will cause that. It happens to me sometimes but that is usually because I misspell, for example export VUNIT_SIMULTAOR=ghdl. If the simulator you give isn't present it would fail.

Anyway, I noticed that my fix isn't good enough. It now fails with GHDL for me locally while ModelSim that used to fail works. At least for tb_axi_stream.

[dalex78]

Yes, that is weird. The commit only changes VHDL files so I can't see that I introduced anything that will cause that

I tested that on the 4.6.0. I will retry later after re-installing ghdl properly (this should have nothing to do with your commit).

[LarsAsplund]

I think I fixed it now. https://github.com/VUnit/vunit/actions/runs/2179374584 looks better but has yet to complete.

[LarsAsplund]

I merged the fix to master so you should be able to test it from there.

[dalex78]

I have tested, with modelsim, everything pass.
I have not tested with GHDL, I am trying to use GHDL with docker (never done before). But, if you tested with GHDL, this issue can be close I guess.

[dalex78]

Test done using the container ghdl/ghdl:bullseye-gcc-9.1.0

GHDL 3.0.0-dev (v2.0.0-93-gff5c35e4) [Dunoon edition]
 Compiled with GNAT Version: 9.3.0
 GCC back-end code generator

With VUnit: issue a0e25830ad28f9ed2009c3e2da426cae3be2e40

Results:

==================================================
pass 500 of 500
==================================================
Total time was 2552.1 seconds
Elapsed time was 2555.5 seconds
==================================================
All passed!

[LarsAsplund]

Great, in that case we can close this issue.