Erroneous cyclic dependency when using VHDL configurations

[masubious]

Hello,

we're using VHDL configurations in our Simulation Setup. With the current VUnit-5.0.0.dev10 this Setup fails with this error:

Re-compile not needed

Traceback (most recent call last):
  File "C:\Python311\Lib\site-packages\vunit\ui\__init__.py", line 737, in main
    all_ok = self._main(post_run)
             ^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\vunit\ui\__init__.py", line 783, in _main
    all_ok = self._main_run(post_run)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\vunit\ui\__init__.py", line 1055, in _main_run
    self._run_test(test_list, report, simulator_if)
  File "C:\Python311\Lib\site-packages\vunit\ui\__init__.py", line 1233, in _run_test
    latest_dependency_updates = self._get_latest_dependency_updates()
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\vunit\ui\__init__.py", line 890, in _get_latest_dependency_updates
    source_files_in_order = self._dependency_graph.toposort()
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\vunit\dependency_graph.py", line 35, in toposort
    self._visit(
  File "C:\Python311\Lib\site-packages\vunit\dependency_graph.py", line 103, in _visit
    visit(node)
  File "C:\Python311\Lib\site-packages\vunit\dependency_graph.py", line 90, in visit
    visit(other_node)
  File "C:\Python311\Lib\site-packages\vunit\dependency_graph.py", line 90, in visit
    visit(other_node)
  File "C:\Python311\Lib\site-packages\vunit\dependency_graph.py", line 90, in visit
    visit(other_node)
  [Previous line repeated 2 more times]
  File "C:\Python311\Lib\site-packages\vunit\dependency_graph.py", line 83, in visit
    raise CircularDependencyException(path_ordered[start:] + [node])
vunit.dependency_graph.CircularDependencyException

This circular dependency is caused by the following construct:
tb_top instantiates testcase_top via component instantiation. testcase_top is split in a file for the enitity and multiple files with different architectures. There is one architecture per file with individual names (the testcases themself). In the architecture file is the needed configuration for the testcase_top instantiation aswell. This is the cause of the apparently false circular dependency. tb_top -> testcase_top -> tb_top.

In the provided example vhdl_configuration -> tb_selecting_test_runner_with_vhdl_configuration.vhd this behaviour does not occur. In this example the configuration is in a separate file (test_reset_behavioral.vhd) and the circular dependency error does not occur.
Sadly this is a worse solution due to the need to edit two files when adding a testcase.

What is the intended behaviour?

I have attached a MVP project displaying the described behaviour.

tb_dff.zip

Thank you for your help and Kind Regards,
~ Raik

[LarsAsplund]

You're right, that example is broken. The configurations can be placed in separate files or maybe in tb_top.

The starting point for VUnit is that there must be a simple and fast approach to unit testing with minimal boilerplate code. We still support that the test code is distributed over many files for more complex testbench scenarios but it will always come at a cost. For example, if the test code is separated from the DUT, there is a need for routing signals. Introducing configurations and components adds another layer of boilerplate and as described in https://vunit.github.io/blog/2023_08_26_vhdl_configurations.html, it doesn't scale well. I've also experienced problems in using configurations in a way that is portable between different simulators which adds to the problems. Personally, I avoid using configurations unless absolutely necessary.

A more capable parser would solve this but we like it simple to keep analysis time = unit testing overhead to a minimum.

It might also be possible to use a combination of the no_parse argument, see https://vunit.github.io/py/vunit.html#vunit.ui.VUnit.add_source_file, and manual dependency adding (https://vunit.github.io/py/vunit.html#vunit.ui.source.SourceFile.add_dependency_on) but I doubt it would be worth the effort.

[masubious]

Hi,

thank you for your reply. I agree with the general sentiments.

My point was that everything was working fine in VUnit 5.0.0.dev3. The PR#1114 (Commit 08b2a6d) seemingly broke that behaviour.
As far as i see it the cause of the cyclic dependency crash is this line:
latest_dependency_updates = self._get_latest_dependency_updates() in vunit/ui/__init__.py

While our approach might have some disadvantages it was the easiest approach to maintain.
Our largest VUnit Simulation has 141 testcases with multiple generic variants, resulting in 613 tests. While this may seem absurd for a unit testing scale, most testcases finish in less than 2 seconds. My personal approach is to test as far up in the design hierarchy as simulation speed allows. This level needs to be tested anyway to avoid module communication conflicts.
The following example illustrates the problem:

Module C containts modules A and B that communicate.
Module A is active low and unit tests for module A are all OK.
Module B is active high and unit tests for module B are all OK.
Without tests for module C I wouldn't see the resulting error when both modules try to communicate.

If I can sufficiently test Modules A and B through Module C everything is fine. This suffices for "single use modules" that are more for structure than re-use. If either Module A or B gets re-used elsewhere there should be a testbench for Module A or B :).

How we tried to structure our tests

The approach of containing all testcase code in the Testbench itself would result in a complete mess.
Thus the "sane" approach is to use different files. Our general approach is to have one testcase_top with minimal interfacing ( Communication to Verification Components is done via Packages) and have each testcase represent an architecture of testcase_top.
The previous approach was to use a generic g_testcase_id to choose which testcase gets loaded via direct instantiation. This resulted in the need to edit 4 different places when adding a new testcase - the testcase itself, the instantiation, the VUnit test list in tb_top and setting the generic in the run.py.
With the proposed solution of configurations being in a separate file / tb_top would still require us to edit multiple files when adding a testcase.
Configurations solved that problem. With some self-written additional code i can simply add a testcase file and VUnit discovers it automatically. I can also structure my testcases in categories (folders) and with some minor python code in the run.py apply the correct set of generics.

The no_parse Argument seems to work, when i apply it to the testcase files. But i don't know yet if it might break something else :)
Simulator support is adapted to ModelSim / QuestaSim, I don't see us switching to a different Simulator anytime soon.

[LarsAsplund]

Ok, I didn't understand that. Seems to be an indication of a bug that needs solving. I will have to dig further.

Having many test cases for a DUT is common so the need to split everything into manageable chunks comes naturally. My approach is to take small groups of related test cases and place them in testbench files of their own. If the infrastructure around the DUT, the test fixture, is small, I simply copy/paste. If not, I can place the test fixture in a reusable entity which is instantiated in the testbench. That way, I can avoid using configurations.

One case where I use configurations is when I have an instantiation of a Verilog IP, for example a CPU. To enable verification under simulators like GHDL and NVC, I need to remove the dependencies on Verilog. To do that I can create a mock in VHDL. For a CPU, it would typically contain a VC for the system bus as that can be used to simulate software interactions with the hardware.

[masubious]

Due to configuration support being in the dev version i would classify it as changed behaviour instead of a bug, but yes, i would prefer our way of using configurations still being supported :).

Okay, to see if I understood you correctly i drew a small image:

As you described

You would structure your tests this way. With different Testbenches that instantiate different Test Fixtures that containt testcases.
The Testbench can control which fixture gets used. The main Problem i see is during development, with frequent changes to modules, I'd have to edit code in multiple places and take a look if i forgot to edit it somewhere.

Our approach

Meanwhile our approach would resort to using a general Testbench (one Testbench for one DUT) and using different testcases while controlling via run.py general modifications in the testbench via generics. With this structure i have no code duplication. I'm free to run testcases in multiple variations aswell .. e.g. Testcase A could run with Generic Set A and Generic Set B.

Are my assumptions for the general test setup correct?

[LarsAsplund]

What I meant was that the test cases are at the top-level along with infrastructure unique to that test suite. The test fixture contains the reusable parts. Data generator, DUT, and data verifier in your example.