-
Notifications
You must be signed in to change notification settings - Fork 258
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
UVM support: allow SV testbenches to specify tests in different approaches #328
Comments
I think it sounds like a good idea to add first-class support of UVM to VUnit . To make it work it would help a lot if an experienced UVM user like yourself provided us with a basic example test bench using UVM so we can start experimenting with integrating it into VUnit. It would be good if the test bench contained tests which should both pass and fail in various ways. Do you think you could contribute with that? We can use this issue to discuss pros and cons of different ways of integrating UVM. |
Ok. I have added a very basic example of a UVM testbench. Please extract the zip file and refer to the README.txt file inside it. The examples have been tested with Questasim 10.6c, but should work in previous versions as well. Please let me know in case of doubts/issues/comment Example uvm_tb_1 is a very Vunit-compliant format, where the testcases are declared within the tb_top module. Basically, the adaptations are done in the SystemVerilog world. The run.py is very similar to most Vunit python run scripts. Example uvm_tb_2 is one where I took the liberty of trying out different features for UVM and also regression/flow support. I did not change the SystemVerilog/UVM world, so I went a little wild with the python code in run.py file and got many of the fancy functionality working fine. I am a novice with Python, so please feel free to modularize it or make it more elegant/user-friendly. Both the examples can be run with 2 testcases - my_test1 (should pass always) and my_test_failing (should fail always). I have added a list of issues with description in ISSUES.txt PS: If you want to try in out with Modelsim (with a SystemVerilog HDL license), you can set SIMULATOR = "MODELSIM" instead of "QUESTASIM" in the run.py file |
Thanks this will make it easier for us. I will have a look. |
Are there any other somewhat established methods of exiting an UVM testbench? This is a bit related to #293
Is the parsing functionality part of UVM or is this something that the simulator vendors provide? Is this the only way to find out the status of a test run? Is there no way to programmatically ask UVM for the number of errors? |
Not that I know of. To prevent the $finish from being executed automatically, you can add this code before the run_test() code line in tb_top.sv. So, the control is going back to the Vunit SV runner. Still, I see this only as a work-around for development purposes. Btw, Vunit still returns a fail here, so this needs to be investigated.
Its not related to #293, which deals with recognizing known failures and flagging them to be seen as an OK instead of a FAIL.
UVM does not do any parsing of the transcript. It has an internal counter/registry which keeps track of all the UVM_<> messages issued within the testbench. Using that, its possible to query this within the UVM/SystemVerilog code towards the end (in a post-simulation check phase) in order to evaluate a PASS/FAIL for the test. This is left to the user BTW, I have two more UVM fundamentals in the list of differences in the first message. . Please have a look. |
Do you think it reasonable to add some VUnit specific stuff to an UVM test bench or do you think VUnit needs to run unmodified UVM test benches for it to be useful? Since it seems UVM does not standardize test pass/fail and leaves it up to the user it seems we should add a standardized mechanism for it. Regarding test case discovery it would be convenient if test case names can be extracted robustly via some simple parsing of the verilog files such that the test case names are known without running any simulation. This is to make for example the Regarding pass fail mechanism it is not good practise to just rely on the absence of errors to say a test passed. There must also be a completion message for a test to be considered a pass. In VUnit this means that any test that does not finish and call the cleanup code will fail. Is there something similar in the UVM world where the test must reach the final statement for a test to be deemed sucessful? |
@krishnan-gopal Your suggestions indicates that it should be possible to do something similar. The relation to #293 is that a UVM testbench would be an "expected failure" that must be handled in a special way if we can't do something similar and this is what I would like to avoid. |
@krishnan-gopal
What is the problem with listing the test inside the test bench in UVM? In this case they are always kept together with the file through which they are run. In VUnit this is parsed so that the test names are also programatically accessible in the Python-world. In this way typical simple test benches would need no Python-configuration.This can be made single source of information such as: // NOTE: Proposal only
import vunit_uvm_pkg::*;
`include "vunit_uvm_defines.svh"
`UVM_TEST_SUITE begin
`UVM_TEST_CASE(my_test1);
`UVM_TEST_CASE(my_test_failing)
end Here If desirable even the seed could be an argument to the macro if a specific seed is desired for the test but I guess. This could then be overridden in the Python-world or a test could be run for several values of the seed. |
Hi. Here are my thoughts on your comments. I have also tried to summarize it to answer your doubts.
In general, its to be noted that the HDL part (tb_top.sv) of SystemVerilog (which also contains the instance of the design) is not OOP-based. This mean that every user will have to write additional Vunit-specific code within the top module for every testbench used. I would prefer writing a single adapter/layer which can be re-used for all class-based OOP testbenches like UVM, and of course Python is the ideal candidate for that code. I am also a fan of having testbench-code independent of the flow/process in case the project decides to move to another tool. However, maybe its possible to put the Vunit-specific UVM-code into a seperate SystemVerilog module/class/scope keeping the rest of the TB code intact (for example using a second tb_top module). Let me try out some options
Each testcase runs in an individual simulation
I would like the --list to work too. Functionally, I would define a testcase as a combination of the test-class (the argument specified with +UVM_TESTNAME) and seed. Remember that the test-class is usually run over several seeds, sometimes thousands, and these seed values may not be known prior to launching the regression run. I would like Vunit to recognize a test-class+seed combination as a single testcase, so that I can get dedicated simulation results, debug waveforms, etc. There's no way I can pre-calculate all possible seed values for each testcase and specify them with compile-time HDL code. Is there someway I can declare only the test-classes in the SystemVerilog code (so the --list works for example) and then specify seed values dynamically during simulation time through the python code ?
There is no completion message explicitly needed in UVM. Its all handled implicitly deep within the base-classes. The testcase is only responsible for creating and sending the stimuli and does not usually check anything. The checking mechanism is done passively by a dedicated component(s) which monitor transactions and issue error messages. |
If you use
In VUnit we also have the concept of phases and it's also possible to raise an objection for the VHDL test runner although we call it locking a phase. However, in VHDL we will not reach the cleanup phase if something happens that stops the simulation. That's why the Python test runner considers an exit without passing the cleanup phase an error. What happens with a UVM testbench if you do something like dereferencing a null pointer? |
You mean that I use SystemVerilog-code to generate randomized seed values at compile-time ? How can I do that ? I thought that a configuration has to be fixed at compile-time in order to be recognized ?
Let me check this, and try to find an easier way. Maybe thats why checks are done externally by parsing the transcript. The report() phase should display a UVM_ERROR : 0 for a PASS which can be seen in the transcript. |
No, I meant configurations created in Python but I think I misunderstood you. Are you looking for the ability to specify a test case that will be listed as a single entry when doing |
exactly. But the simulation doesn't need to run n times. It only has to take the seed from the command line and treat it as a dedicated test case to run a single simulation. However in regression mode, each test case would be run over each specified/random seed, and that case the test report will list all those seeds. |
@krishnan-gopal |
I think in the example in uvm_tb_2 (thought its rather dirty python code) I have tried to add the basic necessities of running a UVM testbench. I dont think anything more is needed for a basic UVM testbench usage. There are a few things left to do:
|
@krishnan-gopal The VUnit |
@krishnan-gopal The philosophy in VUnit has always been to stop on errors and failures such that the test is failed due to not reaching the test runner cleanup. Maybe that is not feasible in SystemVerilog. |
Sorry, I think its --log-level then.
Exactly. I need to look into different options here. Maybe we can make it more Vunit-compatible or UVM-compatible, but I feel we should leave this to the user to decide. Any way to do that ? maybe using hooks/callbacks ? |
@krishnan-gopal |
@kraigher |
@krishnan-gopal It is really easy to add custom command line arguments (https://vunit.github.io/python_interface.html#adding-custom-command-line-arguments) to your run.py script if you want to have a CLI option to for example set a sim_option that controls the UVM_VERBOSITY. |
I wouldn't mind having a standard option to do verbosity control from the command line. Ideally that option would accept a fixed set of values but the problem is that VUnit supports the creation of custom levels in addition to the standard ones and our levels differs from the levels in UVM. Another issue is that VUnit has started to move away from the concept that log levels has an order. The reason is that our log levels contains error related levels (warning, error, failure) and non-error related levels (trace, debug, info). It's a bit awkward to think of orders when you're mixing different things. Errors are more severe than warnings but there is no severity order between info and debug. Debug contains more details than info but it's not clear if warnings is more detailed information than debug or vice versa. And what happens if you add a custom status level? We still support the concept of a verbosity level but the primary concept of verbosity control is that levels at shown or hidden on an individual basis. What is the UVM approach? Coming back to the discussion regarding exit strategy. Do you see a fundamental drawback with handing over the exit to the top level? Doing that doesn't prevent you from parsing the transcript in a post-hook and failing if errors are found. Did you test what happens when dereferencing a null pointer? I'm sure a transcript parser could find that but wouldn't it be simulator specific what error message to look for? |
Log-levels do have an order. In UVM all error, warning and fatal messages (and messages which dont use the UVM messaging methodology like $display) will be displayed out regardless of verbosity.
I am trying out different means of doing that without writing any additional invasive code, but I dont see a drawback of doing that either. I feel its a reasonable deviation from the regular UVM approach, but then I am a lone UVM user
I did. In such a case, the cleanup phases are not invoked, and the simulator exits with a simulator-specific error message. So, UVM is not involved here anymore. |
By removing warning, error and fatal from verbosity control UVM avoids the the ordering concern I mentioned. I guess you may end up with many warnings though. Makes me think about Vivado which can issue warnings for making optimizations which in most cases is exactly what I want it to do. They've solved it by limiting the number of warnings of the same type. A thought for the future maybe...
VUnit also support hierarchical logging so it similar to UVM in that respect.
From what you explain it seems like you would improve over the regular UVM approach by relying less on parsing to catch error messages which you might not have seen before. I understand your concern though and I've seen it before. Since VUnit is free and non-invasive it doesn't require a corporate decision to start using it. The normal case is that an individual starts using it, then it spreads to the rest of the team members and then the company. A common concern for these individuals (in addition to not having to make their testbenches too "weird looking") is how they can make use of VUnit, commit such code to the code repository without forcing others to learn about the Python test runner. The answer has been that the testbench must be "well-behaved" if you run it without the Python test runner. This is something you can do with a VUnit testbenches in VHDL but currently that's not possible with the SV testbench. What are your thoughts on this? |
In the end, the runner setup must be done one way or another. Its possible to encapsulate this additional SV code into a dedicated module, so that the rest of the testbench still looks the same. I have been trying out different approaches. Let me get back to you with a suggestion once I've figured it out. |
I am working now on making Vunit understand my errors suring the clean-up phase. The way I see it, errors are of 3 categories:
It seems that I can make Vunit's PASS/FAIL recognize category 3 only by parsing the transcript. Meaning, I have to do this in the Python-world. Do you have an example where this is done ? I see that the file test_output/<Some long name+HASH>/output.txt is the name assigned to the simulation log. I am looking for a way to access this from the Python-script. Is there some vendor-independent way I can get the pointer to this file within the Python-world ? |
The post_check hook takes an output_path argument which is the folder containing the output.txt file. You can read about it on the website. |
@kraigher |
It is because the output from the post check itself is also goes to the output.txt. Maybe we need to change that by moving the post check to an outer layer. |
I created a new issue for that #332 |
@kraigher , @LarsAsplund
|
@krishnan-gopal That would be inline with the default behavior for how we do it in the VHDL version. We support counting the different log levels, globally as well as on subsystem level, but the default is to stop on the first error or failure but continue on all other levels. For those wanting to continue on an error you can always extract the logs programmatically or maybe in your case parse a log file and then assert on those counters just before you end the simulation. This is how it's done when other non-VUnit error checking mechanisms for VHDL are used. |
@kraigher, @LarsAsplund |
By the way, one general question. Is there a specific purpose for using parameters in the Verilog/SystemVerilog test-configuration to pass arguments from the command-line ? |
@krishnan-gopal I am not against using plusargs instead of generics. In our current System Verilog test benches the parameter is a private implementation detail so it could be changed without any test bench modifications. The reason we used a parameter was that it would work the same as in VHDL and parts of the Python code can be unaware if it is a VHDL or Verilog test bench since the same mechanism is used. If we use plusargs we need to introduce a special handling of VHDL and Verilog test benches. |
@krishnan-gopal So what more help do you need to achieve a workable solution? |
@krishnan-gopal It is important to note there is loose coupling between the Python and HDL world in VUnit. Basically the Python part provides a string to the test bench containing all configuration in a dictionary like structure. The HDL parts writes the results to an output file which is specified by the configuration string. Once simulation is done the Python side reads the output file and determines if test cases where passed, failed or skipped. The format of the output file is:
All test pass if all tests start and test_suite_done is reached. A test is considered failed if it is the last test started and test_suite_done was not reached. A test is considered skipped if it was not started. |
The only change I had to do was in the vunit_pkg.sv in live 143. This was suggested by Lars.
|
@krishnan-gopal That was a bit of quick fix to get forward. However, the ability to run a test suite without test cases is a change that we need. |
@LarsAsplund Yes, I agree with that. |
@krishnan-gopal I think the minimal thing we need to add is "test less" SystemVerilog test benches. Then you can create your test cases in Python using add_config and set simulator flags with correct +UVM_TESTNAME |
@krishnan-gopal I added support for "test less" SystemVerilog test benches now. You can have a look at this example to see how you could use it: https://github.com/VUnit/vunit/tree/uvm/examples/verilog/uvm |
I just tried it out. It works great. Thanks! |
@kraigher, @LarsAsplund The problem with constrained-random testbenches is that its not possible to predict which testcase (test+seed combination) will generate a known error message. So, known error messages are rather testbench-specific and not testcase-specific in UVM. So, #293 doesnt help here. I was able to use the output file stream in my post_check function and parse it for UVM-specific as well as HDL-$error generated error patterns from an "error pattern" file. |
@krishnan-gopal You should set the sim_option (http://vunit.github.io/python_interface.html?highlight=sim_options#simulation-options) vhdl_assert_stop_level to failure. On modelsim this also affects $error in verilog causing simulation to stop and inhibit the test_suite_done criteria from being fulfilled. |
@krishnan-gopal Had any success with the vhdl_assert_stop_level sim option? |
Yes. It works as expected. Now I am able to have full control over error patterns and excluding them for a PASS/FAIL decision, and this is a more elegant alternative to #293 |
I am also working on a way to build testbenches for uVCs. This is another use-case where Vunit could help out a lot. I will come back to you once I have a good methodology/flow. |
@krishnan-gopal Can we close this issue for now and maybe open a new if your uVC experiments come up with new issues? The initial use case seems covered. |
I agree. We can close this issue.
|
Good. I think it looks clean, it sticks with the public APIs of UVM and it's idiomatic VUnit. |
UVM testbenches are widely used for verification, and Vunit could be used as a flow-controller for such methodologies, if some adaptations are made.
UVM testbenches have a top-level SystemVerilog module which has a slightly different style than a Vunit-based SystemVerilog testbench. Nevertheless, it would be nice to add the support for this style either using SystemVerilog or Python-based adaptations.
Here are the differences:
The text was updated successfully, but these errors were encountered: