[vpi] Structure test case

[AndrewNolte]

Uses golden files to easily compare verilog symbol structure between simulators. Split off from #3756

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.70%. Comparing base (53a5f5e) to head (94cfe4d).
Report is 99 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3781      +/-   ##
==========================================
- Coverage   73.54%   72.70%   -0.84%     
==========================================
  Files          49       49              
  Lines        7916     8050     +134     
  Branches     2195     2208      +13     
==========================================
+ Hits         5822     5853      +31     
- Misses       1581     1682     +101     
- Partials      513      515       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[AndrewNolte]

I'd like to land this before putting back up the vpi iteration fixes for verilator, the last of those PRs is almost landed: verilator/verilator#4965

[AndrewNolte]

Any update on this? verilator/verilator#4965 was landed recently @ktbarrett

[ktbarrett]

I've been very busy with my move to Colorado. I probably won't have time to work on cocotb until next week at the earliest.

My question is why do we need to add a new test and source code for this? It does something g very similar to test_iteration and test_discovery (albeit in a lot better way IMO). This is just more code that's going to have to be refactored eventually when someone takes a cracked at replacing all discovery-type and value access tests with a more holistic approach.

That refactor was next on my radar after my current PRs are in.

[ktbarrett]

I guess I'm fine with this coming in if it's blocking your other work. Just know it will probably be rm -rf'd in the near future.

[AndrewNolte]

I agree, these tests try to achieve something similar. I initially just wanted to measure the progress on verilator vpi, so didn't thinking recreating this on sample_module or endian_swapper would be worth my time due to expected ci breaking of other tests.

I think we should move towards golden files for a lot of the cocotb tests for a few reasons:

• It makes it easier to write tests
• It expands the amount we're checking between simulators
• CI can automatically commit the golden output of a simulator that an individual user doesn't have access to

I think this provides a start for moving more of the structure / simulator differentiating tests to use golden files

[ktbarrett]

Questa failure.

#      0.00ns INFO     cocotb.regression                  running vpi_structure.test_structure (1/2)
#                                                             Tests that name, fullname, handle type, and length match across simulators.
# t: HierarchyObject                                 t
#   LONGSTART_a_very_long_name_which_will_get_hashed_a_very_long_name_which_will_get_hashed_a_very_long_name_which_will_get_hashed_a_very_long_name_which_will_get_hashed_LONGEND: LogicObject[1] t.LONGSTART_a_very_long_name_which_will_get_hashed_a_very_long_name_which_will_get_hashed_a_very_long_name_which_will_get_hashed_a_very_long_name_which_will_get_hashed_LONGEND
#   a: LogicObject[8]                                t.a
#   clk: LogicObject[1]                              t.clk
#   count: LogicObject[32]                           t.count
#   do_generate: LogicObject[32]                     t.do_generate
#   fourthreetwoone: ArrayObject[2]                  t.fourthreetwoone
#   half_count: LogicObject[32]                      t.half_count
#   long_int: LogicObject[64]                        t.long_int
#   onebit: LogicObject[1]                           t.onebit
#   quads: ArrayObject[2]                            t.quads
#   real1: RealObject[None]                          t.real1
#   status: IntegerObject[None]                      t.status
#   str1: StringObject[0]                            t.str1
#   text: LogicObject[512]                           t.text
#   text_byte: LogicObject[8]                        t.text_byte
#   text_half: LogicObject[16]                       t.text_half
#   text_long: LogicObject[64]                       t.text_long
#   text_word: LogicObject[32]                       t.text_word
#   twoone: LogicObject[2]                           t.twoone
#   x: LogicObject[8]                                t.x
#   arr: HierarchyArrayObject                        t.arr
#     arr[1]: HierarchyObject                        t.arr[1]
#       i: LogicObject[32]                           t.arr[1].i
#       arr: HierarchyObject                         t.arr[1].arr
#         LENGTH: LogicObject[32]                    t.arr[1].arr.LENGTH
#         check: LogicObject[1]                      t.arr[1].arr.check
#         rfr: LogicObject[1]                        t.arr[1].arr.rfr
#         sig: LogicObject[1]                        t.arr[1].arr.sig
#         verbose: LogicObject[1]                    t.arr[1].arr.verbose
#     arr[2]: HierarchyObject                        t.arr[2]
#       i: LogicObject[32]                           t.arr[2].i
#       arr: HierarchyObject                         t.arr[2].arr
#         LENGTH: LogicObject[32]                    t.arr[2].arr.LENGTH
#         check: LogicObject[1]                      t.arr[2].arr.check
#         rfr: LogicObject[2]                        t.arr[2].arr.rfr
#         sig: LogicObject[2]                        t.arr[2].arr.sig
#         verbose: LogicObject[1]                    t.arr[2].arr.verbose
#   bus1: HierarchyObject                            t.bus1
#     adr: LogicObject[32]                           t.bus1.adr
#     dat: LogicObject[32]                           t.bus1.dat
#     sel: IntegerObject[None]                       t.bus1.sel
#   cond_scope: HierarchyObject                      t.cond_scope
#     scoped_wire: LogicObject[32]                   t.cond_scope.scoped_wire
#     scoped_sub: HierarchyObject                    t.cond_scope.scoped_sub
#       redundant: LogicObject[1]                    t.cond_scope.scoped_sub.redundant
#       subsig1: LogicObject[1]                      t.cond_scope.scoped_sub.subsig1
#       subsig2: LogicObject[1]                      t.cond_scope.scoped_sub.subsig2
#     sub_wrap_gen: HierarchyObject                  t.cond_scope.sub_wrap_gen
#       gen_wrap: HierarchyObject                    t.cond_scope.sub_wrap_gen.gen_wrap
#         LENGTH: LogicObject[32]                    t.cond_scope.sub_wrap_gen.gen_wrap.LENGTH
#         gen_loop: HierarchyArrayObject             t.cond_scope.sub_wrap_gen.gen_wrap.gen_loop
#           gen_loop[0]: HierarchyObject             t.cond_scope.sub_wrap_gen.gen_wrap.gen_loop[0]
#             i: LogicObject[32]                     t.cond_scope.sub_wrap_gen.gen_wrap.gen_loop[0].i
#             wrapped_sub: HierarchyObject           t.cond_scope.sub_wrap_gen.gen_wrap.gen_loop[0].wrapped_sub
#               redundant: LogicObject[1]            t.cond_scope.sub_wrap_gen.gen_wrap.gen_loop[0].wrapped_sub.redundant
#               subsig1: LogicObject[1]              t.cond_scope.sub_wrap_gen.gen_wrap.gen_loop[0].wrapped_sub.subsig1
#               subsig2: LogicObject[1]              t.cond_scope.sub_wrap_gen.gen_wrap.gen_loop[0].wrapped_sub.subsig2
#       my_sub: HierarchyObject                      t.cond_scope.sub_wrap_gen.my_sub
#         redundant: LogicObject[1]                  t.cond_scope.sub_wrap_gen.my_sub.redundant
#         subsig1: LogicObject[1]                    t.cond_scope.sub_wrap_gen.my_sub.subsig1
#         subsig2: LogicObject[1]                    t.cond_scope.sub_wrap_gen.my_sub.subsig2
#   intf_arr[0]: HierarchyObject                     t.intf_arr[0]
#     addr: LogicObject[32]                          t.intf_arr[0].addr
#   intf_arr[1]: HierarchyObject                     t.intf_arr[1]
#     addr: LogicObject[32]                          t.intf_arr[1].addr
#   outer_scope: HierarchyArrayObject                t.outer_scope
#     outer_scope[1]: HierarchyObject                t.outer_scope[1]
#       i: LogicObject[32]                           t.outer_scope[1].i
#       scoped_param: LogicObject[32]                t.outer_scope[1].scoped_param
#       inner_scope: HierarchyArrayObject            t.outer_scope[1].inner_scope
#         inner_scope[1]: HierarchyObject            t.outer_scope[1].inner_scope[1]
#           j: LogicObject[32]                       t.outer_scope[1].inner_scope[1].j
#           scoped_param_inner: LogicObject[32]      t.outer_scope[1].inner_scope[1].scoped_param_inner
#           arr: HierarchyObject                     t.outer_scope[1].inner_scope[1].arr
#             LENGTH: LogicObject[32]                t.outer_scope[1].inner_scope[1].arr.LENGTH
#             check: LogicObject[1]                  t.outer_scope[1].inner_scope[1].arr.check
#             rfr: LogicObject[3]                    t.outer_scope[1].inner_scope[1].arr.rfr
#             sig: LogicObject[3]                    t.outer_scope[1].inner_scope[1].arr.sig
#             verbose: LogicObject[1]                t.outer_scope[1].inner_scope[1].arr.verbose
#         inner_scope[2]: HierarchyObject            t.outer_scope[1].inner_scope[2]
#           j: LogicObject[32]                       t.outer_scope[1].inner_scope[2].j
#           scoped_param_inner: LogicObject[32]      t.outer_scope[1].inner_scope[2].scoped_param_inner
#           arr: HierarchyObject                     t.outer_scope[1].inner_scope[2].arr
#             LENGTH: LogicObject[32]                t.outer_scope[1].inner_scope[2].arr.LENGTH
#             check: LogicObject[1]                  t.outer_scope[1].inner_scope[2].arr.check
#             rfr: LogicObject[3]                    t.outer_scope[1].inner_scope[2].arr.rfr
#             sig: LogicObject[3]                    t.outer_scope[1].inner_scope[2].arr.sig
#             verbose: LogicObject[1]                t.outer_scope[1].inner_scope[2].arr.verbose
#     outer_scope[2]: HierarchyObject                t.outer_scope[2]
#       i: LogicObject[32]                           t.outer_scope[2].i
#       scoped_param: LogicObject[32]                t.outer_scope[2].scoped_param
#       inner_scope: HierarchyArrayObject            t.outer_scope[2].inner_scope
#         inner_scope[1]: HierarchyObject            t.outer_scope[2].inner_scope[1]
#           j: LogicObject[32]                       t.outer_scope[2].inner_scope[1].j
#           scoped_param_inner: LogicObject[32]      t.outer_scope[2].inner_scope[1].scoped_param_inner
#           arr: HierarchyObject                     t.outer_scope[2].inner_scope[1].arr
#             LENGTH: LogicObject[32]                t.outer_scope[2].inner_scope[1].arr.LENGTH
#             check: LogicObject[1]                  t.outer_scope[2].inner_scope[1].arr.check
#             rfr: LogicObject[5]                    t.outer_scope[2].inner_scope[1].arr.rfr
#             sig: LogicObject[5]                    t.outer_scope[2].inner_scope[1].arr.sig
#             verbose: LogicObject[1]                t.outer_scope[2].inner_scope[1].arr.verbose
#         inner_scope[2]: HierarchyObject            t.outer_scope[2].inner_scope[2]
#           j: LogicObject[32]                       t.outer_scope[2].inner_scope[2].j
#           scoped_param_inner: LogicObject[32]      t.outer_scope[2].inner_scope[2].scoped_param_inner
#           arr: HierarchyObject                     t.outer_scope[2].inner_scope[2].arr
#             LENGTH: LogicObject[32]                t.outer_scope[2].inner_scope[2].arr.LENGTH
#             check: LogicObject[1]                  t.outer_scope[2].inner_scope[2].arr.check
#             rfr: LogicObject[5]                    t.outer_scope[2].inner_scope[2].arr.rfr
#             sig: LogicObject[5]                    t.outer_scope[2].inner_scope[2].arr.sig
#             verbose: LogicObject[1]                t.outer_scope[2].inner_scope[2].arr.verbose
#   sub: HierarchyObject                             t.sub
#     redundant: LogicObject[1]                      t.sub.redundant
#     subsig1: LogicObject[1]                        t.sub.subsig1
#     subsig2: LogicObject[1]                        t.sub.subsig2
#   sub_wrap: HierarchyObject                        t.sub_wrap
#     gen_wrap: HierarchyObject                      t.sub_wrap.gen_wrap
#       LENGTH: LogicObject[32]                      t.sub_wrap.gen_wrap.LENGTH
#       gen_loop: HierarchyArrayObject               t.sub_wrap.gen_wrap.gen_loop
#         gen_loop[0]: HierarchyObject               t.sub_wrap.gen_wrap.gen_loop[0]
#           i: LogicObject[32]                       t.sub_wrap.gen_wrap.gen_loop[0].i
#           wrapped_sub: HierarchyObject             t.sub_wrap.gen_wrap.gen_loop[0].wrapped_sub
#             redundant: LogicObject[1]              t.sub_wrap.gen_wrap.gen_loop[0].wrapped_sub.redundant
#             subsig1: LogicObject[1]                t.sub_wrap.gen_wrap.gen_loop[0].wrapped_sub.subsig1
#             subsig2: LogicObject[1]                t.sub_wrap.gen_wrap.gen_loop[0].wrapped_sub.subsig2
#     my_sub: HierarchyObject                        t.sub_wrap.my_sub
#       redundant: LogicObject[1]                    t.sub_wrap.my_sub.redundant
#       subsig1: LogicObject[1]                      t.sub_wrap.my_sub.subsig1
#       subsig2: LogicObject[1]                      t.sub_wrap.my_sub.subsig2
#      0.00ns WARNING  cocotb                             No expected output file found for modelsim. Pass plusarg +update to update output file
#      0.00ns INFO     cocotb.regression                  vpi_structure.test_structure passed
#      0.00ns INFO     cocotb.regression                  running vpi_structure.test_name_matches_iter (2/2)
#                                                             Test name accessibility and handle lengths.
#      0.00ns INFO     cocotb                             t
#      0.00ns INFO     cocotb.regression                  vpi_structure.test_name_matches_iter failed
#                                                         Traceback (most recent call last):
#                                                           File "/opt/actions-runner/_work/cocotb/cocotb/tests/test_cases/test_vpi_structure/vpi_structure.py", line 138, in test_name_matches_iter
#                                                             raise DirectLenMismatch(
#                                                         vpi_structure.DirectLenMismatch: len of direct object does not match iterated object direct_obj=HierarchyObject(t)[0], obj=HierarchyObject(t)[28]
#      0.00ns INFO     cocotb.regression                  **********************************************************************************************
#                                                         ** TEST                                  STATUS  SIM TIME (ns)  REAL TIME (s)  RATIO (ns/s) **
#                                                         **********************************************************************************************
#                                                         ** vpi_structure.test_structure           PASS           0.00           0.01          0.16  **
#                                                         ** vpi_structure.test_name_matches_iter   FAIL           0.00           0.00          1.33  **
#                                                         **********************************************************************************************
#                                                         ** TESTS=2 PASS=1 FAIL=1 SKIP=0                          0.00           0.02          0.12  **
#                                                         **********************************************************************************************

[ktbarrett]

So only GHDL and NVC work here. Interestingly, they are VHDL-only simulators, I wonder if there's some reason for that? EDIT: Big dumb. This is Verilog only. So ALL simulators fail this test.

Is there really any reason to test or enforce this? Why do we care if iteration handles and access by name handles being the same? Is there some reason to not just remove this test?

[AndrewNolte]

I feel like this is just a good way to test it, it would be very surprising if these weren't equivalent

[ktbarrett]

But would it matter?

[ghost]

@marlonjames @AndrewNolte @ktbarrett:
Will the addition of this PR ensure simple struct elaboration into LogicObject in python is working for each simulator? I currently use xcelium, which elaborates the fields of a simple (no nesting) SV struct into dut.<some struct port>.<some member>. It works as you would expect, and I can loop through all the attributes available in Python for a dut.<some struct port> and see all the members.

Verilator though, using cocotb latest main and verilator 5.022 is not elaborating the struct members.

@marlonjames is this expected? I had thought you fixed this in past PRs - in all your work in all the comments and PRs linked here: #1275

[ktbarrett]

@ian-l-kennedy

Will the addition of this PR ensure simple struct elaboration into LogicObject in python is working for each simulator?

No.

As stated in the issue you linked, this is a Verilator issue, and moreover a IEEE standards issue. Indexing into packed objects, structure or array, through the VPI is probably not standards compliant behavior (I'd have to find the exact section). Additionally, indexing into packed arrays doesn't work in any simulator (AFAIK), so any kind of complex packed data structure such as a packed array of packed structures can't be indexed into; only the very trivial case of a top-level packed struct.

#3608 attempts to allow the behavior in the other direction: getting the value of a top-level packed structure as a single value. Regardless of how that PR falls out, if you want to write portable standards compliant code, I'd stop relying on the ability to index into packed structures.

[ghost]

@ktbarrett,

Thanks for getting back to me.

If you follow all the links to understand the breadth of the work, you will see mentioned that @marlonjames also closed out issues in verilator/verilator. I was aware this would start as a verilator issue, then if resolved in verilator become a cocotb issue.

For example, there is confirmation of this here: #3237

Great work @marlonjames, thanks a lot for the many issue references and the work you did upstream with Verilator as well!

There is a wide net of linking from issue to issue, so I was having trouble finding the definitive answer to the question I had posted, so I thought to reach out.

I now see some more explicit evidence in the makefiles that verilator does not support struct constructs in the portmap/port signature.

# Verilator doesn't support structs (gh-1275) in this commit Verilator: Expect failures when accessing structs (see https://github.com/cocotb/cocotb/issues/1275)

@ktbarrett if you have a moment could you please point me to the standards that you are referencing here: Regardless of how that PR falls out, if you want to write portable standards compliant code, I'd stop relying on the ability to index into packed structures.. I am especially interested in removing SystemVerilog code constructs from my synthesizable code. Especially ones that are "against the grain" of any portability standard.

[ktbarrett]

@ian-l-kennedy You can simply remove the packed from the structure definition to treat them like regular structures. If you are still having issues, that's probably just an issue with Verilator not supporting something it should yet, or hitting some corner case. It happens. #1275 is about packed structs as you can see in the example in the OP.

[ghost]

@ktbarrett if you have a moment could you please point me to the standards that you are referencing here: Regardless of how that PR falls out, if you want to write portable standards compliant code, I'd stop relying on the ability to index into packed structures.. I am especially interested in removing SystemVerilog code constructs from my synthesizable code. Especially ones that are "against the grain" of any portability standard.

[ktbarrett]

Hmmm. So IEEE 1800-2017 section 37.18 states that we should be able to use vpiElement to dive into packed objects. We just aren't using vpiElement yet. Currently the few simulators that allow finding struct members are discovering them as a different kind of VPI object. I guess we need to detect packed objects and switch the kind of iteration we are doing.

I wonder how it's supposed to work with getting/setting values. The standard doesn't mention anything wrt to that =/

[ktbarrett]

I think this is a step in the right direction, but it'd be nice to add support for logging bounds, constness, etc. The idea being that it could eventually be used to generate typing stubs for DUTs.

[AndrewNolte]

First off I think this pr is getting pretty long and off topic, this pr is more about whether we want golden files and these types of test in the repo. This pr doesn't really matter for me anymore since I achieved the goal of Verilator vpi to match Xcelium, aside from packed structs. But this PR was instrumental in debugging and verifying that, and I think these golden file tests are especially useful given that not everyone has access to every simulator.

^ re: type generation, I don't think you'd want type bounds (low/high) since different params can change these, then you'd have to have a type for each parameterization. It'd best to keep these to just name: LogicObject etc. I can add bounds and constness to this PR, but would it make more sense to have that in LogicObject.__repr__ ?