vpi_wrapper_*.{ch} files... where? (and why?)

[rossc719g]

I'm a little confused about the correct location for vpi_wrapper_*.{ch} files when compiling using DBPI.

It seems like bsc will place them in the cwd, unless I specify -vdir, which is odd, because the docs say that -vdir defaults to the location of the source file being compiled. This seems to be true for all other files governed by -vdir, but not these?

And then when linking, it seems like they all need to be in the same directory where the output file is being created, even if that is not the directory that contained the code that generated them.

E.g,
If I have this structure below my cwd:

foo/File1.bsv
bar/File2.bsv
baz/

And both files contain BDPI imports, then without -vdir, both sets of vpi_wrapper_ files will end up in the cwd.
If I do specify the -vdir to be the location of each source file when I compile them, then they end up in their respective directories.
If I then try to link them into an executable in the baz directory (with the -p path including both foo and bar), then I first need to move or copy them into the baz dir before linking.

It seems like if you're linking into a non-cwd dir that does not contain your BDPI-containing source file, then there is no way to avoid needing the mv/cp.

Similarly, when linking using VPI, bsc creates a directc_*.so file that always seems to end up in the cwd, even if the executable is being built elsewhere. If I do not move it to the same place as the executable after linking, then it wont be able to find it at runtime.

The reason I am asking is that I'm running bsc from bazel, and bazel is very persnickety about knowing exactly where each file is, so I need to know all of these behaviors, and explain them to bazel. What I would normally do on the command line is just find the files and put them where they need to go manually, but I can't do that here.

Anyway, I'm wondering if there is something I am missing about how these files behave that makes it all make sense?

[quark17]

It seems like bsc will place them in the cwd, unless I specify -vdir, which is odd, because the docs say that -vdir defaults to the location of the source file being compiled. This seems to be true for all other files governed by -vdir, but not these?

I see what you mean. The issue is at lines 465-6 in src/comp/bsc.hs:

then genDPIWrappers errh flags "./" blurb ffuncs
else genVPIWrappers errh flags "./" blurb ffuncs

Where it has "./" (the current directory), that should be replaced with prefix (the directory part of the source file name being compiled). It appears that in 2013 someone changed it, and I don't know what the motivation was. I agree that it's inconsistent and should be changed back, but I'll see what other people think.

And then when linking, it seems like they all need to be in the same directory where the output file is being created, even if that is not the directory that contained the code that generated them.

I don't know what your linking process has for requirements, because you haven't said what you're doing for linking. Are you using BSC to perform the linking step or are you directly executing a downstream tool? In either case, it would help to see the command-line flags you are using for compile and link.

If you are using BSC to link, make sure to provide the -elab flag when doing the BSC compile step. For example:

bsc -v \
   -verilog \
   -elab \
   -p bsv_src:+ \
   bsv_src/Foo.bsv

This flag will cause BSC to generate .ba files for each module. This is a database of information that BSC's linking step can use to reconstruct the entire module hierarchy, including knowing which imported C functions are called. With .ba files, the linking step can be just this:

bsc \
   -verilog \
   -p bsv_src:+ \
   -e mkFoo \
   <.c/.o/.so files>

There is no mention of the .ba files for the imported C functions because BSC has worked out which ones are needed and where to find them. If you didn't provide the -elab flag to generate .ba files for the entire design, then you will need to manually tell BSC which foreign functions to include in the simulation:

bsc \
   -verilog \
   -p bsv_src:+ \
   -e mkFoo \
   bsv_src/my_func.ba \
   <.c/.o/.a files>

If I then try to link them into an executable in the baz directory (with the -p path including both foo and bar), then I first need to move or copy them into the baz dir before linking.

It seems like if you're linking into a non-cwd dir that does not contain your BDPI-containing source file, then there is no way to avoid needing the mv/cp.

This confuses me, because you shouldn't need to move anything. And nothing needs to be in the same directory as the source files for the imported functions -- if you're linking with BSC, you just provide the files on the command line. But maybe it's because I don't understand what "linking into a non-cwd dir" means. Can you show the command lines?

I have written a test that uses BSC to compile and link in multiple configurations (with and without vdir, etc) and it works for me. The source file is in a subdirectory (as shown in the commands above), but I am calling BSC from the same directory in both the compile and link steps. The C file is also in another directory, separate from everything else. (In my case, I am providing the .c file for the imported function, on BSC's command line, but note that you can also precompile into .o or whatever, and provide that; but if you provide .c, BSC will compile it for you, which can be easier.)

[rossc719g]

Thanks! Replies inline.

I see what you mean. The issue is at lines 465-6 in src/comp/bsc.hs ... I agree that it's inconsistent and should be changed back, but I'll see what other people think.

Excellent. Thanks. Is there an issue open for this now?

I don't know what your linking process has for requirements, because you haven't said what you're doing for linking. Are you using BSC to perform the linking step or are you directly executing a downstream tool? In either case, it would help to see the command-line flags you are using for compile and link.

I am using bsc to link. Sorry about not giving a more complete example to start with. I've put together some examples here:

https://github.com/rossc719g/bsc_examples

But also have some example command lines below.

If you are using BSC to link, make sure to provide the -elab flag when doing the BSC compile step.

I have not used the -elab flag. To be honest, I am not sure I fully understand what a .ba file contains exactly. I always just thought of it as the counterpart to a .v file when compiling in -sim mode, but it also seems to pop up in other places. I've never fully understood it though. Even .bo files are not entirely obvious to me. *See my last comment below.

But, creating a .ba for every module in my design might be tricky for me, because bazel needs to know about every consumed and produced file at every step, before executing the tools, and without inspecting the source. So, I'd need to enumerate every module in the BUILD file which compiles it. Not impossible, but certainly not ideal.

If you didn't provide the -elab flag to generate .ba files for the entire design, then you will need to manually tell bsc which foreign functions to include in the simulation

This is what I am currently doing. It is creating the .ba files during the initial compilation, and I provide them during linking explicitly. Whenever I need to synthesize a module, I mention it in the BUILD file, and it passes a -g flag to bsc to generate the .ba (or .v, depending on wether it is building in -sim or -verilog mode). Similarly, when using BDPI, I also mention that in the BUILD, and it will collect the resulting .ba (and other files) for linking later. It is a lot easier to only mention the modules that I am synthesizing, or importing, rather than every module in the design.

This confuses me, because you shouldn't need to move anything.

Then I am probably doing something wrong. The example repo I mentioned above will probably help. Sorry I did not take the time to make it initially.

Can you show the command lines?

The example scripts in my repo all print the command lines when they run, and the README.md does a better job of explaining it all, but briefly, given this example source tree:

beef/Beef.bsv
beef/beef.c
beef/test/BeefTest.bs
cafe/Cafe.bsv
cafe/cafe.c
cafe/test/CafeTest.bs
test/BeefCafeTest.bs

Beef.bsv and Cafe.bsv both have BDPI imports, and the *Test.bs files use them.

Here are the command lines that show what I mean.

bsc -verilog -p %/Libraries -bdir ./beef -simdir ./beef -vdir ./beef -info-dir ./beef ./beef/Beef.bsv
bsc -verilog -p %/Libraries -bdir ./cafe -simdir ./cafe -vdir ./cafe -info-dir ./cafe ./cafe/Cafe.bsv
bsc -verilog -p %/Libraries:./beef:./cafe -bdir ./test -simdir ./test -vdir ./test -info-dir ./test -g mkBeefCafeTest ./test/BeefCafeTest.bs
bsc -verilog -p %/Libraries:./beef:./cafe -bdir ./test -simdir ./test -vdir ./test -info-dir ./test -o ./test/BeefCafeTest.exe -e mkBeefCafeTest ./test/mkBeefCafeTest.v ./beef/beef.c ./beef/beef.ba ./cafe/cafe.c ./cafe/cafe.ba

That last linking step fails with:

VPI registration array file created: ./test/vpi_startup_array.c
User object created: ./beef/beef.o
User object created: ./cafe/cafe.o
clang: error: no such file or directory: './test/vpi_wrapper_beef.c'
clang: error: no input files

Since the vpi_wrapper_*.{c,h} files are actually in ./beef/ and ./cafe/, not ./test/ where it is looking.

Again, the repo does a better job of explaining my issues.

And, regarding my confusion of .ba and .bo files, I have discovered (by just looking at file sizes) that the content of a .bo file apparently changes if you add -g flags on the command line or not... but I don't think -sim vs -verilog make a difference. (right?) So, When I have a package with no synthesized modules, or BDPI or VPI imports, I am just compiling without -sim or -verilog, and using the resulting .bo for both bluesim and verilog linking later. (Is that going to bite me later?) What about .ba files? Does -sim vs -verilog change their content? I assume it would?

I am considering completely segregating all my -sim generated files from my -verilog generated files even when they have the same names... but then I would end up possibly compiling the same source multiple times.

I hope I am explaining myself better this time! And thanks again!

[quark17]

Excellent. Thanks. Is there an issue open for this now?

I've submited PR #576, which I'll merge soon. (Unless the following issue requires rethinking where VPI files get written.)

Since the vpi_wrapper_*.{c,h} files are actually in ./beef/ and ./cafe/, not ./test/ where it is looking.

It's looking in test because you said the vdir is test. For linking, vdir is more an input directory than an output directory, as it's assumed to be the vdir you used when you compiled the files (and is therefore the first directory in the Verilog search path). We've never anticipated that someone would run BSC multiple times with different vdir.

We do anticipate that someone might compile files as a library, that they then provide to others. (In a way, you're compiling libraries, that you then use yourself.) If someone makes a library containing imported C functions, it will have VPI wrapper functions that need to be handed off, too. It seems like there's not a good way for that to happen.

So probably BSC's linking step needs a better way to find the wrapper files than by using a single vdir. I can see three approaches:

• Instead of a predicted location based on vdir, the wrappers should be found in a predictable location based on something else that is provided on the command-line -- for example, relative to the location of the .ba file for the imported function.
• Don't predict a location, but instead walk a search path looking for the wrapper files (and, if not found, BSC can provide a helpful error message to the user)
• Don't go looking for the wrapper files at, and just require the user to specify them on the BSC command line (or otherwise make sure that it's compiled into whatever C object files are provided on the BSC command line)

I don't think the third option is reasonable. And I think the first option would require placing the wrappers in the same directory with the .ba file, which is probably also too restrictive. So we should probably go with the search path option. Then the question is which search path(s) to use (and whether any new path arguments need to be added to BSC).

Since the BSC compile places the wrapper with the Verilog, it would make sense to use the Verilog search path. If compilation has been done into several different vdir, then typically those directories would be specified as a Verilog search path to the linking step, with -vsearch. (I guess someone could manually list all the Verilog files on the BSC command-line, but usually one just specifies paths and lets the Verilog tool find the source code for modules by searching. And, ah, I see that's what you do: in example_B_bad_beef_test.sh, instead of adding ./beef/test to the vsearch path, you just list ./beef/test/mkBeefTest.v on the command-line.)

If adding -vsearch ./beef/test:+ to your link command fixed the problem, would that satisfy you?

And, regarding my confusion of .ba and .bo files, I have discovered (by just looking at file sizes) that the content of a .bo file apparently changes if you add -g flags on the command line or not... but I don't think -sim vs -verilog make a difference. (right?) So, When I have a package with no synthesized modules, or BDPI or VPI imports, I am just compiling without -sim or -verilog, and using the resulting .bo for both bluesim and verilog linking later. (Is that going to bite me later?) What about .ba files? Does -sim vs -verilog change their content? I assume it would?

Let me first say that I avoid using -g. I much prefer to put the synthesis directive in the source file. (In BH, that's done with an attributes pragma, which is a little more wordy than the attribute that's used in BSV.) If you have multiple synthesized modules in one source file, that will result in multiple compiled output modules, and I don't know if that's a problem for how you're compiling separately with Bezel?

Also, the src/comp/ directory has tools called dumpba and dumpbo, which you can build and run, to see what's inside the files. I don't know how understandable the output is, but if you're curious you can have a look. I'll also explain:

The .bo file represents the result of compiling a file. When you compile a file that has an import statement, BSC reads the .bo for that import, to pull in all the definitions (types, functions, modules) that were in that file. If you compile Foo.bs, you will get one Foo.bo.

The .ba file represents the result of synthesizing a module. If Foo.bs contains synthesized modules mkA and mkB, then you will get outputs mkA.ba and mkB.ba from compiling Foo.bs with a backend.

If module mkA is synthesized, that is visible in the Foo.bo file that is generated. Specifically: if you compile without a backend or without specifying that mkA should be synthesized, then Foo.bo contains a definition for mkA which is just the raw source code -- that will be inlined into a module in a later file that instantiates mkA; but if mkA is synthesized, then in Foo.bo the definition for mkA is just a mention of the synthesized module (and the boundary info, like clocks and ports and scheduling, that was figured out during synthesis). This would explain why Foo.bo doesn't differ if you don't use -g. And, yes, it's fine to compile with no backend, and reuse the .bo file for either Verilog or Bluesim.

You can even reuse .ba files for both Verilog or Bluesim. The one exception is if the source for the module used a function like genVerilog (which returns True if Verilog backend is in use), to generate different code for Verilog vs Bluesim. In that case, the .ba file encodes that it is specific to one backend, and BSC will give you an error if you try to use it for another backend. (Most code doesn't use genVerilog or genC, so this shouldn't come up often.)

So if you want to avoid synthesizing a module twice, you can compile with -verilog and include the -elab flag to generate .ba files, which can then be used for Bluesim linking without a second BSC compile step.

Each Verilog module has its own scheduler, and they work in parallel with each other by using the EN and RDY ports, to schedule a whole design. As a result, BSC can generate Verilog modules separately. Bluesim works by gathering up the entire design and making one global scheduler, that tells the rules in each module when to execute. Bluesim, therefore, needs to see the entire design hierarchy. The .ba file contains the information about the module, including what submodules it instantiates. So by providing just the .ba file for the top-level module, BSC can see what submodules it instantiates and automatically determine the .ba files that are needed and go find those files -- which is why you don't need to specify all the .ba files on the command line (or even the top .ba file, since that can be figured out from the name of the top module). BSC's Verilog linking can also benefit from seeing the entire hierarchy: by walking the .ba files, BSC can see which imported C functions are used, and automatically load those .ba files, without them needing the to be specified explicitly on the command line. And I guess that brings up a point: a .ba file can be one of two types, either a module or a foreign function.

BSC can even take a .ba file generate for Bluesim and produce the Verilog files from it, if necessary, for Verilog linking. If you compiled a design for Bluesim, I think you can link for Verilog and BSC will generate the missing Verilog files -- or at least in theory it's possible, I forget if it's actually implemented. There is one small difference between .ba files generated for Verilog vs those generated for Bluesim: when Verilog is generated, BSC computes properties of the ports, such as whether they are registers, or constant, or unused. This info is then recorded in the .ba file. For Bluesim, this info isn't available, so it's not in the .ba file. For that reason, it's better to generate .ba while synthesizing for Verilog, if you plan to use it for both.

[rossc719g]

I've submited PR #576, which I'll merge soon. (Unless the following issue requires rethinking where VPI files get written.)

Great, sounds good.

If adding -vsearch ./beef/test:+ to your link command fixed the problem, would that satisfy you?

Absolutely. I agree with your comments 100%. It feels like -vdir and -vsearch already handle this exact problem for the .v files, and since the wrappers (should) end up in -vdir, it makes perfect sense to piggy-back on that.

And, regarding my confusion of .ba and .bo files, ...

Let me first say that I avoid using -g. I much prefer to put the synthesis directive in the source file.

Until recently, I did this as well, and I agree it is a bit cleaner. The reason I stopped was just because bazel needs me to enumerate the files that get produced, and I much prefer to have a single source of truth whenever possible. Using -g allows me to only put that information in one place: the BUILD file. Even with multiple modules in one file, I can just pas multiple -g flags. It also allows me to do a very fast compilation test by compiling without a backend, and no -g flags, which is nice when modifying flies that take multiple minutes normally.

So if you want to avoid synthesizing a module twice, you can compile with -verilog and include the -elab flag to generate .ba files, which can then be used for Bluesim linking without a second BSC compile step.

This is interesting... but from your other comments I understood that this will generate a .ba for every module, not just the synthesized ones, right? I.e., if only a subset are synthesized, I might end up with more .ba files than .v files? Would it be harmful to the later steps to just discard the ones that do not have a corresponding .v? It feels a tad wasteful to spend the cpu to generate them only to discard them.. but I am not sure if generating an unneeded .ba might be expensive or not. My designs are not small, and I often need to consider compilation time as a factor.

For that reason, it's better to generate .ba while synthesizing for Verilog, if you plan to use it for both.

Makes sense. That matches well with my setup. I am already using the -verilog run to generate the .bo files, and discarding the .bo from the -sim run. If I used the .ba files from the -verilog run as well, I could skip the -sim run entirely (until linking). Although... I do use genVerilog & genC in my code in a few places (Thanks for pointing out that caveat). Maybe I should just not poke this particular bear. :-)

And thanks for the .ba and .bo overview explanation. That helped me a lot.

Did you see the third issue (C) in my example repo about the directc_*.so files? Assuming that the changes for -vdir and -vsearch are adopted, the directc_*.so files are the only thing left where I feel like I need to do any file-shuffling sleight of hand.

[quark17]

This is interesting... but from your other comments I understood that this will generate a .ba for every module, not just the synthesized ones, right? I.e., if only a subset are synthesized, I might end up with more .ba files than .v files?

No. You only get .ba files for synthesized modules. There are exactly as many .ba files as .v files.

The .ba file just captures a snapshot of the state of BSC after it has elaborated a module. When you generate Verilog, BSC is already reaching this point, it's just not dumping a snapshot by default, on its way to further crunching and generating a .v file. Providing the -elab flag just tells BSC to dump the snapshot; it creates no additional work besides the writing of the file.

Did you see the third issue (C) in my example repo about the directc_*.so files?

Oh, I did not. Thanks for pointing that out.

In one case, you are running this command:

bsc \
  -verilog \
  -p %/Libraries:./beef \
  -bdir ./beef/test \
  -vdir ./beef/test \
  -info-dir ./beef/test \
  -vsearch beef:+ \
  -o ./beef/test/BeefTest.exe \
  -e mkBeefTest \
  ./beef/beef.c \
  ./beef/beef.ba

(I've added the -vsearch and removed the explicit mention of mkBeefTest.v, since that can be found in the search path. And also removed the simdir.)

The only thing that makes the executable be placed in a directory is that the filename given to -o has a directory component. You're asking that the generated directc_mkBeefTest.o be placed in that same directory -- which can only be gleaned by taking the directory of the -o argument, yes? That's the behavior that BSC has for bsc -sim -o, where the generated so is placed alongside the executable script. So I think that's fair to do for Verilog. However, in the Bluesim case, the so is just a suffix onto the same name as the executable, and that's not the case here (unless we changed it). But I think that's fair. And it's doable, but it requires a change in each separate link script for the different supported simulators (since each one compiles that object as needed for the specific simulator).

Separately, are you concerned about work files that are generated by the Verilog simulator linking? For different simulators, there may be additional files that are placed in the current directory (where BSC is invoked), which are not needed after the executable is created. For VPI, different simulators have different requirements for files that need to be provided during linker. Older versions of Icarus Verilog need a .sft file, which gets placed in the current directory. Regardless of VPI, Verilator and Modelsim create work directories. I've recently updated the Verilator link script to delete the work directory -- we could in theory do the same with all other generated files (like .sft).

[quark17]

I should add that issue C is a little more complicated: Right now, the generated BeefTest.exe executable has to be run from a directory containing directc_mkBeefTest.so. You're changing into the directory beef/test/ where the executable is found; moving the shared object file to that directory will make your use work. But if you change to any other directory and tried to run .../beef/test/BeefTest.exe (using an appropriate relative or absolute path), that wouldn't work, because the shared object isn't in the directory you are invoking from. For that to work, the BeefTest.exe executable would need to know to look for the object in its own directory. Making that work probably depends on whether each specific simulator supports flags for making that happen. (A few things I've tried with Icarus Verilog on macos haven't worked yet.)

[rossc719g]

No. You only get .ba files for synthesized modules. There are exactly as many .ba files as .v files.

Ah, I misunderstood. Then, it sounds like, except for the genC/genVerilog thing, maybe I could get away with a single compilation. I'll think about how I might be able to get rid of those...

You're asking that the generated directc_mkBeefTest.o be placed in that same directory -- which can only be gleaned by taking the directory of the -o argument, yes?

Yep. That does make the most sense to me.
Like all of things in this thread, they're easy enough to work around if you know about them. They just feel odd or inconsistent or were surprising to me when I discovered them.

Separately, are you concerned about work files that are generated by the Verilog simulator linking?

As long as they are not needed at runtime and I can ignore them, then they do not cause any issues for my workflow since bazel will ignore anything that you do not explicitly mention as a product of a rule, regardless of what directory they end up in. They just get quietly swept under the rug. That's currently happening with a lot of .o files, and things like vpi_startup_array.c, etc. Unless I manually run bsc for debugging the toolchain, I never even see them.

Personally, I think the least surprising thing would be for these files to be created in a temp dir somewhere (e.g., mktemp -d), and then delete the entire dir if/when the linking succeeds. The second-least surprising would be to put them in the same place as the executable. The cwd would be third on my list. Any other locations would be very strange indeed. But that's all just my personal preferences, and you should give them no weight, unless you share the same opinions. ;-)

That said... if they are needed at runtime, then I feel like they should be next to the executable, and that the executable should then look for them there. Otherwise, your cwd must always be the same as the dir that contains the executable, or you need to copy files or make symlinks before running it. Doable... but awkward.

I should add that issue C is a little more complicated...

Yeah, your comments sum it up perfectly. It is a bit of an awkward thing to deal with.
I am certainly able to work around it by copying files or making symlinks, but I must admit that the comments for the code in my build system that implement the workarounds are not "happy" comments, if you know what I mean. :-)

Assuming these .so files could not be statically linked into the executable, the trick I have used in the past is to wrap an executable in something like a .shar or .par file which bundles the real executable and everything needed at runtime into a single wrapper executable. When executed, it will will extract the needed files, including the real executable, into the right places (ideally into a temp dir), execute the real executable (without changing the cwd), and then (if desired) clean up afterwards. As long as it takes care to never overwrite a file that already exists, nobody needs to worry about runtime files at all, and it should be applicable to any simulator. Just a thought.

[rossc719g]

Ah, I misunderstood. Then, it sounds like, except for the genC/genVerilog thing, maybe I could get away with a single compilation. I'll think about how I might be able to get rid of those...

Just to circle back (and to thank you again!)...
I switched to using a single invocation of bsc to generate both the .ba and .v files using -verilog -elab (except for files which make use of genC / genVerilog), and it is working great. Less duplicated work... much faster build times. Yay!

Initially, I was concerned about subtle consequences of forgetting to add those exceptions for cases where I use genC / genVerilog, but it turns out, that I needn't have worried. After being extremely careful to mark all my exceptions, I went ahead and un-marked one, just to see what a mistake would look like. And, it seems like bsc is smarter than I expected. I get errors like this:

Error: Unknown position: (S0042)
  Backend mismatch for file `whatever.ba'. The file was compiled for the
  Verilog backend, while the current compilation is for the Bluesim backend.

But, as you said, for all cases where I don't use genC / genVerilog it works without complaint.

So, it seems to know the difference between a -verilog -elab generated .ba which is safe to use, and one which is not. So, I can't really get it wrong! Which is to say, it is doing exactly the perfect thing in all cases and the world is filled with joy and love. :-)

Again, truly.... Thanks for the tip!