nvc wants to write to standard libraries when invoked

[filmil]

I noticed that NVC wants to write to _NVC_LIB in the standard library directory when invoked.

This seems to be by design.

Unfortunately, this makes the use of NVC in sandboxed build environments, such as bazel, harder than necessary.
Namely, when NVC is invoked in a bazel sandbox, the standard library directory will be available on a read only file system. This means, NVC's attempt to lock the standard libraries will fail.

Is there a way to disable this behavior (by a setting), and at least only for standard libraries. Most of the time users don't actually need to add to them.

[nickg]

Is there a way I can easily reproduce this? I'd prefer to detect this situation and then skip the lock rather than never lock the standard libraries as that would cause problems when building them with make -j.

[nickg]

OK, it happens with any read-ony file system. I think this is a regression caused by the fix for #417 where we open the lock file in read-write mode to work around some NFS issues. We were checking for EACCES or EPERM and then falling back to read-only mode but not EROFS which is what you get if the file-system is read-only.

Could you test again either with the latest master branch or with the commit above applied to the release branch?

[filmil]

Thank you for the fix, I'll report back when I try it.

[...] that would cause problems when building them with make -j.

Hm, frankly the need to serialize writes to any library as it is built, seems like a bug in the build approach.
If make -j produces a broken library, then it's a problem with the Makefile not capturing all build dependencies.

But I understand if nvc is already set in its ways. I just note that it's an unusual approach to take.

[nickg]

The library maintains an index of design units, the lock file is needed to serialise updates to it.

[filmil]

The library maintains an index of design units, the lock file is needed to serialise updates to it.

Understood. What confuses me is, serializing updates on the index only works correctly if the compilation units are independent of each other. I suppose that works for the standard library since it's a bunch of independent standardized elements. I would have expected that if you wanted parallel compilation to work correctly, you'd have each compilation produce an independent output file set, then use another pass to tie all those outputs together into a library. Similar to how parallel C compilation produces a bunch of `.o` files then the linker ties them together into a library. This, of course, requires the build system to capture the dependencies correctly. That said, it's a very minor point in the grand scheme of things. The major point is that `nvc` is a great VHDL simulator that works today, and is very easy to compile. Thank you for creating it and supporting it over the years.

…

Message ID: ***@***.***>

[nickg]

Understood. What confuses me is, serializing updates on the index only works correctly if the compilation units are independent of each other. I suppose that works for the standard library since it's a bunch of independent standardized elements.

Sure, but the dependencies are handled separately by automatically generated makefile fragments, see lib/ieee.08/deps.mk for example. (These come from the --print-deps command BTW.) The problem with VHDL is that a single source file may contain multiple design units and there's no notion of a file being a single translation unit like C/C++, or any direct mapping between file name and design unit name (like Java).

Anyway the problem here is really nothing to do with the build system as such since you hit the same problem if you do nvc --work=foo a.vhd manually and at the same time nvc --work=foo b.vhd where a.vhd and b.vhd are completely independent. Without the lock file both processes race to update the index and one or the other update could be lost. I guess it would be a lot simpler to not use the index at all, but it does save a lot of time compared to opening each file to see what's in it (some of the Xilinx libraries have 1000s of design units in them).

[filmil]

Anyway the problem here is really nothing to do with the build system as such since you hit the same problem if you do nvc --work=foo a.vhd manually and at the same time nvc --work=foo b.vhd where a.vhd and b.vhd are completely independent. Without the lock file both processes race to update the index and one or the other update could be lost. I guess it would be a lot simpler to not use the index at all, but it does save a lot of time compared to opening each file to see what's in it (some of the Xilinx libraries have 1000s of design units in them).

Noted, this and the idiosyncrasies of VHDL compilation as well.

As a complete tangent, do consider building a library index separately from the build process as a more general approach for some future time.

This design decision makes bazel compilation harder than necessary. In bazel, you can't even guarantee that nvc --work=foo a.vhd and nvc --work=foo b.vhd would be compiled on the same machine if remote build ("RBE") is used. That both compilations expect to update a singleton library index makes it impossible to parallelize the compilation in this model, since there is no shared access to this singleton index: the two invocations are physically on two different computers.

I understand this approach to compilation is removed from most people's experience. Most don't use bazel, and even those who do mostly don't use RBE. I just wanted to illustrate the interesting issues introduced by this particular design decision.

I worked around the issue by forcing each library to be compiled as a unit, sacrificing the possible parallelism, and allowing for easy definition of many small libraries.
You might give this a thought for some future edition, if you are interested in shortening the compilation times of 1000s of units. It might be worth doing for large designs.

[nickg]

As a complete tangent, do consider building a library index separately from the build process as a more general approach for some future time.

This design decision makes bazel compilation harder than necessary. In bazel, you can't even guarantee that nvc --work=foo a.vhd and nvc --work=foo b.vhd would be compiled on the same machine if remote build ("RBE") is used. That both compilations expect to update a singleton library index makes it impossible to parallelize the compilation in this model, since there is no shared access to this singleton index: the two invocations are physically on two different computers.

OK I see, thanks for explaining. I'll have a think about it.