CI: add support for "custom" dependencies in the codebase? Case in question: how to best handle modbus extensions for NUT drivers?

[jimklimov]

I have a nagging feeling from the olden days, when many autotools projects came with contrib, custom or similar sources of third-party stuff like zlib, etc. so a single tarball could provide most of the build-required ecosystem in days before packaging... IIRC autotools were largely made because of such use-case, to consistently build such mixed codebases using either system-provided or custom-smuggled code.

Lately there are a couple of efforts relying on libmodbus features which are not (yet) in its mainline code and so packages - PR #2063 (for issue #139) new support for USB, and PR #1671 relies on support for ASCII vs. binary ModBus protocols. Neither is so far included into the upstream library project, and given how slow it is lately about even merging PRs or answering to issues/questions, I am not fully sure we should hold our fingers crossed and hopes to merge new drivers depending on such new features.

On the other hand, specifically for the modbus-ascii part of the question, there is some development simmering from a still-open PR stephane/libmodbus#275 from 2015, and a https://github.com/stephane/libmodbus/tree/ascii-support branch in the main project with last commits in 2022, so maybe "they" as domain experts had reasons (security? stability?..) to not merge this - and we would somehow compromise NUT installations by taking in "random-quality" code as a dependency?

WDYT: Would it make sense to expand NUT codebase to rely on potentially custom-built e.g. libmodbus, depending on configure-time tests whether the system packages provide the needed feature(s) already? There could be different ways to provide that - e.g. most likely a git submodule to our fork of libmodbus with added features (becoming a directory with a copy as part of make dist), but might technically be a full directory copy in the NUT sources, or a tarball to pull from location X, etc.

I suppose such extra source build would yield a libmodbus.so/a library file, which we may have to juggle in the installation footprint to not conflict with a system-provided one - neither to overwrite it, nor to auto-ldload into the same namespace by unsuspecting consumers; in this regard using a static library to build into the concerned drivers and/or nut-scanner (libnutscan?) may be a least-conflicting option for full-library custom builds (potentially irky for libnutscan and whatever its consumers might pull in - such as the system's libmodbus again).

Another alternative is to follow the path our PR #1671 took -- to provide the new abilities with source files in NUT codebase (modbus-ascii.{c,h}) which otherwise rely on standardly available libmodbus packages, thus with minimal long-term conflict against the OS-provided library. I guess this approach could be extended with configure-time checks to use either this implementation or one that hopefully eventually appears in the upstream library. It is not unlike some common/*.c files we have to provide string and other functions available on some platforms but not on others.

A year ago I was wary in that PR review about adding a custom semi-fork in NUT. Now that so little has changed in the library itself over the year, this in fact seems like a viable option. (There is still a question about making this secure somehow, to not introduce unreviewed errors on the HW/protocol support side...)

This partially overlaps with issue #1491 which is about general automation of such prerequisite builds (whether to facilitate NUT for Windows or other platforms with questionable support of pre-packaged build dependencies) but does not delve into customizing such dependencies' sources for NUT.

CC @EchterAgo @asperg @aquette @clepple

[clepple]

I haven't had time to follow this too closely, but when I looked at libmodbus a few years ago, the extra code needed to communicate with USB-based APC hardware was almost worthy of a fork (so as to clearly indicate the additional USB focus). I haven't looked at how hard it would be to make sure that the fork doesn't conflict with a system-provided serial-only libmodbus.

I think the more modern equivalent of a contrib directory is a separate Git repository for the dependency, with a tag and/or submodule reference.

[EchterAgo]

I wasn't aware we could add drivers to libmodbus like in #1671, I like that approach and I would have used it for #2063 if I had figured that out. Adding a modbus backend like that does rely on internals of libmodbus that could change at any time though.

Adding our own fork would avoid this and having our changes battle-tested would make a good case for upstream to accept changes. The lack of upstream updates and the stability of the protocol also make it attractive to use our own fork.

What we also have to consider is that libmodbus is LGPL licensed, so I'm not sure we can actually distribute binaries statically linked to libmodbus.

[aquette]

Static link with LGPL is not an issue here, since nut is gpl (code available), our derivative of libmodbus is also available and everything is provided to users to be able to recompile / relink and get the modifications made for nut.

The essence of LGPL is to ensure that the libs code gets the same attention as gpl (any mods must be contributed back!), but allowing for linking to proprietary code. As long as we fit, no issues. And we would fit here.

You may want to check https://copyleft.org/guide/comprehensive-gpl-guidech11.html#x14-10800010.8 and look for
“work based on the library.” around §10.5

Also fully aligned with our mod that would help testing and giving confidence to the upstream. Hopefully, our changes could be merged back, so that we don't have to maintain our fork in the long run. If that helps, I can contact the upstream, he's french 😁

[jimklimov]

FWIW, added https://github.com/networkupstools/libmodbus to at least bookmark the useful custom code involved in enhanced NUT custom builds...