gnutls: bump to 3.4.14 + 3.5.2, drop lib*.la, add TEST().

[fbrosson]

• Bump to 3.4.14 (current stable) and add 3.5.2, marked untested.
• Fix paths of dependency_libs in develop/lib/lib*.la to make sure they will still be valid even if these dependencies are updated.
• Move $binDir to a _bin sub-package, for more flexibility with regards to package co-installation. 3.4.x and 3.5.x seem to be causing problems for other recipes, so this layout change, with some other changes (e.g. package basename renames) can help.
• Update the layout of 2.8.x and add x86 and x86_64 as untested.
• Change the basename of the recipe of 3.4.x to gnutls34.
• Use gnutls35 as basename for the recipe of 3.5.x.
• Declare CONFLICTS, CONFLICTS_{bin,devel} in a way that allows gnutls-2.8.x to be installed with either gnutls-3.4.x or gnutls-3.5.x. On top of that, a single _bin package may be installed (provided the base package is also installed.) Same thing for _devel.
• Add cmd:{cvs,git} to BUILD_PREREQUIRES in order to make gl/tests/test-vc-list-files-{cvs,git} succeed.
• Patch gl/tests/test-float.c to enable that test.
• Add cmd:{perl,makeinfo} to BUILD_PREREQUIRES.

[fbrosson]

When I run haikuports -S gnutls_x86-3.4.13 I do get valid and working packages that I can install and use.
And when I run haikuports --test gnutls_x86-3.4.13 it starts building some stuff and then runs the tests in /sources/gnutls-3.4.13/gl/tests:

============================================================================
Testsuite summary for GnuTLS 3.4.13
============================================================================
# TOTAL: 60
# PASS:  57
# SKIP:  3
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[7]: Leaving directory '/sources/gnutls-3.4.13/gl/tests'
make[6]: Leaving directory '/sources/gnutls-3.4.13/gl/tests'

For the record, the 3 tests that got skipped are:

SKIP: test-float
SKIP: test-vc-list-files-git.sh
SKIP: test-vc-list-files-cvs.sh

But it then tries to build more stuff in another directory, /sources/gnutls-3.4.13/tests, and fails with:

  CC       resume.o
resume.c: In function 'hsk_hook_cb':
resume.c:151:6: warning: implicit declaration of function 'memmem' [-Wimplicit-function-declaration]
  if (memmem(msg.data, msg.size, "\x00\x17\x00\x00", 4) == 0) {
      ^
resume.c:151:2: warning: nested extern declaration of 'memmem' [-Wnested-externs]
  if (memmem(msg.data, msg.size, "\x00\x17\x00\x00", 4) == 0) {
  ^
  CCLD     resume
resume.o: In function `hsk_hook_cb':
/sources/gnutls-3.4.13/tests/resume.c:151: undefined reference to `memmem'
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:3176: resume] Error 1
make[3]: Leaving directory '/sources/gnutls-3.4.13/tests'
make[2]: *** [Makefile:4767: check-am] Error 2
make[2]: Leaving directory '/sources/gnutls-3.4.13/tests'
make[1]: *** [Makefile:3511: check-recursive] Error 1
make[1]: Leaving directory '/sources/gnutls-3.4.13/tests'
make: *** [Makefile:1408: check-recursive] Error 1
Command '['bash', '-c', '. /wrapper-script']' returned non-zero exit status 2

Does someone have an idea of how the undefined reference to memmem can be fixed? I tried building with cmd:gnulib_tools in BUILD_PREREQUIRES but it didn't change anything, so I commented it out.
Is there something wrong in the patch I added ?
Since 3.4.10, 3.4.11, 3.4.12 also produce the same error (or even before if I don't include any patch), I think it would be worth merging. What do you think?
BTW, shall I remove the cmd:gnulib_tools (which is already commented out) ?
Thanks!

[waddlesplash]

BTW, shall I remove the cmd:gnulib_tools (which is already commented out) ?

Yes.

Does someone have an idea of how the undefined reference to memmem can be fixed?

It can't; it's a C library function we apparently don't implement for whatever reason. Just don't run the tests, I suppose 😒

[waddlesplash]

Unless you know of something using the .la files, please drop them from here too.

[fbrosson]

net-libs/libsoup needs net-libs/glib_networking which in turn needs net-libs/gnutls.
But libsoup does not mention it needs gnutls. So if we removed the libtool files in gnutls then we would need to add a devel:libgnutls to the BUILD_REQUIRES of libsoup. But we don't want to do that because libsoup should not care about which libraries glib_networking depends on.

package-links abstraction + libtool files = much simpler dependencies

[waddlesplash]

But libsoup does not mention it needs gnutls. So if we removed the libtool files in gnutls then we would need to add a devel:libgnutls to the BUILD_REQUIRES of libsoup. But we don't want to do that because libsoup should not care about which libraries glib_networking depends on.

Um, libtool files do not help us with what's in BUILD_REQUIRES, at all? If the files are required and libtool knows that, that's fine and dandy, but it doesn't help us one bit with HaikuPorter dependencies. So I fail to see how this is relevant.

[waddlesplash]

But we don't want to do that because libsoup should not care about which libraries glib_networking depends on.

In the wonderful world of Linux, stuff does care because it has to care. And so we have to care, because we don't have native software that can replace all of this (yet) :).

[fbrosson]

• The BUILD_REQUIRES of libsoup mentions lib:libglib_networking_2.0 (although it should mention devel:libglib_networking_2.0).
• The BUILD_REQUIRES of glib_networking mentions devel:libgnutls
• When haikuporter activates the packages needed to build libsoup, it pulls lib:libglib_networking_2.0 which in turn pulls lib:libgnutls.
  libtool finds libgnutls thanks to libglib_networking_2.0.la, but if we don't fix that file then the full path of libgnutls it has recorded depends on the version and revision of gnutls at the time glib_networking is built.
  So if gnutls gets upgraded after glib_networking was built then libglib_networking_2.0.la won't allow to find libgnutls.

When I was asked a few weeks ago to evaluate whether we could get rid of the libtool files, I started building some packages without the libtool files to see if they could still be built. I found that it was OK because:
a) the packages I tested did list all the dependencies, even indirect;
b) my packages were fresh builds.
It was only later that I noticed the full implications of dropping the libtool files.
If HaikuPorts wants to drop the libtool files, I won't go against it. But I won't have much motivation for contributing anymore.

Keeping and fixing the libtool files allows to narrow the dependencies to just the set of direct dependencies. Who would want to maintain recipes that need to list all indirect dependencies?

[waddlesplash]

Cc'ing @korli here. I still don't understand how libtool files help us with recipes themselves at all.

If there are indirect dependencies, won't they be pulled in automatically? E.g. to build libsoup, one needs devel:libglib_networking_2.0, and that pulls in devel:libgnutls, which it needs.

So I don't see how libtool affects the packages that get pulled into the chrooted environment at all. HaikuPorter doesn't parse libtool files, or know or care about them, and so the activated packages are still the exact same ones that are listed in BUILD_REQUIRES & BUILD_PREREQUIRES (and the complete dependency tree built from them, of course.)

[fbrosson]

Using package-links abstractions for the paths in the libtool files is what allows libtool to find the libs that are part of the set of indirect dependencies even after an indirect dependency gets updated.

[waddlesplash]

Yes, but what does this break? Have we had to patch anything because we didn't have libtool files? I'm having trouble seeing this, your logic seems to be confusing REQUIRES*/PROVIDES* for what is passed to the linker...

I'm still trying to establish where we have to do more work, or something is more broken, or whatever due to not having libtool files.

[waddlesplash]

This is the statement I'm having trouble with, I suppose:

Keeping and fixing the libtool files allows to narrow the dependencies to just the set of direct dependencies. Who would want to maintain recipes that need to list all indirect dependencies?

How exactly does keeping the libtool files change what we put in the recipe? While this may help with linking static libraries, it won't help us with the PROVIDES/REQUIRES sections at all, and that's where the extra work would come in. And for dynamic libraries, .la files are altogether pointless.

[fbrosson]

If I'm not mistaken, libtool files are also useful for building programs that use dynamic linking. The paths that libtool passes to ld are embedded in the runtimes. And since every package has its set of package-links, the stable path can be used and remains valid even if the indirect dependencies are updated.

BTW, thanks to this discussion I have just understood that the BUILD_REQUIRES of libsoup which mentions lib:libglib_networking_2.0 is OK. I though it had to mention devel:libglib_networking_2.0 but now I understood that I was wrong.
So this is very good news, because it means that the fixLibtoolArchives helper function @mmuman wrote is probalbly 100% correct! I'm now almost sure I'll be able to use it to fix the libtool files, because I've just understood that it just needs lib: dependencies, and not the devel: ones.
I'll replace all the complex sed commands by calls to fixLibtoolArchives and I'm pretty sure it will do the job perfectly. BTW, sorry @mmuman for having written the helper did not work for all cases. It probably works perfectly in all cases if called from within recipes with correct BUILD_REQUIRES.

[pulkomandy]

Here is what we would want to achieve in the ideal world. Let's take a practical and simple example.

Here is a game using SDL and SDL_image. SDL_image in turn depends on libpng version 16.
The game links against -lSDL and -lSDL_image, and its package depends on these two libs (with correct lib: entries). The game does NOT link against libpng directly.

This means, if we rebuild SDL_image against a new, ABI-incompatible version of libpng, we don't need to rebuild all games using SDL_image. The runtime_loader will load the new sdl_image lib, and see it depends on libpng, and load that as well.

Note that this is all runtime problems. At runtime, the .la files are not even available, so they are of no help in achieving this.

Now let's see what we need at build time for this to work properly:

• Each .so elf file should have a list of dependencies. You can check that this is already the case: readelf -aW libSDL_image-1.2.so.0 | grep NEEDED
• The dependencies need no absolute path, because our runtime_loader is able to find libraries on its own and does not need the handholding.
• The dependencies should have the correct name, in our case, the runtime loader will search in its library path for files with the name indicated in the NEEDED list. This is different from Linux, which compares not the filename, but the soname embedded inside the ELF file (which is why Linux needs ldconfig to update its library database, and Haiku doesn't).

So, this is all already working properly, without the .la files.

We don't need or want any absolute path in the runtime .so. We want just the name required to lookup the library, and let the runtime_loader figure things out.

Since we already get everything working as it should, without any .la files involved, we should just remove the .la files. If there are problems with doing that, we can fix said problems directly in libtool and/or the runtime_loader, instead of hacking around them with .la files.

In our case, the information libtool gets from the .la files is already available in the .so files (you can check with readelf + grep NEEDED as above if you don't believe me :)). You can get a problem with static linking however, because .a file may not have this information. But, this is irrelevant, because if you statically link something, you have to dynamically link its dependencies (if any) directly.

[waddlesplash]

Yep. (thanks for taking the time to write that out, @pulkomandy!)

[fbrosson]

OK. In a few hours I'll drop the libtool files in this PR (and also in #665 (comment)) although I'm not yet convinced that doing so won't trigger other problems, e.g. for building against static libs of indirect dependencies.

[fbrosson]

Dropped libtool files ;-)