Implement a proper build system

[artob]

Right now, the hdt-lib build process is pretty brittle with regards to platform quirks, and produces only tool binaries and static libraries, not any shared library. Further, building with custom paths or with a selection of features to be enabled/disabled (e.g., Serd, Raptor, zlib, KCB) requires manually editing Makefiles--not at all a welcome practice from a downstream user's point of view.

On POSIX systems, the currently produced static libraries aren't portably linkable with shared libraries that would like to make use of libhdt; moreover, on e.g. x86-64 the individual object files that constitute the static libraries would generally need to be compiled with -fPIC for the libraries to be usable from modern projects. All this requires manually fiddling with flags in both libcds-v1.0.12/src/Makefile and hdt-lib/Makefile.

Additionally, no actual installation facility is currently provided, meaning that users have to manually hunt down built artifacts--which are currently placed in different output directories--and copy them one by one to some installation destination.

All this argues for converting the project to use a proper configuration and build system. The two obvious and realistic choices are Autotools and CMake. Doing this would also greatly facilitate packaging the project for downstream distribution (#19).

I would myself volunteer to convert the project to using standard Autotools, which would give users the familiar ./configure && make && sudo make install workflow that works reliably on all platforms and resolves all aforementioned problems.

[RubenVerborgh]

Autotools would be great!

[mielvds]

Yes, I fully support that! Let me know if I can help (although I don't know much about Autotools)

[artob]

Thanks @RubenVerborgh and @mielvds for weighing in, sounds like we have a go-ahead for this. I'll be looking to begin work on it in the next week or so and will report back on progress. I suspect the difficult bit will be handling the bundled libcds library; otherwise this is fairly straightforward to do.

[kjetilk]

Great stuff! Indeed, I think this would also greatly help distros! :-) And adds another reason to fix #19.

[artob]

Just getting back to this now after the holidays. As per @kjetilk's comment, above, it would indeed be great to fix #19 one way or another. The rest of this is straightforward enough, but the libcds bit not so much...

[kjetilk]

Yeah... One month before Ubuntu closes. But I've also seen stuff getting stuck in the approval queue for a long time, so it may not work out anyway. Sorry I don't have any time to help out, but I think distros are important for adoption.

[artob]

@kjetilk Right, and 16.04 will be the LTS release, which does mean it is a once-in-two-years window. I agree that it would be valuable to get the library included and distributed.

[kjetilk]

@bendiken, yes, that is correct, it is a once-in-two-years window. I'm using Debian on my systems myself, which has a projected freeze date late 2016, but arguably, Ubuntu is an important deployment platform.

[wouterbeek]

I ran into problems when compiling with GCC 6.1.1 #34 and also the -fPIC flag was necessary for me #38.

[artob]

@wouterbeek Thanks for the reminder re: #34. I've been tied up on other fronts, but am catching up and will get back to this soon.

[kjetilk]

This is also seems like a good candidate for the debian-package label.

[RubenVerborgh]

-fPIC support added in #71

[RubenVerborgh]

@bendiken Any status update on this?

[ptorrestr]

Hi. Last year I set an independent repo with libhdt using Autotools. Basically, I created a python wrapper that call libhdt in the backend. Since the current building system produces only static libraries, I decided to create a completely new building system with Autotools. Once it was done, I produce .so/.dylib libraries and I could link python with hdt.

I copy part of that code in this PR and adapted a bit. Now, one can produce libraries using any desirable property using configure.

I had to do some changes in the code:

1. folder hdt-lib was rename to libhdt, as well as libcds
2. libcds was incorporated as en embedded project. It produces its own library (libcds) but is built from the hdt configure script. I think libcds eventually needs to have its own repo, or you can use: https://github.com/fclaude/libcds, https://github.com/fclaude/libcds2
3. libcds is exporting all *.h files when installing. Since I wasn't very short which headers are used by libhdt.
4. Optional libraries are configured in the hdt configure script.

There are some issues that had to be addressed/considered:

1. Since git breaks the timestamps produced by autotools files, there is no point to add them in the repo since you will have to re-generate anyway with autoreconf each time you clone the repo (more info here ). I know this can be annoying for final users. There are ways to make autotools less strict in that regard, but this is not a very good practice. My solution would be generating a final-user release distro using something like ./configure dist and make it public for final-users somewhere.
2. There are several tests that are reporting errors in hdt and cds. I disable them in travis, but you can try it locally and see what is going on. If some tests are not valid any longer, I think they should be removed.

[RubenVerborgh]

Thanks @ptorrestr, this is great!

Why was the rename to libhdt needed?

[ptorrestr]

@RubenVerborgh Just for convention. It makes more sense in my head call the folder "libhdt" than "hdt-lib". It can be changed course.

[LaurensRietveld]

Assuming this issue is resolved (feel free to correct me though), as the autotools PR is merged to develop