Support for the Autotools build system

[guilhermeferreira]

---

Advantages

1. Choose whether to build static, dynamic or both kinds of libraries;
2. Choose whether to build synchronous (MQTTClient), asynchronous
   (MQTTAsync), or both types of libraries;
3. Choose whether or not to build SSL-aware libraries;
4. Choose whether or not to build documentation, test or samples;
5. Link the samples statically with the library, so the library can
   be profiled;
6. Easier cross compilation;
7. Automatically find the dependencies to link (pthread, libcrypto
   and libssl);
8. Automatic handling of portability issues. For example, library
   naming differs accross OS.

---

Build instructions

The first step is to run the bootstrap script:

$ ./bootstrap

Next, in order to build only the static library:

$ ./configure --disable-shared --enable-static
$ make

the commands above build:

libpaho-mqtt3a.a
libpaho-mqtt3c.a

Or, in order to build only the dynamic library:

$ ./configure --enable-shared --disable-static
$ make

the previous commands build:

libpaho-mqtt3a.so.1.0.3
libpaho-mqtt3c.so.1.0.3

and their respective links and sonames.

In order to build static only and link the test samples to static,
so it can be profiled, issue the following command:

$ ./configure --enable-samples --disable-shared
CFLAGS=-pg LDFLAGS=-pg

In order to build the SSL-aware libraries, use the --with-ssl option:

$ ./configure --disable-shared --enable-static --with-ssl
$ make

which build the synchronous and asynchronous secure versions:

libpaho-mqtt3a.a
libpaho-mqtt3as.a <- SSL synchronous
libpaho-mqtt3c.a
libpaho-mqtt3cs.a <- SSL asynchronous

Also, there is an option to enable/disable high level of warnings
(--enable-peak-warnings).

---

Remarks

The regular Make version is around 2.5 times faster than the Autotools
version. The overall time for Autotools include the time of bootstrap,
configure and make. The regular Make time include only the make command.

Successfully built on and for:

• Ubuntu 10.04 (x86 32-bit) with gcc 4.4.3 and GNU libc 2.11.1;
• Ubuntu 10.04 (x86 32-bit) with clang 1.1 and GNU libc 2.11.1;
• Ubuntu 12.04 (x86 64-bit) with gcc 4.6.3 and GNU libc 2.15;
• Ubuntu 12.04 (x86 64-bit) with clang 3.4 and GNU libc 2.15;
• Fedora 23.10 (x86 64-bit) with gcc 5.3.1 and GNU libc 2.22;
• Fedora 23.10 (x86 64-bit) with clang 3.7.0 and GNU libc 2.22;
• FreeBSD 10.3 (AMD 64-bit) with gcc 4.8.5;
• FreeBSD 10.3 (AMD 64-bit) with clang 3.4.1;

Signed-off-by: Guilherme Maciel Ferreira guilherme.maciel.ferreira@gmail.com

[ralight]

I don't believe this should be merged. There are already three separate ways of building the code, adding another does not seem like a good idea.

This is especially the case if nobody else has the necessary automake/autoconf skills.

[guilhermeferreira]

Hi Roger, I understand your concerns. However, Autotools provides a more portable build system than GNU Make. And, as far as I know, it's more common on Unix than CMake.

Moreover, this Autotools version provides some features that CMake currently doesn't provide. For example, static library build, default make targets, and a more customized build of Paho library.

[ralight]

CMake is very widely supported (and allows Windows support, which autotools do not), whether or not it is less common shouldn't be the overriding factor.

Likewise, if there are features lacking in the CMake support then a more reasonable approach would be to add those features.

[icraggs]

I am following this conversation :-)

In principle I don't mind including extra ways of building, as long as it doesn't mean a lot of extra maintenance work (for me). The automated Hudson builds (https://hudson.eclipse.org/paho/view/C%20clients/) use Ant, Make and Visual Studio. I used Ant because we used it for the Java client too, so that may not have been the best choice, but I use it at the top level on all platforms to kick off Make or Visual Studio and then run the regression tests.

I wouldn't want to discard the existing build mechanism before making sure the new was a suitable replacement for the automated builds. Surely they could co-exist anyway?

The Windows build hasn't completely recovered from the CMake contribution, but it's close, and I have had to do it. I'm wondering where would this stop, with multiple ways of building? How many are there? It feels like I could spend a lot of time maintaining multiple build options. (I have declined to explicitly make allowances for cygwin on Windows before now because I thought that Visual Studio would be more popular with Windows users.)

I am no expert in CMake or Autotools. Or Make in fact. So I have no particular axe to grind, just looking for the best way forward.

[ralight]

Build scripts co-existing is of course possible (I admit I also do this with mosquitto using bare make as the main script but having cmake for mac/windows support) but it is a maintenance burden. At the moment there are three ways of building - ant+make, cmake and custom visual studio project files. My suggestion would be to reduce this to at most two, not to add more. It would be fairly straightforward to have cmake do everything for all platforms, including running tests.

autotools is one of those things that is often done badly, using copy-pasted code because people don't take the time to learn it. I'm not suggesting that is the case here. I certainly couldn't maintain it without effort but have found cmake easy to deal with.

It should definitely depend on the preference of the main developers though - I keep bare makefiles for mosquitto because it works better for me on a day to day basis than cmake.

[guilhermeferreira]

First, I apologize for the long answer.

Like I said before, I clearly understand your concerns guys. And I won't deny that one more building system to be maintained adds overhead.

Both CMake and Autotools are build system generators. Their purpose is to create native compilation files (Makefiles, Ant's XMLs or Visual Studio solutions). In my humble opinion, these native compilation systems should not be maintained. But they should be generated for different distributions. Because maintaining ifdefs in build scripts is very error prone, and is handled automatically by Autotools or CMake.

Though I agree it takes more time to learn Autotools than CMake, Autotools provides more powerful options to generate build scripts for Unix than CMake. For example, the dependency discovery with Autotools (on Unix systems) is more detailed than those in CMake. You have built in support to look for compatible libraries, headers, or even function within a systems.

Another reason (a little bit personal) is that Autotools's variables are plain Unix variables. On the other hand, CMake uses these CMAKE_VERY_LONG_WEIRD_VARIABLES that make integration with other systems, like Buildroot, counterintuitive.

I don't have knowledge regarding Mac OS building systems. But in my experience, CMake is good to generate Windows's compilation files, while Autotools exceeds in Unix systems.

In the end, it makes sense to include this build system only if the project maintainer feels it can be maintained in the long term.

[fpagliughi]

Apologies for being so late to the discussion... the past few months
have been crazy busy.

On this topic, I sort of agree with everyone, and would hate to see a
proliferation of build tools. If forced to chose just one for a C/C++
library, I would probably go with CMake as it seems to cover the most
ground and has been slowly gaining popularity the last few years. But...

Autotools is still the de facto way to distribute open-source C/C++
projects on Linux, and for us it still has one overwhelming advantage:
nearly any Autotools project can be cross-compiled to virtually any
embedded GCC/Linux board with a single, standard option, '--host', like:

./configure --host=arm-linux-gnueabihf

You don't need to look anything up. You don't need to read through any
make files to figure out how to cross-compile. And you don't need to
write any extensions or specific support for the target. This is
especially helpful if you have a few dozen libraries to cross-compile,
or if you want to add the project to a buildroot or yocto system.

That said, I haven't the foggiest idea of how to create Autotools
projects, either. I've pulled the books out a number of times to get it
done for the Paho C++ library, but always wound up in the garage working
on the dirt bike an hour later. So I'm ecstatic that Guilherme has done
the hard work here. But the advantage is lost for the C++ library if the
Paho C library does not also use Autotools.

So although there is an overlap between the two, I would vote that we
support both CMake and Autotools for these libraries. It would place a
burden on us, the maintainers, but might vastly simplify things for the
users.

Frank Pagliughi

[JuergenKosel]

Hello,

from my user of this library point of view, I like the way how it is done in some other projects,
like wireshark :
They use cmake for building with visual studio, because MSVC is not easily usable with autoconf/automake.
And for UNIX/Linux they use autoconf/automake, which has its stength for portability and cross-compiling.

Greetings
Juergen

[fpagliughi]

Agreed, autotools is inapplicable for Windows builds, but remember that projects like Wireshark are targeted specifically at desktop computers. I would assume that the vast majority of MQTT clients are remote embedded systems, and even though the C/C++ libraries target the desktop OS's, my (biased) assumption is that many of the deployments for these libraries will be on deeply embedded Linux gateways that aggregate sensor data and send it back to a remote broker.

My typical target system is a 200MHz ARM9 with limited memory that that sleeps 98% of the time. These boards can not support native builds, so easy cross-compiling is mandatory. But even on my current prototype that is using a RaspberryPi 2, which is capable of native builds, compiling the whole project on the board takes 10 minutes, whereas the cross build takes less than 3 sec!

So, I acknowledge the need for CMake to support non Linux/Unix systems, but for my own selfish needs, I would also like to preserve simple cross-compiling. But I'm open to suggestions about how to accomplish that.

[JuergenKosel]

Hello,

I think, that the following changes should be made:

1. Replace the recursive Makefile.am, because recursive Makefile may be harmful (see Peter Miller´s paper http://aegis.sourceforge.net/auug97.pdf ).
2. The Makefile.am and CMakeLists.txt should include some Makefile.common, which contains variable definition for automake and cmake, as it is done in the wireshark project. This may reduce the maintenance effort.

Greetings
Juergen

[guilhermeferreira]

Thank you Frank for your support. And thank you Juergen for the great suggestions. I'll test and evaluate them in this weekend.

[icraggs]

Juergen, there appears to be a problem with the IP validation for your pull request:

"The following users do not have valid CLA. Please see this page for details, or to sign a CLA.

noreply@github.com"

Can this be sorted out?

I don't think I want to get rid of the standard GNU make file just yet, but have the other build mechanisms available too. As I've never seriously used CMake or Autotools, I'll have to try them out to see how it all fits together.

The contribution is quite big as well, although most of that is consumed by Doxygen configuration files. I'm not sure whether due to its size it needs to go through the Eclipse IP legal review, or as it is not code, and in the case of the the Doxygen files, mostly comments, that's not needed. I'll have to check. Roger, do you have a view on that?

[JuergenKosel]

Hello Ian,

Am 07.09.2016 um 13:45 schrieb Ian Craggs:

Juergen, there appears to be a problem with the IP validation for your
pull request:

"The following users do not have valid CLA. Please see this page for
details, or to sign a CLA.

|noreply@github.com" |

Can this be sorted out?

Yes, this can be sorted out. It is a github merge-commit.
Unfortunately I do not know, how this single commit could be removed
from the pull request.

I don't think I want to get rid of the standard GNU make file just yet,
but have the other build mechanisms available too. As I've never
seriously used CMake or Autotools, I'll have to try them out to see how
it all fits together.

I would remove the Visual Studio projects, because they can only be
used, if the user has the matching Visual Studio version.
But with cmake the user could always create the Visual Studio files for
his version.
So these *.sln and vcxproj files are candidates for removal.

Greetings
Juergen

[ralight]

Ian, I'm not sure about the IP side of things. The doxyfile*.cmake files are mostly documentation rather than code.

There does seem to be some discussion about the license of these files though. The doxyfile templates (which have then been modified here to include the cmake bits) are generated by doxygen from a template file that is licensed under the GPL - so one argument could be that the output is GPL as well. The doxygen documents state that the output of doxygen is not affected by its license, but it still might be worth getting legal opinion. In the case of the Apache Foundation, their decision is to allow the use of generated doxyfiles, but with the comments removed

https://issues.apache.org/jira/browse/LEGAL-224
http://doxygen.10944.n7.nabble.com/Doxyfile-license-td7411.html
https://lists.debian.org/debian-legal/2009/07/msg00044.html

[JuergenKosel]

Hello,

Am 08.09.2016 um 10:25 schrieb Roger Light:

Ian, I'm not sure about the IP side of things. The doxyfile*.cmake files
are mostly documentation rather than code.

the doxyfiles have been added long ago in commit
3aafa73

They are just modified (renamed/duplicated) in this pull request.

There does seem to be some discussion about the license of these files
though. The doxyfile templates (which have then been modified here to
include the cmake bits) are generated by doxygen from a template file
that is licensed under the GPL - so one argument could be that the
output is GPL as well. The doxygen documents state that the output of
doxygen is not affected by its license, but it still might be worth
getting legal opinion. In the case of the Apache Foundation, their
decision is to allow the use of generated doxyfiles, but with the
comments removed
http://www.apache.org/legal/resolved.html#can-we-use-doxygen-generated-config-files

If there is an issue with the doxygen files, than it should be solved
independently of this pull request.

Greetings
Juergen

[ralight]

Juergen,

You're right, I wasn't looking closely enough.

[guilhermeferreira]

I apologize for the time I was absent. I was working on another issue in Paho MQTT C++.

Regarding the CLA problem:

there appears to be a problem with the IP validation for your pull request:
"The following users do not have valid CLA. Please see this page for details, or to sign a CLA.
noreply@github.com"
Can this be sorted out?

Yes, this can be sorted out. It is a github merge-commit. Unfortunately I do not know, how this single commit could be removed from the pull request.

The rebasing removes this single merge commit. And a "forced" push overwrites the remote branch, keeping the same pull request:

git rebase --interactive origin/develop
git push --force origin build-autotools

Juergen I noticed you've requested another pull #174. If you requested that pull because of the merge commit, we can keep this one #148, because I removed that merge commit.

[guilhermeferreira]

Ian,

I don't think I want to get rid of the standard GNU make file just yet, but have the other build mechanisms available too. As I've never seriously used CMake or Autotools, I'll have to try them out to see how it all fits together.

I should have changed the name long time ago. With the Juergen's help, this change now performs out-of-tree builds. Now it doesn't overwrite the root GNU Makefile.

[guilhermeferreira]

Guys,

There's a problem with the out-of-tree documentation generation. And now that I'm done with Paho MQTT C++, I'll work again on that matter.

Best regards,

[tbeu]

Coming from https://dev.eclipse.org/mhonarc/lists/paho-dev/msg03607.html. Static libraries via autotools is exactly what I was looking for. Thank you @guilhermeferreira

[tbeu]

@guilhermeferreira Your branch build-autotools-feature-test-macros gives

automake: warnings are treated as errors
Makefile.am:304: warning: compiling 'src/samples/MQTTAsync_publish.c' in subdir requires 'AM_PROG_CC_C_O' in 'configure.ac'
Makefile.am: installing 'autotools_build/depcomp'
parallel-tests: installing 'autotools_build/test-driver'
autoreconf: automake failed with exit status: 1

and your other branches build-autotools and build-autotools-ac-define give

config.status: error: cannot find input file: `Makefile.in'

on CentOS 7 when trying to configure the static libs. Do you have an idea what is going wrong here? Additionally, do you want to rebase on official master HEAD? Thanks.

[JuergenKosel]

@tbeu wrote:

on CentOS 7 when trying to configure the static libs. Do you have an idea what is going wrong here? Additionally, do you want to rebase on official master HEAD? Thanks.

You might have forgotten to call the bootstrap script, which calls autoreconf --install.
Rebasing an already published branch may lead to confusing, as the git manuals say. And furthermore it will lead to ip-validation check to fail.

[tbeu]

I called ./bootstrap followed by ./configure --disable-shared --enable-static as you suggested. I can give you the complete log if you like.

What about merging the master HEAD?

[tbeu]

Does autoreconf required admin rights?

[JuergenKosel]

@tbeu : autoreconf does not need admin rights. But autoconf, automake and libtool needs to be properly installed.

[tbeu]

Do you see this warning treated as error (your branch build-autotools-feature-test-macros)?

[guilhermeferreira]

Hello @tbeu, sorry for the long absence. The branch build-autotools-feature-test-macros isn't supposed to be used yet. It is one of my testing branches.

Thanks @tbeu for the error report and thank you @JuergenKosel for the support. I installed CentOS and tested on it. I updated the branch with a fix. It seems CentOS's autotools required another macro in order to work.

Also, I'm using develop as a base, not master.

Successfully built on and for:

• Ubuntu 10.04 (x86 32-bit) with gcc 4.4.3 and GNU libc 2.11.1.
• Ubuntu 10.04 (x86 32-bit) with clang 1.1 and GNU libc 2.11.1.
• Ubuntu 12.04 (x86 64-bit) with gcc 4.6.3 and GNU libc 2.15.
• Ubuntu 12.04 (x86 64-bit) with clang 3.4 and GNU libc 2.15.
• Fedora 23.10 (x86 64-bit) with gcc 5.3.1 and GNU libc 2.22.
• Fedora 23.10 (x86 64-bit) with clang 3.7.0 and GNU libc 2.22.
• FreeBSD 10.3 (AMD 64-bit) with gcc 4.8.5.
• FreeBSD 10.3 (AMD 64-bit) with clang 3.4.1.
• CentOS 7 (x86 64-bit) with gcc 4.8.5 and GNU libc 2.17.

[icraggs]

Given the conversation, I don't mind adding this (probably) as long as a) it's not viewed as a primary build mechanism (at least for now) b) it's not going to significantly add to my maintenance burden and c) it doesn't affect the other build mechanisms.

This PR seems to have accumulated a bunch of extra changes not directly to do with building with Autotools, unless I'm mistaken (a lot of doc build updates for instance. It would be easier for me to evaluate and merge if the different aspects were separated into different, more focussed PRs.

[guilhermeferreira]

Hello @icraggs,

Regarding (a), I don't know how to enforce to the users to use one building system over another. About (b), again, I'll take responsibility for this building system. And of course, other people probably are going to help. And regarding (c), @JuergenKosel improved this PR so it can build the out-of-tree.

All commits here are related to the Autotools. The commit about documentation fixes the Autotools' Doxygen documentation generation. Also, I wouldn't like to squeeze the commits into one because part of the work is made by @JuergenKosel. And it's fair to keep the original author's contribution history.

[dawagner]

When doing an out-of-tree build (as done in the travis build), the binaries and generated files will not clutter the source dir; as a result, I don't think there's any need to add any gitignore entry.

[icraggs]

After a long time just leaving this here because I was uncomfortable supporting (yet) another build method, I'm going to close it. Some recent posts and efforts to move away from Autotools, notably by Eric S. Raymond, support this.