RFC: Add minimal support for autotools subproject

[xclaesse]

Some projects like GStreamer has many external dependencies that does not have native meson build systems.

Subprojects are really useful to build a project on platforms that doesn't have any of its deps, like cross-building GStreamer for Android. Adding them all in wrapdb is an infinite wast of time. It would be a huge time saving of meson could just run configure/make itself for subprojects.

In the master project (e.g. GStreamer):

libfoo = dependency('libfoo', fallback : ['libfoo', 'libfoo_dep'])

In subproject's meson.build:

project('libfoo', 'c')

autotools = import('autotools')

# This cause meson to run `./configure` in `<build dir>/subprojects/libfoo_build`,
# and add ninja rules to run:
# `make -C <build dir>/subprojects/libfoo_build"` and
# `make DESTDIR=<build dir>/subprojects/libfoo_installed install`
autotools.configure()

# This will create a dependency object that has `-I<build dir>/subprojects/libfoo_installed/<prefix>/<includedir>`, `-L<build dir>/subprojects/libfoo_installed/<prefix>/<libdir>`, and `-lfoo`
libfoo_dep = autotools.declare_dependency()

Would be nice to also be able to use .pc file generated by make in autotools.declare_dependency(), but that would mean running make at meson's configure time... Maybe that would be acceptable since this autotools module is really just a "best effort" to help developers, not something nice and clean.

Also, I wouldn't be too worried about properly rebuild autotools subprojects if they change, IMHO the use-case here is really to provide fallback for deps that won't change. Again, this is just a "best effort" case, developers can always to rebuild from scratch if something goes wrong (often the case with autotools itself anyway).

And finally, since it's easier to use vanilla upstream tarball/git, the small subproject's meson.build could be generated from the .wrap file, by adding build_type=autotools.

[xclaesse]

I'll try to work on a PoC PR to see how it can looks like in practice.

There are 2 key elements I had in mind when doing this proposal:

1. subproject's build system should be transparent to the master project.
2. All autotools specific code in meson should be in a meson module, we don't want to pollute the core code with crap from other build systems.

[ocrete]

I don't think this should be autotools specific.. Instead you could do something much more generic along the lines of CMake's ExternalProject.. https://cmake.org/cmake/help/latest/module/ExternalProject.html

[xclaesse]

Yeah, of course here I say "autotools" but it could be anything that has "configure/build/install" kind of steps. But I would like to keep it simple at least for a PoC, it can always be generalized later by creating a new meson module and share common code.

So here I would like to start small, and grow from there, to see how it goes.

[jpakkane]

The declaration should be outside of meson.build if possible, such as in the wrap file. And further this should only be shorthand for "get this dependency, build it with its own build system, install and then use the .pc files it produced to do the actual build configuration".

Attempting to do anything within the same build dir such as using libraries without installing them is not ok. It will not work.

That being said maybe we could do something simpler to achieve the same thing. Something in the vein of what Flatpak manifest files do. That is, we would have a file that looks like this:

[
  {
    "name" : "foobar",
    "url": "https://...",
    "sha256": "...",
    "buildsystem": "autotools"
  },
  {
    ...
  }
]

And then have a command such as meson prepare-external-dependencies that downloads, builds and installs all the things.

[ocrete]

I like Jussi's plan, just one extra request, I assume those older deps may not have .pc files, so having a nice way to generate them from this would be a nice plus.

[jpakkane]

Presumably the build definition would have all the cc.find_library magic needed to find dependencies not providing .pc files already.

The "correct" fix for any project missing .pc files is to submit a PR upstream to add them.

[xclaesse]

About the JSON example you posted, that's basically a .wrap file, we can just reuse that for the download part.

About extra prepar-external-dependencies command, I don't think it's useful, it can be part of the normal meson configure when master project calls subproject()

[xclaesse]

Btw, it's going to install in master project's build dir, not on the system, so you'll have to tell cc.find_library() or pkg-config to search there.

[xclaesse]

Note that one important idea here is using external project as fallback if it's not provided by the distro, that can only be known at configure time of master project.

[jpakkane]

it's going to install in master project's build dir

No! They should be installed in some sort of final destination (preferably the same where the current project will end up eventually). Otherwise you'd have to reinstall them again upon final install and all things pointing to the file system such as .pc files and install time rpaths will be wrong.

[ocrete]

No! They should be installed in some sort of final destination (preferably the same where the current project will end up eventually). Otherwise you'd have to reinstall them again upon final install and all things pointing to the file system such as .pc files and install time rpaths will be wrong.

You probably don't want to install the headers, etc of the subproject publicly in all cases. We have the somewhat common case where you want to "hide" the dependency by building it statically and linking it into the final project.

[xclaesse]

No! They should be installed in some sort of final destination

No, the whole point here is to not have to install in a prefix, otherwise why would I use meson to do that? I could just run configure & make install myself, or use jhbuild/cerbero, that's what they do. The whole point here is really to have autotools projects as fallback when the dependency is not found on the system.

Otherwise you'd have to reinstall them again upon final install

Yes, and why is that a problem? Also in my usage at least, there is no installation step, when I develop with GStreamer, I run the code uninstalled.

and all things pointing to the file system such as .pc files

.pc files are correct because I compile with the final --prefix, but install with a DESTDIR. Those pc files are usable by meson by setting PKG_CONFIG_SYSROOT_DIR I think (didn't work on that part yet).

install time rpaths will be wrong

That's actually the only issue I see in my WIP pull request, meson sets the rpath correctly to run uninstalled, but upon install the rpath is not rewritten/removed. I'm sure that can be fixed, but I haven't looked at rpath handling yet.

[nirbheek]

No, the whole point here is to not have to install in a prefix, otherwise why would I use meson to do that? I could just run configure & make install myself, or use jhbuild/cerbero, that's what they do

One important use-case is to build everything once and install it into a prefix which is then distributed to people, with just meson && ninja && install. This is very different from doing ./autogen.sh && make && install then setup env vars for the prefix, then do meson && ninja && install which is suddenly a lot more complicated and you've reinvented jhbuild or cerbero.

[Piping]

Hi, I would like to know if there is way to link static_library with autotools generated archive files. link_with keyword argument does not work with custom_target.

I have two autotool dependencies that one will be static linked to main library and the other one(python) will need to be installed into final destination and link with application. Both dependencies are fairly large and complex autotools build system.

This RFC feature is really needed.