external-project: New module to build configure/make projects

This adds an experimental meson module to build projects with other
build systems.

Closes: #4316

CODEOWNERS

@@ -1,5 +1,6 @@
 * @jpakkane
 /mesonbuild/modules/pkgconfig.py @xclaesse
 /mesonbuild/modules/cmake.py @mensinda
+/mesonbuild/modules/unstable_external_project.py @xclaesse
 /mesonbuild/ast/* @mensinda
 /mesonbuild/cmake/* @mensinda

docs/markdown/External-Project-module.md

@@ -0,0 +1,116 @@
+# External Project module
+
+**Note**: the functionality of this module is governed by [Meson's
+  rules on mixing build systems](Mixing-build-systems.md).
+
+*This is an experimental module, API could change.*
+
+This module allows building code that uses build systems other than Meson. This
+module is intended to be used to build Autotools subprojects as fallback if the
+dependency couldn't be found on the system (e.g. too old distro version).
+
+The project will be compiled out-of-tree inside Meson's build directory. The
+project will also be installed inside Meson's build directory using make's
+[`DESTDIR`](https://www.gnu.org/prep/standards/html_node/DESTDIR.html)
+feature. During project installation step, that DESTDIR will be copied verbatim
+into the desired location.
+
+External subprojects can use libraries built by Meson (main project, or other
+subprojects) using pkg-config, thanks to `*-uninstalled.pc` files generated by
+[`pkg.generate()`](Pkgconfig-module.md).
+
+External build system requirements:
+- Must support out-of-tree build. The configure script will be invoked with the
+  current workdir inside Meson's build directory and not subproject's top source
+  directory.
+- Configure script must generate a `Makefile` in the current workdir.
+- Configure script must take common directories like prefix, libdir, etc, as
+  command line arguments.
+- Configure script must support common environment variable like CFLAGS, CC, etc.
+- Compilation step must detect when a reconfigure is needed, and do it
+  transparently.
+
+Known limitations:
+- Executables from external projects cannot be used uninstalled, because they
+  would need its libraries to be installed in the final location. This is why
+  there is no `find_program()` method.
+- The configure script must generate a `Makefile`, other build systems are not
+  yet supported.
+- When cross compiling, if `PKG_CONFIG_SYSROOT_DIR` is set in environment or
+  `sys_root` in the cross file properties, the external subproject will not be
+  able to find dependencies built by meson using pkg-config. The reason is
+  pkg-config and pkgconf both prepend the sysroot path to `-I` and `-L` arguments
+  from `-uninstalled.pc` files. This is arguably a bug that could be fixed in
+  future version of pkg-config/pkgconf.
+
+*Added 0.56.0*
+
+## Functions
+
+### `add_project()`
+
+This function should be called at the root directory of a project using another
+build system. Usually in a `meson.build` file placed in the top directory of a
+subproject, but could be also in any subdir.
+
+Its first positional argument is the name of the configure script to be
+executed (e.g. `configure` or `autogen.sh`), that file must be in the current
+directory and executable.
+
+Keyword arguments:
+- `configure_options`: An array of strings to be passed as arguments to the
+  configure script. Some special tags will be replaced by Meson before passing
+  them to the configure script: `@PREFIX@`, `@LIBDIR@` and `@INCLUDEDIR@`.
+  Note that `libdir` and `includedir` paths are relative to `prefix` in Meson
+  but some configure scripts requires absolute path, in that case they can be
+  passed as `'--libdir=@PREFIX@/@LIBDIR@'`.
+- `cross_configure_options`: Extra options appended to `configure_options` only
+  when cross compiling. special tag `@HOST@` will be replaced by
+  `'{}-{}-{}'.format(host_machine.cpu_family(), build_machine.system(), host_machine.system()`.
+  If omitted it defaults to `['--host=@HOST@']`.
+- `verbose`: If set to `true` the output of sub-commands ran to configure, build
+  and install the project will be printed onto Meson's stdout.
+- `env` : environment variables to set, such as `['NAME1=value1', 'NAME2=value2']`,
+  a dictionary, or an [`environment()` object](Reference-manual.md#environment-object).
+
+Returns an [`ExternalProject`](#ExternalProject_object) object
+
+## `ExternalProject` object
+
+### Methods
+
+#### `dependency(libname)`
+
+Return a dependency object that can be used to build targets against a library
+from the external project.
+
+Keyword arguments:
+- `subdir` path relative to `includedir` to be added to the header search path.
+
+## Example `meson.build` file for a subproject
+
+```meson
+project('My Autotools Project', 'c',
+  meson_version : '>=0.56.0',
+)
+
+mod = import('unstable_external_project')
+
+p = mod.add_project('configure',
+  configure_options : ['--prefix=@PREFIX@',
+                       '--libdir=@LIBDIR@',
+                       '--incdir=@INCLUDEDIR@',
+                       '--enable-foo',
+                      ],
+)
+
+mylib_dep = p.dependency('mylib')
+```
+
+## Using wrap file
+
+Most of the time the project will be built as a subproject, and fetched using
+a `.wrap` file. In that case the simple `meson.build` file needed to build the
+subproject can be provided by adding `patch_directory=mysubproject` line
+in the wrap file, and place the build definition file at
+`subprojects/packagefiles/mysubproject/meson.build`.

docs/markdown/snippets/external_project.md

@@ -0,0 +1,24 @@
+## External projects
+
+A new experimental module `unstable_external_project` has been added to build
+code using other build systems than Meson. Currently only supporting projects
+with a configure script that generates Makefiles.
+
+```meson
+project('My Autotools Project', 'c',
+  meson_version : '>=0.56.0',
+)
+
+mod = import('unstable_external_project')
+
+p = mod.add_project('configure',
+  configure_options : ['--prefix=@PREFIX@',
+                       '--libdir=@LIBDIR@',
+                       '--incdir=@INCLUDEDIR@',
+                       '--enable-foo',
+                      ],
+)
+
+mylib_dep = p.dependency('mylib')
+```
+

docs/sitemap.txt

@@ -50,6 +50,7 @@ index.md
 			Windows-module.md
 			Cuda-module.md
 			Keyval-module.md
+			External-Project-module.md
 		Java.md
 		Vala.md
 		D.md

mesonbuild/compilers/compilers.py

@@ -96,6 +96,9 @@
                   'vala': 'VALAFLAGS',
                   'rust': 'RUSTFLAGS'}
 
+cexe_mapping = {'c': 'CC',
+                'cpp': 'CXX'}
+
 # All these are only for C-linkable languages; see `clink_langs` above.
 
 def sort_clink(lang):

mesonbuild/dependencies/base.py

@@ -642,28 +642,35 @@ def _call_pkgbin_real(self, args, env):
         mlog.debug("Called `{}` -> {}\n{}".format(call, rc, out))
         return rc, out, err
 
-    def _call_pkgbin(self, args, env=None):
-        # Always copy the environment since we're going to modify it
-        # with pkg-config variables
-        if env is None:
-            env = os.environ.copy()
-        else:
-            env = env.copy()
-
-        extra_paths = self.env.coredata.builtins_per_machine[self.for_machine]['pkg_config_path'].value
-        sysroot = self.env.properties[self.for_machine].get_sys_root()
+    @staticmethod
+    def setup_env(env, environment, for_machine, extra_path=None):
+        extra_paths = environment.coredata.builtins_per_machine[for_machine]['pkg_config_path'].value
+        if extra_path:
+            extra_paths.append(extra_path)
+        sysroot = environment.properties[for_machine].get_sys_root()
         if sysroot:
             env['PKG_CONFIG_SYSROOT_DIR'] = sysroot
         new_pkg_config_path = ':'.join([p for p in extra_paths])
         mlog.debug('PKG_CONFIG_PATH: ' + new_pkg_config_path)
         env['PKG_CONFIG_PATH'] = new_pkg_config_path
 
-        pkg_config_libdir_prop = self.env.properties[self.for_machine].get_pkg_config_libdir()
+        pkg_config_libdir_prop = environment.properties[for_machine].get_pkg_config_libdir()
         if pkg_config_libdir_prop:
             new_pkg_config_libdir = ':'.join([p for p in pkg_config_libdir_prop])
             env['PKG_CONFIG_LIBDIR'] = new_pkg_config_libdir
             mlog.debug('PKG_CONFIG_LIBDIR: ' + new_pkg_config_libdir)
 
+
+    def _call_pkgbin(self, args, env=None):
+        # Always copy the environment since we're going to modify it
+        # with pkg-config variables
+        if env is None:
+            env = os.environ.copy()
+        else:
+            env = env.copy()
+
+        PkgConfigDependency.setup_env(env, self.env, self.for_machine)
+
         fenv = frozenset(env.items())
         targs = tuple(args)
         cache = PkgConfigDependency.pkgbin_cache

mesonbuild/envconfig.py

@@ -339,6 +339,7 @@ def __init__(
         'cmake': 'CMAKE',
         'qmake': 'QMAKE',
         'pkgconfig': 'PKG_CONFIG',
+        'make': 'MAKE',
     }  # type: T.Dict[str, str]
 
     # Deprecated environment variables mapped from the new variable to the old one

mesonbuild/interpreter.py

@@ -2505,6 +2505,8 @@ def holderify(self, item):
             return ExternalProgramHolder(item, self.subproject)
         elif hasattr(item, 'held_object'):
             return item
+        elif isinstance(item, InterpreterObject):
+            return item
         else:
             raise InterpreterException('Module returned a value of unknown type.')
 
@@ -2530,6 +2532,8 @@ def process_new_values(self, invalues):
                 # FIXME: This is special cased and not ideal:
                 # The first source is our new VapiTarget, the rest are deps
                 self.process_new_values(v.sources[0])
+            elif isinstance(v, InstallDir):
+                self.build.install_dirs.append(v)
             elif hasattr(v, 'held_object'):
                 pass
             elif isinstance(v, (int, str, bool, Disabler)):

mesonbuild/mesonlib.py

@@ -971,6 +971,17 @@ def get_cmake_define(line: str, confdata: 'ConfigurationData') -> str:
     else:
         raise MesonException('#mesondefine argument "%s" is of unknown type.' % varname)
 
+def get_variable_regex(variable_format: str = 'meson') -> T.Pattern[str]:
+    # Only allow (a-z, A-Z, 0-9, _, -) as valid characters for a define
+    # Also allow escaping '@' with '\@'
+    if variable_format in ['meson', 'cmake@']:
+        regex = re.compile(r'(?:\\\\)+(?=\\?@)|\\@|@([-a-zA-Z0-9_]+)@')
+    elif variable_format == 'cmake':
+        regex = re.compile(r'(?:\\\\)+(?=\\?\$)|\\\${|\${([-a-zA-Z0-9_]+)}')
+    else:
+        raise MesonException('Format "{}" not handled'.format(variable_format))
+    return regex
+
 def do_conf_str (data: list, confdata: 'ConfigurationData', variable_format: str,
                  encoding: str = 'utf-8') -> T.Tuple[T.List[str],T.Set[str], bool]:
     def line_is_valid(line : str, variable_format: str) -> bool:
@@ -982,14 +993,7 @@ def line_is_valid(line : str, variable_format: str) -> bool:
                 return False
         return True
 
-    # Only allow (a-z, A-Z, 0-9, _, -) as valid characters for a define
-    # Also allow escaping '@' with '\@'
-    if variable_format in ['meson', 'cmake@']:
-        regex = re.compile(r'(?:\\\\)+(?=\\?@)|\\@|@([-a-zA-Z0-9_]+)@')
-    elif variable_format == 'cmake':
-        regex = re.compile(r'(?:\\\\)+(?=\\?\$)|\\\${|\${([-a-zA-Z0-9_]+)}')
-    else:
-        raise MesonException('Format "{}" not handled'.format(variable_format))
+    regex = get_variable_regex(variable_format)
 
     search_token = '#mesondefine'
     if variable_format != 'meson':