Provide access to environment variables in build files

[cxj]

How would I include the value of an environment variable in either my meson.build or meson_options.txt files? Is that even possible?

E.g.
option('someoption', type : 'string', value : ${SANDBOX})

[jpakkane]

At the moment this is not possible. Something about this needs to be designed at some point.

If you wish to set a default, you can do so with -D the first time you run Meson. So something like this

meson .. -Dsomeoption=foobar

or even

meson .. -Dsomeoption=$SANDBOX

[germandiagogomez]

👍

I am also in a need to use this.

[doraskayo]

Not having this feature is a bit of an issue for me. I'm currently porting a build system from CMake to Meson, and in this use case I need to build against an installed SDK in Windows. In order to be able to automatically detect its location in the filesystem, the SDK sets its installed path as an environment variable during installation.

The ability to set the default value in meson_options.txt to an existing environment variable (alongside an optional fallback value, perhaps) would be the only way to keep the current simplicity of building the project.

[nirbheek]

It makes sense to allow default values of meson_options.txt to be set via environment variables, but if I may ask, what SDK is this? If it's an open-source one, perhaps we can add support for it to dependency().

A workaround is also to use run_command() to run Python 3 and print the environment variable. You can get an object representing the Python interpreter used by Meson and use that safely.

pymodule = import('python3')
python3 = pymodule.find_python()

res = run_command(python3, '-c', 'import os; print(os.environ["SDK_VARIABLE"])')
if res.returncode() == 0
  sdk_var = res.stdout
else 
  # not found, do fallback
endif

[doraskayo]

@nirbheek, I'm using the LunarG's Vulkan SDK, and it's technically assembled from a few open-source repositories. This is its main repository: KhronosGroup/Vulkan-LoaderAndValidationLayers.

As for my use case, I only need to include the headers from the Include folder of the SDK, and link against vulkan-1.lib from either the Bin or Bin32 folders, depending on the platform. Frankly, I'm not quite sure where the source for vulkan-1.lib is hosted. It seems to contain symbols for the public API of vulkan.dll, which the shared library that the above repository builds. I haven't actually had a chance to build from source for Windows (only for Linux), so perhaps it's also one of the outputs.

And thanks a lot for the interim solution, I'll try it out.

[nirbheek]

It would be cool if we added support for the Vulkan SDK to Meson. Would you be interested in working on that? You know more about it than anyone else. It's really simple: inside mesonbuild/dependencies.py look at class BoostDependency for a good example of a dependency that is looked up via an environment variable.

I'd be happy to review and help you refine your PR of course. :)

[doraskayo]

Sure, I guess I can give it a try. I'll get to it once I finish converting my CMake project to Meson.

[nirbheek]

Great! Looking forward to your PR. 👍

[anordal]

For finding packages, is the problem that the package doesn't ship a pkg-config file, or is it that pkg-config is a Unix thing and not used on Windows?

If it's a missing pkg-config file problem, why not write it for them? Should be preferrable to adding special casing to Meson. In the case of Vulkan, I see the dependency documentation has you covered now:

Vulkan can be located using pkg-config

If it's a Windows problem (of not having something akin to pkg-config), then how on Earth are you supposed to find build dependencies on Windows in general? pkg-config supports Windows, but that of course only means that it could be used.

[diam]

For access to some environment variable from a meson.build, I've found this trick:

    cmd = run_command('sh', '-c', 'echo $ILOG_ROOT')
    ILOG_ROOT = cmd.stdout().strip()
    message('ILOG_ROOT=' + ILOG_ROOT)

It is a little too verbose. But as it is possible, I think meson should propose some
command to do that.
e.g:

    ilog_root = get_env('ILOG_ROOT')

-- Maurice

[jpakkane]

Accessing environment variables from Meson scripts is intentionally not supported and will not be added in the foreseeable future.

Environment variables are terrible for any sort of configuration because they are mutable global state. If your setup depends on environment variables being set, then running any sort of command that causes reconfiguration from a different terminal that does not have those envvars set (or from an IDE or any of a thousand of different options) breaks your setup in unobvious ways that are at worst incredibly difficult to debug.

Using envvars for configuration is a code smell. Don't use them whenever possible. Convert those to project options or something similar instead. It is the only reliable solution.

[nirbheek]

Accessing environment variables from Meson scripts is intentionally not supported and will not be added in the foreseeable future.

To add to this, if there are standard frameworks that require configuration via environment variables, the best way forward is:

1. Add a Meson module for it; this has already been done for Qt5 and Qt4, for instance, which do some configuration via environment variables
2. Work with upstream to find a better mechanism to do configuration (perhaps pkg-config files)
3. Once (2) succeeds, change the module to use both mechanisms and all meson users benefit from it transparently

[diam]

Thank you for this guideline.

In fact I discovered Meson thank to the very nice presentation from Jussi at INRIA-Saclay (France) last friday. Then I really wanted to try it, then if possible switch from cmake to meson ;-).

I'm trying to expose here one typical use case I try to solve with meson.
(not sure it is the good place for that: perhaps a new FAQ or howto entry?)

Let be a proprietary closed software (CPLEX from IBM) which propose some executables cplex, cplexamp, ... some static libraries et some includes directories. The same need could arise from some open source projet (say xlife++ for finite elements).

We want to install several versions of cplex in a non intrusive manner:

• should be not available by default in the user environment,
• we want to be able to use one version in a terminal and another one in another terminal (in the same session).

In fact, we vant to switch compilation of one project from which use one cplex library version to another cplex library version without modifying the compiled project code (no cplex pathes should be hard coded in the project).
CPLEX propose some Makefile examples which position some variables. So I proposed one environment file for each cplex version which the user can use to choose its cplex version.

For exemple let:

  use_cplex1
  use_cplex2

be two shell commands which update the standard PATH, LD_LIBRARY_PATH variables and position some others specifics environment variables like:

ILOG_CPLEX_INCLUDE_DIR
ILOG_CPLEX_LIB_DIR

(and much others with unambiguous names)

I can provide some Makefile or some CmAkElIsTs.txt (or CMakeLists.txt ;-) for that.
Now, I'would like to test some Meson.build file to do the same purpose (and if possible, without having to wait that all softwares in the world beeing Meson-compatible).

In fact, I probably want to use a command like this in meson.build:

find_library('cplex')

or:

cplex_dep = dependency('cplex')

But I don't know how to make this working. Perhars I should write one pkg-config for each
cplex version? But how can I tell Meson where this file is?

-- Maurice

[jpakkane]

The "correct" solution is to get upstream to add pkg-config files to their build system. This is not guaranteed to work or even if it does it takes a long time for this to appear. The same can be said about convincing upstream to build with Meson so you can use the dependency as a subproject. :)

In the mean time there are a few choices you can do:

• write your own pkg-config files for the installs and select which one you want with PKG_CONFIG_PATH envvar when first setting up the build dir
• write your own pkg-config files but give each of them a unique name so you can choose between them with the dependency name (this works but goes against the grain of how pkg-config "should" be used)
• set up search dirs with a project option

The latter works roughly by having a project option with the name of, say, cplex_root. It is a string pointing to the root of where cplex is installed. Then you can get the include paths with something like:

cplex_inc = include_directories(join_paths(get_option('cplex_root'), 'include'))

And get the library with something like:

cc = meson.get_compiler('c')
cplex_dep = cc.find_library('cplex', dirs : [join_paths(get_option('cplex_root'), 'lib')])

You can change the value of cplex_root from the command line with meson configure.

[diam]

Hmm, getting IBM to use Meson to build CPLEX would be great, but
it might take a bit of time ;-)

So what I understand is that the (my) best way to process is to write a pkg-config file for each cplex installed version (and other tools).

Then when a user want to do some development with a specific set of tools, I'll ask them to call explicitly that set of tools in their shell:

   use_toolset
   # or use_toolsetA, for other variants of this toolset)

This could add some variables (and update PATH, ...) but above all add pathes to the PKG_CONFIG_PATH variable pour some config file (cplex-X.Y.Z.cp, ...)

Now I suppose a meson.build file can make use of it simply by calling:

   cplex_dev = dependency('cplex')

Is it right?
-- Maurice

[jpakkane]

Pretty much apart from the fact that dependency('cplex') searches for a file called cplex.pc, not cplex-X.Y.Z.pc. So you need to either:

• have each pc file be called cplex.py and choose which one to use by setting the PKG_CONFIG_PATH envvar
• have each dependency have its unique name and search for them with `dependency('cplex-X.Y.X')

[diam]

Thank you versus much Jussi, I finally managed to make it work!

For every platform, I'll have a directory with all versions of my package plus a link (e.g @cplex.pc) to prefered one (e.g cplex1280.pc). I need yet to clean up installation, clean up the .pc files and find good positions for pkgconfig paths.

The shell commands:

use_cplex
use_cplexA
use_cplexB

(among other) complete the PKG_CONFIG_PATH variable with the right .pc file.

-- Maurice

[ptomato]

It would be nice to provide access to $DESTDIR specifically when adding an install script, in case the install script was only one command that you had to pass a path to. It's certainly possible, but annoying, to create a separate shell script for that.

[nirbheek]

It would be nice to provide access to $DESTDIR specifically when adding an install script

That value will generally only be available when the install script is run, not when meson is being run, so I don't see how that is useful?

[ptomato]

e.g.,

update_some_cache = find_program('update-some-cache')
meson.add_install_script(update_some_cache, join_paths('$DESTDIR', mypkgdatadir))

and have $DESTDIR be expanded.

[nirbheek]

It is a common idiom to do something like:

$ ./configure && make
$ DESTDIR=/foo/bar/dir make install

And the meson equivalent is:

$ meson _build && ninja -C _build
$ DESTDIR=/foo/bar/dir ninja -C _build install

In this case your install script will fail. You should ideally use a wrapper script (python, preferably since that will always be available), and fetch DESTDIR inside it.

[ptomato]

It wouldn't fail if Meson expanded $DESTDIR when invoking the install script...

I can certainly (and already have) made a wrapper script, but I have to duplicate a bunch of path computation in it since I don't have access to Meson variables there.

[elboulangero]

I have another use-case for reading an environment variable (maybe).

I use gtk-builder-tool validate to validate the ui files of my applications. The snippet look like that:

gtk_builder_tool = find_program('gtk-builder-tool', required: false)
if gtk_builder_tool.found()
  foreach ui_file : ui_files
    test('Validate ' + ui_file, gtk_builder_tool,
      args: [ 'validate', join_paths(meson.current_source_dir(), ui_file) ]
    )
  endforeach
endif

The issue arises when I try to build in a Fedora container.

• The gtk development package is gtk3-devel, and it contains the tool gtk-builder-tool, which trigger the test mentioned above.
• Being in a container, there's no graphics, and the test fails with Unable to init server: Could not connect: Connection refused.

There's probably more than one solution to that, but the one that seems the most straightforward to me would be to test if the environment variable DISPLAY is set, and run the test only then.

If anyone has a better idea btw, you're welcome :)

[nirbheek]

@elboulangero in this use-case you want the env variable check to be done when the test is run, not when the configuration is done, because in theory the configuration and the running could be done in different environments.

So I'd recommend using a script that skips the test when you know that gtk-builder-tool can't be run. We should probably add a kwarg to test() that allows you to specify a script to run that can decide whether to run a test or skip it.

[jpakkane]

The "mesonic" and portable way of doing this is to have a standalone Python script that does all the inspection and validation needed and then either runs the test or returns error code 77 (which means the test was skipped). Then you would use it something like:

validator = find_program('ui_validator.py')
foreach ui_file : files(ui_files)
  test('Validate ' + ui_file, validator, args: [ ui_file])
endforeach

This has the added benefit that if you want to add more validation steps, adding them to the validator script is easy.

[elboulangero]

@nirbheek @jpakkane I followed your suggestion and used a standalone script, it works great.

https://gitlab.com/goodvibes/goodvibes/commit/6acdf51

Thanks a bunch!

[scivision]

It appears that it's a long-standing Meson design design to not directly read environment variables, so it seems this issue should be closed.

[Makogan]

I ended up here trying to find a way to enable validation layers on a vulkan project. Normally you do that by exporting an env var. I saw in this thread that official support was added. The docs do shortly mention Vulkan but I assume this is for linking. To enable validation layers one must set the VK_LAYER_PATH env var. Now, I can just export the path to my .profile. However that only works on Linux and adds some cognitive load on the user of the system should they chose to build the project.

Is it possible to have some support for the layers path in a similar fashion to pkg_config_path? i.e. passing it as an argument to the the build directory setup args?

I can try adding it myself with some guidance. It seems the vulkan class in ui.py would be the place to send the flag into, but trying to sneak around meson;s code I am not sure how to get the flag down there to enable the env var.

[Jan200101]

Accessing environment variables from Meson scripts is intentionally not supported and will not be added in the foreseeable future.

Environment variables are terrible for any sort of configuration because they are mutable global state. If your setup depends on environment variables being set, then running any sort of command that causes reconfiguration from a different terminal that does not have those envvars set (or from an IDE or any of a thousand of different options) breaks your setup in unobvious ways that are at worst incredibly difficult to debug.

Using envvars for configuration is a code smell. Don't use them whenever possible. Convert those to project options or something similar instead. It is the only reliable solution.

I honestly agree, but sometimes there is no choice.

For example the devkitPro toolchains have no official fixed location and are instead identified by the environment variables they set ($DEVKITPRO, $DEVKITARM, $DEVKITPPC, etc)

Since its only distributed via pacman and the maintainer has a history of blocking access to direct download if he deems the machine downloading it is not trustworthy its not possible to simply download the latest version on the fly.

Building the toolchain from scratch is not an option either since the buildscripts are, according to the lead developer, not used for creating the pacman packages, are licenseless and build incomplete versions of the toolchain.

Their official solution for using meson is a shell script that generates a cross file on the fly, but that is not a reliable solution since upstream does not commit to keeping it working nor can it easily be detected within Meson

[xclaesse]

Their official solution for using meson is a shell script that generates a cross file on the fly,

Sounds like a job for meson env2mfile: https://mesonbuild.com/Release-notes-for-0-62-0.html#experimental-command-to-convert-environments-to-cross-files