Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how to set environment variables in subprocesses #1002

Open
evmar opened this issue Aug 9, 2015 · 10 comments
Open

Document how to set environment variables in subprocesses #1002

evmar opened this issue Aug 9, 2015 · 10 comments

Comments

@evmar
Copy link
Collaborator

@evmar evmar commented Aug 9, 2015

See discussion on
https://groups.google.com/d/topic/ninja-build/ZGZ2Ewxsxaw/discussion
and previous thread linked there.

@nirbheek

This comment has been minimized.

Copy link

@nirbheek nirbheek commented May 24, 2016

Reading through that discussion, it seems that environment variables aren't supported in Ninja because CreateProcess doesn't support setting PATH to find the executable to run when using lpCommandLine instead of lpApplicationName? The thing is, execvpe and execle have the same restriction, so I don't understand what the issue is here.

It's really simple to set the environment inherited by a process on all platforms, so I'd like to understand what the blocker is here. Cheers!

@nirbheek

This comment has been minimized.

Copy link

@nirbheek nirbheek commented May 24, 2016

FWIW, we would really make good use of this feature in Meson: mesonbuild/meson#266, mesonbuild/meson#384, etc.

@nirbheek

This comment has been minimized.

Copy link

@nirbheek nirbheek commented Nov 7, 2016

Would the project be interested in a PR that implements this? It's a troublesome blocker downstream for https://github.com/mesonbuild/meson/ since we have to add wrapper scripts for any command that needs environment variables set, which is slowly turning out to be quite a few.

Even Makefiles and Visual Studio project files support this, so Ninja stands out in not supporting this. :)

We'd be happy to implement this though if there is interest in having this feature in Ninja.

@nico

This comment has been minimized.

Copy link
Collaborator

@nico nico commented Jun 20, 2017

On POSIXy platforms, you can do

rule foo
  command = ENV1=env1 ENV2=env2 my_command

or

rule foo
  command = $env my_command
foo out: in
  env = ENV1=env1 ENV2=env2

Is that not enough for you? Most commands shouldn't need env vars, and generally supporting less stuff keeps ninja simple.

@nirbheek

This comment has been minimized.

Copy link

@nirbheek nirbheek commented Jun 20, 2017

Yes, that part is documented, but Meson is a cross-platform build system (meant to have feature-parity across all supported platforms) and we currently have to use Python script wrappers whenever we need to set environment variables or have arguments with special characters (especially newlines).

At first, we also thought that setting env vars is a niche use-case and pushed back when users asked for it, but a surprisingly large number of people need it. For instance, gobject-introspection in GNOME requires you to set CC/CXX/etc while invoking its tools, otherwise it uses an auto-detection mechanism that ends up using the wrong compiler. "Custom targets" in Meson can run arbitrary commands, and people often show up asking how they can set environment variables while running some tool that they can't change the behaviour of.

The Python script workaround is ok, but invoking the python interpreter for each command really slows things down sometimes.

@nico

This comment has been minimized.

Copy link
Collaborator

@nico nico commented Jun 21, 2017

Maybe gobject-introspection could accept those via command line args instead? (In my experience, builds that rely on env vars instead of flags tend to be much more vulnerable to devs screwing up their local env, etc.)

MSVC's cl.exe requires a few env vars to find includes and whatnot; for that ninja has ninja -t msvc which has a -e flag that can load an environment from a file. So meson could theoretically use that on Windows, and the posix stuff elsewhere.

But using python wrappers for the rare, "needs to be flexible" case and making the common commands work without reliance on the env seems like the best approach to me.

@sam-github

This comment has been minimized.

Copy link

@sam-github sam-github commented Jul 12, 2017

I think I've run into this issue. I'm trying to build a FIPS executable on window, and need to follow these directions:

https://www.openssl.org/docs/fips/UserGuide-2.0.pdf, section 5.3.2:

For the Windows®
 environment a perl script fipslink.pl is provided which performs a
function similar to fipsld for Unix®
/Linux®
. Several environment variables need to be set:
FIPS_LINK is the linker name, normally “link”
FIPS_CC is the C compiler name, normally “cl”
FIPS_CC_ARGS is a string of C compiler arguments for compiling fips_premain.c
PREMAIN_DSO_EXE should be set to the path to fips_premain_dso.exe if a DLL is
being linked (can be omitted otherwise)
PREMAIN_SHA1_EXE is the full path to fips_standalone_sha1.exe
FIPS_TARGET is the path of the target executable or DLL file

Even though FIPS_CC is defined in my environment, it looks like ninja is not passing the env to the child process (perl fipslink.pl ...). I'll see if -e works for me.

@nirbheek

This comment has been minimized.

Copy link

@nirbheek nirbheek commented Jul 20, 2017

Another use-case for this is setting LD_LIBRARY_PATH before running executables built in the project. This is often necessary with executables linked to shared libraries built in the project to ensure that already-existing versions in a custom prefix are not used (-Wl,-rpath doesn't work on distros that set DT_RUNPATH instead of DT_RPATH such as Debian and Ubuntu). Autotools (libtool) uses shell scripts for this. It would be a significant slowdown of the build if we had to spawn a Python interpreter or run a shell script for such cases.

As another datapoint, cmake came across this use-case so much that they have a C wrapper for it which sets the environment and then calls the process. They need it anyway for their make backend, but ninja-only build systems would greatly benefit from this feature.

@evmar

This comment has been minimized.

Copy link
Collaborator Author

@evmar evmar commented Jul 22, 2017

I'm not really sure what this bug is discussing anymore, since it looks like it was originally about updating the docs. :)

For LD_LIBRARY_PATH, setting it via the command line already works as Nico mentioned above. It doesn't involve another process; we must use a shell to parse the command line anyway. (Subprocesses on Linux are oh-so-cheap anyway; you can't even measure /bin/sh -c "echo hello" because it is so fast.)

On Windows we have ninja -t msvc. You might try writing your meson feature using that and then once we have some experience with that we could figure out how to improve it.

@joemfb joemfb mentioned this issue Mar 20, 2018
@marc-h38

This comment has been minimized.

Copy link

@marc-h38 marc-h38 commented Jan 15, 2020

Kconfiglib has a hard $srctree requirement for out of source builds. It uses a few other variables:
https://github.com/ulfalizer/Kconfiglib/blob/424d0d38e7/kconfiglib.py#L108

we currently have to use Python script wrappers whenever we need to set environment variables

From: https://github.com/ulfalizer/Kconfiglib/tree/424d0d38e7be15c5#overview

The entire library is contained in kconfiglib.py. The bundled scripts are implemented on top of it. Implementing your own scripts should be relatively easy, if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.