DYLD_LIBRARY_PATH gets unset when using pyenv activate

[nevsan]

I spent a lot of time tracking down a very complex bug. The behavior is that when I use pyenv activate and then launch python, my DYLD_LIBRARY_PATH environment variable gets erased (actually, any DYLD_* environment variable gets erased). This is not the case if I launch the python executable from ~/.pyenv/versions directly or source the activate script for the virtual environment directly.

What's stranger is that this behavior only exhibits itself on OS X El Capitan, and only in later versions of pyenv-virtualenv. After much debugging, searching, and git bisecting, I found out that commit 64ceee5 in pyenv-virtualenv is what broke things (currently using 'master' from pyenv repo). This change explicitly removed usage of the virtualenv activate scripts (which for me is the only way it doesn't clobber DYLD_*).

The following test script will test for good/bad behavior

eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
pyenv activate test-env
export DYLD_LIBRARY_PATH='test'
python -c 'import os; os.environ["DYLD_LIBRARY_PATH"]'

It turns out that this is due to a new 'feature' in System Integrity Protection. It will delete DYLD_* variables when launching subprocesses. Something in the commit I mentioned is now incompatible with SIP, probably because it launches a child process somewhere.

[kersson]

I worked with @nevsan on this issue. From the SIP docs:

Spawning children processes of processes restricted by System Integrity Protection, such as by launching a helper process in a bundle with NSTask or calling the exec(2) command, resets the Mach special ports of that child process. Any dynamic linker (dyld) environment variables, such as DYLD_LIBRARY_PATH, are purged when launching protected processes.

[yyuu]

@nevsan @kersson Thanks for reproduction procedure! I confirmed the reproduction with latest master, and I also confirmed it doesn't reproduce with v20151103. Actually I'm still not sure what is causing this difference. I'll try digging into this....

[yyuu]

I suspect that the SIP problem happens always when using pyenv's shims script. In 64ceee5, I replaced virtualenv's activate script by some impl in pyenv-sh-activate, and the PATH manipulation will never be done after that.

I prepared some script and I confirmed that DYLD_LIBRARY_PATH is not visible from python invoked via shim script. Because the shims are the heart of pyenv, it's really difficult to deal with.... 😞

#!/usr/bin/env bash
set -e
set -x
WORKDIR="$(mktemp -d "/tmp/${BASH_SOURCE##*/}.XXXXXXXX")"
teardown() {
  rm -fr "${WORKDIR}"
}
trap teardown EXIT

if [ -z "${PYENV_ROOT}" ]; then
  echo "no PYENV_ROOT defined" 1>&2
  exit 1
fi

cp -a "${PYENV_ROOT}" "${WORKDIR}/pyenv"
export PYENV_ROOT="${WORKDIR}/pyenv"
export PYENV_DEBUG=1

# remove non-default plugins
for plugin in "${PYENV_ROOT}/plugins/"*; do
  if [[ "${plugin##*/}" != "python-build" ]]; then
    rm -fr "${plugin}"
  fi
done

export PATH="${PYENV_ROOT}/bin:${PATH}"
eval "$(pyenv init -)"

export DYLD_LIBRARY_PATH="${WORKDIR}"
TEST_VALUE="$(python -c 'from __future__ import print_function;import os;print(os.getenv("DYLD_LIBRARY_PATH"))' || true)"

if [[ "${DYLD_LIBRARY_PATH}" != "${TEST_VALUE}" ]]; then
  echo "${DYLD_LIBRARY_PATH} != ${TEST_VALUE}" 1>&2
  exit 1
fi

[yyuu]

Actually this is a restriction imposed by the operating system rather than a bug of pyenv. At least, this must not be a bug of pyenv-virtualenv.

Basically I have no idea how to deal with this other than just disabling SIP completely at least for now. Because even shims scripts cannot read the environment variables of DYLD_, pyenv cannot deal with this by design.

[kersson]

Unfortunately, I have to agree with you. We need to change our code to not rely on DYLD_LIBRARY_PATH since Apple clearly doesn't want us to 😄 Thanks for your help in investigating this and hopefully you've found it useful to at least be aware of this restriction.

[yyuu]

I'll close this because basically we cannot do nothing for this. Feel free to open related issue on pyenv if there is something feasible workaround in pyenv.

[nevsan]

If anyone needs a workaround, just checkout v20151103 of pyenv-virtualenv, and things will work until you get around to changing your code so that it doesn't rely on DYLD_*.

[yyuu]

As @nevsan mentioned, adding Python's prefix in front of shims PATH might be a workaround for some cases. The variable might be eliminated once the process spawns some child(ren)?, though.