Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LD_LIBRARY_PATH and DYLD_LIBRARY_PATH not imported on OS X #1523

Open
currymj opened this issue Mar 3, 2017 · 8 comments
Labels

Comments

@currymj
Copy link

@currymj currymj commented Mar 3, 2017

I tried to import GPU Tensorflow; it failed to find the CUDA libraries. I checked the process.env environment variable and found that LD_LIBRARY_PATH and DYLD_LIBRARY_PATH had not been added. After further digging, I found that the shell-env package doesn't import them because it just calls the env command, which doesn't include them in its output for some reason.

Manually going to the console, adding the paths to process.env, and restarting the kernel fixed the problem.

I'm not sure what the solution is here. Either import these separately, allow them to be configured somewhere, or something else? I think there are enough people who want to call CUDA from Python that it's probably worth doing though.

@rgbkrk rgbkrk added macOS GPU labels Mar 3, 2017
@rgbkrk

This comment has been minimized.

Copy link
Member

@rgbkrk rgbkrk commented Mar 3, 2017

We definitely need to figure out why those aren't loaded properly.

Are you setting LD_LIBRARY_PATH in your startup scripts for your shell (.bashrc)?

One annoyance on OS X is that it invokes Applications with launchctl and does not create a shell for you. The shell-env was put in place to work around this issue by spawning a shell for the user.

I'm installing Tensorflow on my mac now, I've only used it on remote systems prior to this.

@lgeiger

This comment has been minimized.

Copy link
Member

@lgeiger lgeiger commented Mar 3, 2017

New versions of OS X enable system integrity protection per default. Meaning that setting the DYLD_LIBRARY_PATH and LD_LIBRARY_PATH will have no effects.

I had this issue before in another context, but it looks like it's a common issue on other Node modules as well: oracle/node-oracledb#231

Here is a guide to turn of SIP: http://osxdaily.com/2015/10/05/disable-rootless-system-integrity-protection-mac-os-x/

@currymj

This comment has been minimized.

Copy link
Author

@currymj currymj commented Mar 3, 2017

I'm doing it in .bash_profile; I put it in .profile as well just to see if it would work.

I've definitely run into the problem of environment variables in non-command-line OS X apps before. But in this case, all other environment variables are loaded fine! I think the root of the problem is that for some reason, the env shell command (which shell-env relies on) doesn't include LD_LIBRARY_PATH or DYLD_LIBRARY_PATH. If there's a good place to just shell out, echo $LD_LIBRARY_PATH, and separately add it, point me at it and I'd be happy to throw together a PR.

I'm confused by the SIP issue, because I have it enabled but 1) everything works fine on command line or in jupyter notebook invoked from there, I'm able to set those variables and tensorflow will find the CUDA libraries; 2) when I manually add the missing fields to process.env everything in nteract works perfectly.

EDIT:

rather than turning off SIP, might it be possible to just allow users to configure nteract to add these fields to process.env?

@rgbkrk

This comment has been minimized.

Copy link
Member

@rgbkrk rgbkrk commented Mar 3, 2017

Side question so that I can experience this - what kind of Mac do you have, what GPU do you use? My Macbook Pro (2015 model) only has the built in Intel GPU, and the newer Macs use AMD GPUs.

As for if you're able to extract the LD_LIBRARY_PATH, my first thought is adding to

.fromPromise(shellEnv())

Off the cuff, I'm guessing it would look something like:

import shellEnv from "shell-env";
import Rx from "rxjs/Rx";

const spawn = require("spawn-rx").spawn;

/**
 * Observable of environment variable mappings
 * @param  {string} name environment variable name e.g. LD_LIBRARY_PATH
 * @return {Object}      { [name]: env_var_value }
 */
function extractEnvVariable(name: string) {
  return spawn("bash", ["-c", `echo $${name}`]).map(
    value => value === "" ? {} : { [name]: value }
  );
}

let env$ = Rx.Observable.fromPromise(shellEnv());

// Possibly a workaround
if (process.platform === "darwin") {
  env$ = Rx.Observable
    .merge(
      env$,
      extractEnvVariable("LD_LIBRARY_PATH"),
      extractEnvVariable("DYLD_LIBRARY_PATH")
    )
    .reduce((env, moreEnv) => Object.assign({}, env, moreEnv));
}

const createEnv$ = env$
  .first()
  .do(env => {
    Object.assign(process.env, env);
  })
  .publishReplay(1);

createEnv$.connect();

export default createEnv$;

However, if I had to guess we probably need to look into how the kernel gets spawned (the above code is for locating kernels and other necessary paths).

@currymj

This comment has been minimized.

Copy link
Author

@currymj currymj commented Mar 3, 2017

15 inch Retina, mid 2014, NVIDIA GeForce GT 750M graphics card. i think this may be a red herring though, b/c everything works fine once the environment variables are set. i'm inferring things in tensorflow right now! (the reason i bring this up is that getting everything compatible among CUDA, Xcode, clang, gcc, Python, and Tensorflow can be extremely frustrating.)

now i've found that manually editing os.environ in python before importing tensorflow also works.

Looking at this, I think that SIP might indeed be the problem; also extractEnvVariable above might not work.

~ curry$ echo $LD_LIBRARY_PATH
/Users/curry/torch/install/lib::/usr/local/cuda/lib
~ curry$ bash
bash-3.2$ echo $LD_LIBRARY_PATH

bash-3.2$ 
@bimsapi

This comment has been minimized.

Copy link

@bimsapi bimsapi commented May 15, 2017

For what it's worth, rather than disable SIP, I've resorted to linking the .dylib objects into /usr/local/lib. E.g., ln -s $ORACLE_HOME/*.dylib* /usr/local/lib

@stale

This comment has been minimized.

Copy link

@stale stale bot commented Apr 13, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Apr 13, 2018
@rgbkrk

This comment has been minimized.

Copy link
Member

@rgbkrk rgbkrk commented Apr 13, 2018

This seems like an issue we should move into some docs. Is anyone game for making a page encapsulating this?

@stale stale bot removed the stale label Apr 13, 2018
@rgbkrk rgbkrk added the docs label Apr 13, 2018
stephengroat added a commit to stephengroat/capstone that referenced this issue May 24, 2018
aquynh added a commit to aquynh/capstone that referenced this issue May 25, 2018
* Cleanup build process

avoiding bash subshells (which happen in for loops) because they
like to selectively inherit environment variables

* address restrictions of osx sip

see nteract/nteract#1523 (comment)
@stale stale bot added the stale label Feb 7, 2019
@nteract nteract deleted a comment from stale bot Feb 10, 2019
@stale stale bot removed the stale label Feb 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.