KeyError: 'PROJ_LIB'

[am-thyst]

Just created a whole new environment and had to reinstall everything. The basemap import works fine on a normal terminal (from the exact same python path), but when submitted as a job via qsub it brings the following issue:

from mpl_toolkits.basemap import Basemap, addcyclic

Traceback (most recent call last):
  File "/var/spool/torque/mom_priv/jobs/63539.rdf-xcat.SC", line 21, in <module>
    from mpl_toolkits.basemap import Basemap, addcyclic
  File "/nerc/n02/n02/amethyst/miniconda3/envs/myenv/lib/python3.7/site-packages/mpl_toolkits/basemap/__init__.py", line 155, in <module>
    pyproj_datadir = os.environ['PROJ_LIB']
  File "/nerc/n02/n02/amethyst/miniconda3/envs/myenv/lib/python3.7/os.py", line 678, in __getitem__
    raise KeyError(key) from None
KeyError: 'PROJ_LIB'

[WeatherGod]

Most likely, the environment isn't being set correctly by qsub. It has been awhile since I used it, but there is a command-line option that will let the submitted job run with all the same environment variables as are in the environment that called qsub.

…

On Sat, Oct 13, 2018 at 2:49 PM Amethyst ***@***.***> wrote: Just created a whole new environment and had to reinstall everything. The basemap import works fine on a normal terminal (from the exact same python path), but when submitted as a job via qsub it brings the following issue: from mpl_toolkits.basemap import Basemap, addcyclic Traceback (most recent call last): File "/var/spool/torque/mom_priv/jobs/63539.rdf-xcat.SC", line 21, in <module> from mpl_toolkits.basemap import Basemap, addcyclic File "/nerc/n02/n02/amethyst/miniconda3/envs/myenv/lib/python3.7/site-packages/mpl_toolkits/basemap/__init__.py", line 155, in <module> pyproj_datadir = os.environ['PROJ_LIB'] File "/nerc/n02/n02/amethyst/miniconda3/envs/myenv/lib/python3.7/os.py", line 678, in __getitem__ raise KeyError(key) from NoneKeyError: 'PROJ_LIB' — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#428>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARy-FYFk3wgYSvXX42wcbZRVNIkE7OZks5ukjWjgaJpZM4XaxQ3> .

[am-thyst]

@WeatherGod thanks for this heads up, I just checked a qsub page and tried adding -V to my qsub command. The next error was different:

Traceback (most recent call last):
  File "/var/spool/torque/mom_priv/jobs/63543.rdf-xcat.SC", line 21, in <module>
    from mpl_toolkits.basemap import Basemap, addcyclic
  File "/nerc/n02/n02/amethyst/miniconda3/envs/myenv/lib/python3.7/site-packages/mpl_toolkits/basemap/__init__.py", line 156, in <module>
    epsgf = open(os.path.join(pyproj_datadir,'epsg'))
FileNotFoundError: [Errno 2] No such file or directory: 'PROJ_LIB/epsg'

[QuLogic]

I'm not sure that did what you thought it might. It's probably easier if in your command script, you activate the environment there instead of relying on qsub to pass along the right stuff.

[am-thyst]

@QuLogic do you mean in the job script? I start it off with #!path/to/python/in/my/env, which seemed to work with my other environment

[QuLogic]

Yes, I don't think that is safe with how packages in conda-forge work. You really need to activate the environment to ensure that environment variables are set correctly. So it would be best if the job script were /bin/bash, activated the environment, and then ran the Python script you really wanted to run.

[WeatherGod]

I find it odd that the name of the directory it was looking for was "PROJ_LIB". When I see stuff like that, I wonder if I missed a dollar sign somewhere and what was supposed to be $PROJ_LIB was written as PROJ_LIB.

…

On Sat, Oct 13, 2018 at 4:19 PM Elliott Sales de Andrade < ***@***.***> wrote: Yes, I don't think that is safe with how packages in conda-forge work. You really need to activate the environment to ensure that environment variables are set correctly. So it would be best if the job script were /bin/bash, activated the environment, and then ran the Python script you really wanted to run. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#428 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARy-OlnC-aqRdwIoIrYLZ4nhvMdy6ZOks5ukkq6gaJpZM4XaxQ3> .

[am-thyst]

@QuLogic thanks for your suggestion, I've made some changes and I'm still getting the same key error. @WeatherGod surely I shouldn't have to delve into the basemap code? I've checked my version, I'm on basemap 1.2.0 with all requirements satisfied.

Tried the classic uninstall and reinstall, to no avail

[ImportanceOfBeingErnest]

I had a similar problem when installing basemap into an existing conda environment via conda install -n myenv basemap=1.2 the other day. Unfortunately I can't remember if the missing environment variable was called PROJ_LIB or something else. I solved this by uninstalling basemap (but not the other dependencies it created), installing an old wheel file I had on disc for basemap 1.1, then installing basemap 1.2 on top of it.
Also I now cannot reproduce this with a fresh environment any more. Still, it might be interesting for you to know.

[am-thyst]

@ImportanceOfBeingErnest thanks for the suggestion, installed basemap 1.1 instead and got the same error message. Weird.

[statiksof]

#419 Is not the same issue?

[am-thyst]

@statiksof thanks for this - looks like I opened this in the wrong place, too. I'll close this and work through the other threads.