Ability to use PYTHONPATH from user bash_profile in macOS application

[mrclary]

It would be nice to be able to import the PYTHONPATH from a user's bash_profile to replace/append to Spyder's PYTHONPATH Manager.

The issue is that many users may require setting PYTHONPATH in their bash_profile for other purposes, but the macOS application cannot make use of this, thus users may be required to manage PYTHONPATH in two locations: in bash_profile as well as in Spyder's PYTHONPATH Manager.

[steff456]

Hi @mrclary,

Yes! This is a really good idea. Have you tested this option to see if this is working with the mac app?

[mrclary]

I haven't tested it yet, though I think it should be fairly easy to incorporate.

However, now I'm wondering if this shouldn't be part of a larger improvement to add the ability to set/get other environment variables for the Consoles, not just PYTHONPATH. For example, similar to the PYTHONPATH Manager, a Console Environment Manager that gives the user the option to "import" environment variables from their .bash_profile and select to include/exclude individual variables, as well as the ability to manually add them.

For background, my company has systems setup that require data analysts to have certain environment variables set (~100 variables!), including PYTHONPATH. Conventionally, these are all set and managed in the user's .bash_profile and will be used by other applications in addition to Python development in Spyder. With the advent of Spyder's macOS application, these are not available for the IPython Console to inherit, which presents a couple of inconveniences.

• We have to use the startup script feature provided by Spyder in order to grab the necessary environment variables and update os.environ.
• The PYTHONPATH environment variable set in .bash_profile is not visible to Spyder, so it cannot be used for command completion in the Editor. This means that we have to use the PYTHONPATH Manager, and we have to manage the PYTHONPATH in two separate places. While the startup script is sufficient to provide the PYTHONPATH for the IPython Console (and command completion therein), only the PYTHONPATH Manager can be used for command completion in the Editor.

To retrieve the environment variables is fairly straightforward:

import os
import subprocess as sp

out, err = sp.Popen('[[ -e /etc/profile ]] && source /etc/profile; '
                    '[[ -e ~/.bash_profile ]] && source ~/.bash_profile; '
                    'printenv',
                    shell=True, stdout=sp.PIPE, stderr=sp.PIPE).communicate()
variables = {}
for kv in out.decode().strip().split('\n'):
    k, v = kv.split('=')
    variables.update({k: v})

[mrclary]

I don't know how environment variables are handled on Windows systems, or whether the Spyder executable has any of the same issues as the macOS application. So consider the previous comment as a brainstorm.

[steff456]

This looks right, can you please test it and let me know if it works as expected? Right now probably the information will not appear in Spyder's python path manager, but you can test it in the consoles.

If it works, could you help us implement this in two PRs?

1. The first PR will be to add support to show the python path in the python path manager, because right now I think it is not displayed there.
2. The second PR will read the bash_profile and will apply the python path in the python path manager.

Please be careful because if any weird site_packages is added it might corrupt the consoles or weird things may happen. At the end as we cannot filter what is inside the bash_profile making it risky.

[mrclary]

@steff456, I don't think I understand your comment.

If we limit the scope to just PYTHONPATH, then I can extract that from the .bash_profile and combine with the list in the PYTHONPATH Manager. This will automatically be sent to consoles upon update. There is no dealing with site_packages; these entries in the PYTHONPATH are never handled by .bash_profile or Spyder, but by the site.py script built into any python distribution. Python always adds these to whatever PYTHONPATH is supplied to it when it is invoked.

We cannot control what is in the .bash_profile, this is true. However, it is invoked anytime a user opens a terminal, so there should not be any security risk there. What is returned by the subprocess, though, is just a string of the environment variables. This could be filtered at the subprocess to return only the PYTHONPATH, rather than using printenv.
We also cannot control what is in the PYTHONPATH returned there, but we can run checks that the paths exist.

I will submit a PR, though and we can go from there.

[steff456]

Perfect! I know weird things happen if site_packages is added, but now I understand your point and I believe it must be safe to do it. Thanks for the explanation, and thanks for working in this :)

[mrclary]

@steff456, What should the user experience be?
I listed some considerations here.

Another possibility is that this feature and PYTHONPATH Manager could be mutually exclusive.

• A preference either in the IPython console->Startup or Python interpreter section of Spyder preferences. This determines whether PYTHONPATH is taken from PYTHONPATH Manager or from the user's .bash_profile.

Or maybe this preference determines whether PYTHONPATH from .bash_profile is always merged into PYTHONPATH upon IPython Console startup, bypassing the PYTHONPATH Manager.

What do you think?

[mrclary]

@steff456 @ccordoba12, I think I prefer #14918 over #14853.

• Users that launch Spyder from a conda terminal will already inherit environment variables and PYTHONPATH.
• Standalone application users will have the option to enable/disable the feature.
• It does not alter the behavior of the PYTHONPATH Manager.

[mrclary]

@steff456, should this issue also be moved over to the ux repo for discussion?