Inconsistent use of system PYTHONPATH in completions and IPython Console

[mrclary]

Statement of Problem

If a user has a system PYTHONPATH environment variable defined (e.g. in ~/.bash_profile on unix, or Environment Variables on Windows), then Spyder inconsistently handles it in the context of completions in the editor, IPython Console's os.environ['PYTHONPATH'], and IPython Console's sys.path. The following table summarizes the different behavior based on platform and Spyder launch mechanism.

• "Completions" indicates whether the system PYTHONPATH paths are included in Jedi's completion search paths
• "os.environ" indicates whether the system PYTHONPATH paths are included in the IPython Console's os.environ['PYTHONPATH']
• "sys.path" indicates whether the system PYTHONPATH paths are included in the IPython Console's sys.path

Platform	Launch	Interpreter	Completions	os.environ PYTHONPATH	sys.path
macOS	bootstrap	internal	Y	N*	N
macOS	bootstrap	external	Y	N*	N
macOS	conda	internal	Y	Y	Y
macOS	conda	external	N	Y	Y
macOS	app	internal	Y	Y	Y
macOS	app	external	N	N	N
Win	bootstrap	internal	Y	N*	N
Win	bootstrap	external	N	N*	N
Win	conda	internal	N	Y	Y
Win	conda	external	N	Y	Y
Win	app	internal	N	Y^	N
Win	app	external	N	N	N

* This is due to os.environ['PYTHONPATH'] being entirely replaced by spyder-kernels under DEV and should be resolved by #17408
^ In the case of Windows application with internal interpreter for the IPython Console, the internal Spyder pkgs directory is also in os.environ['PYTHONPATH']

Discussion

@spyder-ide/core-developers , what is the desired treatment of the system PYTHONPATH environment variable with regard to completions in the editor and the IPython Console's os.environ['PYTHONPATH'] and sys.path?

My opinion is that the system PYTHONPATH environment variable should be treated the same across platforms and launch mechanisms: either included in/excluded from each of completions paths, IPython Console's os.environ['PYTHONPATH'], and IPython Console's sys.path.

I propose the following.

1. Exclude the system PYTHONPATH paths from completions paths, IPython Console's os.environ['PYTHONPATH'], and IPython Console's sys.path.
2. Provide a mechanism whereby the user can import the system PYTHONPATH paths into the PYTHONPATH Manager.
3. (bonus points) Provide a mechanism whereby the user can export the PYTHONPATH Manager paths to the system PYTHONPATH environment variable.

I think that item 1 is important because the user should not have "hidden" paths affecting completions in the Editor or executing code in the IPython Console. All relevant paths should be defined by the PYTHONPATH Manager and the executable environment's site.py module.

However, the system PYTHONPATH environment variable may be useful to users. By providing the mechanism in item 2, we provide access to those paths while maintaining transparency (no "hidden" paths). Additionally, the full power to arrange or enable/disable paths is available to the user.

Spyder already provides item 3 for the Windows platform, but it is lacking for the Unix platforms (macOS and Linux). This feature can be added to these platforms as well by cleverly handling shell source scripts (e.g. .bash_profile). This is certainly more complicated for Unix platforms than for Windows, but I think it may be possible to achieve in a safe and stable way.

[mrclary]

Another consideration regarding os.environ['PYTHONPATH'] in IPython Console is that this is the mechanism for transporting paths to subprocesses. Currently, subprocesses in IPython Console (user code calls Popen or the like) will inherit the same inconsistencies shown in the table above for "os.environ PYTHONPATH".
Is it possible to update the Console's os.environ['PYTHONPATH'] via the same mechanism that we update the Console's sys.path?