Mac OS - Execute in external system terminal not working

[vancromy]

Issue Report Checklist

• Searched the issues page for similar reports
• Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
• Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
• Could not reproduce inside jupyter qtconsole (if console-related)
• Tried basic troubleshooting (if a bug/error)
  • Restarted Spyder
  • Reset preferences with spyder --reset
  • Reinstalled the latest version of Anaconda
  • Tried the other applicable steps from the Troubleshooting Guide
• Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description

Execution in an external system terminal not working for macOS Big Sur & Spyder 5.1.1 (standalone Mac Installer).

I'm working with miniconda environments and have changed python interpreter to the correct environment. Running the code in a dedicated or open console works just fine.

What steps reproduce the problem?

1. Go to Preferences
2. Under Run -> Console -> Select Execute in an external system terminal
3. Run code
4. External system terminal opens but get a fatal Python error

What is the expected output? What do you see instead?

Running code in external system terminals works just like running in console within Spyder

Paste Traceback/Error Below (if applicable)

PASTE TRACEBACK HERE

Versions

• Spyder version: 5.1.1
• Python version: Python 3.9.5 64-bit
• Qt version: 5.12.10
• PyQt version: 5.12.3
• Operating System name/version: Darwin 20.5.0

Dependencies

PASTE DEPENDENCIES HERE

[ccordoba12]

Hey @vancromy, thanks for reporting. We haven't tested this functionality with our Mac installer, so it could be broken indeed.

@mrclary, are you able to reproduce this?

[mrclary]

I get a slightly different traceback, but I can reproduce this error.
However, this only occurs if Python Interpreter is set to "Default (i.e. same as Spyder's)". If I use a different environment (e.g. a conda environment), then I can execute successfully in an external terminal.

Same as Spyder Env

Last login: Tue Aug 10 21:39:03 on ttys002
/var/folders/5v/28jqvwxs2cd5fj93gvwnykdrqc926z/T/spyder-rclary/run_spyder_6hvyy2_z.sh ; exit;

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
>> /var/folders/5v/28jqvwxs2cd5fj93gvwnykdrqc926z/T/spyder-rclary/run_spyder_6hvyy2_z.sh ; exit;
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = '/usr/local/mdsplus/pydevices:/Users/rclary/Documents/Python/mypython:/Users/rclary/Documents/Python/c2w:/Users/rclary/Documents/Python/c2w_psp:/Users/rclary/Documents/Python/dap_jscopes:/Users/rclary/Documents/Python/vessel:/Users/rclary/Documents/Python/msdb-csv:/usr/local/mdsplus/python'
  program name = '/Users/rclary/Applications/Spyder.app/Contents/MacOS/python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/Users/rclary/Applications/Spyder.app/Contents/MacOS/python'
  sys.base_prefix = '/Users/runner/hostedtoolcache/Python/3.9.5/x64'
  sys.base_exec_prefix = '/Users/runner/hostedtoolcache/Python/3.9.5/x64'
  sys.platlibdir = 'lib'
  sys.executable = '/Users/rclary/Applications/Spyder.app/Contents/MacOS/python'
  sys.prefix = '/Users/runner/hostedtoolcache/Python/3.9.5/x64'
  sys.exec_prefix = '/Users/runner/hostedtoolcache/Python/3.9.5/x64'
  sys.path = [
    '/usr/local/mdsplus/pydevices',
    '/Users/rclary/Documents/Python/mypython',
    '/Users/rclary/Documents/Python/c2w',
    '/Users/rclary/Documents/Python/c2w_psp',
    '/Users/rclary/Documents/Python/dap_jscopes',
    '/Users/rclary/Documents/Python/vessel',
    '/Users/rclary/Documents/Python/msdb-csv',
    '/usr/local/mdsplus/python',
    '/Users/runner/hostedtoolcache/Python/3.9.5/x64/lib/python39.zip',
    '/Users/runner/hostedtoolcache/Python/3.9.5/x64/lib/python3.9',
    '/Users/runner/hostedtoolcache/Python/3.9.5/x64/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x0000000115c5ae00 (most recent call first):
<no Python frame>
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

[Process completed]

Conda Env

Last login: Tue Aug 10 21:40:13 on ttys002
/var/folders/5v/28jqvwxs2cd5fj93gvwnykdrqc926z/T/spyder-rclary/run_spyder_csral3ro.sh ; exit;

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
>> /var/folders/5v/28jqvwxs2cd5fj93gvwnykdrqc926z/T/spyder-rclary/run_spyder_csral3ro.sh ; exit;
P(A1) = 0.0132; P(A2) = 0.0529; P(A3) = 0.0661; P(A4) = 0.8677; Total = 1.0000
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

[Process completed]

What is interesting to me is that my error shows no PYTHONHOME set. I would think that this is the reason for the problem, except that the OP error shows the correct PYTHONHOME set.

[mrclary]

I can poke around and see what the issue might be...

[vancromy]

I get a slightly different traceback, but I can reproduce this error.

However, this only occurs if Python Interpreter is set to "Default (i.e. same as Spyder's)". If I use a different environment (e.g. a conda environment), then I can execute successfully in an external terminal.

Same as Spyder Env

Last login: Tue Aug 10 21:39:03 on ttys002

/var/folders/5v/28jqvwxs2cd5fj93gvwnykdrqc926z/T/spyder-rclary/run_spyder_6hvyy2_z.sh ; exit;



The default interactive shell is now zsh.

To update your account to use zsh, please run `chsh -s /bin/zsh`.

For more details, please visit https://support.apple.com/kb/HT208050.

>> /var/folders/5v/28jqvwxs2cd5fj93gvwnykdrqc926z/T/spyder-rclary/run_spyder_6hvyy2_z.sh ; exit;

Could not find platform independent libraries <prefix>

Could not find platform dependent libraries <exec_prefix>

Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]

Python path configuration:

  PYTHONHOME = (not set)

  PYTHONPATH = '/usr/local/mdsplus/pydevices:/Users/rclary/Documents/Python/mypython:/Users/rclary/Documents/Python/c2w:/Users/rclary/Documents/Python/c2w_psp:/Users/rclary/Documents/Python/dap_jscopes:/Users/rclary/Documents/Python/vessel:/Users/rclary/Documents/Python/msdb-csv:/usr/local/mdsplus/python'

  program name = '/Users/rclary/Applications/Spyder.app/Contents/MacOS/python'

  isolated = 0

  environment = 1

  user site = 1

  import site = 1

  sys._base_executable = '/Users/rclary/Applications/Spyder.app/Contents/MacOS/python'

  sys.base_prefix = '/Users/runner/hostedtoolcache/Python/3.9.5/x64'

  sys.base_exec_prefix = '/Users/runner/hostedtoolcache/Python/3.9.5/x64'

  sys.platlibdir = 'lib'

  sys.executable = '/Users/rclary/Applications/Spyder.app/Contents/MacOS/python'

  sys.prefix = '/Users/runner/hostedtoolcache/Python/3.9.5/x64'

  sys.exec_prefix = '/Users/runner/hostedtoolcache/Python/3.9.5/x64'

  sys.path = [

    '/usr/local/mdsplus/pydevices',

    '/Users/rclary/Documents/Python/mypython',

    '/Users/rclary/Documents/Python/c2w',

    '/Users/rclary/Documents/Python/c2w_psp',

    '/Users/rclary/Documents/Python/dap_jscopes',

    '/Users/rclary/Documents/Python/vessel',

    '/Users/rclary/Documents/Python/msdb-csv',

    '/usr/local/mdsplus/python',

    '/Users/runner/hostedtoolcache/Python/3.9.5/x64/lib/python39.zip',

    '/Users/runner/hostedtoolcache/Python/3.9.5/x64/lib/python3.9',

    '/Users/runner/hostedtoolcache/Python/3.9.5/x64/lib/lib-dynload',

  ]

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding

Python runtime state: core initialized

ModuleNotFoundError: No module named 'encodings'



Current thread 0x0000000115c5ae00 (most recent call first):

<no Python frame>

logout

Saving session...

...copying shared history...

...saving history...truncating history files...

...completed.



[Process completed]

Conda Env

Last login: Tue Aug 10 21:40:13 on ttys002

/var/folders/5v/28jqvwxs2cd5fj93gvwnykdrqc926z/T/spyder-rclary/run_spyder_csral3ro.sh ; exit;



The default interactive shell is now zsh.

To update your account to use zsh, please run `chsh -s /bin/zsh`.

For more details, please visit https://support.apple.com/kb/HT208050.

>> /var/folders/5v/28jqvwxs2cd5fj93gvwnykdrqc926z/T/spyder-rclary/run_spyder_csral3ro.sh ; exit;

P(A1) = 0.0132; P(A2) = 0.0529; P(A3) = 0.0661; P(A4) = 0.8677; Total = 1.0000

logout

Saving session...

...copying shared history...

...saving history...truncating history files...

...completed.



[Process completed]

What is interesting to me is that my error shows no PYTHONHOME set. I would think that this is the reason for the problem, except that the OP error shows the correct PYTHONHOME set.

Thanks for the quick reply @mrclary. For me I'm experiencing the behaviour in a conda env. I'm happy to help you with troubleshooting if there's any logs and things you'd like me to try on my side. :)

[mrclary]

Thanks for the quick reply @mrclary. For me I'm experiencing the behaviour in a conda env. I'm happy to help you with troubleshooting if there's any logs and things you'd like me to try on my side. :)

Do you mean when you set Preferences->Python interpreter to use a Conda environment, you still get the error?

[vancromy]

Thanks for the quick reply @mrclary. For me I'm experiencing the behaviour in a conda env. I'm happy to help you with troubleshooting if there's any logs and things you'd like me to try on my side. :)

Do you mean when you set Preferences->Python interpreter to use a Conda environment, you still get the error?

Yes that's right. Here's a screenshot to show that:

[mrclary]

I think that #16199 may resolve the issue.
However, this should have been an issue for any version of macOS, not just Big Sur.

Here is what I've found:

1. If Terminal.app is already running and was launched by the user (not Spyder)
   • External system terminal succeeds if an external interpreter is used
   • External system terminal fails if the default interpreter is used (same as Spyder)
2. If Terminal.app is not running
   • External system terminal fails if an external interpreter is used
   • External system terminal succeeds if the default interpreter is used (same as Spyder)

The reason for the above behavior is that if the Terminal.app is launched by the user, the PYTHONHOME environment variable will not be inherited when Spyder attempts to launch Termina.app using a subprocess (because it is already running). This results in a failure when using the default interpreter, which must have the PYTHONHOME environment variable defined properly.
Conversely, if the Terminal.app is launched by Spyder, then the PYTHONHOME environment variable is inherited (the default behavior of Popen is to inherit os.environ). But this results in the a failure when using an external interpreter because an external interpreter cannot have PYTHONPATH PYTHONHOME defined.