Interpreter hangs on macOS when executing any shell command

[appenz]

Describe the bug

Interpreter always hangs on macOS when executing shell commands which makes it practically unusable as it disables both shell commands and apple script. Bug may be the same as #556 or #557 , but as both are old, opening a new one. Tested with different shells and different versions of Python on macOS.

Reproduce

On macOS 13.6.2 Ventura running on an M1 MacBook Pro:

• Install python via brew (tested with 3.7 and 3.12)
• Install open-interpreter with pip
• Configure interpreter to use gpt-4-turbo and run "interpreter -y"
• Ask "list the files in my home directory." and confirm
• Interpreter attempts to execute "ls ~" but hangs
• CTRL-C aborts the attempt, error below.

Expected behavior

No hanging.

Screenshots

list the files in my home directory.

Plan:

1 Execute a shell command to list the files in your home directory.

Now, I will execute the shell command to list the files.

Do it.

[EDIT: At this point it hangs, I have to interrupw tih CTRL-C]

ls ~

Traceback (most recent call last):
File "/opt/homebrew/lib/python3.11/site-packages/interpreter/core/respond.py", line 205, in respond
for line in interpreter.computer.run(language, code):
File "/opt/homebrew/lib/python3.11/site-packages/interpreter/core/computer/terminal/terminal.py", line 40,
in run
for chunk in self._active_languages.run(code):
File
"/opt/homebrew/lib/python3.11/site-packages/interpreter/core/computer/terminal/languages/subprocess_language.
py", line 128, in run
output = self.output_queue.get(timeout=0.3) # Waits for 0.3 seconds
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/homebrew/Cellar/python@3.11/3.11.7/Frameworks/Python.framework/Versions/3.11/lib/python3.11/queue.py",
line 180, in get
self.not_empty.wait(remaining)
File
"/opt/homebrew/Cellar/python@3.11/3.11.7/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.p
y", line 331, in wait
gotit = waiter.acquire(True, timeout)

Open Interpreter version

0.2.0

Python version

3.12.1 (also tested with 3.7 with identical results)

Operating System name and version

macOS 13.6.2 Ventura

Additional context

Also tested zsh, bash and fish. All shells produce the same result.

[Notnaton]

To install Open Interpreter, you’ll need Python 3.10 or 3.11. Run python --version to check your version.

https://docs.openinterpreter.com/getting-started/setup

[appenz]

Re-tested with Python 3.11 with identical results. Full output below.

gappenzeller@GAppenzellerMBP14 /o/h/bin (stable)> interpreter -y

show a list of the files in the /tmp directory.

ls /tmp

Traceback (most recent call last):
File "/opt/homebrew/lib/python3.11/site-packages/interpreter/core/respond.py", line 205, in respond
for line in interpreter.computer.run(language, code):
File "/opt/homebrew/lib/python3.11/site-packages/interpreter/core/computer/terminal/terminal.py", line 40, in run
for chunk in self._active_languages.run(code):
File "/opt/homebrew/lib/python3.11/site-packages/interpreter/core/computer/terminal/languages/subprocess_language.py", line 128, in
run
output = self.output_queue.get(timeout=0.3) # Waits for 0.3 seconds
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.7/Frameworks/Python.framework/Versions/3.11/lib/python3.11/queue.py", line 180, in get
self.not_empty.wait(remaining)
File "/opt/homebrew/Cellar/python@3.11/3.11.7/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 331, in
wait
gotit = waiter.acquire(True, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

[Notnaton]

Can you run:

interpreter --debug

and paste the output here?
Make sure to remove any api-keys should they appear...

[ibehnam]

Can you run:

interpreter --debug

and paste the output here? Make sure to remove any api-keys should they appear...

interpreter --debug                                                                                      (oi) 21:47:22
usage: interpreter [-h] [-ci CUSTOM_INSTRUCTIONS] [-s SYSTEM_MESSAGE] [-y] [-v] [-m MODEL] [-t TEMPERATURE] [-lsv]
                   [-lsf] [-c CONTEXT_WINDOW] [-x MAX_TOKENS] [-b MAX_BUDGET] [-ab API_BASE] [-ak API_KEY]
                   [-av API_VERSION] [-xo MAX_OUTPUT] [-fc] [-dt] [-o] [-sm] [-safe {off,ask,auto}] [-cf CONFIG_FILE]
                   [-f] [-l] [-vi] [-os] [--config] [--reset_config] [--conversations] [--version]
interpreter: error: unrecognized arguments: --debug

I have the same problem as the OP. Python=3.11.

[hewigovens]

@Notnaton @ibehnam it should be -v now: --debug_mode / -d has been renamed to --verbose / -v.

[ibehnam]

@Notnaton @ibehnam it should be -v now: --debug_mode / -d has been renamed to --verbose / -v.

Yeah this showed the debug messages but all of them stop as soon as OI is asked to execute the AppleScript command. This is the last message I see (I have to CTRL-C twice to close OI; when I do CTRL-C once, I see the error message below and OI starts talking about the fact that the operation was not successful, so I must CTRL-C again.

  Traceback (most recent call last):
    File "/Users/behnam/opt/anaconda3/envs/oi/lib/python3.11/site-packages/interpreter/core/respond.py", line 205, in
  respond
      for line in interpreter.computer.run(language, code):
    File
  "/Users/behnam/opt/anaconda3/envs/oi/lib/python3.11/site-packages/interpreter/core/computer/terminal/terminal.py",
  line 40, in run
      for chunk in self._active_languages.run(code):
    File
  "/Users/behnam/opt/anaconda3/envs/oi/lib/python3.11/site-packages/interpreter/core/computer/terminal/languages/subpro
  cess_language.py", line 128, in run
      output = self.output_queue.get(timeout=0.3)  # Waits for 0.3 seconds
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/behnam/opt/anaconda3/envs/oi/lib/python3.11/queue.py", line 180, in get
      self.not_empty.wait(remaining)
    File "/Users/behnam/opt/anaconda3/envs/oi/lib/python3.11/threading.py", line 331, in wait
      gotit = waiter.acquire(True, timeout)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

OI's message after this (I CTRL-C-interrupted it):

  The traceback you've posted indicates that there was aKeyboardInterrupt which is...

[appenz]

Ran it with -v -y. Output is about 100k lines and non-trivial to scrub. Below the lines before/after the shell call. I agree with @ibehnam that there isn't much to work with here.

[...]
completion obj content: It
model_response: ModelResponse(id='chatcmpl-e2ef1d19-5cc9-4a65-a769-f834e7e58db2', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=None, role=None))], created=1705030918, model='gpt-4-1106-preview',
object='chat.completion.chunk', system_fingerprint=None, usage=Usage()); completion_obj: {'content': 'It'}
model_response finish reason 3: None
hold - False, model_response_str - It
model_response: ModelResponse(id='chatcmpl-8g2ibbmwfYeRb6HDvgxw2wm6yLNyS', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(tool_calls=None, function_call=None, content='It', role='assistant'))], created=1705030918,
model='gpt-4-1106-preview', object='chat.completion.chunk', system_fingerprint='fp_168383a679', usage=Usage())
PROCESSED CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-8g2ibbmwfYeRb6HDvgxw2wm6yLNyS', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(tool_calls=None, function_call=None, content='It', role='assistant'))],
created=1705030918, model='gpt-4-1106-preview', object='chat.completion.chunk', system_fingerprint='fp_168383a679', usage=Usage())
Logging Details LiteLLM-Success Call
Chunk in terminal_interface: {'role': 'computer', 'type': 'console', 'end': True}
success callbacks: []

ls /tmp

Traceback (most recent call last):
File "/opt/homebrew/lib/python3.11/site-packages/interpreter/core/respond.py", line 205, in respond
for line in interpreter.computer.run(language, code):
File "/opt/homebrew/lib/python3.11/site-packages/interpreter/core/computer/terminal/terminal.py", line 40, in run
for chunk in self._active_languages.run(code):
File "/opt/homebrew/lib/python3.11/site-packages/interpreter/core/computer/terminal/languages/subprocess_language.py", line 128, in run
output = self.output_queue.get(timeout=0.3) # Waits for 0.3 seconds
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.7/Frameworks/Python.framework/Versions/3.11/lib/python3.11/queue.py", line 180, in get
self.not_empty.wait(remaining)
File "/opt/homebrew/Cellar/python@3.11/3.11.7/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 331, in wait
gotit = waiter.acquire(True, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

Chunk in terminal_interface: {'role': 'assistant', 'type': 'message', 'start': True}
Chunk in terminal_interface: {'role': 'assistant', 'type': 'message', 'content': 'It'}
value of chunk: ChatCompletionChunk(id='chatcmpl-8g2ibbmwfYeRb6HDvgxw2wm6yLNyS', choices=[Choice(delta=ChoiceDelta(content=' seems', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)],
created=1705030917, model='gpt-4-1106-preview', object='chat.completion.chunk', system_fingerprint='fp_168383a679')
PROCESSED CHUNK PRE CHUNK CREATOR: ChatCompletionChunk(id='chatcmpl-8g2ibbmwfYeRb6HDvgxw2wm6yLNyS', choices=[Choice(delta=ChoiceDelta(content=' seems', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0,
logprobs=None)], created=1705030917, model='gpt-4-1106-preview', object='chat.completion.chunk', system_fingerprint='fp_168383a679')

[antonme]

I have exactly the same bug. Everything hangs on any shell exec

[Azunyan1111]

I couldn't get the shell to work, but sudo interpreter got it working, just FYI.

[leo4life2]

same issue.

[leo4life2]

I couldn't get the shell to work, but sudo interpreter got it working, just FYI.

adding sudo doesn't fix it for me, hangs on any command execution as well. ipython and python code execution works.

[alexeyeryshev]

I have the same issue; adding sudo helps, though.
OS: Sonoma 14.4.1

[Steve235lab]

With some tests I found that when OI try to execute some sudo command, the password-input prompt from the OS may be removed unexpectedly from the context interface, which makes it looks like hanging there, however if you just input your password, it will execute. This issue doesn't occurs when starting OI with sudo interpreter because you have already permitted the process of OI run in sudo mode and all commands run by OI are actually its sub-process, then there's no need to ask for password again. I think the problem we should focus is why the password-input prompt from the OS was removed unexpectedly from the command line interface, I think this is a bug probably.

[Steve235lab]

I will open a PR to fix this issue this weekend.

[ryota-murakami]

@Steve235lab I've never used to sudo with interpreter, thank you for the information and contribution!

[edwardjarchibald]

I ran into the same or a related issue. I have some hosts in my environment set up for ssh keyed access i.e. no password required. I wanted to use OI to execute a series of commands on those hosts using ssh but I got the same hang. To diagnose the issue I used the simplest possible command i.e. 'ssh user@myhost "echo Hello!"' If I execute this in the terminal it works as expected - ssh just exits after the remote system returned "Hello!" But executing this in OI results in the hang. The issue here is that the subprocess keeps running because stdin is open and the remote host, therefore, waits for further input. I experimented with closing stdin immediately after sending the code (in core/computer/terminal/languages/subprocess.py and then: poll for process exit, make sure to read to EOF for stdout and stderr. This works as expected. The problem is that this will only work properly for interactive code which waits for additional input once it start up in the subprocess. Not really sure that this is even desirable. In any event, in case it's useful, I have attached my changes to subprocess.py that works very well for me. Perhaps there's some thought needed for making a distinction between running shell commands for interactive code versus non-interactive? Here's the updated code:

def terminate(self):
        if self.process:
            self.process.terminate()
            self.process.stdout.close()
            self.process = None

    def start_process(self):
        if self.process:
            self.terminate()

        my_env = os.environ.copy()
        my_env["PYTHONIOENCODING"] = "utf-8"
        self.process = subprocess.Popen(
            self.start_cmd,
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True,
            bufsize=0,
            universal_newlines=True,
            env=my_env,
            encoding="utf-8",
            errors="replace"
        )
        threading.Thread(
            target=self.handle_stream_output,
            args=(self.process.stdout, False),
            daemon=True,
        ).start()
        threading.Thread(
            target=self.handle_stream_output,
            args=(self.process.stderr, True),
            daemon=True,
        ).start()

    def run(self, code):
        try:
            """
                Always start a new process. Once we close stdin
                we can't send more code to the process.
            """
            self.start_process()
            self.process.stdin.write(code + "\n")
            """
                Close stdin after sending the command.
                This allows the subprocess to exist when the
                code is done and stdout, stderr have been
                fully read elsewhere in the code.
            """
            self.process.stdin.close()  
          

            """
                Check for process termination and full
                drain of output.
            """
            while True:
                if self.process.poll() is not None: 
                    break
                if not self.output_queue.empty():
                    yield self.output_queue.get()
                else:
                    time.sleep(0.1)

            # Drain any remaining output
            try:
                while True:
                    output = self.output_queue.get(timeout=0.3)
                    yield output
            except queue.Empty:
                pass

        except Exception as e:
            yield {
                "type": "console",
                "format": "output",
                "content": f"Error: {str(e)}"
            }

[sam1am]

When oi runs a sudo command, I usually do not see the first password prompt. If it hangs after sudo, type in your password. Or if you want to make sure, just hit enter and it will re-prompt you for sudo password.