FileNotFoundError exception when trying to execute micropython code

[rdlaner]

I'm running Jupyter via the VS Code extension and just started trying out your package. I can load the extension without issue, but as soon as I try to run some micropython code, it fails trying to run the mpremote command:

Below is the full exception output. I tried running the mpremote command manually on the command line and it seemed to run without issue, so perhaps this is an issue within VS Code and its access to the file system?

I'm running on MacOS version 14.2.1.

Hopefully we can get this worked out, I'm excited to use this extension to help debug one of my micropython projects!

{
	"name": "FileNotFoundError",
	"message": "Failed to start m",
	"stack": "---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
File ~/Documents/MicroPython/Projects/MicroPython_CO2/.venv/lib/python3.11/site-packages/micropython_magic/interactive.py:100, in ipython_run(cmd, stream_out, timeout, shell, hide_meminfo, store_output, log_errors, tags, follow)
     99 try:
--> 100     process = subprocess.Popen(
    101         cmd,
    102         shell=False,
    103         stdout=subprocess.PIPE,
    104         stderr=subprocess.PIPE,
    105         start_new_session=False,
    106     )  # ,  universal_newlines=True)
    107 except FileNotFoundError as e:

File /opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py:1026, in Popen.__init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, user, group, extra_groups, encoding, errors, text, umask, pipesize, process_group)
   1023             self.stderr = io.TextIOWrapper(self.stderr,
   1024                     encoding=encoding, errors=errors)
-> 1026     self._execute_child(args, executable, preexec_fn, close_fds,
   1027                         pass_fds, cwd, env,
   1028                         startupinfo, creationflags, shell,
   1029                         p2cread, p2cwrite,
   1030                         c2pread, c2pwrite,
   1031                         errread, errwrite,
   1032                         restore_signals,
   1033                         gid, gids, uid, umask,
   1034                         start_new_session, process_group)
   1035 except:
   1036     # Cleanup if the child failed starting.

File /opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py:1950, in Popen._execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, gid, gids, uid, umask, start_new_session, process_group)
   1949         err_msg = os.strerror(errno_num)
-> 1950     raise child_exception_type(errno_num, err_msg, err_filename)
   1951 raise child_exception_type(err_msg)

FileNotFoundError: [Errno 2] No such file or directory: 'mpremote connect auto resume  run /var/folders/zz/4_j849n104j0ztbqh1mc34100000gn/T/tmpb667n9to.py'

The above exception was the direct cause of the following exception:

FileNotFoundError                         Traceback (most recent call last)
Cell In[8], line 1
----> 1 get_ipython().run_cell_magic('micropython', '', 'from micropython import const\
TEST_CONST = const(5)\
')

File ~/Documents/MicroPython/Projects/MicroPython_CO2/.venv/lib/python3.11/site-packages/IPython/core/interactiveshell.py:2517, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
   2515 with self.builtin_trap:
   2516     args = (magic_arg_s, cell)
-> 2517     result = fn(*args, **kwargs)
   2519 # The code below prevents the output from being displayed
   2520 # when using magics with decorator @output_can_be_silenced
   2521 # when the last Python token in the expression is a ';'.
   2522 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):

File ~/Documents/MicroPython/Projects/MicroPython_CO2/.venv/lib/python3.11/site-packages/micropython_magic/octarine.py:211, in MpyMagics.micropython(self, line, cell)
    209 if not cell:
    210     raise UsageError(\"Please specify some MicroPython code to execute\")
--> 211 output = self.MCU.run_cell(
    212     cell, timeout=args.timeout, follow=args.follow, mount=args.mount
    213 )

File ~/Documents/MicroPython/Projects/MicroPython_CO2/.venv/lib/python3.11/site-packages/micropython_magic/mpr.py:108, in MPRemote2.run_cell(self, cell, timeout, follow, mount)
    105     run_cmd = f'mount \"{Path(mount).as_posix()}\" ' + run_cmd
    107 # TODO: detect / retry / report errors copying the file
--> 108 result = self.run_cmd(
    109     run_cmd,
    110     stream_out=True,
    111     timeout=timeout,
    112     follow=follow,
    113 )
    114 # log.info(_)
    115 # log.info(f.name, \"copied to device\")
    116 Path(f.name).unlink()

File ~/Documents/MicroPython/Projects/MicroPython_CO2/.venv/lib/python3.11/site-packages/micropython_magic/mpr.py:61, in MPRemote2.run_cmd(self, cmd, auto_connect, stream_out, shell, timeout, follow)
     59         log.warning(f\"cmd is not a string: {cmd}\")
     60 log.debug(cmd)
---> 61 return ipython_run(
     62     cmd, stream_out=stream_out, shell=shell, timeout=timeout or self.timeout, follow=follow
     63 )

File ~/Documents/MicroPython/Projects/MicroPython_CO2/.venv/lib/python3.11/site-packages/micropython_magic/interactive.py:108, in ipython_run(cmd, stream_out, timeout, shell, hide_meminfo, store_output, log_errors, tags, follow)
    100     process = subprocess.Popen(
    101         cmd,
    102         shell=False,
   (...)
    105         start_new_session=False,
    106     )  # ,  universal_newlines=True)
    107 except FileNotFoundError as e:
--> 108     raise FileNotFoundError(f\"Failed to start {cmd[0]}\") from e
    110 assert process.stdout is not None
    111 if follow == False:
    112     # we do not need to follow the output of the command
    113     # just return

FileNotFoundError: Failed to start m"
}

[Josverl]

Indeed it appears that the tmpfile.py cannot be accessed by mpremote

the sequence is :

1. create tempfile using tempfile.NamedTemporaryFile
2. save cell content to tempfile
3. subprocess.run mpremote run tempfile
4. delete tempfile

I have only tested this on Windows and Linux so it could be something with tempfile.NamedTemporaryFile but it should be supported on MacOS.

What you could do to troubleshoot is to comment out the
Path(f.name).unlink() and see if you can see anything specific in the security / ownership of that file

[Josverl]

Another thought : I recently (yesterday) updated this code to add a feature and simplify things a bit.
version 0.7.0 should have the prior behaviour.

• not sure if that is any better but a simple test %pip install micropython-magic==0.7.0 & restart kernel

• create tempfile using tempfile.NamedTemporaryFile

• save cell content to tempfile

• subprocess.run mpremote copy tempfile :__mpmagic.py

• subprocess.run mpremote exec "exec(__mpmagic.py)" #

• delete tempfile

[Josverl]

also added an option in 0.8.1 to crank up logging by setting %config MicroPythonMagic.loglevel = 'TRACE'

[rdlaner]

Ok, I think I figured it out, at least initially. It looks like the command being passed to subprocess.Popen() is not being formatted properly as Popen is supposed to accept a list of arguments. I made a quick hack to use shlex and now everything seems to be working as expected:

    log.trace(f"per char , with timeout of {timeout} seconds")
    log.trace(f"cmd: {cmd}")
    import shlex
    try:
        process = subprocess.Popen(
            shlex.split(cmd),
            shell=False,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            start_new_session=False,
        )  # ,  universal_newlines=True)
    except FileNotFoundError as e:
        raise FileNotFoundError(f"Failed to start {cmd}") from e

[Josverl]

Thanks, that helps understand what the root cause is.
Although according to the docs 'args should be a sequence of program arguments or else a single string or path-like object.'

probably the devil is in the details : On POSIX, if args is a string, the string is interpreted as the name or path of the program to execute. However, this can only be done if not passing arguments to the program.

Then it makes more sense to work with a list of strings rather than a single command, but that may take a while as i'm currently
busy with other things, and want to make sure that that does create regressions. As you have a workaround i hope that that is good enough for a few weeks.

[rdlaner]

Yeah, no problem, I'm unblocked.

For what it's worth, the design pattern I've used in most of my projects that use subprocess is to have the api take the command as a string and then use shlex to split that string up when passing it to a subprocess function call. Seems to work pretty reliably.

[Josverl]

there are a few fineky quoting escapes to handle evals inside evals with spaces parameters,
Also shlex only works for Posix , it has an explicit warning on that , and my personal pref is for Windows.
So I do rather have something that works for all, should not be to much effort.
It would be good if you could run the test suite on Mac to verify at that time

[rdlaner]

Makes sense and sounds good. Let me know when you have an update and I can run the test suite.

[Josverl]

Prerelase 0.8.2a0 works on my machine ...

you can install that and see if you still need to patch

to run the test use branch : https://github.com/Josverl/micropython-magic/tree/Josverl/issue21

[rdlaner]

Great, just installed v 0.8.2a0 and am able to send cells to my micropython board without issue!

On a different note, do you know if it is possible to get the full backtrace when a micropython exception occurs on the device? The only pain point I had in my development flow was when I ran into an exception, I would have to manually copy over my files and run them on device in order to get the full backtrace.

[Josverl]

if it is possible to get the full backtrace when a micropython exception occurs on the device?
I created a new issue to track that.

Could you add an example to that issue of some failing code and the traceback that you would like to see in Jupyter ?