Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP adding a new tool to access a MicroPython target with host filesystem mounted on the board #3034

Open
wants to merge 5 commits into
base: master
from

Conversation

@dpgeorge
Copy link
Member

commented Apr 21, 2017

The aim of this tool is to allow easy and rapid development of programs that run on pyboards (in the general sense, ie anything that runs MicroPython and has a REPL available via a serial port).

The main feature is that the host PC's filesystem is mounted on the target pyboard. This is kind of reverse to what @dhylands' rshell currently does (and also mpfshell and ampy). Essentially the PC becomes a (huge) filesystem on the pyboard (with data coming in over the serial connection) and all filesystem commands work transparently.

Using the tool is very easy: you just run it and optionally specify the serial port to connect to. Then you get a normal REPL running directly on the board, just like with picocom/miniterm/etc. But before you get access to the REPL a script is run on the pyboard which installs a hook in the pyboard's filesystem so that the pyboard has access to the PC's filesystem (the directory that you run the tool from).

So at the REPL if you do os.listdir() you'll see the contents of the PC's current directory. You can also stat and open and read and close files.

Ultimately this can be extented to allow you to import scripts from your PC without ever copying them to the pyboard. This would allow for very fast workflow to develop an entire program on your PC and run it on the pyboard without ever copying any scripts.

This tool is currently only a proof-of-concept and there are a few things that need to be implemented on the firmware side before everything works (eg importing doesn't yet work).

@dpgeorge

This comment has been minimized.

Copy link
Member Author

commented Apr 21, 2017

Because this tool creates Python-level file objects it requires something like #2824 to get to a fully working state. Alternatively the RemoteFile object could be implemented in C but that's a pretty dirty solution.

@dhylands

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2017

Sounds interesting. I'm going to have to play with this.

@dpgeorge

This comment has been minimized.

Copy link
Member Author

commented Apr 21, 2017

Here is an example session:

$ ./mprepl.py
Connected to MicroPython at /dev/ttyACM0
Local directory ./ is mounted at /remote
Use Ctrl-X to exit this shell
>
MicroPython v1.8.7-646-g584677099-dirty on 2017-04-21; PYBLITEv1.0 with STM32F411RE
Type "help()" for more information.
>>> 1+2
3
>>> os.listdir('/')
['flash', 'remote']
>>> os.chdir('/remote')
>>> os.listdir()
['.gitattributes', 'mprepl.py', 'tinytest', 'cc1', 'mpy-tool.py', 'gen-changelog.sh', 'make-frozen.py', 'gendoc.py', 'upip_utarfile.py', 'file2h.py', 'dfu.py', 'bootstrap_upip.sh', '.gitignore', 'check_code_size.sh', 'upip.py', 'pyboard.py', 'codestats.sh', 'build-stm-latest.sh', 'pydfu.py', 'tinytest-codegen.py', '__pycache__', 'insert-usb-ids.py', 'gen-cpydiff.py']
>>> os.stat('blah')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 87, in stat
OSError: [Errno 2] ENOENT
>>> os.stat('pyboard.py')
(33261, 13766726, 2054, 1, 1000, 1000, 14932, 1492755867, 1491968296, 1491968296)
>>> os.stat('tinytest')
(16877, 13895970, 2054, 2, 1000, 1000, 4096, 1492753666, 1434813688, 1448726688)
>>> f=open('mprepl.py')
>>> f.read(40)
'#!/usr/bin/env python3\n\n"""\nThis script '
>>> f.read(100)
'gives you a MicroPython REPL prompt and provides a hook on the target\nboard so that the current dire'
>>> f.close()
>>> 
@dpgeorge

This comment has been minimized.

Copy link
Member Author

commented Apr 22, 2017

I've now pushed some more commits to this branch which add io.IOBase so that files can be implemented in pure Python, and improvements to extmod/vfs.c to allow importing of files to go via a Python hook.

I've also updated the mprepl.py script to allow writing to files on the host PC, and also importing files from the host PC.

It now enables the following workflow:

  • run mprepl.py from the directory you are developing a program in
  • edit your scripts/datafiles/etc on the host PC using your favourite editor/IDE
  • in mprepl prompt run execfile('main.py'); this will execute the code from your PC (assuming the main file is main.py)
  • watch the code run!
  • your code can read/write datafiles from/to the PC
  • your code can import other scripts from the PC (even packages should work)
  • you can then make changes to your code and re-run execfile (although any imports won't be reloaded)
  • if needed you can do a soft reset (Ctrl-D) at the mprepl prompt to get back to a fresh state (soft reset will reload the filesystem hook); this will let you import modified modules

Any feedback on this new tool is welcome.

@HenrikSolver

This comment has been minimized.

Copy link
Contributor

commented Apr 22, 2017

Really nice! I have played around with it a little bit and have not found any issues. A future improvement could be to keep the command history in the tool, that way it would survive a soft reboot.

@pfalcon

This comment has been minimized.

Copy link
Contributor

commented Apr 22, 2017

So, I'm surprised to see such tool, I would think it as not a natural direction to want to move "PC" into a "pyboard", the normal direction, of making "pyboard" seamless accessible from "PC", i.e. elaborating pyboard.py tool as a more natural.

It's also pretty cunning way to sneak in the stream protocol implementation for Python classes ;-).

But my only real concern is the following: how about we don't propagate usage of unfinished VFS API, with known drawbacks? As mentioned in the previous reviews, listdir() should NOT be in the basis of a VFS class. Instead, ilistdir() should be, which allows to enumerate files in a fixed amount of RAM. listdir() then should be implemented in terms of ilistdir() in a generic way in VFS support code.

Summing up, it would be nice to finalize VFS API first, which also includes #2950 (if something there depends on me, please push me, though I'm in general +1 to merge it if it provides needed external functionality, we can optimize implementation later).

Thanks.

@chrismas9

This comment has been minimized.

Copy link
Contributor

commented Apr 22, 2017

+1 +1

  1. I have always been concerned about the fragility of the mp filesystem when shared with the host over USB as there is no file sharing protection. Commercial products often need to expose the USB port for maintenance, eg uploading log files or downloading new programs and with MSC disabled this provides a safe way with the onus on the host to manage file sharing. Curious people can't just plug in a USB cable and mess with the files on the target. Developers with mprepl.py can, or you could include the functionality of mprepl.py into a host side maintenance program that automates whatever maintenance is required.

So, I'm surprised to see such tool, I would think it as not a natural direction to want to move "PC" into a "pyboard", the normal direction, of making "pyboard" seamless accessible from "PC", i.e. elaborating pyboard.py tool as a more natural.

  1. This provides a way to develop and debug programs on small systems without enough flash for a file system. You can execute programs from the host until they are finalised, then build and burn a frozen image. Using the reverse direction won't work when there is no file system to share.
@peterhinch

This comment has been minimized.

Copy link
Contributor

commented Apr 23, 2017

Nice! A couple of oddities. If an SD card is fitted I'm seeing

$ ./mprepl.py 
Traceback (most recent call last):
  File "./mprepl.py", line 384, in <module>
    main()
  File "./mprepl.py", line 379, in main
    main_loop(console, dev)
  File "./mprepl.py", line 321, in main_loop
    pyb.exec_(fs_hook_code)
  File "/mnt/qnap2/data/Projects/MicroPython/micropython/tools/pyboard.py", line 336, in exec_
    raise PyboardError('exception', ret, ret_err)
pyboard.PyboardError: ('exception', b'', b'Traceback (most recent call last):\r\n  File "<stdin>", line 1, in <module>\r\n  File "io.py", line 1, in <module>\r\nImportError: no module named \'uasyncio\'\r\n')

As the error suggests, there is no copy of uasyncio available on /flash, /sd, or as a frozen module.

If I make uasyncio available on the SD card I get

$ ./mprepl.py 
Traceback (most recent call last):
  File "./mprepl.py", line 384, in <module>
    main()
  File "./mprepl.py", line 379, in main
    main_loop(console, dev)
  File "./mprepl.py", line 321, in main_loop
    pyb.exec_(fs_hook_code)
  File "/mnt/qnap2/data/Projects/MicroPython/micropython/tools/pyboard.py", line 336, in exec_
    raise PyboardError('exception', ret, ret_err)
pyboard.PyboardError: ('exception', b'About to yield\r\n', b'Traceback (most recent call last):\r\n  File "<stdin>", line 1, in <module>\r\n  File "io.py", line 36, in <module>\r\n  File "uasyncio/core.py", line 129, in run_forever\r\n  File "uasyncio/core.py", line 102, in run_forever\r\n  File "uasyncio/__init__.py", line 21, in add_reader\r\nOSError: stream operation not supported\r\n')

Secondly, if I import a file it sometimes appears to hang but a second enter runs it.

@dpgeorge dpgeorge force-pushed the dpgeorge:mprepl branch from 057ef2c to 6b020da May 10, 2017

@dpgeorge

This comment has been minimized.

Copy link
Member Author

commented May 17, 2017

@pfalcon I'm thinking it would be nice to merge the C-level patches from this PR (addition of uio.IOBase and VFS being able to import-stat a generic FS) for v1.9. Why? because it goes that "last mile" with the VFS and rounds it off nicely, allowing one to implement a full FS -- and files -- in pure Python.

@pfalcon

This comment has been minimized.

Copy link
Contributor

commented May 17, 2017

If it's configurable and off by default - sure, why not.

@dpgeorge

This comment has been minimized.

Copy link
Member Author

commented May 26, 2017

Ok, I decided not to merge the C-level patches from this PR just yet. There're some things to discuss and get right before it can go it. So leaving it until after v1.9.

@dpgeorge dpgeorge force-pushed the dpgeorge:mprepl branch from fad6a0c to 9490d7d May 14, 2019

@cwalther

This comment has been minimized.

Copy link

commented May 14, 2019

Wow, that’s a very cool tool! Why have I never heard of this before? Now if it would work over WebREPL that would be even cooler!

It doesn’t totally work for me yet on macOS with an ESP8266 board running MicroPython v1.10-8-g8b7039d7d. I can open(), read(), and write() files, but importing doesn’t work. If I have a file hello.py on the host containing print('hello') and issue import hello, it just gets stuck without any output, ctrl-C does not interrupt it. When I ctrl-X out of mprepl, reconnect with screen, and send a couple of ctrl-Cs (one isn’t sufficient), it returns to a functional REPL saying

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "hello.py", line 2
SyntaxError: invalid syntax
>>> 
>>>

Also, files can’t be used in a with statement:

>>> with open('hello.py', 'rb') as f: f.read()
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'RemoteFile' object has no attribute '__exit__'

I haven’t tried debugging or fixing any of this yet.

@cwalther

This comment has been minimized.

Copy link

commented May 15, 2019

👍 That appears to have fixed the import case!

Here’s how I added the context manager and seek methods that my test program happened to need (take with a grain of salt, it’s past midnight and I only lightly tested this):

--- mprepl.py
+++ mprepl.py
@@ -27,6 +27,7 @@
 CMD_CLOSE = 5
 CMD_READ = 6
 CMD_WRITE = 7
+CMD_SEEK = 8
 
 fs_hook_code = """\
 import os, io, select, ustruct as struct, micropython
@@ -37,6 +38,7 @@
 CMD_CLOSE = 5
 CMD_READ = 6
 CMD_WRITE = 7
+CMD_SEEK = 8
 class RemoteCommand:
     def __init__(self):
         try:
@@ -136,6 +138,18 @@
         n = self.cmd.rd_int32()
         self.cmd.end()
         return n
+    def seek(self, offset, whence=0):
+        self.cmd.begin(CMD_SEEK)
+        self.cmd.wr_int32(self.fd)
+        self.cmd.wr_int32(offset)
+        self.cmd.wr_uint32(whence)
+        pos = self.cmd.rd_int32()
+        self.cmd.end()
+        return pos
+    def __enter__(self):
+        return self
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.close()
 
 class RemoteFS:
     def mount(self, readonly, mkfs):
@@ -221,7 +235,7 @@
         self.fin = fin
         self.fout = fout
     def rd_uint32(self):
-        return struct.unpack('<I', self.rd(4))[0]
+        return struct.unpack('<I', self.fin.read(4))[0]
     def wr_uint32(self, i):
         self.fout.write(struct.pack('<I', i))
     def rd_int32(self):
@@ -315,6 +329,13 @@
     n = data_files[fd][0].write(buf)
     cmd.wr_int32(n)
 
+def do_seek(cmd):
+    fd = cmd.rd_int32()
+    offset = cmd.rd_int32()
+    whence = cmd.rd_uint32()
+    pos = data_files[fd][0].seek(offset, whence)
+    cmd.wr_int32(pos)
+
 cmd_table = {
     CMD_STAT: do_stat,
     CMD_ILISTDIR_START: do_ilistdir_start,
@@ -323,6 +344,7 @@
     CMD_CLOSE: do_close,
     CMD_READ: do_read,
     CMD_WRITE: do_write,
+    CMD_SEEK: do_seek,
 }
 
 def main_loop(console, dev):
@cwalther

This comment has been minimized.

Copy link

commented May 19, 2019

Heads-up: I almost have it running over WebREPL (with some hacks and open questions). The current roadblock is that WebREPL cannot transfer binary data that is not UTF-8, because it just packs it into WebSocket text messages without checking, which causes conformant recipients to close the connection. Will have to find a way to fix that first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.