rshell cp fails if connection is via UART #10

peterhinch · 2015-10-29T09:50:26Z

SD card contains only boot.py:

import pyb
pyb.usb_mode(None)
uart = pyb.UART(6, 115200)
pyb.repl_uart(uart)

Typical rshell session copying a 2.9KB file (I don't always get trace data: sometimes it just hangs)

/mnt/qnap2/data/Projects/MicroPython/micropython-micropower> cp ttest.py /flash
Traceback (most recent call last):
  File "/mnt/qnap2/python/rshell/rshell.py", line 1785, in <module>
    main()
  File "/mnt/qnap2/python/rshell/rshell.py", line 1782, in main
    shell.cmdloop(cmd_line)
  File "/mnt/qnap2/python/rshell/rshell.py", line 1084, in cmdloop
    cmd.Cmd.cmdloop(self)
  File "/usr/local/opt/python-3.4.3/lib/python3.4/cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "/mnt/qnap2/python/rshell/rshell.py", line 1115, in onecmd
    return cmd.Cmd.onecmd(self, line)
  File "/usr/local/opt/python-3.4.3/lib/python3.4/cmd.py", line 217, in onecmd
    return func(arg)
  File "/mnt/qnap2/python/rshell/rshell.py", line 1353, in do_cp
    if not cp(src_filename, dst_filename):
  File "/mnt/qnap2/python/rshell/rshell.py", line 364, in cp
    filesize, xfer_func=send_file_to_remote)
  File "/mnt/qnap2/python/rshell/rshell.py", line 933, in remote
    xfer_func(self, *args, **kwargs)
  File "/mnt/qnap2/python/rshell/rshell.py", line 622, in send_file_to_remote
    sys.stdout.write(chr(ord(char)))
TypeError: ord() expected a character, but string of length 0 found

This seems similar to using rshell with the WiPy: most functionality works except for cp.

The text was updated successfully, but these errors were encountered:

dhylands · 2015-10-29T17:12:36Z

ok - thanks - I'll look into this

dhylands · 2015-10-29T17:13:44Z

Is this over a wireless link? Or using something like an FTDI connection?

dhylands · 2015-10-30T05:08:19Z

Initial analysis suggests that the uart read buffer is getting messed up. The host side wrote 512 bytes.

I tried: modifying your boot.py to look like:

import pyb
pyb.usb_mode(None)
uart = pyb.UART(6, 115200, timeout_char=200, read_buf_len=600)
pyb.repl_uart(uart)

it got the first 198 bytes fine, and then it dropped 8 characters. I confirmed (using my logic analyzer) that the correct characters arrived on the Rx line.

I probably won't get much further tonight, so I thought I'd write this up to let you know what I found.

dhylands · 2015-10-30T06:26:18Z

And it definitely looks like a software issue. I toggled a GPIO for each character received and the dropped characters were received.

dhylands · 2015-10-30T06:32:51Z

Nope - I see some dropped IRQs. So that tells me that interupts were disabled or some higher priority handler was running.

peterhinch · 2015-10-30T09:26:46Z

Sorry for the delay getting back to you but we are separated by a few degrees of longitude :) I was testing using an FTDI adaptor lead connected by short jumpers to Pyboard UART6. The link seems reliable.

dhylands · 2015-10-30T16:53:47Z

I opened: micropython/micropython#1555

From my logic analyzer traces it looks like interupts are being disabled to do sdcard I/O and that's causing dropped characters.

I'm going to see if I can figure out how to do UART Rx using DMA.

dhylands · 2015-11-09T01:26:05Z

This should be addressed by micropython/micropython#1608

peterhinch · 2016-02-10T13:07:31Z

I'm still seeing this with a Pyboard connected via an FTDI adaptor. Rshell works fine except for copying files to the Pyboard SD card where it hangs. The card in a card reader shows a zero length file. This is with current firmware.

dhylands · 2016-02-10T16:42:42Z

I stumbled across this last night as well on my NUCLEO 401 board. In that
case, it was copying to internal flash, and also over a non-USB serial link.

On Wed, Feb 10, 2016 at 5:07 AM, Peter Hinch notifications@github.com
wrote:

I'm still seeing this with a Pyboard connected via an FTDI adaptor. Rshell
works fine except for copying files to the Pyboard SD card where it hangs.
The card in a card reader shows a zero length file. This is with current
firmware.

—
Reply to this email directly or view it on GitHub
#10 (comment).

Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

dhylands · 2016-02-11T13:50:08Z

@peterhinch Are you using rshell from this repository? This repository is now obsolete and has been replaced by: https://github.com/dhylands/rshell and you can now install rshell using pip.

peterhinch · 2016-02-11T18:04:51Z

Installing with pip sounds so easy. Unless you have boxes running rather old versions of Debian. I'll tackle that can of worms tomorrow.

On this relatively up to date laptop (Mint 17.1) I managed to install with pip3 (pip threw errors). The following presumably implies I'm running the right version:

axolotl adminpete # pip3 install rshell --upgrade
Requirement already up-to-date: rshell in /usr/local/lib/python3.4/dist-packages
Requirement already up-to-date: pyserial in /usr/local/lib/python3.4/dist-packages (from rshell)
Requirement already up-to-date: pyudev>=0.16 in /usr/local/lib/python3.4/dist-packages (from rshell)
Requirement already up-to-date: six in /usr/local/lib/python3.4/dist-packages (from pyudev>=0.16->rshell)
Cleaning up...
axolotl adminpete #

Version I'm running

[adminpete@axolotl]: ~
$ which rshell
/usr/local/bin/rshell
[adminpete@axolotl]: ~

Attempt to run

[adminpete@axolotl]: ~
$ rshell -p /dev/ftdi
Connecting to /dev/ftdi ...
Welcome to rshell. Use Control-D to exit.
/home/adminpete> cp rats9.py /sd
^CTraceback (most recent call last):
  File "/usr/local/bin/rshell", line 9, in <module>
    load_entry_point('rshell==0.0.1', 'console_scripts', 'rshell')()
  File "/usr/local/lib/python3.4/dist-packages/rshell/command_line.py", line 4, in main
    rshell.main.main()
  File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 2254, in main
    shell.cmdloop(cmd_line)
  File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 1311, in cmdloop
    cmd.Cmd.cmdloop(self)
  File "/usr/lib/python3.4/cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 1342, in onecmd
    return cmd.Cmd.onecmd(self, line)
  File "/usr/lib/python3.4/cmd.py", line 217, in onecmd
    return func(arg)
  File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 1690, in do_cp
    if not cp(src_filename, dst_filename):
  File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 452, in cp
    filesize, xfer_func=send_file_to_remote)
  File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 1107, in remote
    xfer_func(self, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 738, in send_file_to_remote
    char = dev.read(1)
  File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 1073, in read
    return self.pyb.serial.read(num_bytes)
  File "/usr/local/lib/python3.4/dist-packages/serial/serialposix.py", line 481, in read
    ready, _, _ = select.select([self.fd], [], [], timeout)
KeyboardInterrupt
[adminpete@axolotl]: ~
$ 
``` (It hung on the attempt to copy). Is there a way to get rshell to issue a version number?

dhylands · 2016-02-11T19:48:27Z

Good question. I don't think that there is, but I'll figure out a way to do that.

And yeah - you need to use pip3 because rshell requires python3. I guess I should make sure I say that :)

dhylands · 2016-02-11T19:50:07Z

I also discovered that if you have python3, you can run python3 -m ensurepip and that will get you pip3.

peterhinch · 2016-02-12T07:56:53Z

Thanks for the tip re pip3.

I'm puzzled you've closed this as I'm still experiencing the hanging problem. To try to settle the version issue I used pip3 to uninstall rshell and deleted the directories in /usr/local/lib/python3.4/dist-packages. I then reinstalled it with pip3 checking that the code had re-appeared in the same location. I then cloned your repository into ~/temp using

git clone https://github.com/dhylands/rshell.git

Finally I ran the following checks, which have convinced me I'm running the right version - but if I'm doing something dumb do please enlighten me!

[adminpete@axolotl]: ~
$ cd /usr/local/lib/python3.4/dist-packages/rshell
[adminpete@axolotl]: /usr/local/lib/python3.4/dist-packages/rshell
$ md5sum *.py
55537e0c645de5b4d323b1fb97873b4b  command_line.py
8671bdf67f2ff1a301d0cd0e49e901f4  getch.py
d41d8cd98f00b204e9800998ecf8427e  __init__.py
7656d1566097065c4250b531093fe2f3  main.py
da18a8d6d57694c55c1751cf53860f1d  pyboard.py
[adminpete@axolotl]: /usr/local/lib/python3.4/dist-packages/rshell

$ cd ~/temp/rshell/rshell/
[adminpete@axolotl]: ~/temp/rshell/rshell
$ md5sum *.py
55537e0c645de5b4d323b1fb97873b4b  command_line.py
8671bdf67f2ff1a301d0cd0e49e901f4  getch.py
d41d8cd98f00b204e9800998ecf8427e  __init__.py
7656d1566097065c4250b531093fe2f3  main.py
da18a8d6d57694c55c1751cf53860f1d  pyboard.py
[adminpete@axolotl]: ~/temp/rshell/rshell

$ which rshell
/usr/local/bin/rshell
[adminpete@axolotl]: ~/temp/rshell/rshell

$ alias rshell
alias rshell='rshell -p /dev/pyboard'
[adminpete@axolotl]: ~/temp/rshell/rshell
$ 
$ alias rsftdi
alias rsftdi='rshell -p /dev/ftdi'
[adminpete@axolotl]: /mnt/qnap2/data/Projects/MicroPython/micropython/docs

dhylands · 2016-02-12T08:40:20Z

The closing wasn't intentional. I must have just clicked on the wrong button.

dhylands · 2016-02-12T08:43:20Z

I just wanted to make sure you were using the pip installed version, because I seem to recall fixing a few issues. I'm homing in on the issue (I think). At least for me on the Nucleo. I suspect that your problem is somehow related. Could you try with --buffer-size=30 and see if that makes any difference?

peterhinch · 2016-02-12T08:47:32Z

Yes! A quick test was successful: I copied a small file to /sd, 'cat'-ed it and deleted it all OK.
[edit]
A test copying a 56KB file to and from /sd with a subsequent verify also worked, as did editing files on /sd. Looking good!

dhylands · 2016-02-12T16:37:03Z

What seems to be happening is that something slowed down a bit and rshell is no longer processing buffers as quickly as it used to. rshell has a default bufer size of 512, however, the UART only has a buffer of 32 (or maybe 64). This host side sends a buffer of data and waits for an ACK before sending another buffer worth of data. If the micropython side processes the data quickly enough then it can deal with more than the UART buffer worth, however, it seems like we were really close to the edge of "running in real-time" and something has slowed down enough that the python side is taking more than a character time to process each character. All it takes is one character to be dropped and the file transfer grinds to a halt.

Now to figure out a workaround.

peterhinch · 2016-02-12T17:57:18Z

ctrl S ctrl Q flow control in the UART driver? Alas it would need a firmware change, but perhaps a useful option.

Another one I get occasionally on a USB connection is this "serial port /dev/pyboard closed". It's benign in that you can restart the link and carry on, but I thought it worth mentioning. The sequence below was just after dropping back from the REPL with ctrl X. After the error I hit ctrl D and fired up rshell again, without resetting the Pyboard.

/mnt/qnap2/data/Projects/MicroPython/micropython-micropower> cp upower.py /flash
serial port /dev/pyboard closed
/mnt/qnap2/data/Projects/MicroPython/micropython-micropower> 
[adminpete@axolotl]: /mnt/qnap2/data/Projects/MicroPython/micropython-micropower
$ rshell
Connecting to /dev/pyboard ...
Welcome to rshell. Use Control-D to exit.
/mnt/qnap2/data/Projects/MicroPython/micropython-micropower> cp upower.py /flash
/mnt/qnap2/data/Projects/MicroPython/micropython-micropower> repl
Entering REPL. Use Control-X to exit.
>
MicroPython v1.6-35-ge372e83 on 2016-02-11; PYBv1.0 with STM32F405RG
Type "help()" for more information.
>>>

dhylands closed this as completed Feb 11, 2016

dhylands reopened this Feb 12, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rshell cp fails if connection is via UART #10

rshell cp fails if connection is via UART #10

peterhinch commented Oct 29, 2015

dhylands commented Oct 29, 2015

dhylands commented Oct 29, 2015

dhylands commented Oct 30, 2015

dhylands commented Oct 30, 2015

dhylands commented Oct 30, 2015

peterhinch commented Oct 30, 2015

dhylands commented Oct 30, 2015

dhylands commented Nov 9, 2015

peterhinch commented Feb 10, 2016

dhylands commented Feb 10, 2016

dhylands commented Feb 11, 2016

peterhinch commented Feb 11, 2016

dhylands commented Feb 11, 2016

dhylands commented Feb 11, 2016

peterhinch commented Feb 12, 2016

dhylands commented Feb 12, 2016

dhylands commented Feb 12, 2016

peterhinch commented Feb 12, 2016

dhylands commented Feb 12, 2016

peterhinch commented Feb 12, 2016

rshell cp fails if connection is via UART #10

rshell cp fails if connection is via UART #10

Comments

peterhinch commented Oct 29, 2015

dhylands commented Oct 29, 2015

dhylands commented Oct 29, 2015

dhylands commented Oct 30, 2015

dhylands commented Oct 30, 2015

dhylands commented Oct 30, 2015

peterhinch commented Oct 30, 2015

dhylands commented Oct 30, 2015

dhylands commented Nov 9, 2015

peterhinch commented Feb 10, 2016

dhylands commented Feb 10, 2016

dhylands commented Feb 11, 2016

peterhinch commented Feb 11, 2016

dhylands commented Feb 11, 2016

dhylands commented Feb 11, 2016

peterhinch commented Feb 12, 2016

dhylands commented Feb 12, 2016

dhylands commented Feb 12, 2016

peterhinch commented Feb 12, 2016

dhylands commented Feb 12, 2016

peterhinch commented Feb 12, 2016