From b68f77952f3896059bccfffa62b2dc32cdf7c83b Mon Sep 17 00:00:00 2001 From: Damien George Date: Tue, 6 Oct 2020 12:11:10 +1100 Subject: [PATCH 1/2] lib/utils/pyexec: Add stdin-reader on raw REPL with flow control. Background: the friendly/normal REPL is intended for human use whereas the raw REPL is for computer use/automation. Raw REPL is used for things like pyboard.py script_to_run.py. The normal REPL has built-in flow control because it echos back the characters. That's not so with raw REPL and flow control is just implemented by rate limiting the amount of data that goes in. Currently it's fixed at 256 byte chunks every 10ms. This is sometimes too fast for slow MCUs or systems with small stdin buffers. It's also too slow for a lot of higher-end MCUs, ie it could be a lot faster. This commit adds a new raw REPL mode which includes flow control: the device will echo back a character after a certain number of bytes are sent to the host, and the host can use this to regulate the data going out to the device. The amount of characters is controlled by the device and sent to the host before communication starts. This flow control allows getting the maximum speed out of a serial link, regardless of the link or the device at the other end. Also, this new raw REPL mode parses and compiles the incoming data as it comes in. It does this by creating a "stdin reader" object which is then passed to the lexer. The lexer requests bytes from this "stdin reader" which retrieves bytes from the host, and does flow control. What this means is that no memory is used to store the script (in the existing raw REPL mode the device needs a big buffer to read in the script before it can pass it on to the lexer/parser/compiler). The only memory needed on the device is enough to parse and compile. Finally, it would be possible to extend this new raw REPL to allow bytecode (.mpy files) to be sent as well as text mode scripts (but that's not done in this commit). Some results follow. The test was to send a large 33k script that contains mostly comments and then prints out the heap, run via pyboard.py large.py. On PYBD-SF6, prior to this PR: $ ./pyboard.py large.py stack: 524 out of 23552 GC: total: 392192, used: 34464, free: 357728 No. of 1-blocks: 12, 2-blocks: 2, max blk sz: 2075, max free sz: 22345 GC memory layout; from 2001a3f0: 00000: h=hhhh=======================================hhBShShh==h=======h 00400: =====hh=B........h==h=========================================== 00800: ================================================================ 00c00: ================================================================ 01000: ================================================================ 01400: ================================================================ 01800: ================================================================ 01c00: ================================================================ 02000: ================================================================ 02400: ================================================================ 02800: ================================================================ 02c00: ================================================================ 03000: ================================================================ 03400: ================================================================ 03800: ================================================================ 03c00: ================================================================ 04000: ================================================================ 04400: ================================================================ 04800: ================================================================ 04c00: ================================================================ 05000: ================================================================ 05400: ================================================================ 05800: ================================================================ 05c00: ================================================================ 06000: ================================================================ 06400: ================================================================ 06800: ================================================================ 06c00: ================================================================ 07000: ================================================================ 07400: ================================================================ 07800: ================================================================ 07c00: ================================================================ 08000: ================================================================ 08400: ===============================================.....h==......... (349 lines all free) (the big blob of used memory is the large script). Same but with this PR: $ ./pyboard.py large.py stack: 524 out of 23552 GC: total: 392192, used: 1296, free: 390896 No. of 1-blocks: 12, 2-blocks: 3, max blk sz: 40, max free sz: 24420 GC memory layout; from 2001a3f0: 00000: h=hhhh=======================================hhBShShh==h=======h 00400: =====hh=h=B......h==.....h==.................................... (381 lines all free) The only thing in RAM is the compiled script (and some other unrelated items). Time to download before this PR: 1438ms, data rate: 230,799 bits/sec. Time to download with this PR: 119ms, data rate: 2,788,991 bits/sec. So it's more than 10 times faster, and uses significantly less RAM. Results are similar on other boards. On an stm32 board that connects via UART only at 115200 baud, the data rate goes from 80kbit/sec to 113kbit/sec, so gets close to saturating the UART link without loss of data. The new raw REPL mode also supports a single ctrl-C to break out of this flow-control mode, so that a ctrl-C can always get back to a known state. It's also backwards compatible with the original raw REPL mode, which is still supported with the same sequence of commands. The new raw REPL mode is activated by ctrl-E, which gives an error on devices that do not support the new mode. Signed-off-by: Damien George --- lib/utils/pyexec.c | 113 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/lib/utils/pyexec.c b/lib/utils/pyexec.c index 2b86af3bbaadd..a4d22b072f7b4 100644 --- a/lib/utils/pyexec.c +++ b/lib/utils/pyexec.c @@ -56,6 +56,7 @@ STATIC bool repl_display_debugging_info = 0; #define EXEC_FLAG_SOURCE_IS_RAW_CODE (8) #define EXEC_FLAG_SOURCE_IS_VSTR (16) #define EXEC_FLAG_SOURCE_IS_FILENAME (32) +#define EXEC_FLAG_SOURCE_IS_READER (64) // parses, compiles and executes the code in the lexer // frees the lexer before returning @@ -91,6 +92,8 @@ STATIC int parse_compile_execute(const void *source, mp_parse_input_kind_t input if (exec_flags & EXEC_FLAG_SOURCE_IS_VSTR) { const vstr_t *vstr = source; lex = mp_lexer_new_from_str_len(MP_QSTR__lt_stdin_gt_, vstr->buf, vstr->len, 0); + } else if (exec_flags & EXEC_FLAG_SOURCE_IS_READER) { + lex = mp_lexer_new(MP_QSTR__lt_stdin_gt_, *(mp_reader_t *)source); } else if (exec_flags & EXEC_FLAG_SOURCE_IS_FILENAME) { lex = mp_lexer_new_from_file(source); } else { @@ -122,6 +125,12 @@ STATIC int parse_compile_execute(const void *source, mp_parse_input_kind_t input // uncaught exception mp_hal_set_interrupt_char(-1); // disable interrupt mp_handle_pending(false); // clear any pending exceptions (and run any callbacks) + + if (exec_flags & EXEC_FLAG_SOURCE_IS_READER) { + const mp_reader_t *reader = source; + reader->close(reader->data); + } + // print EOF after normal output if (exec_flags & EXEC_FLAG_PRINT_EOF) { mp_hal_stdout_tx_strn("\x04", 1); @@ -170,6 +179,94 @@ STATIC int parse_compile_execute(const void *source, mp_parse_input_kind_t input } #if MICROPY_ENABLE_COMPILER + +#ifndef MICROPY_REPL_STDIN_BUFFER_MAX +#define MICROPY_REPL_STDIN_BUFFER_MAX (256) +#endif + +typedef struct _mp_reader_stdin_t { + bool eof; + uint16_t window_max; + uint16_t window_remain; +} mp_reader_stdin_t; + +STATIC mp_uint_t mp_reader_stdin_readbyte(void *data) { + mp_reader_stdin_t *reader = (mp_reader_stdin_t *)data; + + if (reader->eof) { + return MP_READER_EOF; + } + + int c = mp_hal_stdin_rx_chr(); + + if (c == CHAR_CTRL_C || c == CHAR_CTRL_D) { + reader->eof = true; + mp_hal_stdout_tx_strn("\x04", 1); // indicate end to host + if (c == CHAR_CTRL_C) { + #if MICROPY_KBD_EXCEPTION + MP_STATE_VM(mp_kbd_exception).traceback_data = NULL; + nlr_raise(MP_OBJ_FROM_PTR(&MP_STATE_VM(mp_kbd_exception))); + #else + mp_raise_type(&mp_type_KeyboardInterrupt); + #endif + } else { + return MP_READER_EOF; + } + } + + if (--reader->window_remain == 0) { + mp_hal_stdout_tx_strn("\x01", 1); // indicate window available to host + reader->window_remain = reader->window_max; + } + + return c; +} + +STATIC void mp_reader_stdin_close(void *data) { + mp_reader_stdin_t *reader = (mp_reader_stdin_t *)data; + if (!reader->eof) { + reader->eof = true; + mp_hal_stdout_tx_strn("\x04", 1); // indicate end to host + for (;;) { + int c = mp_hal_stdin_rx_chr(); + if (c == CHAR_CTRL_C || c == CHAR_CTRL_D) { + break; + } + } + } +} + +STATIC void mp_reader_new_stdin(mp_reader_t *reader, mp_reader_stdin_t *reader_stdin, size_t buf_max) { + // Make flow-control window half the buffer size, and indicate to the host that 2x windows are free. + size_t window = buf_max / 2; + char reply[3] = { window & 0xff, window >> 8, 0x01 }; + mp_hal_stdout_tx_strn(reply, sizeof(reply)); + + reader_stdin->eof = false; + reader_stdin->window_max = window; + reader_stdin->window_remain = window; + reader->data = reader_stdin; + reader->readbyte = mp_reader_stdin_readbyte; + reader->close = mp_reader_stdin_close; +} + +STATIC int do_reader_stdin(int c) { + if (c != 'A') { + // Unsupported command. + mp_hal_stdout_tx_strn("R\x00", 2); + return 0; + } + + // Indicate reception of command. + mp_hal_stdout_tx_strn("R\x01", 2); + + mp_reader_t reader; + mp_reader_stdin_t reader_stdin; + mp_reader_new_stdin(&reader, &reader_stdin, MICROPY_REPL_STDIN_BUFFER_MAX); + int exec_flags = EXEC_FLAG_PRINT_EOF | EXEC_FLAG_SOURCE_IS_READER; + return parse_compile_execute(&reader, MP_PARSE_FILE_INPUT, exec_flags); +} + #if MICROPY_REPL_EVENT_DRIVEN typedef struct _repl_t { @@ -203,6 +300,13 @@ void pyexec_event_repl_init(void) { STATIC int pyexec_raw_repl_process_char(int c) { if (c == CHAR_CTRL_A) { // reset raw REPL + if (vstr_len(MP_STATE_VM(repl_line)) == 2 && vstr_str(MP_STATE_VM(repl_line))[0] == CHAR_CTRL_E) { + int ret = do_reader_stdin(vstr_str(MP_STATE_VM(repl_line))[1]); + if (ret & PYEXEC_FORCED_EXIT) { + return ret; + } + goto reset; + } mp_hal_stdout_tx_str("raw REPL; CTRL-B to exit\r\n"); goto reset; } else if (c == CHAR_CTRL_B) { @@ -388,6 +492,15 @@ int pyexec_raw_repl(void) { int c = mp_hal_stdin_rx_chr(); if (c == CHAR_CTRL_A) { // reset raw REPL + if (vstr_len(&line) == 2 && vstr_str(&line)[0] == CHAR_CTRL_E) { + int ret = do_reader_stdin(vstr_str(&line)[1]); + if (ret & PYEXEC_FORCED_EXIT) { + return ret; + } + vstr_reset(&line); + mp_hal_stdout_tx_str(">"); + continue; + } goto raw_repl_reset; } else if (c == CHAR_CTRL_B) { // change to friendly REPL From 18414379ae79665a70cb0c80bf04af088816626b Mon Sep 17 00:00:00 2001 From: Damien George Date: Thu, 8 Oct 2020 23:58:02 +1100 Subject: [PATCH 2/2] tools/pyboard.py: Add fast raw-paste mode. This commit adds support to pyboard.py for the new raw REPL paste mode. Note that this new pyboard.py is fully backwards compatible with old devices (it detects if the device supports the new raw REPL paste mode). Signed-off-by: Damien George --- tools/pyboard.py | 55 +++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/tools/pyboard.py b/tools/pyboard.py index c97ddbe487be3..9b0497d383c69 100755 --- a/tools/pyboard.py +++ b/tools/pyboard.py @@ -253,6 +253,7 @@ def inWaiting(self): class Pyboard: def __init__(self, device, baudrate=115200, user="micro", password="python", wait=0): + self.use_raw_paste = True if device.startswith("exec:"): self.serial = ProcessToSerial(device[len("exec:") :]) elif device.startswith("execpty:"): @@ -359,6 +360,41 @@ def follow(self, timeout, data_consumer=None): # return normal and error output return data, data_err + def raw_paste_write(self, command_bytes): + # Read initial header, with window size. + data = self.serial.read(2) + window_size = data[0] | data[1] << 8 + window_remain = window_size + + # Write out the command_bytes data. + i = 0 + while i < len(command_bytes): + while window_remain == 0 or self.serial.inWaiting(): + data = self.serial.read(1) + if data == b"\x01": + # Device indicated that a new window of data can be sent. + window_remain += window_size + elif data == b"\x04": + # Device indicated abrupt end. Acknowledge it and finish. + self.serial.write(b"\x04") + return + else: + # Unexpected data from device. + raise PyboardError("unexpected read during raw paste: {}".format(data)) + # Send out as much data as possible that fits within the allowed window. + b = command_bytes[i : min(i + window_remain, len(command_bytes))] + self.serial.write(b) + window_remain -= len(b) + i += len(b) + + # Indicate end of data. + self.serial.write(b"\x04") + + # Wait for device to acknowledge end of data. + data = self.read_until(1, b"\x04") + if not data.endswith(b"\x04"): + raise PyboardError("could not complete raw paste: {}".format(data)) + def exec_raw_no_follow(self, command): if isinstance(command, bytes): command_bytes = command @@ -370,7 +406,24 @@ def exec_raw_no_follow(self, command): if not data.endswith(b">"): raise PyboardError("could not enter raw repl") - # write command + if self.use_raw_paste: + # Try to enter raw-paste mode. + self.serial.write(b"\x05A\x01") + data = self.serial.read(2) + if data == b"R\x00": + # Device understood raw-paste command but doesn't support it. + pass + elif data == b"R\x01": + # Device supports raw-paste mode, write out the command using this mode. + return self.raw_paste_write(command_bytes) + else: + # Device doesn't support raw-paste, fall back to normal raw REPL. + data = self.read_until(1, b"w REPL; CTRL-B to exit\r\n>") + if not data.endswith(b"w REPL; CTRL-B to exit\r\n>"): + print(data) + raise PyboardError("could not enter raw repl") + + # Write command using standard raw REPL, 256 bytes every 10ms. for i in range(0, len(command_bytes), 256): self.serial.write(command_bytes[i : min(i + 256, len(command_bytes))]) time.sleep(0.01)