Process console command history

[CosminGGeorgescu]

Pull Request Overview

This pull request adds an easy-to-use Bash-like command history for the process console.

Added features

• User can enable/disable the command history from the board's main.rs.
• If enabled, user can scroll up or down by pressing up and down arrow keys respectively.
• Pressing on accident up or down arrow saves the unfinished command and can be fetched back by going to the bottom of command history.
• Modifying a command from history is also saved (if accidentally pressed up or down arrow).
• Empty commands (whitespace-filled) don't get registered in the command history.
• Invalid commands also are saved into the history structure.

Code size for IMIX binary build with enabled/disabled command history:

Build	.text	.data	.bss	.dec
Tock master	174468	0	29592	204060
Enabled with default size	175492	0	28944	204436
Disabled	174292	0	28576	202868

If disabled, the code size decreases.

Testing Strategy

The implementation of the command history was tested on the nrf52840dk and esp32-c3-devkitM-1 boards.

Files changed

capsules/process_console.rs
~ implementation for command history

boards/components/process_console.rs
~ added a new branch to process_console_component_static macro for easily enable/disable the command history

Documentation Updated

Updated the documentation in doc/Process_Console.md accordingly :

• Code support on how to enable/disable the command history
• How to use command history
• Support for selecting a custom size for the command history

TODO or Help Wanted

Feedback is much appreciated.

Documentation Updated

• Updated the relevant files in /docs.

Formatting

• Ran make prepush.

[alexandruradovici]

This feature is very helpful in education, most of our students tried immediately to scroll up and down. I think this can be added, as it does not increase code size.

I curious though why the code size actually decreases with the feature turned off.

[alexandruradovici]

Kind reminder to review this, we need this the for Rust Nation presentation. :)

[lschuermann]

I'm by no means an expert on the process console code, but from a coarse review this looks good to me. I'm not too worried about the code size or memory usage impact of this (as long as it doesn't grow astronomically), given it's intended solely for debugging and experimentation.

[hudson-ayers]

The decreased code size when this is disabled is just an indication that LLVM was making some suboptimal inlining decisions regarding the process console before this was added, but that may not apply universally across boards and could be fixed with explicit inline directives anyway.

Since this can be disabled from main I am indifferent to the addition -- I personally would probably advocate for it being off-by-default (i.e. unless you specify a length when creating the component no commands will be saved) but will not block this on that.

[alexandruradovici]

I think that for educational purposes and for getting started with Tock, the history should be on by default. Most developers do expect this feature from the console, all of our students being frustrated that it was not available.

[phil-levis]

As the original process console author, I agree with this improvement conceptually. I haven't done a code review.

[jettr]

I meant to collaborate with Alex on this since we have a downstream version of process console that handles command history.

I want to share the small state machines that we use down stream that you can either choose to use or not (your call). I just wanted it to be an option. Consider this last minute collaboration.

/// Start of terminal escape sequence
const CHAR_ESC_START: u8 = b'\x1B'; 

/// The FSM for processing terminal escape sequences
#[derive(Copy, Clone)]
enum EscState {
    /// Most recent character is not part of an escapee sequence, so process it elsewhere
    Bypass,
    /// The escape sequence completed with a recognized escape key
    Complete(EscKey),
    Started,
    Bracket,
    Bracket1,
    Bracket3,
    Bracket4,
    Unrecognized,
    UnrecognizedDone,
}

impl EscState {
    /// Process the data for this state and returns the next state in the FSM
    fn next_state(self, data: u8) -> Self {
        use self::{EscKey::*, EscState::*};
        match (self, data) {
            (Bypass, CHAR_ESC_START)
            | (UnrecognizedDone, CHAR_ESC_START)
            | (Complete(_), CHAR_ESC_START) => Started,
            (Bypass, _) | (UnrecognizedDone, _) | (Complete(_), _) => Bypass,
            (Started, b'[') => Bracket,
            (Bracket, b'A') => Complete(Up),
            (Bracket, b'B') => Complete(Down),
            (Bracket, b'C') => Complete(Right),
            (Bracket, b'D') => Complete(Left),
            (Bracket, b'1') => Bracket1,
            (Bracket1, b'~') => Complete(Home),
            (Bracket, b'3') => Bracket3,
            (Bracket3, b'~') => Complete(Delete),
            (Bracket, b'4') => Bracket4,
            (Bracket4, b'~') => Complete(End),
            _ => {
                if EscState::terminator_esc_char(data) {
                    UnrecognizedDone
                } else {
                    Unrecognized
                }
            }
        }
    }

    /// Returns true if the processing next state logically consumed the data and it should not
    /// be processed elsewhere
    pub fn data_consumed(&self) -> bool {
        !matches!(self, EscState::Bypass)
    }

    fn terminator_esc_char(data: u8) -> bool {
        data.is_ascii_alphabetic() || data == b'~'
    }
}

And when it is used, we do something like the follow. Note that cursor is the position of the cursor and len is the total pending command length.

        let esc_state = self.esc_state.get().next_state(ch);
        self.esc_state.set(esc_state);

        if let EscState::Complete(key) = esc_state {
            match key {
                EscKey::Right if cursor < len => {
                    Direction::Right.move_cursor(1);
                    cursor += 1;
                }
                EscKey::Left if cursor > 0 => {
                    Direction::Left.move_cursor(1);
                    cursor -= 1;
                }
                EscKey::End if cursor < len => {
                    Direction::Right.move_cursor(len - cursor);
                    cursor = len;
                }
                EscKey::Home if cursor > 0 => {
                    Direction::Left.move_cursor(cursor);
                    cursor = 0;
                }
                EscKey::Delete if cursor < len => {
                    // Remove the character under the cursor
                    try_copy_within(&mut cmd_buf, cursor + 1..len, cursor)?;
                    len -= 1;
                    // Print the updated buffer with a clearing space at end
                    for c in cmd_buf.iter().skip(cursor).take(len - cursor) {
                        console_output_no_eol!("{}", *c as char);
                    }
                    console_output_no_eol!(" ");
                    Direction::Left.move_cursor(len - cursor + 1);
                }
                EscKey::Up | EscKey::Down => {
                    let res: Result<(), ErrorCode> = self.mru.map_or(Ok(()), |mru| {
                        // Clear the current line on console
                        Direction::Left.move_cursor(cursor);
                        for _ in 0..len {
                            console_output_no_eol!(" ");
                        }
                        Direction::Left.move_cursor(len);

                        // Copy in the next data from mru circular buffer slices
                        let (b1, b2) = if matches!(key, EscKey::Up) {
                            mru.get_next_older()
                        } else {
                            mru.get_next_newer()
                        };
                        let total = b1.len() + b2.len();
                        cmd_buf
                            .get_mut(..b1.len())
                            .ok_or(ErrorCode::INVAL)?
                            .copy_from_slice(b1);
                        cmd_buf
                            .get_mut(b1.len()..total)
                            .ok_or(ErrorCode::INVAL)?
                            .copy_from_slice(b2);
                        cursor = total;
                        len = total;

                        // Print all new characters
                        for c in cmd_buf.iter().take(len) {
                            console_output_no_eol!("{}", *c as char);
                        }
                        Ok(())
                    });
                    res?
                }
                _ => {}
            };
        }
        // If the escape sequence consumed the character, we are done processing this character
        if esc_state.data_consumed() {
            return Ok(());
        }

Also here is the Direction helper

/// Direction to move a cursor
enum Direction {
    Left,
    Right,
}

impl Direction {
    fn move_cursor(&self, count: usize) {
        if count != 0 {
            console_output_no_eol!(
                "\x1b[{}{}",
                count,
                match self {
                    Direction::Left => "D",
                    Direction::Right => "C",
                }
            );
        }
    }
}

Lastly, we have a Most Recently Used (mru in code above) circular buffer cache for console history that compactly stores previous commands with an upper bound limit on the totally number of bytes and stored commands, which ever comes first. I can also share this too with its unit tests.

Also don't feel like you have to make these changes before landing this PR. If you want to incorporate this feedback in a follow up PR, I would be happy to help out. I just wanted to start talking about this.

[alexandruradovici]

I agree, @MATrix22 @CosminGGeorgescu you can prepare a new pull request as soon as this gets merged. Thank you @jettr for the feedback.

[hudson-ayers]

bors r+

(On the Friday call, there was agreement that we could merge this as-is then seperately incorporate Jett's suggestion in a new PR, so as to ensure this makes it in before RustNation)

[bors]

Build succeeded:

• ci-build (macos-latest)
• ci-build (ubuntu-latest)
• ci-format (ubuntu-latest)
• ci-qemu
• ci-tests (macos-latest)
• ci-tests (ubuntu-latest)