Skip to content

UI Design

Autumn edited this page Feb 14, 2022 · 22 revisions

UI Software Design

See screencast of small-poc is on youtube:


Semantic Widgets

All visible objects on the screen (widgets) should carry semantic information with them whenever possible. Semantic information will allow various interactions with the widgets, such as appropriate context menu.

Widgets should have unambigous references to the underlying objects they represent. For example, for EC2 instance that would be AWS account id + region + instance id. ARN format would probably be good for interoperability.

  • Sharing a widget with another person should be possible (more about this in Timeline Design - Sharing
  • All objects/widgets must have unique ID to assist cross-session correlation.


Support of the "session" concept - a work on particular task by the operator.

  • Ability to switch to a session at any time, even after exiting and starting the shell.
  • Separate per-session history, so it could be shared, removed, etc. See also Timeline Design.
  • Ability to re-play session, step by step, optionally modifying commands.
  • Continuing session after exiting and restarting the shell would be possible


All operations made via the UI, including mouse operations in GUI must have and display textual representation, allowing: copy / paste / save to a file / send to friend / log in history

  • All widgets displayed in the UI must be results of commands with textual representation
    • Example: if a user sorts a table using the UI, a command must be generated and run to display the sorted results
    • All widgets must carry with them the commands that generated them. More specifically a history entry which includes whatever is necessary to reproduce the command (working directory, environment variables, etc). Maybe standard input... (and output for history purposes).


Ability to support several i/o modules such as terminal (CLI) and web.


Provide API to NGS scripts (and other scripts via CLI) to create/update widgets.

  • Example: API/protocol to report the progress
    • Current task (Copying mydata.txt to /tmp/)
    • Overall progress (70% or File 7 out of 10)
    • ETA maybe

View / Interaction - General

TODO: Review this section.

  • "Training wheels" - configurable amount of help for beginners (similar to Lynx browser). Examples:
    • Shortcuts can be shown
  • Interactivity
    • Keep in mind that many of the widgets will be interactive, not just a static text dump
    • Interactive data types
      • Protocol for specifying operations on the interactive data types
      • Should gracefully degrade when go to file or piped to another program instead of interactive UI
    • Allow navigation on the screen, choosing operations from a menu for the objects
    • Allow selection of multiple objects and performing an operation on all of them
      • The generated code will have a loop or use a facility for parallel execution (contextually derived or user provided which one)
    • History of operations in context menu - all actions must have textual representation. This enables history.
    • Interoperability with non-NGS scripts. External programs should be able:
      • Detect running under NGS
      • Return interactive objects
  • Display structured results as such (JSON, Yaml, ...)
    • Most of the data dealt with is tables. List of files, list of instances in a cloud, list of load balancers. Amazing that none of current UNIX shell tools (I heard of) don't treat the data as such. The closest you get is set of records in awk. Well, if the fields in records are the same it's actually a table. $1 in awk could be id or name, referencing the data by column name and not by field number. Yes, you have jq and it's close but it still works (in best case) with list of records with same fields. PowerShell has something in that direction (Get-Process | Where-Object {$_.handles -gt 200}) in combination with Format-Table.
    • (Maybe) Allow editing it and saving to file.
    • (Maybe) Allow write jq filters in (G)UI by selecting the elements
  • Easy navigation to related objects (EC2 instance <-> Containing VPC; EC2 instance <-> attached EBS; EC2 instance <-> attached subnet)
  • Smart output grouping for large outputs. For example, if there are more than X EC2 instances are listed, they will not be shown by default but rather a summary. The summary will allow further navigation using additional filtering on the original query. Example output for large number of EC2 instances: ""You have listed 1000 instances. Suggested filters: (A) 900 t1.small , 100 c3.medium (B) 800 env=prod , 200 env=Dev, ...". Low cardinality filters first?
  • Tables should support
    • different types of resources in one table
    • different parts of table generated by different commands?
    • grouping (including by things like cost)

Context Area

  • There should be one area which would hold widgets for current contexts
  • Examples of contexts
    • Current directory
    • Current AWS account/profile (open issue - support for multiple)
    • Current Git repository?
    • Resource(s) in a cloud?
  • Context widget should have a history (LRU) so that switching to previous context would be easy. This is per context type history.
  • Bookmarks?
    • Including to point in the Timeline
  • Copy/paste buffers?
  • Temporary and/or open files?
  • Contexts should have stable/predictable order on the screen

See Context Design

Visualization Area

  • Visualization area shows diagrams (SVG? Text?) for easy navigation to related objects.
  • Visualization area can contain different views. Examples:
    • Network (EC2 instance -> subnets, VPC)
    • Security (EC2 instance -> security groups, instance profile)
    • Relevant CloudWatch graphs

Commands Area and Processes handling

TODO: Review this section.

  • Commands prompt and input area
    • The prompt should be interactive (for example, click on git branch name in prompt -> context menu -> "change branch" -> branches menu -> pick one)
    • The prompt should be dynamically-updatable (for example would poll something and display result)
    • Notification widgets, so that different plugins/components have a place to report the status. Notification widgets must be interactive. Example of notifications widgets which could be implemented:
      • Build (in CodeBuild or Travis CI) succeeded/failed.
      • CloudWatch alarms
      • CloudTrail events notifications
      • Remote git branch got new commits
      • SNS subscription
      • Polling (SSH) a server till some condition is true, such as provision finished
      • Polling a server till SSH connection is possible
    • Commands scroll up, new commands are added at the bottom. When a command that haven't completed yet, reaches top of the screen, it can be converted to a mini-area at the top (right?) of the screen, representing the command and current progress (and exit status later).
    • Dynamic update command output, think ps / top when their output is in the stream
    • "pin" feature so user defined command sticks to the screen and being re-run and the output updated, essentially making it a widget
  • Each process will have its own area on the screen
  • Not to block, allow typing next commands even if previous command is still running
    • By default, consider commands related and only queue next commands but do not execute
    • Upon successfull completion of a command, execution of the next one starts
    • If an error occures, pending commands are blocked, until the user resolves the situation and reruns them
    • There should be an option not to queue the newly entered command but to start the execution immediately
  • Output
    • Truncated stdout/stderr: Commands' outputs displayed below the commands up to max N lines then scroll the output in a small window below the command. When there is more output than a human can process - don't display it and suggest saving it to a file (maybe).
    • Option to view full stdout/stderr in a pager
    • Ability to mark "seen until here" in output, similar to pressing enter several times in a terminal while a command is running.
  • Provide good feedback. In GUI for example, this can be green / red icon near a completed command to show exit status. Tweaking prompt to include such info or typing echo $? all the time is not what I dream about.
    • External, unaware programs:
      • Easy way to write hooks that will provide additional information about running process, such as progress.
      • For external programs that will be unaware of any progress reporting protocol (currently all existing programs), use heuristics such as look at open files + position in files to guess the progress with high probability.
  • Commands history: among duplicate commands all but last should be grayed out, so that non-grayed out commands are unique.
  • When hover over an object, highlight the same object everywhere it appears on the screen.
  • [later] Confirmation mode. One user in collaboration mode gives the command to execute, another user must approve the command for execution.
  • [later] Underline red/green for existing/non-existing files? Fish shell does it for the commands.
  • Maybe: allow automatic chaining of monitoring commands. Example: after call to async command, automatically execute polling command until the status change from semantic "in progress" to something else (success/failure).


TODO: Review this section.

  • Smart completion, context sensitive

    • Command switches and values
    • Complete objects (file names, urls, etc) depending on context. Think wget .../x.tgz, tar [COMPLETION_KEY] [Maybe some choice keys] -> xzf x.tgz
    • Maybe API to insert objects to completion history
    • Auto-detect completion history objects for existing commands (by wrappers and/or hooks probably)
  • "Mentioned" completion

    • Complete objects from output of previous commands. Example: apt-cache search ... , apt-get install ... Isn't this copy+paste annoying? It's already on the screen, it's a package name, and still the system can't complete...

Open issues

  • How to deal with a command that requires regular stdin/stdout interaction.
  • Maybe: API/protocol for interaction
    • Maybe using file descriptors (must be stdin/stdout in case of remote shell?)


  • UI representation of a job. A map with all involved processes, their open files, sockets, pipes, resource usage (CPU, disk, network), process running time, accumulative CPU time, ...

Terminal integration

TODO: Review this section.

  • Bidirectional metadata communications channel with the shell (for iTerm2 integration and possibly others). Most of the points below are rephrased email from George Nachman, creator of iTerm2.
    • Check Protocol Buffers
    • New DCS code to communicate capability of the shell and then switch protocols
      • Need to make sure that all output is captured and goes through the new protocol
    • Features
      • Possible marking of output metadata such as
        • Belonging to a particular process
        • Output file descriptor (stdout/stderr) on which the data came out of the process
        • Mark objects such as files so that terminal could provide context menu with file operations for example.
        • In general, pass all possible metadata about the objects on the screen
      • Tell terminal about running jobs so it could present a task manager (plus maybe UI to select active task on the terminal side)
      • Typing a command while a command is running in cooked mode could have a nice UI rather than mixing command output with echoed keystrokes
      • File transfer between terminal and the shell
      • Status bar
      • Semantic history (need to check how relates to History Design)
      • Interactive commands on the objects on the screen
        • Shell sends context menu for each object (or semantic object type) to the terminal
        • Terminal sends the shell menu interaction result

Confined output of a program

External programs' outputs will be confined. The main readme contains many ideas about this (which should be move from readme to this page).

Related Projects

Design (WIP)

TODO: Overall description

Components (WIP)

TODO: List of components. For each: description of the role, API, etc.

  • Session manager - anything pertaining to history and managing the whole list of widgets
  • Layout manager - organizes widgets on a canvas

Types and methods (WIP)

  • UI::SessionManager

    • Fields
      • TBD
    • Methods
      • push(SessionManager, Widget) ?
  • Widget type. (CommandsPipelineWidget, CommandWidget, etc subtypes)

    • Fields
      • id - globally unique widget id, probably UUIDv4, maybe URI (ngs:widget:SESSION-ID:WIDGET-ID)
      • uoid - underlying object id
      • meta - a Namespace
        • created - Time type object
        • updated - Time type object
      • data - a Namespace of fields, subtype-specific
      • children - sub-widgets. Note that displaying order of children is independently configurable, somewhat like CSS.
    • Methods
      • render(Widget, ...). Parameters TBD. Candidates: constraints (for dimensions etc), target (for ), context. Returns target-specific data structure. For a terminal that would be the text lines for example. The output might be even more specific depedending on for example whether ther terminal supports UTF-8.


Terminal Libraries / Applications

  • - a library facilitating complex TUIs on modern terminal emulators with image support (sixel, Kitty). Very fast sixel encoder. C + bindings to others
  • - a tool and library that can convert images into either Unicode pictures or terminal image formats (sixel, iTerm2, Kitty). Very fast sixel encoder. C
  • - libsixel, a library for encoding/decoding sixel images, with good support on real hardware terminals. Fast sixel encoder. C + bindings to others


These terminals have good working image support with either sixel or iTerm2 image formats. (Kitty images are only supported by kitty and wezterm, and as of February 2022 do not always behave the same way.)

Widget Examples