Runtime Introspection

[iamrecursion]

Summary

The main selling point of Enso and Enso Studio is the language's interactivity. However, enabling this interactivity requires deep runtime support for introspection of running code. This allows the GUI to obtain values and type information to display to users, but also allows for the implementation of more complex functionality such as debuggers and a REPL.

Value

Without the support for runtime introspection it is impossible to provide the promised Enso GUI, so this is inherently very necessary.

Specification

The following elements are presented in order as a plan for the introspection functionality into the interpreter in a way that allows it to take place in parallel with the work on the new syntax (#806). Each of these phases should not break interpreter behaviour on master. Additionally, it is very important that the runtime and the engine only talk via org.graalvm.polyglot. There should never be an explicit runtime dependency on runtime by either gateway or language_server.

• Implement passing of source-code location information through the interpreter (Absolute Spans in the AST #752)
• Take the interpreter's handling of dependencies and the execution flow and streamline it to make it work with interactive workflows (Re-Architect the Interpreter Lifecycle #756).
• Design the language server protocol (Design the Language Server Protocol #912).
• Extract values from scopes based on source locations (Implement Value Extraction from Scopes #779), and use this to enable arbitrary code execution on these values (Allow Arbitrary Code Execution on Extracted Values #778).
• Implement support for 'entering' functions in the interpreter (Enable Entering Arbitrary Functions #726).
• Implement stub services for the gateway and language server components (Stub Out Gateway and Language Server Akka Services #742).
• Implement caching of values in the interpreter (Implement Value Caching #782).
• Clarify the language server and gateway architecture (LSP Design Clarification #700, JSON-RPC WebSocket Server Library #699, Re-Architect the Language Server Codebase #698).
• Design support for multi-client connections in the engine, and record this design in the documentation (Design Rudimentary Multi-Client Support #736). From now on, all future functionality should account for multi-client development.
• Implement the required runtime functionality to support the basic language server workflow (Implement the LSP Starting Point Runtime Functionality #739).
• Implement support for capabilities and write-lock in the language server (Implement Capability Registration #701, Implement Multiclient Write Lock #714).
• Text buffer management (Determine the Buffer Representation #667, Implement the Text Buffer Structure #666).
• Implement the text editing functionality of the language server (Implement text/openFile #665, Implement text/closeFile #664, Implement text/save #663, Implement text/applyEdit #662, Implement text/didChange #661).
• Implement the file-manager functionality of the language server (Writing Files #679, Reading Files #678, Creating Files and Directories #677, Deleting Files and Directories #676, Copying Files and Directories #675, Moving Files and Directories #674, Exists for Files and Directories #673, Directory Tree Listing #672, Revisit Compatibility with GraalVM Java 11 Builds #476, Requests for File/Directory Information #670, Filesystem Events #669 ). Care must be taken to avoid conflicts with these operations in a multi-client situation.
• Implement support for execution management (Design the Execution Management Functionality #710, Integrate the Language Server and Runtime #734, Implement executionContext/create #658, Implement executionContext/push #657, Implement executionContext/expressionValuesComputed #656, Execution Context Integration with Text Change Events #655, Implement executionContext/destroy #654, Implement executionContext/pop #653, Implement executionContext/recompute #652, Implement executionContext/fork #481).
• Implement the runtime server support for execution management (Provide the ID Map to the Interpreter #649, Execution Service for the Runtime Server #648, Runtime Server Support for Code Diffs #647, Runtime Server Internal Context Management #646).
• Implement stopgap JSON Serialisation (Add Stopgap JSON Serialisation #637).
• Implement visualisation support (Flatbuffers Learning #639, Truffle Integration for the Binary Protocol #640).
• Support for working with binary files (Binary File Reading and Writing Support #668).
• Fix flatbuffers stuff (Fix Flatbuffers Issues for IDE #609).
• Project manager uberjar (Build an Uberjar with the Project Manager and the Language Server #614).
• I/O stream redirection (Input/Output Stream Redirection #705).
• Introduce threaded execution to the runtime server instrument (Introduce Threaded Executions to the Runtime Server Instrument #606).
• Add support for dynamic content roots (Support for Dynamic Content Roots #486).
• Execution context enhancements (Specify and create an executionContext/pause and unpause methods #471, Implement executionContext/fork #481).
• Bug fixing ( Vulnerability: Path Traversal attack #474, Asynchronous Stopping of a Language Server Causes ProjectManagementApiSpec to Fail #479).

Acceptance Criteria & Test Cases

• The above requirements have been met.
• They have been rigorously tested, and are accompanied by benchmarks designed to detect regressions.