APL-Sheets

A sample Spreadsheet style program in APL.

The following sections are a sort of "Project Record" for the development of this application, to give the reader context in the development process of making this application as well as guidance on how other such applications can be developed as well. These sections are designed to be augmented by the commit log of the repository.

Much of this overall process is an adaptation of Cleanroom Software Engineering to a particular view of APL; see [1] for more information.

The Design Principles

1. APL is an executable specification language
2. Try to implement your system as a specification
3. Make the system "provable/verifiable" first and foremost
4. Emphasize stochastic, black-box testing to certify system behavior
5. Favor linear data flow architectures
6. Minimize entanglement, dependencies, and abstraction
7. Keep state global, isolated, sequential, and potentially serialized
8. Utilize mathematical APL for as much as is easily verifiable using such methods
9. Utilize sequence-based function specification for defining inherently event-driven, branchy code
10. Utilize the immediate-mode GUI paradigm [3] for user-interface interactions

Project Phases

• Project Plan
• User Requirements Analysis
• Initial Architecture Specification
• Increment Planning
• Increment 1
• Increment 2
• Increment 3
• Increment 4
• Increment 5

User Requirements

1. Displays a spreadsheet-like visualization of an APL relation/table.
2. Data is stored efficiently for typical data workflows.
3. Spreadsheet can be scrolled around dynamically.
4. User can edit a cell by clicking on it.
5. Editing can support long contents.
6. User can save the sheet to a file.
7. User can load a sheet from a file previously saved.
8. User can import a sheet from a CSV.
9. Edits should auto-save
10. Sheets can be sorted by columns.
11. Sorting can ignore headers, optionally.
12. User can open a remotely shared file between 2 or more users.
13. User can live-share a sheet to a remote share server.
14. Must use an server-wide authentication token to share a sheet or access one.
15. Two people can edit different cells simultaneously.
16. Two people cannot edit the same cell simultaneously.
17. Remotely shared sheets disappear from the remote server when all users unlink.

Architecture

There are two primary sub-systems:

• Sheets Application
• Document Server

Each syb-system is contained in a single namespace with Run entry point.

The sheets application uses Conga to communicate over a single websocket to an HTMLRenderer instance for its UI. We will use plain HTML for the GUI plus a JavaScript engine to render state changes in the system.

Data is represented as a standard inverted table representation with column types and optional header names.

Data is persisted by serializing to component files, with one component file per inverted table/sheet.

Communication with the document server is accomplished by an always on binary conga socket.

Client UI Websocket protocol

The HTMLRenderer front-end UI interface communicates with the Dyalog APL session via a single websocket connection.

When an event occurs on the interface side, the interface will send an event to the session of the form:

[event_type, data]

Where the data and the event_type sets are to be defined.

The session will send messages to the interface in the form of an array of "rendering" messages, which take the form of:

[node_id, command, data]

Where node_id is the node being manipulated, and the following set of commands and data pairs are possible:

command	data
html	"html_string"
text	"text_string"
value	"data"
set_attr	["attribute", "contents"]
attr	"attribute"
focus	""

Each of these updates the appropriate contents of the node identified by node_id.

In the case of html commands, the innerHTML contents are updated with the data string.

In the case of text commands, the innerText property is updated.

The value command sets the value property of a node.

In the case of set_attr commands, the "attribute" will be set with setAttribute to "contents".

With del_attr, an attribute is deleted.

The focus command will put the focus on the node_id.

Document Server socket protocol

Document server will communicate over a binary data pipe transmitting compressed and serialized APL arrays where each message comes in the form of

('command' data)

And the following messages are supported, either sent by the server or the client.

client/server	command	data
client	'link'	('share_name' 'user')
client	'doc'	document
client	'reshape'	(rows cols)
client	'edit'	((row col) value)
client	'lock'	(row col)
client	'unlock'	''
server	'doc'	document
server	'empty'	''
server	'reshape'	(rows cols)
server	'edit'	((row col) value)
server	'lock'	((row col) user)
server	'unlock'	(row col)

The client is expected to initiate contact with a link command first, and send a doc command on receiving an empty command.

The server will be presumed to have ultimate state for the document status, meaning that lock, unlock, and edit commands are all to be dependent on the state of the server, and a state discrepancy will resolve in favor of the server.

Core Conga Protocols

We are using Conga to do network communication, which forms the basis for all events that will be sent and received. The documentation for Conga is located here:

https://docs.dyalog.com/latest/Conga%20User%20Guide.pdf

We will be using two primary protocols:

1. WebSockets: This is the protocol used for the connection between the HTMLRenderer and the Dyalog APL service. We will depend on the WSAutoUpgrade option for the server.
2. Raw: To communicate with the Document Server, we will communicate over raw bytes with the RawAsByte option enabled on the server.

The important sections in the Conga Documentation relevant to us are:

• Sec. 9.8: Options
• Sec. 4.2: Root Objects
• Sec. 4.1.3: Conga Modes
• Sec. 6.5: WebSocket Protocol
• Sec. A.32: DRC.Wait
• Sec. A.29: DRC.Srv
• Sec. A.3: Conga.Init
• Sec. A.5: Conga.New
• Sec. A.9: DRC.Close
• Sec. A.10: DRC.Clt
• Sec. A.16: DRC.Init
• Sec. A.28: DRC.SetProp

Architectual background information

Much of this architectural design is based on the SAM pattern [2].

Increments

These increments divide up the work into feedback cycles of implementation, certification, and evaluation.

Increment 1: Initial display

Goal: Get the basic architecture wiring complete, implement the core specification loop, and get rendering and UI layout stuff to work. Basic minimal integration, without any core logic or interactivity.

Requirements:

• R#01. Displays a spreadsheet-like visualization of an APL relation/table.
• R#02. Data is stored efficiently for typical data workflows.
• R#03. Spreadsheet can be scrolled around dynamically.

Technical Tasks:

• Specify Client Update protocol
• Implement JS Render Engine
• Define UI Layout
• Define UI Style
• Specify sheet table encoding
• Specify core Sheets event loop
• Define core responses and stimuli
• Define minimal state
• Define Conga uplink
• Enable defining row and column size of table

Increment 2: Cell Editing

Goal: To get core interactivity working and ensure that the JS rendering protocol is sufficient to the task.

Requirements:

• R#04. User can edit a cell by clicking on it.
• R#05. Editing can support long contents.

Technical Tasks:

• Define stimuli and responses for cell edits
• Update specification for handling cell edits
• Ensure state updates are consistent
• Define per cell updates in the UI rendering engine
• Define UI styling for actively edited cells

Increment 3: Persistence

Goal: To enable local file persistence. We want to save and load files of the correct formats.

Requirements:

• R#06. User can save the sheet to a file.
• R#07. User can load a sheet from a file previously saved.
• R#08. User can import a sheet from a CSV.
• R#09. Edits should auto-save

Technical Tasks:

• Define persistence functions
• Define stimuli and responses for loading and saving
• Define UI element for identifying where a file is saved
• Specify new states in the specification table
• Define CSV Import Format

Increment 4: Live sharing

Goal: To enable live sharing and editing

Requirements:

• R#12. User can open a remotely shared file between 2 or more users.
• R#13. User can live-share a sheet to a remote share server.
• R#14. Must use an server-wide authentication token to share a sheet or access one.
• R#15. Two people can edit different cells simultaneously.
• R#16. Two people cannot edit the same cell simultaneously.
• R#17. Remotely shared sheets disappear from the remote server when all users unlink.

Technical Tasks, Document Server:

• Define the document server protocol
• Specify and implement the document server stimuli and responses
• Define the document server specification
• Specify the document server state representation
• Define Conga uplink for the document server

Technical Tasks, Client application:

• Add stimuli and responses for document sharing
• Update specification to include document sharing events
• Add UI elements to handle sharing
• Specify remote sharing functions

Increment 5: Sorting

Goal: To enable editable sorting

Requirements:

• R#10. Sheets can be sorted by columns.
• R#11. Sorting can ignore headers, optionally.

Technical Tasks:

• Enable a permutation vector on state
• Map permutation vector on edits
• Define reactive UI elements to sort columns
• Define sorting responses and stimuli

References

[1] https://www.amazon.com/Cleanroom-Software-Engineering-Technology-Process/dp/0201854805/

[2] https://sam.js.org/

[3] https://caseymuratori.com/blog_0001