Docs

Table of Contents

1. Project Structure
2. Shortcuts
3. Understanding the Data Model
   3.1 Data Types
   3.2 Working with the Data Types
   3.3 Views
4. Data Storage/Persistence
5. Metaprogramming
6. Cursor and Caret

1. Project Structure

The main directory structure is organized as follows. Tests are located in a subdirectory named __tests__ in in each directory.

• /src/App.css - All the styles. New components should use styled components (particularly @emotion/styled), but legacy styles are still kept in one giant stylesheet.
• /src/constants.ts - Constant values. For constants that are only used in a single module, start by defining them in the module itself. They can be moved to constants.js if they need to be used in multiple modules or if there is a strong case to define them separately, e.g. an app-wide configuration that may need to be changed or tweaked.
• /src/action-creators - Redux action creators. Prefer reducers; only define an action creator if it requires a side effect. Use strings directly rather than defining an action type; I haven't found any benefit in practice for separate action type definitions as typos are immediately exposed. Use action creators rather than dispatching actions directly in order to gain type safety.
• /src/components - React functional components.
• /src/hooks - Custom React hooks. The project started before hooks were released, so we don't leverage them a lot, but we would like to use them more.
• /src/reducers - Redux reducers. Use util/reducerFlow to compose reducers.
• /src/redux-enhancers - Redux enhancers
• /src/redux-middleware - Redux middleware
• /src/selectors - Select, compute, and possibly memoize slices from state.
• /src/shortcuts - Keyboard and gesture shortcuts
• /src/util - Miscellaneous

2. Shortcuts

There are more than 50 shortcuts that are available to the user for editing and navigating. Each of these shortcuts has a corresponding action-creator, reducer, and unit test.

bindContext	indentOnSpace	pinOpen
bumpThoughtDown	index	pinSubthoughts
clearThought	join	proseView
collapseContext	moveCursorBackward	redo
cursorBack	moveCursorForward	search
cursorDown	moveThoughtDown	splitSentences
cursorForward	moveThoughtUp	subcategorizeAll
cursorNext	newGrandChild	subcategorizeOne
cursorPrev	newSubthought	toggleContextView
cursorUp	newSubthoughtTop	toggleHiddenThoughts
delete	newThoughtAbove	toggleSidebar
deleteEmptyThoughtOrOutdent	newThoughtOrOutdent	toggleSort
exportContext	newUncle	toggleSplitView
extractThought	note	toggleTableView
home	openShortcutPopup	undo
indent	outdent

3. Understanding the Data Model

3.1. Data Types

3.1.0. Thought

We describe em to users in terms of creating, editing, and organizing their thoughts.

Thought is the data type for a specific thought in the tree. Thoughts are stored in the Redux state as state.thoughts.thoughtIndex. Individual thoughts can be retrieved with the selector getThoughtById.

rank

The rank of a Thought is a number used to determine the thought's sort order among its siblings.

Ranks are unique within a single context. There is no relationship between ranks across contexts.

Ranks are relative; the absolute value does not matter. What matters is only if a rank is greater than or less than other ranks in the same context.

A new thought will be assigned a rank depending on where it is inserted:

• at the end of a context → rank of last thought + 1
• at the beginning of a context → rank of first thought - 1 (may be negative!)
• in the middle of a context → rank halfway between surrounding siblings (may be fractional!)

Negative ranks allow new thoughts to be efficiently inserted at the beginning of a context without having to modify the ranks of all other siblings. e.g. If a thought is placed before a thought with ranks 0, it will be assigned a rank of -1.

Fractional ranks allow new thoughts to efficiently be inserted between any two siblings without having to modify the ranks of other siblings. e.g. If a thought is placed between thoughts with ranks 5 and 6, it will be assigned a rank of 5.5.

importJSON autoincrements the ranks of imported thoughts across contexts for efficiency and may result in different ranks that would be produced by manually adding the thoughts, but the sibling-relative ordering will be the same.

3.1.1. Context

The word context refers to the ancestor path of a thought. e.g. c is in the context a/b:

- a
  - b
    - c

3.1.2. Path

/** A sequence of thoughts from contiguous contexts. */
type Path = ThoughtId[]

e.g. ['kv9a-vzva-ac4n', '2mv0-atk3-tjlw', 'vkwt-ftz1-094z'] may represent the Path for a/b/c:

- a
  - b
    - c
  - d

A Path always starts at the ROOT thought. The ROOT thought is not part of the Path itself, but is implied as the starting point. i.e. The first id of a Path (e.g. kv9a-vzva-ac4n) is a child of the ROOT thought. The Path representing the ROOT thought itself is a special case defined as [HOME_TOKEN].

An important Path in em is the thought that is being edited: state.cursor. When the user clicks on thought A, state.cursor will be set to [idOfA]. Navigating to a subthought will append the child's id to the cursor. So hitting ArrowDown on A will set the cursor to [idOfA, idOfB], etc.

Circular Paths are allowed. This is possible because of the Context View, described below, which allows jumping across the hierarchy.

3.1.3. SimplePath

/** A contiguous Path with no cycles. */
export type SimplePath = Path & Brand<'SimplePath'>

A SimplePath is a Path that has not crossed any Context Views, and thus has no cycles. Typescript is not expressive enough to capture this property in a type, but we can use brand types to require explicit casting, thus minimizing the chance of using a Path with cycles when a SimplePath is required. A Brand type is a nominal type that disallows implicit conversion. See: https://spin.atomicobject.com/2018/01/15/typescript-flexible-nominal-typing/.

3.1.4. Lexeme

/** An object that contains a list of contexts where a lexeme appears in 
    different word forms (plural, different cases, emojis, etc). */
export interface Lexeme {
  contexts: ThoughtId[],
  created: Timestamp,
  lastUpdated: Timestamp,
  updatedBy: string
}

A Lexeme stores all the contexts where a thought appears in identical and near-identical word forms (ignoring case, plurality, emojis, etc). Lexemes are stored in state.thoughts.lexemeIndex and keyed by a hash of the thought value. Think of these as the inbound links to a thought.

e.g. The Lexeme for cat contains two contexts, Animals and Socrates:

- Animals
  - Cats
  - Dogs
- My Pets
  - Socrates
    - cat

Usage Tip: Use the getLexeme selector to get the Lexeme for a thought value.

3.2 Working with the Data Types

There are several common selectors that are used to access a parent, child, or sibling of a thought. Consider the following thought structure:

- a
  - b
    - c
      - x
      - y
      - z

• To get the parent Thought of Thought c: thoughtB ≅ getThoughtById(state, thoughtC.parentId). See: getThoughtById.
• To get the parent Path of Path a/b/c: pathAB ≅ rootedParentOf(state, pathABC). See: rootedParentOf.
  • Note: rootedParentOf always return a valid Path. If passed a child of the ROOT context, rootedParentOf returns [HOME_TOKEN]. If parentOf is passed a child of the ROOT context, it returns an empty array, which is not a valid Path. parentOf should only be used if one or more additional thoughts are immediately appended, e.g. appendToPath(parentOf(path), thoughtId). This is not currently reflected in the return type of parentOf.
• To get the children of Thought c: childrenOfC ≅ getAllChildrenAsThoughtsById(state, thoughtC.id). If you have the Context of c rather than its ID, you can use: childrenOfC ≅ getAllChildren(state, contextABC). See: getChildren.ts.
• To get the next sibling of Thought y: thoughtZ ≅ nextSibling(state, 'y', contextABC). See: nextSibling.
• To get the previous sibling of Thought y: thoughtX ≅ prevSibling(state, 'y', contextABC). See: prevSibling.

Those are just the most basic. There are many selectors and util functions which can be used to traverse, navigate, and convert between Thought, ThoughtId, Context, and Path:

• basic traversal
  • parentOfThought - Returns the parent Thought of a given ThoughtId.
  • nextSibling/prevSibling - Gets the next/previous sibling of a thought, according to its parent's sort preference.
  • rootedParentOf - Gets the parent Context/Path of a given Context/Path. If passed a child of the root thought, returns [HOME_TOKEN] or [ABSOLUTE_TOKEN] as appropriate.
  • parentOf - Gets the parent of a Context or Path. Use rootedParentOf instead unless you are immediately appending an additional thoughtnn
  • appendToPath - Appends one or more child nodes to a Path or SimplePath. Ensures the root thought is removed.
• children
  • getChildren - Gets all visible children of a Context, unordered.
  • getAllChildren - Returns the subthoughts (as ThoughtIds) of the given context unordered. If the subthoughts have not changed, returns the same object reference.
  • getAllChildrenAsThoughts - Returns the subthoughts (as Thoughts) of the given context unordered.
  • getAllChildrenAsThoughtsById - Returns the subthoughts (as Thoughts) of the given ThoughtId unordered.
  • getAllChildrenSorted - Gets all children of a Context sorted by rank or sort preference.
  • getChildrenRanked - Gets all children of a Context sorted by their ranking. Returns a new object reference even if the children have not changed.
  • getChildrenRankedById - Gets all children of a ThoughtId sorted by their ranking. Returns a new object reference even if the children have not changed.
  • getChildrenSorted - Gets all visible children of a Context sorted by rank or sort preference.
• lookup/conversion
  • getThoughtById - Gets a Thought by its ThoughtId.
  • pathToThought - Gets the head Thought of a path.
  • pathToContext - Converts a Path to a Context.
  • contextToPath - DEPRECATED: Converts a Context to a Path. This is a lossy function! If there is a duplicate thought in the same context, it takes the first. It should be removed. Build up the Path from information that is already in scope, or use thoughtToPath instead.
  • thoughtToPath - Generates the Path for a Thought by traversing upwards to the root thought.
  • head - Gets the last ThoughtId or value in a Path or Context.
  • childIdsToThoughts - Converts a list of ThoughtIds to a list of Thoughts. If any one of the thoughts are not found, returns null.
  • thoughtToContext - Generates the Context for a Thought by traversing upwards to the ROOT thought.
  • contextToThought - Gets the head Thought of a context.
  • contextToThoughtId - Recursively finds the thought represented by the context and returns the id. This is the part of the independent migration strategy. Will likely be changed to some other name later.
  • unroot - Removes the root token from the beginning of a Context or Path.
• ancestors/descendants
  • getAncestorBy - Traverses the thought tree upwards from the given thought and returns the first ancestor that passes the check function.
  • getDescendantContexts - Generates a flat list of all descendant Contexts. If a filterFunction is provided, descendants of thoughts that are filtered out are not traversed.
  • getDescendantThoughtIds - Generates a flat list of all descendant Paths. If a filterFunction is provided, descendants of thoughts that are filtered out are not traversed.
  • ancestors - Returns a subpath of ancestor children up to the given thought (inclusive).
  • isDescendant
  • isDescendantPath
• predicates
  • isHome - Returns true if the Thoughts or Path is the home context.
  • isRoot - Returns true if the Thoughts or Path is the one of the root contexts.
  • hasChild
  • hasLexeme
  • equalPath
  • equalThoughtRanked
  • equalThoughtSorted
  • equalThoughtValue
• attributes
  • attribute
  • attributeEquals
• lexemes
  • getLexeme - Gets the Lexeme of a given value.
  • getLexemeById - Gets the Lexeme at the given lexemeIndex key.
  • getContexts - Returns all of the Thoughts of a thought's Lexeme.
• context view
  • appendChildToPath - Appends the head of a child SimplePath to a parent Path. In case of a parent with an active context view, it appends the head of the parent of the childPath.
  • getChildPath
  • simplifyPath - Infers the path from a path that may cross one or more context views.

3.3. Views

3.3.1. Normal View

In normal view (default), a thought's children are rendered in a collapsible tree.

- a
  - m [cursor] 
    - x
    - y
- b
  - m
    - y
    - z

3.3.1.1. Normal View Recursion [DEPRECATED]

3.3.2. Context View

Warning: Context View functionality is currently disabled and under redesign.

Enter Ctrl + Shift + S, or click the button in the toolbar to activate the Context View on a given thought. This will show all the contexts that a thought appears in.

e.g. Given a normal view...

- a
  - m
    - x
    - y
- b
  - m
    - y
    - z

Activating the context view on m (indicated by ~) renders:

- a
  - m~ [cursor]
    - a
    - b
      - y
      - z
- b
  - m
    - y
    - z

a and b are listed under a/m~ because they are the contexts that m appears in. They are the inbound links to m, as opposed to the outbound links that are rendered from a context to a child.

Note: The ranks of the contexts are autogenerated and do not correspond with the rank of the thought within its context, but rather the sorted order of the contexts in the context view.

Usage Tip: Use getContexts() or getThought(...).contexts to get the contexts for a thought value.

3.3.2.1. Context View Recursion

Descendants of contexts within a context view are rendered recursively. The Child thoughts that are generated from the list of contexts mentioned above can render their own Child thoughts (defaulting to Normal View). But what Context to use? When the parent is in Normal View, a Path is converted to a Context. When the parent, is in Context View, e.g. ['a', 'm'], we have direct access to the Context not from a Path but from getContexts('m'): [{ context: ['a'], rank: 0 }, { context: ['b'], rank: 1 }]. We then combine the desired Context with the head thought to render the expected Child thoughts. See the following example.

Note: The cursor here is circular. The underlying data structure allows for cycles. This is possible because only a fixed number of levels of depth are shown at a time.

(~ indicates context view)

- a
  - m~
    - a [cursor]
      - x
      - y
    - b
- b
  - m
    - y
    - z

The ThoughtContexts for m are [{ context: ['a'], rank: 0 }, { context: ['b'], rank: 1 }]. Where do x and y come from? They are the children of ['a', 'm']. When the Context View of m is activated, and the context ['a'] is selected, it renders the children of ['a', 'm'].

3.3.2.2. contextChain

When working with Context Views, it is necessary to switch between the full Path that crosses multiple Context Views, and the contiguous SimplePath segments that make it up. This is the only way to get from a Path that crosses multiple Context Views to a single Context, which does not allow cycles.

This more verbose and explicit representation of a transhierarchical Path and its different Context View boundaries is called a contextChain. contextChain is not stored in state, but derived from the cursor via splitChain(state, cursor). Consider the following:

• a
  • m
    • x
• b
  • m
    • y

When cursor is a/m~/b/y, then contextChain is (ranks omitted for readability):

[
  ['a', 'm'],
  ['b', 'y']
]

That is, the cursor consists of the initial segment a/m, then we enter the Context View of m, then b/y.

This allows the cursor to move across multiple Context Views. A more complicated example (copy and paste into em to test):

  - Books
    - Read
      - C. S. Peirce
        - Philosophical Writings
          - Three Categories
          - Philosophy of Math
          - Philosophy Logic
          - Semiotics
  - Personal
    - Influences
      - Gregory Bateson
      - Michael Polanyi
      - C. S. Peirce
  - Philosophy
    - Philosophy of Math
      - Statistical Inference
      - Probability as Potential
    - Philosophy of Science
    - Metaphysics
      - Sri Aurobindo
      - Forrest Landry
        - Potentiality
          - Probability as Potential
            - Potentiality vs Actuality

The cursor /Books/Read/C.S. Peirce/Philosophical Writings/Philosophy of Math~/Philosophy/Probability as Potential~/Potentiality/Potentiality vs Actuality spans two Context Views (Philosophy of Math and Probability as Potential), thus there are three segments in the contextChain:

[
  ['Books', 'Read', 'C. S. Peirce', 'Philosophical Writings', 'Philosophy of Math'],
  ['Philosophy', 'Probability as Potential'],
  ['Potentiality', 'Potentiality vs Actuality'],
]

Other functions related to contextChain are:

• chain
• contextChainToPath
• lastThoughtsFromContextChain
• splitChain
• thoughtsEditingFromChain

4. Data Storage/Persistence

Thoughts are stored in the underlying objects lexemeIndex and thoughtIndex, which map hashed values to Lexemes and ids to Thoughts, respectively. Only visible thoughts are loaded into state. The pullQueue is responsible for loading additional thoughts into state that need to be rendered.

• state (Redux)
• local (IndexedDB via Dexie)
• remote (Firebase) [optional; if user is logged in]

The syncing and reconciliation logic is done by pull, push, reconcile, and updateThoughts.

5. Metaprogramming

Metaprogramming provides the ability to alter em's behavior from within em itself through hidden subthoughts called metaprogramming attributes. Metaprogramming attributes begin with = and are hidden unless showHiddenThoughts is toggled on from the toolbar. Generally an attribute will affect only its parent context.

Note: User settings are stored as metaprogramming thoughts within [EM, 'Settings']. See INITIAL_SETTINGS for defaults.

List of possible metaprogramming attributes:

• =bullets Hide the bullets of a context. Options: Bullets, None.
• =children Apply attributes to all children. Currently only works with =style and =bullets. e.g. This would make b and c the color tomato:

  - a
    - =children
      - =style
        - color
          - tomato
    - b
    - c
  

• =focus When the cursor is on this thought, hide parent and siblings for additional focus. Options: Normal, Zoom.
• =hidden The thought is only displayed when showHiddenThoughts === true.
• =immovable The thought cannot be moved.
• =label Display alternative text, but continue using the real text when linking contexts. Hide the real text unless editing.
• =note Display a note in smaller text underneath the thought.
• =options Specify a list of allowable subthoughts.
• =pin Keep a thought expanded. Options: true, false.
• =pinChildren Keep all thoughts within a context expanded. Options: true, false.
• =readonly The thought cannot be edited, moved, or extended.
• =style Set CSS styles on the thought. May also use =children/=style or =grandchildren/=style.
• =uneditable The thought cannot be edited.
• =unextendable New subthoughts may not be added to the thought.
• =view Controls how the thought and its subthoughts are displayed. Options: List, Table, Prose.

6. Cursor and Caret

6.1. Cursor

The cursor is a dark gray circle surrounding the bullet of the active thought. It is stored as a Path in state.cursor. Only one thought can have the cursor at a time. All shortcuts operate on the cursor thought or its children, so it serves as the main point of interaction for the user while editing. The cursor is not the browser selection (see below), however the cursor thought contains the browser selection (caret) while editing.

You can call setCursor to set state.cursor. setCursor does not set the browser selection, although it does maintain some state for the position of the caret in caretOffset.

6.2. Caret / Browser Selection

The caret is the native browser selection, i.e. window.getSelection(). We use the name "caret" because it is shorter, and is distinguishable from "cursor". Unless otherwise specified, the caret refers to a browser selection that is collapsed, i.e. no text is selected.

Caret position is set by selection.set(...). This is typically handled automatically by the Editable component. Each Editable instance checks if the caret should be active on that thought when it is rendered. That is, it maintains the browser selection even when thoughts are re-rendered during navigation. There are other checks related to edit mode on mobile, drag-and-drop, etc.

6.2.1. Desktop

On desktop, the caret is always on the cursor thought.

6.2.1. Mobile

On mobile, the caret is only set when in edit mode. Otherwise the cursor changes without any browser selection. This allows the user to navigate thoughts without opening the virtual keyboard. Edit mode is stored in state.editing. When the user closes their mobile keyboard, state.editing is set to false.

• To enter editing mode, the user taps on the cursor thought or activates a shortcut that modifies a visible thought, such as newThought, clearText, subcategorizeOne, etc.
• Tapping on a non-cursor thought while not in edit mode will not activate edit mode.
• To close editing mode, the user closes the virtual keyboard or navigates to the root.