Overview over lively.next Internals

The Lively World

• Conceptual:
  • A world is a "workbench". The place where everything is built.
  • Offers multiple workspaces between which can be navigated.
• Technical:
  • The whole state is defined within the world.
    --> When saving and loading a world, the current state of the world is unaltered (in theory, see exceptions in section on Saving a world).
  • A world is independent from the browser.

Object Graph

• Contains all JavaScript objects in the world.
• Objects reference each other.
• Is a cyclic structure.
  --> Cannot be directly serialized to JSON.

Saving a world

• Takes a snapshot of the whole world state (in contrast to e.g. just saving changes).
• Use starting point (the World).
• Object Graph is traversed by serializer:
  • Serialize primitives directly (bool, string, number, ...)
  • Objects and their referenced objects are serialized recursively:
    • Newly created objects receive a unique Morphic ID from the serializer. This one will be unaltered for the lifetime of the object, even beyond saving and (re-)loading a world.
    • Objects alrady visited during the serialization process are remembered (via the object identity of the JS VM); recursion will be terminated at these, they will now be referenced via their Morhpic ID.
    • Traversion results in linearized graph. Cyclic dependencies are solved via Morphic IDs.
    • Linearized graph was serialized to JSON string while traversing.
• Caveats: Not everything can or should be serialized
  • Custom objects are usually small and can be fully serialized.
  • Morphs are tricky; they contain references to renderer, meta state, ...
    • By default, all properties will be serialized --> not desirable
      --> Serializer checks for different getters, e.g. get __serialize_only__(), which either define properties to serialize or properties that should not be serialized
      --> Referenced objects that should not be serialized (that should be "pruned") can be added to the pruning process by implementing those special getters
    • Morphs with epiMorph = true will not be serialized, e.g. the toolbar
  • Types and classes should not be serialized recursively, since they may come from a large third-party module --> only the type info (path to class) is serialized together with the object
  • Closures (i.e. their contexts) cannot be serialized and will be lost
    • Module lively.lang/closures supports saving closures partially --> Types, classes, closures, ... need to be re-initialized upon loading of the world (see details at respective section).

Loading a world

• Deserialization of JSON string to objects with Morphic IDs and properties. Starts at a given ID (usually the world).
• Recursive (re-)construction of objects using the constructors of the respective classes --> results in world's object graph
• Closures (i.e. their contexts) were not saved --> need to be re-initialized via property setters
• Type definitions/classes were not saved (to give flexibility and optimize file size)
  • Module system of Lively allows to find respective classes easily (path to class was saved by serializer)
  • Breaks if path is no longer valid --> Migrations.js: Pre-Process of deserialization. Patchs unreferenced/no longer present classes within the given snapshot.

Coyping objects

• When copying objects, the original JS object is serialized.
• The copy is then instantiated from this JSON string.
• During the serialization process, the Serializer creates a new Morphic ID for the copied object.

Synchronous work on same world

• See section on versioning.

Versioning and data storage

Git

• Core elements of Lively are versioned via Git.
• Changes in core modules need to be persisted via commits.

MorphicDB

• Everything within a world (everything that is serialized) is stored and versioned within a MorphicDB.
• MorphicDB: LevelDB which supports interactions using an interface identical to CouchDB thanks to PouchDB
  --> ergo: MorphicDB is used like a CouchDB, but works using a LevelDB internally. All thanks to the magic of the JS library PouchDB.
• When running Lively locally: Usage of LevelDB (with CouchDB interface)
• When collaborating with others: Usage of remote CouchDB (set up on a shared server)
  • MorphicDBSynchronizer allows one to use remote CouchDB
  • Everything is still saved in local LevelDB
  • CouchDB runs on server
  • Synchronizer checks LevelDB periodically

Snapshot

• Serialized world (JSON file), created by serializer.
• Saved at:
  • lively.morphic/objectdb/morphicdb/snapshots
  • Directory contains sub-directories named after combinations of two hex digits.
  • These are buckets of all commits whose hash start with thes two hex digits.

Commits

• A commit consists of:
  • Commit ID (path to JSON file containing the snapshot)
  • Commit info
    • Predecessor --> creates commit history
    • Author
    • Hash
    • Timestamp
    • Name
• Saved in database at: lively.morphic/objectdb/morphicdb-commits (.ldb-file, LevelDB-file)

Discussion of used DB types

• The conception MorphicDB seems rather complicated. We want to discuss the reasons.

Why CouchDB

• CouchDB can detect differences between DBs --> synchronisation of worlds possible
  --> Usable for collaboration between multiple worlds
  --> Usually, a remote CouchDB (without underlying LevelDB) is set up on shared server
• Caveat: Only compares commit IDs
  --> Can detect possible conflicts, but cannot merge/show diffs between commits
  --> Allows for sequential work in one world, only (as of now)

Why LevelDB

• The setup of CouchDB is difficult.
• Lively should be rather easy to set up.
• LevelDB is easy to set up.
• It's a match.
• Especially useful for local instances.

Synchronous work on same world

• Loading same world as two instances A and B at a time works without issues, since same snapshot is loaded.
• Saving world A works without a problem, commit (and thus snapshot) is created as usual.
• Saving world B:
  • Would raise warning, because commit from world A is not a predecessor and already present.
  • Options:
    • Cancel new commit.
    • Overwrite old commit from world A: New commit is appended to history. Old commit is not deleted (just overlaid by new commit) and could be restored.
    • Merge conflict resolution currently not possible.

Freezing

• Freezing a Morph/Object creates own world around it
• Makes it loadable on it's own, without meta information on how to edit the object
• Used to publish finished results

Rendering

• Lively works on a Virtual DOM, comparable to VueJS or React
• Workflow: Browser requests animation frame --> render pass / render frame
  • Traverse Object Graph
  • Generate Virtual DOM Nodes
  • Render()
• Advantages:
  • Rendering code is easy to write:
    • No concerns of sycnhronizity
    • Unidirectional data flow
  • Slight performance boost in certain situations:
    • Changes can are queued and then applied at once.
    • Only objects that changed need to be updated.
• Disadvantages:
  • On every render pass, a big tree of objecs must be generated. --> New objects must be created, others deleted --> Garbage collector needs to run often --> power intensive
  • Problems with changes of objects:
    • Are not instantanious
    • Code may no be certain that changes have already been applied (e.g. that making a window fullscreen has already changed the object's extent in the next line) --> Requires waiting on next render frame (async)
      • whenRendered gives promise
      • await(this.whenRendered())
      • execute next code with JS promise syntax .then

Animations

• All JS based, no CSS transitions
• Change properties during animation via withAnimationDo, e.g.
  • change vertices of a path (e.g. for a sad smiley)
  • colors

CSS

• Changing CSS in lively is possible
  • Don't do this!
  • CSS is overwritten anyway (since lively applies the properties using Inline-CSS)
    • Can be prevented with important, but causes side effects, as a Morph's properties are having no effect anymore
  • Demolishs the layer of abstraction that should have been achieved --> Does not allow the advantages of lively/morphic

FastLoad

• Experimental feature
• Normally, lively also loads all modules when loading a snapshot --> Requires a lot of meta information to be loaded (Source, Classes, AST, Depedencies, ...)
  • Takes a lot of time
  • But: Actually only required for objects that are changed in the system (i.e. newly developed features or adjusted objects) --> Not required for most objects (e.g. parts of the development system, like windows)
• Idea: Similar to freezing
  • Only load everything needed to start, no meta information --> Does not allow editing of source code of those objects (e.g. system parts)
• Current state:
  • Viewing source code possible
  • Load meta information on demand impossible (e.g. you cannot change that window behaviour, if you have used FastLoad)