pnpm monorepo, lazy loading stores onto ctx object & separation of domain data into separate stores using ctx-core

[btakita]

This PR is still a work in progress. I created this to continue the discussion over the architectural direction of the ui, particularly in separating domain objects from the ui. I feel like it's at a point to where the general architectural direction I proposed is articulated in code & ready for feedback. @guillemcordoba's concern about using MobX (or any other store library) can be addressed through composing the svelte stores into a MobX (or other store libary) object. This is not done, but the possibility of such an integration should be evident.

This pull request refactors the ui & tests projects into a pnpm monorepo of separate libraries. I chose @syn-ui as a npm library prefix as a placeholder

• test: @syn-ui/test
• ui/apps/app: @syn-ui/app
• ui/libs/model: @syn-ui/model
• ui/libs/utils: @syn-ui/utils
• ui/libs/zome-client: @syn-ui/zome-client

This PR demonstrates the usage of lazily instantiated svelte stores with ctx objects as a way to decouple data flow and have declarative (opposed to imperative) dependency instantiation.

The stores are lazily instantiated in _b functions. Actions are in ui/libs/model/actions

import { toggle_session, me_b, am_i_scribe_b } from '@syn-ui/model'
const app_port = 8888, app_id = 'syn'
const ctx = await toggle_session({ app_port, app_id })
console.info({
  $me: me_b(ctx).$,
  $am_i_scribe: am_i_scribe_b(ctx).$,
})

@syn-ui/models (ui/libs/model) is currently split up into:

• actions: called from the ui to drive toggling, joining, or leaving a session
• colors: color logic
• content: the document content data
• delta: document delta logic & data
• session: session related logic & data
• signals: real time Signal handlers
• timers: setInterval timers used to poll incoming zome data

The tests utilize rpc_ calls in ui/libs/zome-client.

---

This PR uses pnpm instead of npm for package management & running scripts. Since separate packages are built using the monorepo, I added ./bin/tsc-build.sh to compile the typescript files used by the app's rollup build.

To install:

cd syn
npm i -g pnpm
pnpm i

To run tsc-build.sh

./bin/tsc-build.sh -w

To start the uit;

(cd ui/apps/app && pnpm run dev)

---

The tests pass. Multiple browers are not exchanging data however. I may need help with troubleshooting the zome-ui interactions here. I just wanted to post this not completed PR to continue the discussion & to discuss any issues w/ the session api.

---

Note: This uses the https://github.com/ctx-core/ctx-core libraries, which I have created, to demonstrate these patterns. While I need to work on the documentation, I have develop these libraries over the last few years to evolve these patterns. The functions used are pretty self-contained, so if you want to use them without the dependency on @ctx-core, it's practical to extract the functions.

cc @zippy @qubist @rayzer42

[pegaltier]

@btakita looks promising! even if I prefer NX monorepo: https://nx.dev/

[btakita]

@pegaltier Thank you. I have not used nx. I built some tooling around pnpm. Are there any open source projects that would serve as a good reference? What do you like about nx?

On a general note, I think this PR has functional parity with the main branch. I'll go ahead & squash the changes. It seems like joining the same session from a different port does not work anymore. Is this a regression in the zome?

[pegaltier]

@btakita with NX you can enjoy a robust CLI (actually the Angular CLI extended) you can secure boundaries between different libs/modules, you can get a dependency graph automatically generated based on your code. You can easily get computation caching locally or even shared cache with others devs,ci using NX Cloud... It fits very well for large projects. I'm not sure here but still it works well even for small codebase

You can check the official vids: https://www.youtube.com/watch?v=mVKMse-gFBI

You will find a lot of information, tutorial, example here: https://github.com/pegaltier/awesome-utils-dev/blob/master/utils-coding/utils-angular-list.md#ecosystem-monorepo

You should be able to install it in your branch in one line: npx add-nx-to-monorepo

source: https://nx.dev/latest/node/getting-started/nx-setup

[guillemcordoba]

Hi @btakita , this all looks great. Just FYI I'll be managing the next steps for Syn, alongside @zippy.

First of all, thanks so much for all this effort, and sorry for lagging in the response.

After some digging, I understood the general pattern that you are proposing, and I like it quite a bit. The main goal for the near future is to have a core syn package exporting custom elements that can be reused across applications. I think this pattern can achieve that quite well, we just need to have a way to pass the context from/to frameworks other than svelte. I'm also hoping to export the svelte stores and have them also work in react or vue.

I also like the pnpm monorepo pattern.

As far as the dependency with ctx-core, it's a bit unclear to me at the moment whether we should maintain this dependency. Could you clarify what benefits we obtain from using it, and whether it would be possible to do the same without it?

With this resolved, I would be inclined to merge in this refactor and maybe restructure a bit the packages to use existing tools, and to have a clearer separation between what should be happ domain concerns (in the case of syntext: title, body and meta) and what should be syn's domain concerns (Commits, etc.).

[zippy]

Indeed thanks @btakita for your contribution. I'm looking forward to seeing it merged and really excited that @guillemcordoba will be picking up momentum on syn to turn it into a first class library/starting-point for many holochain apps!

[guillemcordoba]

Hi @btakita I have merged this into develop, and will be applying the necessary changes. Thanks :D

[btakita]

I appreciate the review & merge. Looking forward to @guillemcordoba having the bandwidth to shepard Syn & the ensuing progress & innovation. I'll take a first pass here & will get into more detail soon.

The goal of ctx-core is to be a library that supports individual developers/teams to develop/maintain many (1000+) libraries & apps. I have been using the ctx-core multirepo (monorepo + git submodules) to support the framework I have been using across my projects. ctx-core includes many utility functions to support type checking, multirepo development, composable naming convention (support abstraction building/maintenance across libraries & apps), functional reactive programming, etc. There is still some innovation & churn in ctx-core, as my understanding & the state of the art changes. I think we should coordinate somehow as our goals of efficiently supporting many apps are similar & we can develop a common system for mutual benefit.

A current issue w/ ctx-core is that it is over 5 years old it was built with churn over the years as the JS->TS ecosystem has evolved & is bespoke according to my needs during that time. I have been doing by best to keep all of the code up to date w/ the current practice but I need a better way maintaining unused (not necessarily obsolete) code. I'm considering extracting a common library that merges core libraries together such as @ctx-core/object, @ctx-core/function, @ctx-core/combinators, etc. At the time these libraries were first created, my understanding was less refined & the patterns were different. I'm happy have this extraction be sheparded with a more public process as I believe the pattern churn has settled with recent changes.

Since this PR, there are some changes to the naming convention & the Ctx. Factory functions now should use a _ suffix instead of a _ prefix. The _ sufix better fits with the existing practice of having a _ prefix mean a private function/variable. Stores now should use a $ suffix & the value should not have a $ prefix in js/ts files. In *.svelte files, the $ prefix is needed, so the a value of a store will be represented as $value$. I created a ticket w/ commentary. Both of these naming convention changes have resolved some naming convention contradictions with my other projects.

ctx-core now supports composible typing on the ctx object itself. Each library can have a Ctx interface which extends dependency Ctx interfaces, ensuring there are no naming & type conflicts within a ctx object.

Re: resource reference cleanup, ctx-core now has the weak_r_ (r means Record), aliased to weak_ctx_, function, which instantiates a ctx where the object values are a WeakRef, ensuring that unreferenced ctx values are cleaned up by the garbage collector. This will allow better support for large apps where the ctx is in flux when navigating from page to page.

I will follow up with a PR upgrade the @ctx-core/* libraries & to roll out those changes.

SolidJS has recently hit the v1 milestore & I am excited to utilize SolidJS (along w/ Svelte) for my client work. Lit is also interesting but I prefer Solid's function based approach, it's reactive context scoping w/ auto cleanup, performance, & ability to scale to many components. The const [getter, setter] = createSignal({}) interface of Solidjs is also more finely grained than Svelte's stores & may be a better fit for a Syn common library. Even though Solid has scoped-based auto-cleanup, the ctx object is still beneficial to handle lazy loading of arbitrary (non-reactive) values, referencing within call siblings or outside of reactive scopes. ctx objects are general purpose while Solid's scoping is limited to Solid itself. A problem w/ Solid is compatibility w/ React may have issues as Solid has it's own tsx/jsx renderer.

I'm happy to do the migration to a simpler & more focused core library instead of having core functionality spread over a few @ctx-core/* libraries. The feedback that you have provided has been important in speeding up the evolution of ctx-core & the overall framework of practices.

If you like the multirepo approach, we can use ctx-core as a scaffold & extract/fork functions as needed into the common library. This is all I have for now but I will let some time pass work through the implications.

[guillemcordoba]

Hum thanks for the comment @btakita. I think it's important to clarify the intended outcome for the syn UI library: a package with custom elements and a state management store(s) to make it as easy as possible to include syn in other apps (that includes other frameworks like vue, react or angular, and even vanilla js).

Also one of the most important topics is the separation of concerns between syn objects and happ defined types. Syn deals with commits, snapshots, folks and sessions, while the consuming happ defines their own types. In the case of syntext which is what is implemented in this repo, that's title, body and meta.

With this in mind, I want to consider carefully the tech stack that we use for it to be flexible to adapt to different use cases. I'm even considering whether to continue using svelte or not, as yes it compiles down to customelements but there are a lot of caveats, gotchas and differences that need to be taken into account in the core library code (eg sveltejs/svelte#3119, https://javascript.plainenglish.io/can-you-build-web-components-with-svelte-3c8bc3c1cfd8).

If you have any clear insight on this it would be appreciated, I've used customelements in other scenarios with amazing success.

[guillemcordoba]

This is an example of what I'm looking for to improve interoperability between frameworks: https://github.com/webcomponents/community-protocols/blob/main/proposals/context.md.

[btakita]

The focused goals that you presented make sense. On one end, it is possible to have a 0 dependency vanilla js library for ultimately interpolarity on the DOM & even for SSR. I can see why you like the lit library due to lit building webcomponents at it's core. Complex app scaling is not a primary concern as the use case is constrained to handle the reactive elements of document sharing. If I understand correctly, we can optimize for a smooth interop & extension experience. It would be useful to have a finely grained api to target libraries using multiple paradigms. Primitives as finely-grained as the most finely-grained use case (or even finer) allow the most flexibility to create the correct interop abstracitons for each target library.

In some of my situations using Svelte & Solid, I want to create interops between the two libraries so the interfaces between the two libraries have a smoother compatability experience. solidjs/solid#504 I commend you for taking a similar ergonomic approach to have a smooth & focused compatability experience. I'd like to have a well designed system where the different abstractions are compatible as well.

re: the webcomponents "Context Protocol": I commend making a standard for common patterns. This standard is scoped to the render tree & decouples from the general DI where "Context API !== Dependency Injection Framework". The framework supported by ctx-core library a the general purpose Dependency Injection framework that can be overlaid onto the webcomponents "Context Protocol". Note thet ctx-core is not the framework but an implementation library supporting the usage patterns & conventions which make up the framework.

Re: ctx-core, it's not needed, but as far as I can tell, is novel & at the leading edge of certain concepts such as working with ctx objects for dependency injection. Each of the functions are relatively self-contained which enables them to be easily forked. The issue with forking is that any innovation that occurs w/ the implementation of the usage patterns would not be carried over. This could be an opportunity to extract out a focused library that handles the lazy load DI ctx patterns, where we can keep the benefits of innovation of the patterns while not including unrelated functions.

Re: multi-repos: one can create a multi-repo using any combination of source repos. This is strategy for individual people & organizations to use, as there can be a single multi-repo or each team/person can have their own multi-repo with the libraries that they are concerned about. To make my development experience smoother, my goal as a consultant is to be able to mix/match composible multi-repos for many clients, where each client has their own multi-repo. Monorepos work great as well & I prefer monorepos in some cases, but multi-repos break out each library & app as a primitive which can be composed to actively work on multiple packages together. There is some development overhead to dealing w/ git submodules, but it does facilitate scaling to many packages across many organizations. Note that having submodules is optional, as one can move between a git modules while actively developing a dependency & a standard library (npm package) relationship once development of the dependency is complete.

I'm throwing many ideas here, but I just want to present the idea to be considered for the Holochain ecosystem. As the app ecosystem grows, this pattern could be a way to reuse code & help make contributions to multiple libraries more effective. My goal is to scale development of individual people & teams to handle 1000+ libraries & apps. If we consider Syn by itself, the simplest solution is the best, but when other libraries & apps are considered, consolidating the development & abstractions of all of these related libraries starts to make more sense. cc: @zippy