DISCUSSION: packaging / release strategy for UI component modules?

[pospi]

Current Problem

We don't as yet have a coherent understanding of how we will name, encapsulate / isolate, bundle and deliver Neighbourhoods UI components to consumers of our modules.

Additionally, we should aim to ensure that:

• Components are "zero dependency" and can be dropped in with a simple import
  • (except perhaps for the prefiguration of some browser polyfills for the older Safari/Webkit underneath the Holochain launcher runtime)
• Light and dark themes exist
• Theming is configurable via custom CSS properties

Proposed Solution

In the Valueflows UI project it seems to be making sense to move toward separate ESModules for each CustomElement (i.e. @vf-ui/component-*). Managing this is straightforward with PNPM, which at this stage I believe almost (if not) all of the NHs projects have been converted to. It's a good way of ensuring that the modules are genuinely zero-dependency because all dependencies must be explicitly recorded in package.json in order to be available to a module in a PNPM-managed workspace.

We then simply need to set "private": true in any package.json files under the workspace which should not be published to NPM; and initialise any "version" fields as needed for those which should.

The other thing to decide on is a module naming convention. In the case of separate modules per UI control, my vote would be @neighbourhoods/component- as a prefix.

From there it is just a case of running pnpm -r publish --access public from a repository root after each update. Changes to each package "version" are compared to what's on NPM and published if the module metadata has been changed.

Alternatives

Bundling all UI components to a single @neighbourhoods/design-system-components NPM module seems to be the other alternative. On one hand it's clean and doesn't pollute the global NPM namespace; on the other it relies on tree-shaking to remove unneeded bloat and can blow out the application size with some bundling configurations.

Care should be taken to standardise module metadata and README files in all packages run through a publication pipeline, such that references to git repositories match that of the workspace. We want to ensure that navigation from NPM to these modules is straightforward whilst they're spread across multiple project repos.

We could make a pass at extracting shared UI components to a separate "storybook" / "styleguide" repository for the NH design system and update all Applet repositories to reference the canonical versions of the code. But I don't know that this is necessary at this stage, so long as a component is being published by only one repository.
For components in use in multiple Applets maybe it's a case-by-case decision as to where that component should live? Some of them could be considered NH Launcher widgets (eg. "create or join NH"). Others are Applet-specific widgets (eg. "time entry"). Still others could come from @holochain-open-dev/profiles. Maybe take the approach of de-duplicating as we go, and if we bump into cases where we can't decide where something lives then that is a pretty good indicator that there's a more abstract CustomElement module that both components could be inheriting from?
Sorta thinking out loud. Reflections appreciated.

Adittional context

I'm feeling some additional push toward this with the implementation of a second Applet. Re-use of generic components already created in Applet 1 for Applet 2 would be nice. But currently the situation is unwieldly use of pnpm link and the social encryption of knowing which of the various repositories to have checked out, because nothing is published to NPM.

I think that despite lingering dependencies on Material UI and Shoelace, what we have is "good enough" to publish as a first alpha and iterate on from there even if some of them look kludgy. Hopefully it will also help us to standardise and update the shared components we already have? (eg. @neighbourhoods/component-create-or-join-nh).

Also to note that both the Timetracking Applet and Marketplace Applet define and export modules to the @vf-ui/ NPM organisation as well as to @neighbourhoods/, as they each implement some backend-agnostic pure-Valueflows UI controls. However, though consistency is desirable there is no real reason that all modules within each of these workspaces must respect the same format so the VF and NH modules need not have the same naming convention.

[weswalla]

Appreciate these reflections @pospi.

I'm not particularly partial to either of the options you've outlined. For me, the important factors to consider are:

• for either approach, are there example component libraries we can take inspiration from
• is it easy for applet developers to use (i.e. does it follow well used conventions for component libraries)
• is it straightforward to maintain (not requiring specialized knowledge about a specific technology)
• is it adaptable (we will undoubtedly be adding new components and iterating on component standards (styling, APIs, framework compatibility, etc)

I've no experience building & maintaining a design system, so I don't think I can offer much more that what I've listed above!

[weswalla]

FWIW, I also see two distinct types of reusable components we want to provide for applet developers:

1. NH design system: includes typical components in applicaitons (drop downs, menu bars, buttons, forms, modals, navigation, etc.)
2. sensemaker widgets (used for generating and displaying assessments along dimensions)

for (1) it makes sense to maintain this lib on npm.js, however for (2), I not sure that makes the most sense? For (2), I think we would be building out own custom component (widget) registration and persistence system. For example, these widgets can be developed by applet developers, and then we want them to get added to a NH wide widget registry where CAs and other NH participants can configure which widgets are used to create assessments (as well as along which dimensions).

Perhaps there is a core set of assessment widgets (actually I think this makes sense) so that applet developer could just make use of those and not have to write their custom implementations if they don't want to. So maybe these would be part of (1)?

The other thought I have is that just as we plan on creating an applet marketplace, we may want to support a widget marketplace as well.

Either way, if we are to handle widget registration/persistence within NHs we will need to decide on how exactly the widgets are serialized. For that, 2 approaches come to mind:

1. expose applet developed widgets through the applet interface. Here, on nh-launcher initialization/page reload, we would load up all the applet uis and pull the widgets from the interface and add them to a store. In this case, the widgets are part of the same bundle as the applet (contained in the ui.zip that gets bundled in the .webhapp file)
2. bundle widgets separately. We would need to define some interface for widgets and then bundle the code as a es module that can be imported (in the same way that applet uis are bundled). but in the ui.zip there would be a separate file (or files, depending on if we keep multiple widgets together or separate) that gets handled by the nh-launcher in a different way.

[pospi]

I think all these concerns are important, but I think mostly a separate-but-related issue? I see content delivery concerns within "native" Neighbourhoods as a second-order concern of UI component bundling and packaging (packaging has to work first).

But- prior to either of these things functioning there needs to be an ability to load and link these components as dependencies through a standardised (ESModule) package manager; in order to get them into a build pipeline for packaging and delivering to NHs. These preliminary (current day, non-NHs) linking / importing / developer ergonomics concerns are the focus of this issue.

For clarity, I want to be able to simply do import CreateOrJoinNh from '@neighbourhoods/component-create-or-join-nh' in order to include the default Neighbourhood joining functionality in an app; or import ErrorDisplay from '@neighbourhoods/component-error-display' in order to use the standardised styles for displaying a field error on a form.

I think it's also important to state that our aim should be to enable composing and reusing NHs UI components elsewhere in other "normal" UI application projects, and not just in NHs Applets or just in our low-code UI builders. Things should be modular, simple and flexible all the way up the stack.

[weswalla]

Thanks for the clarification,

I think all these concerns are important, but I think mostly a separate-but-related issue?

Yeah after reading this I agree, what I brought up was a bit off the mark of this specific discussion.

These preliminary (current day, non-NHs) linking / importing / developer ergonomics concerns are the focus of this issue.

Got it.

Things should be modular, simple and flexible all the way up the stack.

Definitely!

[pospi]

After doing some experimentation with Valueflows UI it seems like there are going to be tradeoffs in different packaging approaches.

What first becomes obvious is that LitElement simplifies things greatly for component library bundling since it only provides a few simple abstractions over the base browser-supplied CustomElement class and is thus a runtime-only framework. This differs from frameworks which operate entirely or partially at compile-time (Svelte, Vue etc) where the source code and compiled ESModule code must differ.

So, though Lit is not the most ergonomic library to work with and can take longer to develop components it may be worth focusing on it as a core component library for simplicity reasons.

It should be noted that LitElement components can of course also be bundled and minified similarly to what occurs when components developed in other frameworks are compiled; though this is actually sub-optimal when such modules are included in a compiled ESModule project for reasons of code duplication.

In fact, there are 4 possible versions of a packaged component that other projects might import:

1. "pure ESModule" where:
   • the CustomElement source is pure JavaScript
   • all dependencies are still external
   • byte size of the CustomElement is optimal, especially if minified
   • Most suited for import into an ESModule or CommonJS project using NPM or similar to manage dependencies. Workable but fiddly in the browser since all dependencies must be known and included.
2. "bundled isolated ESModule" where:
   • the CustomElement source is pure JavaScript
   • all dependencies are bundled within the module
   • Byte size is largest, since every component must include code for all its dependencies. (For Svelte apps this adds 307 lines of unminified code per component and duplicates child component module code within parent UI controls.)
   • Most suited for UMD or asynchronous loading into the browser. Workable with a package manager but leads to unnecessary bloat in bundle sizes.
3. "bundled combined ESModules" where:
   • the CustomElement source is pure JavaScript
   • dependencies are bundled, minified and chunked for re-use across all components in the library (which is only possible if they are published as a single ESModule)
   • byte size of the CustomElement is kept reasonably in check by shared framework code being de-duplicated through all components in the library
   • Most suited for import into an ESModule or CommonJS project using NPM or similar to manage dependencies. Very workable but inefficient in the browser since the entire component library must be downloaded in order to import just one of them.
4. "raw source module" where:
   • the CustomElement source is TypeScript, Svelte, Vue or some other 'compile to JS' format
   • all dependencies are still external
   • byte size of the CustomElement is determined by build pipeline and potentially smallest possible, since shared framework code can be de-duplicated across an entire Applet project
   • Most suited for use in projects using the same build frameworks where developers wish to take advantage of all intellisense and optimisation features of the UI framework.

I am probably going to stop experimenting for a while and await feedback on omissions or various combinations of these options before proceeding on polishing any build systems, since the shared components I am focused on presently are built with Lit and packaging in format 1 is all that's needed currently.

For more complex components with build steps my current thought for Svelte components already a part of VF-UI is to aim for a combination of options 2-4. With some work we should be able to get all 4 options if we spend some time on using PNPM to script per-component Vite configs; but at face value 3 seems to have the best tradeoffs even if we need to be somewhat cautious of overall library size (read: let's not be putting binary assets in there). To get a combination of 2 & 3 requires duplicating component builds on NPM but I think that's not so bad if scripted within a single repo.

For clarity, here's what dependent code would look like for all 4 options:

1. import AgentProfileCheck from '@vf-ui/component-agent-profile-check'
2. usage in a project is the same as 1. Usage in the browser would be direct load via UMD script tag and then referencing global.AgentProfileCheck.
3. import { AgentProfileCheck } from '@vf-ui/core-components'
4. import AgentProfileCheck from '@vf-ui/component-agent-profile-check/AgentProfileCheck.svelte'

Note that we can include format 4 along with any other module formats if we simply include the raw uncompiled source file at a known path within any module and ensure it appears in package.exports.

What do we think? Are these assumptions making sense so far? Am I trying to get the best of too many worlds at once? Would anybody like to express a strong preference or objections to any formats outlined above?

[pospi]

Some notes on Svelte component libraries, after doing a little research on Svelte 4 today:

• Svelte 4.x includes features for the advanced configuration of CustomElements. We should be able to leverage this to inject ScopedRegistryHost into our Svelte UI component constructors such that they can be built for direct compatibility with our existing LitElement applets.
• There are very few breaking changes between Svelte 3.x & 4.x. Updating existing components should be mostly as straightforward as changing peerDependencies to include the 4.x version series.
• I also remembered that I once wrote this utility. Updating it to Svelte 4.x should allow me to get rid of the last workarounds. Some stuff is probably now unnecessary since Svelte's further maturation (eg. Sapper output). Should be straightforward to combine this tool with tsc in order to get a combined Lit + Svelte build system working in component library repositories, so that we can mix formats.

[pospi]

I guess there is a fifth format, one which is optimal for modern build systems and now configured for the NH Design System Components repository:

5. "combined pure ESModules" which is alike (1) and (3) in that:
   • all components are packaged into the same ESModule bundle
     • components may be imported from a single entrypoint and optionally tree-shaken by bundling processes (eg. import { NHTabButton } from '@neighbourhoods/design-system-components')
     • components may (with a little work) be imported from isolated entrypoints such that tree shaking is not necessary (eg. import NHTabButton from '@neighbourhoods/design-system-components/TabButton')
   • the CustomElement source is pure JavaScript
   • all dependencies are still external
   • byte size of the CustomElement is optimal, especially if minified
   • Most suited for import into an ESModule or CommonJS project using NPM or similar to manage dependencies. Workable but fiddly in the browser since all dependencies must be known and included.

In other words, "what the TypeScript compiler gives you by default".

[pospi]

Ya I think we can maybe close this too @nick-stebbings. I wanted to put a little more work into my build toolchain in order to be able to have a mixed-framework compilation strategy but I think that's distinct from packaging decisions; which I think we have agreed on as "combined pure ESModules" +

components may (with a little work) be imported from isolated entrypoints such that tree shaking is not necessary (eg. import NHTabButton from '@neighbourhoods/design-system-components/TabButton')

...correct?

I think in many ways this is actually less work in managing build artifacts & packaging data. My strategy was going to be:

• root-level package.json with private: true and pnpm-workspace.yaml with all the Storybook dependencies for development
• dist/ directory which contains a package.json which has the metadata for the final transpiled ESModule package as delivered to NPM. The directory itself (and all other contents) are ignored by git.
  • Note that this manifest file does not need any exports, main etc fields declared since the default behaviour of NPM when packaging is to include all *.js files and allow all of them to be imported directly if their paths are known.
• custom build steps which copy LICENSE, README.md etc from the repository root to dist/ after executing tsc & equivalents to populate the build directory

[pospi]

I'd like to ask a package organising / naming question here, maybe for @nick-stebbings @weswalla @adaburrows

I am thinking that there are:

• some set of modules @neighbourhoods/dev-util-components which could include things like the applet test harness (if converted into a generator function that parameterises _todoStore); or the default 'create or join NH' component.
• some other set of modules including @weswalla's new render-block that will be required in all contexts (Dashboard, Applet, dev env, test harness).
  • is @neighbourhoods/nh-launcher-applet the place for these? Or somewhere else?
• maybe some set of @neighbourhoods/core-assessment-widgets which are distinct from @neighbourhoods/design-system-components in that they would add inference and logic based on the way the bound Dimension/s are inferred?
• perhaps some other set of modules containing test mocks etc useful in third party development, to come out of what @nick-stebbings has been working on in the storybook?

What do you think about this suggested naming & grouping? We can still develop all of these components as part of https://github.com/neighbour-hoods/design-system-components if we want to, and publish multiple packages from different directories. Or we can finalise our Storybook / UI component library architectural patterns & scripts and have separate repos. (If we do that, we should provide a scaffolding tool for component libs and create protocol around mirroring updates from there back onto all repos.)

[pospi]

Moving forward with some decisions:

• "design system components" repository to be extended to build & publish multiple packages
• @neighbourhoods/dev-util-components to be added for the test harness and 'create or join' components
• the render-block component doesn't do much and can be coded straight into the test harness
• other UI modules to be added later as needed

[adaburrows]

So one of the ideas that I had in mind when creating the one repo for the design system was being able to structure the repo like most of the other ones out there, which makes it a bit easier for both devs and designers to work on the same repo. There's tons of examples, so I'll include a few just so you all can get a flavor.

Note, since we're not using the paid version of Token Studio for Figma, it produces a giant tokens.json file that I had to write a script to parse the themes out of. In many of the following examples, there is a design-tokens directory that contains further directories that correspond to the nested entries in the tokens.json file.

• https://github.com/didoo/style-dictionary-demo
  • https://didoo.medium.com/how-to-manage-your-design-tokens-with-style-dictionary-98c795b938aa
• https://github.com/salesforce-ux/design-system

However, there is a really great example of moving everything into a separate design org and breaking everything out into separate repos over and American Airlines:

• https://github.com/AuroDesignSystem
• https://github.com/AlaskaAirlines/AuroDesignTokens/blob/master/package.json
• https://github.com/AlaskaAirlines/auro-button/blob/master/package.json
• A component generator: https://github.com/AlaskaAirlines/WC-Generator

Mozilla's corporate design system however, is probably more similar in scope to ours and they've taken the approach of breaking things down into tokens, assets, and the main design system with components and scss.

• https://github.com/mozilla/protocol-tokens/blob/main/package.json
• https://github.com/mozilla/protocol-assets/blob/main/package.json
• https://github.com/mozilla/protocol/blob/main/package.json

I happen to prefer the way Mozilla structured it, but I'm not opposed to putting everything into one giant repository like salesforce. I thought I had more examples of design systems, but I can't find them at the moment.

[adaburrows]

This might also provide more context for those who might want a more comprehensive view of a design system:

• https://uxdesign.cc/everything-you-need-to-know-about-design-systems-54b109851969
• https://medium.com/bumble-tech/from-zero-to-cosmos-part-1-2d080fe35bf2
• https://github.com/sturobson/Awesome-Design-Tokens

[nick-stebbings]

I have held back on commenting so far as I am not too knowledgeable about different ways of packaging things up but I will take a look at some of these examples this week.

I would also like to make a library for easy mocking of a store in testing (basically consolidate some of mine and @pospi 's work and hide the gory bits), and perhaps create a simple API - just pass on options object to a constructor to define the form of the returned mock data store like e.g. {applets: 2, dimensions : {subjective: 1, objective: 3} ...}

... and with realistic (i.e. better than mine) factory objects, as mine are a bit rushed and now we are getting to more complex testing requirements they are not cutting it so much.