Define the first-class external module architecture and authoring workflow

[MehmetSecgin]

Summary

External modules are already part of the product story for dispatch, but the current architecture and docs still feel like a minimal runtime path rather than a fully first-class authoring/build/packaging system.

This issue tracks the intended long-term direction for external modules as real, dispatch-native modules authored and distributed outside core.

Current state

Today:

• external modules are discovered from filesystem directories containing module.json
• the loader reads each manifest and dynamically imports the runtime entry
• built-ins do not go through the same discovery/import path
• current docs and scaffolding lean toward plain runtime JS entries like index.mjs

Relevant code:

• src/modules/loader.ts
• src/modules/index.ts
• src/modules/builtin/index.ts
• src/commands/module.ts
• docs/modules/conventions.md
• README.md

Architectural concerns

1. External modules currently feel runtime-compatible, but not yet authoring-first

The runtime contract works, but the overall experience still reads as:

• author index.mjs directly
• point module.json at that entry

rather than:

• author in TypeScript
• build to runtime JS
• validate/pack/install through a dispatch-native workflow

2. The docs blur three different concerns

We should distinguish more explicitly between:

• authoring format
• runtime entry format
• packaging/bundle format

Those are not the same thing.

3. Per-command eager loading may become a scaling problem

dispatch is a fresh process on every CLI invocation. Today that means each command may pay for:

• module directory scanning
• manifest reads
• dynamic imports of external module entries
• any top-level work those modules do at import time

That is probably acceptable for a small number of lightweight modules, but may not age well as teams add heavier external modules.

Direction this issue proposes

Product stance

External modules should eventually have a built-in-quality authoring experience, while still preserving a runtime boundary from core.

Desired model

• external modules are authored in TypeScript
• modules compile to built JS artifacts
• module.json points to the built runtime entry, e.g. dist/index.mjs
• dispatch provides first-class validation, packaging, and installation support
• runtime loads compiled JS only
• top-level import work stays lightweight

Canonical workflow to aim toward

Potential long-term flow:

• dispatch module init scaffolds a TS-based module
• dispatch module build compiles to dist/
• dispatch module validate validates the built module contract
• dispatch module pack creates a shareable module bundle
• dispatch module install installs a packed module into the user module area

Performance direction

Near-term guidance

Even before lazy-loading exists, external module guidance should explicitly recommend:

• no network work at import time
• no expensive filesystem work at import time
• no heavy top-level initialization
• lazy dependency setup inside handlers where possible

Longer-term improvement

Consider splitting:

• metadata discovery
• handler loading

That would allow dispatch to build an action index from manifests first, and import a module entry only when one of its actions is actually needed.

Suggested phased path

Phase 1

• clarify docs: authoring format vs runtime format vs package format
• explicitly bless TS-authored modules compiled to built JS
• keep runtime loading contract as-is

Phase 2

• improve scaffolding so external modules default to TS authoring
• keep module.json as runtime manifest
• strengthen validate/inspect around compiled modules

Phase 3

• add first-class pack/install workflow
• add lazy-loading or partial-loading strategy for external modules

Open questions

• Should dispatch own module builds directly, or only define the contract and let teams use standard TS tools at first?
• How much of the module toolchain should live in core CLI versus a separate ecosystem package?
• Should external module packaging be filesystem bundle based, tarball based, or both?
• What should the canonical external module directory layout be once TS authoring is first-class?

Recommendation

Treat external modules as a first-class product surface, not a sidecar escape hatch.

That means the long-term dispatch way should be:

• TS-authored
• compiled ahead of time
• runtime-loaded as built JS
• validated and packed with dispatch-native tooling
• eventually loaded more lazily than the current eager-import model

[MehmetSecgin]

The external module architecture has been designed and is being implemented. The canonical module repo layout, monorepo structure, TypeScript-only source convention, and job co-location pattern are all settled. Closing in favour of the concrete issues that came out of this work: #27, #28, and the workbench design.