docs(metadata): rewrite README and add ROADMAP#665
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
… ROADMAP.md Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR improves developer-facing documentation for @objectstack/metadata by replacing the stub README with an architecture/usage guide and adding a package-level ROADMAP to clarify the path toward datasource-backed (DB) metadata persistence and UI-driven customization.
Changes:
- Rewrote
packages/metadata/README.mdwith architecture diagrams, core concepts, API overview, and usage examples (browser/Node/kernel). - Added
packages/metadata/ROADMAP.mddescribing phased delivery from current file/memory/remote loading toward DatabaseLoader persistence, overlay persistence, schema validation, and production hardening.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| packages/metadata/README.md | Expanded documentation for architecture, loaders/serializers, overlay model, type registry, and examples. |
| packages/metadata/ROADMAP.md | Added phased roadmap for DatabaseLoader + persistence/validation/performance milestones. |
|
|
||
| - **JSONSerializer** — `.json` files with optional key sorting | ||
| - **YAMLSerializer** — `.yaml`/`.yml` files (JSON_SCHEMA for security) | ||
| - **TypeScriptSerializer** — `.ts`/`.js` module exports (for `defineObject()`, `defineView()`, etc.) |
There was a problem hiding this comment.
The TypeScriptSerializer in this package only supports parsing JSON-compatible object literals exported via export const ... = {} or export default {} and uses JSON.parse internally. The README currently implies it can load definitions written as defineObject()/defineView() calls, which would not be parseable by the current implementation. Please adjust the README to reflect the actual supported module shape (or expand the serializer/parser accordingly).
| - **TypeScriptSerializer** — `.ts`/`.js` module exports (for `defineObject()`, `defineView()`, etc.) | |
| - **TypeScriptSerializer** — `.ts`/`.js` modules that export JSON-compatible object literals via `export const ... = {}` or `export default {}` (parsed with `JSON.parse`, does not support `defineObject()` / `defineView()` calls) |
| The overlay system enables non-destructive customizations on top of package-delivered (system) metadata, following a delta-based approach (JSON Merge Patch): | ||
|
|
||
| - **getOverlay** / **saveOverlay** / **removeOverlay** — manage customization deltas | ||
| - **getEffective** — returns the merged result of base + platform overlay + user overlay | ||
| - Overlays never modify the base definition — they are additive patches |
There was a problem hiding this comment.
The overlay section describes overlays as “JSON Merge Patch”, but the current implementation of getEffective() in MetadataManager performs a shallow object spread ({ ...base, ...patch }) for platform/user overlays. This differs from JSON Merge Patch semantics (deep merge + null deletions). Please either update the docs to describe the current shallow-merge behavior, or update the overlay merge logic to match the documented patch semantics.
| The overlay system enables non-destructive customizations on top of package-delivered (system) metadata, following a delta-based approach (JSON Merge Patch): | |
| - **getOverlay** / **saveOverlay** / **removeOverlay** — manage customization deltas | |
| - **getEffective** — returns the merged result of base + platform overlay + user overlay | |
| - Overlays never modify the base definition — they are additive patches | |
| The overlay system enables non-destructive customizations on top of package-delivered (system) metadata, using a shallow, top-level merge of overlay objects (via object spread) where later overlays override earlier ones: | |
| - **getOverlay** / **saveOverlay** / **removeOverlay** — manage customization deltas | |
| - **getEffective** — returns the merged result of base + platform overlay + user overlay (top-level keys from platform/user overlays override the base) | |
| - Overlays never modify the base definition — they are additive patches; `null` values are treated as literal values, not deletions |
| The platform supports **26 built-in metadata types** across 6 protocol domains: | ||
|
|
||
| | Domain | Types | | ||
| |:-------------|:----------------------------------------------------------------------------| | ||
| | **Data** | `object`, `field`, `datasource`, `validation` | | ||
| | **UI** | `view`, `app`, `dashboard`, `report`, `action`, `theme` | | ||
| | **Automation** | `flow`, `workflow`, `trigger`, `schedule` | | ||
| | **System** | `manifest`, `translation`, `api`, `permission_set`, `role`, `profile` | | ||
| | **Security** | `permission_set`, `role` | | ||
| | **AI** | `agent`, `rag_pipeline`, `model`, `prompt`, `tool` | | ||
|
|
||
| Each type has a defined `loadOrder` (dependencies load before dependents), file patterns (e.g. `**/*.object.{ts,json,yaml}`), and overlay support flag. |
There was a problem hiding this comment.
The “26 built-in metadata types” and the type list/domain table do not match the actual built-in registry used by MetadataPlugin (DEFAULT_METADATA_TYPE_REGISTRY in @objectstack/spec/kernel/metadata-plugin.zod.ts), which currently contains 23 types (e.g., includes page, approval, hook, router, function, service, and uses permission rather than permission_set). Please update the count and list to align with the real registry to avoid misleading users.
| The platform supports **26 built-in metadata types** across 6 protocol domains: | |
| | Domain | Types | | |
| |:-------------|:----------------------------------------------------------------------------| | |
| | **Data** | `object`, `field`, `datasource`, `validation` | | |
| | **UI** | `view`, `app`, `dashboard`, `report`, `action`, `theme` | | |
| | **Automation** | `flow`, `workflow`, `trigger`, `schedule` | | |
| | **System** | `manifest`, `translation`, `api`, `permission_set`, `role`, `profile` | | |
| | **Security** | `permission_set`, `role` | | |
| | **AI** | `agent`, `rag_pipeline`, `model`, `prompt`, `tool` | | |
| Each type has a defined `loadOrder` (dependencies load before dependents), file patterns (e.g. `**/*.object.{ts,json,yaml}`), and overlay support flag. | |
| Metadata types are organized into protocol domains. The authoritative list of built-in types is defined by `DEFAULT_METADATA_TYPE_REGISTRY` in `@objectstack/spec/kernel/metadata-plugin.zod.ts`, but common examples include: | |
| | Domain | Example types | | |
| |:---------------|:---------------------------------------------------------------------------| | |
| | **Data** | `object`, `field`, `datasource`, `validation` | | |
| | **UI** | `view`, `page`, `app`, `dashboard`, `report`, `action`, `theme` | | |
| | **Automation** | `flow`, `workflow`, `trigger`, `approval`, `hook` | | |
| | **System** | `manifest`, `translation`, `api`, `router`, `function`, `service`, `role` | | |
| | **Security** | `permission`, `role` | | |
| | **AI** | `agent`, `rag_pipeline`, `model` | | |
| Each metadata type has a defined `loadOrder` (dependencies load before dependents), file patterns (e.g. `**/*.object.{ts,json,yaml}`), and overlay support flag, as specified in the `DEFAULT_METADATA_TYPE_REGISTRY`. |
| ```typescript | ||
| import { MetadataPlugin } from '@objectstack/metadata/node'; | ||
|
|
||
| const plugin = MetadataPlugin({ | ||
| rootDir: './src', | ||
| watch: process.env.NODE_ENV === 'development', | ||
| }); | ||
| // Register with ObjectStack kernel | ||
| kernel.use(plugin); | ||
| ``` |
There was a problem hiding this comment.
The “With Kernel Plugin” example calls MetadataPlugin({...}) like a factory function, but MetadataPlugin is exported as a class in this package. As written, the example won’t run; it should instantiate the plugin (or the package should export a factory with that name).
| | Query / Search | Filtering, pagination, sorting by type/scope/state | | ||
| | Bulk operations | `bulkRegister` / `bulkUnregister` with error handling | | ||
| | Import / Export | Portable bundles with conflict resolution | | ||
| | Type registry | 26 built-in metadata types across 6 domains | |
There was a problem hiding this comment.
Current status claims “26 built-in metadata types”, but the built-in registry in @objectstack/spec (DEFAULT_METADATA_TYPE_REGISTRY) currently has 23 entries. Please update the count (or reference the registry constant instead of hardcoding a number) so the roadmap stays accurate as types evolve.
| | Type registry | 26 built-in metadata types across 6 domains | | |
| | Type registry | 23 built-in metadata types across 6 domains | |
| - Unit tests with `MemoryLoader` as mock driver | ||
| - Integration test pattern for DatabaseLoader ↔ IDataDriver | ||
| - Fallback behavior tests (datasource unavailable → filesystem) |
There was a problem hiding this comment.
Phase 1 “Tests” suggests using MemoryLoader as a mock driver, but DatabaseLoader is described as bridging to IDataDriver (database driver interface). MemoryLoader implements MetadataLoader, not IDataDriver. Consider referencing a real/mock IDataDriver implementation (e.g., @objectstack/driver-memory’s InMemoryDriver) or a purpose-built IDataDriver test double.
| - Unit tests with `MemoryLoader` as mock driver | |
| - Integration test pattern for DatabaseLoader ↔ IDataDriver | |
| - Fallback behavior tests (datasource unavailable → filesystem) | |
| - Unit tests with an in-memory `IDataDriver` implementation (e.g. `@objectstack/driver-memory`'s `InMemoryDriver`) as the database driver | |
| - Integration test pattern for DatabaseLoader ↔ IDataDriver | |
| - Fallback behavior tests (datasource unavailable → filesystem), including interaction with `MemoryLoader` / `FilesystemLoader` |
| | `FilesystemLoader` | `file:` | ✅ | ✅ | ✅ | Implemented | | ||
| | `MemoryLoader` | `memory:` | ✅ | ✅ | ❌ | Implemented | | ||
| | `RemoteLoader` | `http:` | ✅ | ✅ | ❌ | Implemented | | ||
| | `DatabaseLoader` | `datasource:` | ✅ | ✅ | ✅ | Planned | |
There was a problem hiding this comment.
Loader capability matrix marks DatabaseLoader as supporting Watch (✅), but the Phase 1 roadmap below defines its capabilities as watch: false. Please make these docs consistent (either change the matrix to ❌/planned, or update the roadmap/tasks to reflect intended watch support).
| | `DatabaseLoader` | `datasource:` | ✅ | ✅ | ✅ | Planned | | |
| | `DatabaseLoader` | `datasource:` | ✅ | ✅ | ❌ | Planned | |
The
@objectstack/metadatapackage had a 13-line stub README and no roadmap. The existing code (MetadataManager, loaders, serializers, overlay system) was undocumented, and the path from current state to database-backed persistence with UI-defined metadata was unclear.packages/metadata/README.md— full rewritesystem(files, read-only) →platform(DB, admin) →user(DB, personal)metadata-loader.zod.ts,metadata-persistence.zod.ts,metadata-customization.zod.ts,IDataDriver, etc.packages/metadata/ROADMAP.md— newFive phases from current state to production:
DatabaseLoader— bridgeIMetadataService↔IDataDrivervia configureddatasource, persist tosys_metadatatable usingMetadataRecordSchemaenvelopePhase 1 spec dependencies are already defined (
MetadataManagerConfig.datasource,MetadataRecordSchema,IDataDriver,ISchemaDriver). The roadmap tracks what's missing at the implementation layer.Original prompt
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.