feat(metadata): implement dual-table projection for queryable metadata

OBJECTOS_IMPLEMENTATION.md

@@ -0,0 +1,185 @@
+# ObjectOS Implementation Summary
+
+## Overview
+
+Successfully implemented the ObjectOS layer - a new `@objectstack/objectos` package containing system runtime object definitions that represent metadata as queryable data.
+
+## Architecture Decision
+
+Based on architectural discussions, we established:
+
+1. **Location**: `packages/objectos` (NOT `packages/plugins/plugin-system`)
+   - Rationale: System objects are core infrastructure, not optional plugins
+   - ObjectOS represents the OS-level primitives of the platform
+
+2. **Dual-Table Pattern**: Keep BOTH systems (do NOT deprecate `sys_metadata`)
+   - `sys_metadata`: Source of truth for package management, version control, deployment
+   - Type-specific tables (`sys_object`, `sys_view`, etc.): Queryable data for UI/reporting
+
+## Package Structure
+
+```
+packages/objectos/
+├── src/
+│   ├── objects/
+│   │   ├── sys-metadata.object.ts     # Generic metadata envelope
+│   │   ├── sys-object.object.ts       # Object definitions
+│   │   ├── sys-view.object.ts         # View definitions
+│   │   ├── sys-agent.object.ts        # AI Agent definitions
+│   │   ├── sys-tool.object.ts         # AI Tool definitions
+│   │   ├── sys-flow.object.ts         # Flow definitions
+│   │   └── index.ts
+│   ├── index.ts                        # Package entry point
+│   └── registry.ts                     # System object registry
+├── package.json
+├── tsconfig.json
+├── tsup.config.ts
+└── README.md
+```
+
+## System Objects Implemented
+
+### 1. sys_metadata (Generic Envelope)
+- **Purpose**: Source of truth for package management
+- **Features**: Version control, checksums, package ownership, deployment tracking
+- **Fields**: 20+ fields including version, checksum, package_id, managed_by, scope
+
+### 2. sys_object (Queryable Object Definitions)
+- **Purpose**: Browse/filter/search object definitions in Studio
+- **Features**: Denormalized data, complex fields as JSON
+- **Fields**: 30+ fields including fields_json, capabilities_json, field_count
+
+### 3. sys_view (Queryable View Definitions)
+- **Purpose**: Manage view metadata through Object Protocol
+- **Features**: View type filtering, object references
+- **Fields**: columns_json, filters_json, sort_json, config_json
+
+### 4. sys_agent (AI Agent Definitions)
+- **Purpose**: AI agent configuration as data
+- **Features**: Model config, tools/skills management
+- **Fields**: model, temperature, system_prompt, tools_json, skills_json
+
+### 5. sys_tool (AI Tool Definitions)
+- **Purpose**: AI tool registry as queryable data
+- **Features**: Parameter schemas, handler code
+- **Fields**: parameters_json, handler_code
+
+### 6. sys_flow (Automation Flow Definitions)
+- **Purpose**: Flow metadata management
+- **Features**: Flow types, trigger configuration
+- **Fields**: flow_type, nodes_json, edges_json, trigger_type
+
+## Key Features
+
+1. **Metadata as Data**
+   - All metadata types are queryable using Object Protocol
+   - Same CRUD operations as business data
+   - Consistent API: `/api/v1/data/sys_object`, `/api/v1/data/sys_view`
+
+2. **Dual-Table Architecture**
+   ```
+   Package Loader
+        ↓
+   sys_metadata (source of truth)
+        ↓ (projection)
+   sys_object, sys_view, etc. (queryable)
+        ↓
+   Studio UI (auto-generated)
+   ```
+
+3. **Version Management**
+   - `sys_metadata` tracks all versions
+   - `sys_metadata_history` table for history
+   - Checksum-based change detection
+   - Package upgrade/downgrade support
+
+4. **Auto-Generated UI**
+   - Studio uses Object Protocol
+   - No custom UI code per metadata type
+   - Leverage grid/form/kanban views
+
+## Industry Alignment
+
+- **Salesforce**: CustomObject, CustomField (queryable metadata)
+- **ServiceNow**: sys_db_object, sys_dictionary (table-based metadata)
+- **Kubernetes**: CRDs as structured resources
+
+## Next Steps
+
+### Phase 1: Integration (Immediate) ✅ COMPLETED
+- [x] Update `packages/metadata` service to support projection
+- [x] Implement dual-table sync logic
+- [x] Register system objects in runtime bootstrap
+
+**Implementation Details:**
+- Created `MetadataProjector` service in `packages/metadata/src/projection/`
+- Integrated projection into `DatabaseLoader.save()` and `DatabaseLoader.delete()`
+- Added projection functions for each metadata type: object, view, agent, tool, flow
+- Updated `MetadataPlugin` to register all system objects from `@objectstack/objectos`
+- Projection is enabled by default, can be disabled via `enableProjection: false` option
+
+### Phase 2: Studio Integration (Next)
+- [ ] Update Studio to query type-specific tables
+- [ ] Use `/api/v1/data/sys_object` for browsing
+- [ ] Auto-generate metadata forms
+
+### Phase 3: Testing & Documentation (Later)
+- [ ] Add comprehensive test coverage
+- [ ] Update documentation
+- [ ] Create migration guides
+
+## Usage Example
+
+```typescript
+import { SystemObjects } from '@objectstack/objectos';
+
+// Register all system objects during bootstrap
+for (const [name, definition] of Object.entries(SystemObjects)) {
+  await kernel.metadata.register('object', name, definition, {
+    scope: 'system',
+    isSystem: true,
+    managedBy: 'platform',
+  });
+}
+
+// Query metadata using Object Protocol
+const objects = await client.data.find('sys_object', {
+  filter: { namespace: 'crm' },
+  sort: 'name',
+});
+
+// Studio auto-generates UI
+<GridView object="sys_object" />
+<FormView object="sys_object" recordId="account" />
+```
+
+## Benefits
+
+1. ✅ **Unified Protocol**: One protocol for both data and metadata
+2. ✅ **Auto-Generated UI**: Studio reuses existing components
+3. ✅ **Better DX**: Consistent API for all entity types
+4. ✅ **Version Control**: Full history via sys_metadata_history
+5. ✅ **Package Management**: Track ownership and deployments
+6. ✅ **Industry Standard**: Follows Salesforce/ServiceNow patterns
+
+## Files Created
+
+- `/home/runner/work/framework/framework/packages/objectos/package.json`
+- `/home/runner/work/framework/framework/packages/objectos/tsconfig.json`
+- `/home/runner/work/framework/framework/packages/objectos/tsup.config.ts`
+- `/home/runner/work/framework/framework/packages/objectos/README.md`
+- `/home/runner/work/framework/framework/packages/objectos/src/index.ts`
+- `/home/runner/work/framework/framework/packages/objectos/src/registry.ts`
+- `/home/runner/work/framework/framework/packages/objectos/src/objects/index.ts`
+- `/home/runner/work/framework/framework/packages/objectos/src/objects/sys-metadata.object.ts`
+- `/home/runner/work/framework/framework/packages/objectos/src/objects/sys-object.object.ts`
+- `/home/runner/work/framework/framework/packages/objectos/src/objects/sys-view.object.ts`
+- `/home/runner/work/framework/framework/packages/objectos/src/objects/sys-agent.object.ts`
+- `/home/runner/work/framework/framework/packages/objectos/src/objects/sys-tool.object.ts`
+- `/home/runner/work/framework/framework/packages/objectos/src/objects/sys-flow.object.ts`
+
+## Conclusion
+
+The ObjectOS package establishes a clean architectural foundation for treating metadata as queryable data. This enables auto-generated Studio UI, unified APIs, and follows industry best practices from Salesforce, ServiceNow, and Kubernetes.
+
+The dual-table architecture preserves the benefits of `sys_metadata` for package management while adding queryability through type-specific tables.

PHASE_1_IMPLEMENTATION.md

@@ -0,0 +1,139 @@
+# Phase 1 Implementation Summary: Dual-Table Metadata Projection
+
+## Overview
+
+Successfully implemented the dual-table projection logic for the metadata service. This establishes the architectural foundation for treating metadata as queryable data following industry best practices from Salesforce, ServiceNow, and Kubernetes.
+
+## Architecture
+
+The dual-table pattern maintains two layers:
+
+1. **sys_metadata** (Source of Truth)
+   - Package management
+   - Version control
+   - Deployment tracking
+   - History via sys_metadata_history
+
+2. **Type-Specific Tables** (Queryable Projections)
+   - sys_object — Object definitions
+   - sys_view — View configurations
+   - sys_agent — AI agent metadata
+   - sys_tool — AI tool registry
+   - sys_flow — Automation flows
+
+## Implementation Details
+
+### 1. MetadataProjector Service
+
+Created `packages/metadata/src/projection/metadata-projector.ts` with:
+
+- **Project transformation functions** for each metadata type
+- **Denormalization logic** to flatten complex structures for querying
+- **Support for both IDataDriver and IDataEngine** (ObjectQL)
+- **Automatic CRUD operations** on projection tables
+
+Key methods:
+- `project(type, name, data)` — Create/update projection
+- `deleteProjection(type, name)` — Remove projection
+- `transformToProjection(type, name, data)` — Type-specific transformation
+
+### 2. DatabaseLoader Integration
+
+Updated `packages/metadata/src/loaders/database-loader.ts`:
+
+- Added `enableProjection` configuration option (default: true)
+- Integrated `MetadataProjector` into save() flow
+- Added projection cleanup in delete() flow
+- Projection occurs AFTER sys_metadata save (async safety)
+
+### 3. System Object Registration
+
+Updated `packages/metadata/src/plugin.ts`:
+
+- Added dependency on `@objectstack/objectos`
+- Registered all system objects from SystemObjects registry
+- Objects registered via manifest service during plugin init()
+- Includes: sys_object, sys_view, sys_agent, sys_tool, sys_flow
+
+## Projection Mapping
+
+Each metadata type is projected with denormalized fields for efficient querying:
+
+### Object Projection (object → sys_object)
+- Complex structures (fields, indexes, validations) → JSON columns
+- Capabilities → individual boolean columns for filtering
+- Denormalized field_count for sorting/filtering
+
+### View Projection (view → sys_view)
+- Columns, filters, sort, config → JSON columns
+- Display options → individual columns
+- Object reference for joins
+
+### Agent Projection (agent → sys_agent)
+- Model configuration → individual columns
+- Tools, skills → JSON columns
+- Memory settings → individual columns
+
+### Tool Projection (tool → sys_tool)
+- Parameters schema → JSON column
+- Handler code → text column
+
+### Flow Projection (flow → sys_flow)
+- Nodes, edges, variables → JSON columns
+- Trigger configuration → individual columns
+- Active status for filtering
+
+## Benefits Achieved
+
+1. **Unified Query Protocol**: Metadata can be queried using Object Protocol API
+2. **Studio Auto-Generation**: UI can use `/api/v1/data/sys_*` endpoints
+3. **Efficient Filtering**: Denormalized fields enable fast queries
+4. **Preserved History**: sys_metadata maintains full version control
+5. **Package Tracking**: All projections include package_id and managed_by
+
+## Files Changed
+
+### New Files
+- `packages/metadata/src/projection/metadata-projector.ts` — Projection service
+- `packages/metadata/src/projection/index.ts` — Module exports
+
+### Modified Files
+- `packages/metadata/src/loaders/database-loader.ts` — Added projection integration
+- `packages/metadata/src/plugin.ts` — Registered system objects
+- `packages/metadata/src/index.ts` — Exported projection module
+- `packages/metadata/package.json` — Added @objectstack/objectos dependency
+- `OBJECTOS_IMPLEMENTATION.md` — Updated Phase 1 status
+
+## Usage Example
+
+```typescript
+// Projection happens automatically when saving metadata
+await metadataService.register('object', 'account', {
+  name: 'account',
+  label: 'Account',
+  fields: { /* ... */ },
+  // ... object definition
+});
+
+// Results in TWO database writes:
+// 1. sys_metadata: Full envelope with JSON payload
+// 2. sys_object: Denormalized projection with queryable columns
+
+// Studio can now query via Object Protocol
+const objects = await client.data.find('sys_object', {
+  filter: { namespace: 'crm' },
+  sort: 'name',
+});
+```
+
+## Next Steps
+
+Phase 2 will focus on Studio integration:
+- Update Studio to use type-specific table queries
+- Auto-generate metadata management UI
+- Leverage existing grid/form/kanban components
+
+Phase 3 will add:
+- Comprehensive test coverage
+- Documentation updates
+- Migration guides for existing deployments

apps/studio/test/ai-chat-panel.test.tsx

@@ -97,10 +97,10 @@ describe('use-ai-chat-panel', () => {
     });
 
     it('does not throw when localStorage is unavailable', () => {
-      const originalSetItem = Storage.prototype.setItem;
-      Storage.prototype.setItem = () => { throw new Error('QuotaExceeded'); };
+      const originalSetItem = localStorage.setItem.bind(localStorage);
+      localStorage.setItem = () => { throw new Error('QuotaExceeded'); };
       expect(() => saveMessages([makeMsg({ id: '1', role: 'user', content: 'A' })])).not.toThrow();
-      Storage.prototype.setItem = originalSetItem;
+      localStorage.setItem = originalSetItem;
     });
   });
 });
@@ -260,10 +260,10 @@ describe('Agent Selector', () => {
     });
 
     it('should not throw when localStorage is unavailable', () => {
-      const originalSetItem = Storage.prototype.setItem;
-      Storage.prototype.setItem = () => { throw new Error('QuotaExceeded'); };
+      const originalSetItem = localStorage.setItem.bind(localStorage);
+      localStorage.setItem = () => { throw new Error('QuotaExceeded'); };
       expect(() => saveSelectedAgent('metadata_assistant')).not.toThrow();
-      Storage.prototype.setItem = originalSetItem;
+      localStorage.setItem = originalSetItem;
     });
   });