diff --git a/agent/791/a.todo.md b/agent/791/a.todo.md index b6ccfb5c4..862db2371 100644 --- a/agent/791/a.todo.md +++ b/agent/791/a.todo.md @@ -35,11 +35,11 @@ - [x] CourseInformation component rendering successfully with admin authentication - [x] Clean UI without debug elements -## Phase 6: Studio-UI Features ⚠️ BLOCKED (Requires Express Integration) +## Phase 6: Studio-UI Features ✅ COMPLETED - [x] Add StudioFlush component with "Flush to Static" functionality -- [ ] **BLOCKED**: Update StudioFlush.vue to use Express HTTP API instead of CLI exec -- [ ] **BLOCKED**: Add progress reporting via HTTP streaming or polling -- [ ] **BLOCKED**: Implement proper error handling for HTTP flush operations +- [x] Update StudioFlush.vue to use Express HTTP API instead of CLI exec +- [x] Add progress reporting via HTTP streaming or polling +- [x] Implement proper error handling for HTTP flush operations - [x] Add flush status/progress feedback in studio-ui (UI complete) - [x] Handle flush errors gracefully with user feedback (UI complete) - [x] Integrate StudioFlush into studio-ui App.vue @@ -71,17 +71,20 @@ - [ ] Create basic studio mode documentation - [x] Add studio-ui package to monorepo build system -## Phase 9.5: Express Backend Integration **CURRENT PRIORITY** -- [ ] **CURRENT**: Add mandatory express backend service to `skuilder studio` command -- [ ] Configure express environment variables to connect to studio CouchDB instance -- [ ] Enable audio normalization processing for studio content (FFMPEG pipeline) -- [ ] Add express service shutdown handling for graceful studio exit -- [ ] Extend wire-format.ts with `FLUSH_COURSE` ServerRequestType -- [ ] Create `FlushCourse` interface in wire-format.ts -- [ ] Add express route for flush operations using `CouchDBToStaticPacker` directly -- [ ] Implement automatic content monitoring and processing for studio sessions -- [ ] Create MCP integration points for content authoring and browsing -- [ ] Document express integration benefits (audio processing, API endpoints, MCP) +## Phase 9.5: Express Backend Integration ✅ COMPLETED (Option A: Bundle Express Subprocess) +- [x] Add express build integration to CLI package.json (similar to studio-ui pattern) +- [x] Create express bundling/embedding process in CLI build scripts +- [x] Create `ExpressManager` class for subprocess management (similar to `CouchDBManager`) +- [x] Add express startup to `launchStudio()` function with dynamic port assignment +- [x] Configure express environment variables to connect to studio CouchDB instance +- [x] Add express to cleanup process in CLI signal handlers (SIGINT/SIGTERM) +- [x] Extend wire-format.ts with `PACK_COURSE` ServerRequestType (renamed from FLUSH_COURSE) +- [x] Create `PackCourse` interface in wire-format.ts (renamed from FlushCourse) +- [x] Add express route for pack operations using `CouchDBToStaticPacker` directly +- [x] Enable audio normalization processing for studio content (FFMPEG pipeline) +- [x] Implement automatic content monitoring and processing for studio sessions +- [ ] **DEFERRED** Create MCP integration points for content authoring and browsing +- [ ] **DEFERRED** Document express integration benefits (audio processing, API endpoints, MCP) ## Phase 10: Testing & Validation - [ ] Test studio mode with various sui course structures diff --git a/agent/a.1.assessment.md b/agent/a.1.assessment.md new file mode 100644 index 000000000..869cbf5c5 --- /dev/null +++ b/agent/a.1.assessment.md @@ -0,0 +1,211 @@ +# Assessment: CLI Studio Express Integration + +## Current State Analysis + +### How CLI Currently Handles Studio-UI + +The CLI's `studio` command currently: + +1. **Bundled Static Assets**: Studio-UI is built as a static Vue.js app and bundled into the CLI package + - Built via `npm run build:studio-ui` in CLI build process + - Assets copied to `dist/studio-ui-assets/` via `embed:studio-ui` script + - Served via Node.js `serve-static` middleware with dynamic config injection + +2. **Process Management**: CLI manages processes directly in Node.js + - **CouchDB**: Uses `CouchDBManager` class to spawn Docker containers + - **Studio-UI Server**: Creates HTTP server using Node.js `http` module + - **Process Lifecycle**: Handles graceful shutdown via SIGINT/SIGTERM handlers + +3. **Configuration Injection**: Dynamic config injection for studio-ui + - Injects CouchDB connection details into `window.STUDIO_CONFIG` + - Modifies `index.html` at runtime with database connection info + - Uses SPA fallback routing for client-side routing + +### Express Backend Architecture + +The Express backend (`@vue-skuilder/express`) is: + +1. **Standalone Service**: Designed as independent Node.js/Express application + - Main entry: `src/app.ts` + - Hardcoded port: 3000 + - Manages own CouchDB connections via `nano` client + - Handles authentication, course management, classroom operations + +2. **External Dependencies**: Requires external CouchDB instance + - Connects to CouchDB via environment configuration + - Manages multiple databases (courses, classrooms, users) + - Includes its own initialization and setup logic + +3. **Heavyweight Service**: Full-featured API server + - Authentication middleware + - File upload processing + - Complex business logic for course/classroom management + - Logging and error handling + +## Integration Options Analysis + +### Option A: Bundle Express and Run as Subprocess + +**Approach**: Bundle express into CLI and spawn it as a child process + +**Pros**: +- Clean separation of concerns +- Express runs in its own process space +- Can leverage existing Express configuration +- Easy to manage process lifecycle (start/stop) +- Familiar process management pattern (similar to CouchDB) + +**Cons**: +- Requires bundling entire Express app with CLI +- Multiple Node.js processes running +- More complex communication between CLI and Express +- Harder to pass configuration dynamically +- Potential port conflicts + +**Technical Implementation**: +```typescript +// Similar to how CLI spawns CouchDB +const expressProcess = spawn('node', [expressDistPath], { + env: { ...process.env, COUCHDB_URL: couchUrl } +}); +``` + +### Option B: Import Express Directly (Same Process) + +**Approach**: Import Express app and run it in the same Node.js process as CLI + +**Pros**: +- Single process - more efficient resource usage +- Direct communication between CLI and Express +- Easy to pass configuration objects +- Simpler deployment (single Node.js process) +- Can share CouchDB connection instances + +**Cons**: +- Tight coupling between CLI and Express +- Harder to isolate Express errors from CLI +- Express initialization could block CLI startup +- More complex to handle Express-specific configuration +- Potential conflicts with CLI's HTTP server + +**Technical Implementation**: +```typescript +// Import Express app and configure it +import { createExpressApp } from '@vue-skuilder/express'; +const expressApp = createExpressApp(couchConfig); +expressApp.listen(3000); +``` + +### Option C: Expect Express Running Separately + +**Approach**: CLI expects Express to be running as separate service + +**Pros**: +- Complete separation of concerns +- Express can be managed independently +- No changes needed to CLI architecture +- Easy to scale Express separately +- Clear service boundaries + +**Cons**: +- Additional setup complexity for users +- Need to coordinate between multiple services +- User must manage Express lifecycle manually +- Harder to provide "one-command" studio experience +- Complex error handling when Express is down + +**Technical Implementation**: +```typescript +// CLI just checks if Express is available +const expressHealthCheck = await fetch('http://localhost:3000/health'); +if (!expressHealthCheck.ok) { + throw new Error('Express server not running'); +} +``` + +### Option D: Hybrid Approach - Express Module + +**Approach**: Refactor Express into a configurable module that CLI can import and control + +**Pros**: +- Best of both worlds - modularity with integration +- CLI maintains control over process lifecycle +- Express can be configured per CLI session +- Clean API boundaries +- Reusable Express module + +**Cons**: +- Requires significant refactoring of Express package +- Breaking changes to Express architecture +- More complex implementation +- Need to maintain backward compatibility + +**Technical Implementation**: +```typescript +// Express as configurable module +import { ExpressService } from '@vue-skuilder/express'; +const expressService = new ExpressService({ + port: 3001, + couchdb: couchConfig, + logger: cliLogger +}); +await expressService.start(); +``` + +## Key Considerations + +### 1. **Process Management Consistency** +- CLI already manages CouchDB via subprocess (Docker) +- Studio-UI runs as HTTP server within CLI process +- Express subprocess would follow CouchDB pattern + +### 2. **Configuration Management** +- CLI injects config into Studio-UI at runtime +- Express needs CouchDB connection details +- Studio-UI needs to know Express endpoint + +### 3. **Port Management** +- CLI finds available ports dynamically (Studio-UI: 7174+) +- Express hardcoded to port 3000 +- Need to avoid port conflicts + +### 4. **Error Handling & Lifecycle** +- CLI handles graceful shutdown for all services +- Express needs to integrate with CLI's process management +- Studio-UI depends on both CouchDB and Express + +### 5. **User Experience** +- Current: Single `skuilder studio` command starts everything +- Goal: Maintain single-command simplicity +- Express adds complexity but provides powerful features + +## Recommendation + +**Option A: Bundle Express and Run as Subprocess** is the best approach because: + +1. **Architectural Consistency**: Matches existing CouchDB subprocess pattern +2. **Clean Separation**: Express runs independently but managed by CLI +3. **Minimal Changes**: Can reuse existing Express code with minimal refactoring +4. **Process Management**: Leverages CLI's existing process lifecycle handling +5. **Configuration**: Can pass config via environment variables (established pattern) + +### Implementation Plan + +1. **Express Modifications**: + - Make port configurable via environment variable + - Add health check endpoint + - Ensure clean shutdown on SIGTERM/SIGINT + +2. **CLI Integration**: + - Add Express process management (similar to CouchDB) + - Bundle Express dist in CLI build process + - Dynamic port allocation for Express + - Update Studio-UI config injection to include Express endpoint + +3. **Process Orchestration**: + - Start CouchDB first (as currently done) + - Start Express with CouchDB connection details + - Start Studio-UI with both CouchDB and Express endpoints + - Coordinate shutdown of all services + +This approach maintains the current architecture's clarity while adding the powerful Express backend capabilities that users need for full studio functionality. \ No newline at end of file diff --git a/packages/cli/package.json b/packages/cli/package.json index a17044dfd..14cbf845b 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -18,9 +18,11 @@ } }, "scripts": { - "build": "rm -rf dist && npm run build:studio-ui && tsc && npm run embed:studio-ui", + "build": "rm -rf dist && npm run build:studio-ui && npm run build:express && tsc && npm run embed:studio-ui && npm run embed:express", "build:studio-ui": "cd ../studio-ui && npm run build", + "build:express": "cd ../express && npm run build", "embed:studio-ui": "mkdir -p dist/studio-ui-assets && cp -r ../studio-ui/dist/* dist/studio-ui-assets/", + "embed:express": "mkdir -p dist/express-assets && cp -r ../express/dist/* dist/express-assets/ && cp -r ../express/assets dist/express-assets/", "dev": "tsc --watch", "lint": "npx eslint .", "lint:fix": "npx eslint . --fix", @@ -50,6 +52,7 @@ "@types/inquirer": "^9.0.0", "@types/node": "^20.0.0", "@types/serve-static": "^1.15.0", + "@vue-skuilder/express": "workspace:*", "@vue-skuilder/studio-ui": "workspace:*", "typescript": "~5.7.2" }, diff --git a/packages/cli/src/commands/studio.ts b/packages/cli/src/commands/studio.ts index e8ec86420..24c46c5b0 100644 --- a/packages/cli/src/commands/studio.ts +++ b/packages/cli/src/commands/studio.ts @@ -7,6 +7,7 @@ import { dirname } from 'path'; import http from 'http'; import { CouchDBManager } from '@vue-skuilder/common/docker'; import serveStatic from 'serve-static'; +import { ExpressManager } from '../utils/ExpressManager.js'; const __filename = fileURLToPath(import.meta.url); const __dirname = dirname(__filename); @@ -27,6 +28,7 @@ interface StudioOptions { // Global references for cleanup let couchDBManager: CouchDBManager | null = null; +let expressManager: ExpressManager | null = null; let studioUIServer: http.Server | null = null; async function launchStudio(coursePath: string, options: StudioOptions) { @@ -62,6 +64,10 @@ async function launchStudio(coursePath: string, options: StudioOptions) { couchDBManager.getConnectionDetails() ); + // Phase 9.5: Launch Express backend + console.log(chalk.cyan(`⚡ Starting Express backend server...`)); + expressManager = await startExpressBackend(couchDBManager.getConnectionDetails(), resolvedPath); + // Phase 7: Launch studio-ui server console.log(chalk.cyan(`🌐 Starting studio-ui server...`)); console.log( @@ -77,6 +83,7 @@ async function launchStudio(coursePath: string, options: StudioOptions) { console.log(chalk.green(`✅ Studio session ready!`)); console.log(chalk.white(`🎨 Studio URL: http://localhost:${studioUIPort}`)); console.log(chalk.gray(` Database: ${studioDatabaseName} on port ${options.port}`)); + console.log(chalk.gray(` Express API: ${expressManager.getConnectionDetails().url}`)); if (options.browser) { console.log(chalk.cyan(`🌐 Opening browser...`)); await openBrowser(`http://localhost:${studioUIPort}`); @@ -200,7 +207,7 @@ async function startStudioCouchDB(_databaseName: string, port: number): Promise< } /** - * Stop entire studio session (CouchDB + UI server) + * Stop entire studio session (CouchDB + Express + UI server) */ async function stopStudioSession(): Promise { // Stop studio-ui server @@ -215,6 +222,18 @@ async function stopStudioSession(): Promise { studioUIServer = null; } + // Stop Express backend + if (expressManager) { + try { + await expressManager.stop(); + console.log(chalk.green(`✅ Express backend stopped`)); + } catch (error: unknown) { + const errorMessage = error instanceof Error ? error.message : String(error); + console.error(chalk.red(`Error stopping Express backend: ${errorMessage}`)); + } + expressManager = null; + } + // Stop CouchDB if (couchDBManager) { try { @@ -285,7 +304,8 @@ async function startStudioUIServer(connectionDetails: ConnectionDetails, unpackR }, database: { name: '${unpackResult.databaseName}', - courseId: '${unpackResult.courseId}' + courseId: '${unpackResult.courseId}', + originalCourseId: '${unpackResult.courseId}' } }; @@ -312,7 +332,8 @@ async function startStudioUIServer(connectionDetails: ConnectionDetails, unpackR }, database: { name: '${unpackResult.databaseName}', - courseId: '${unpackResult.courseId}' + courseId: '${unpackResult.courseId}', + originalCourseId: '${unpackResult.courseId}' } }; @@ -384,6 +405,40 @@ async function openBrowser(url: string): Promise { } } +/** + * Phase 9.5: Start Express backend server + */ +async function startExpressBackend(couchDbConnectionDetails: ConnectionDetails, projectPath: string): Promise { + const expressManager = new ExpressManager( + { + port: 3001, // Start from 3001 to avoid conflicts + couchdbUrl: couchDbConnectionDetails.url, + couchdbUsername: couchDbConnectionDetails.username, + couchdbPassword: couchDbConnectionDetails.password, + projectPath: projectPath + }, + { + onLog: (message) => console.log(chalk.gray(` Express: ${message}`)), + onError: (error) => console.error(chalk.red(` Express Error: ${error}`)) + } + ); + + try { + await expressManager.start(); + + const connectionDetails = expressManager.getConnectionDetails(); + console.log(chalk.green(`✅ Express backend ready`)); + console.log(chalk.gray(` URL: ${connectionDetails.url}`)); + console.log(chalk.gray(` Port: ${connectionDetails.port}`)); + + return expressManager; + } catch (error: unknown) { + const errorMessage = error instanceof Error ? error.message : String(error); + console.error(chalk.red(`Failed to start Express backend: ${errorMessage}`)); + throw error; + } +} + /** * Phase 4: Unpack course data to studio CouchDB */ diff --git a/packages/cli/src/utils/ExpressManager.ts b/packages/cli/src/utils/ExpressManager.ts new file mode 100644 index 000000000..f763db06f --- /dev/null +++ b/packages/cli/src/utils/ExpressManager.ts @@ -0,0 +1,209 @@ +import { spawn, ChildProcess } from 'child_process'; +import { fileURLToPath } from 'url'; +import { dirname, join } from 'path'; +import fs from 'fs'; +import { createRequire } from 'module'; + +const require = createRequire(import.meta.url); + +const __filename = fileURLToPath(import.meta.url); +const __dirname = dirname(__filename); + +export interface ExpressManagerOptions { + port: number; + couchdbUrl: string; + couchdbUsername: string; + couchdbPassword: string; + projectPath?: string; // Path to the project for studio mode +} + +export interface ExpressManagerCallbacks { + onLog?: (message: string) => void; + onError?: (error: string) => void; +} + +export class ExpressManager { + private process: ChildProcess | null = null; + private readonly options: ExpressManagerOptions; + private readonly callbacks: ExpressManagerCallbacks; + + constructor(options: ExpressManagerOptions, callbacks: ExpressManagerCallbacks = {}) { + this.options = options; + this.callbacks = callbacks; + } + + async start(): Promise { + if (this.process) { + throw new Error('Express server is already running'); + } + + const expressAssetsPath = join(__dirname, '..', 'express-assets'); + + if (!fs.existsSync(expressAssetsPath)) { + throw new Error('Express assets not found. Please rebuild the CLI package.'); + } + + const expressMainPath = join(expressAssetsPath, 'app.js'); + + if (!fs.existsSync(expressMainPath)) { + throw new Error('Express main file not found at: ' + expressMainPath); + } + + // Find available port starting from requested port + const availablePort = await this.findAvailablePort(this.options.port); + + // Get version from package.json + let version = '0.0.0'; + try { + const packageJsonPath = join(__dirname, '..', '..', 'package.json'); + const packageJson = require(packageJsonPath); + version = packageJson.version || '0.0.0'; + } catch (error) { + // Fallback version if package.json not found + version = '0.0.0'; + } + + // Set environment variables for express + const env = { + ...process.env, + PORT: availablePort.toString(), + COUCHDB_SERVER: this.extractServerFromUrl(this.options.couchdbUrl), + COUCHDB_PROTOCOL: this.extractProtocolFromUrl(this.options.couchdbUrl), + COUCHDB_ADMIN: this.options.couchdbUsername, + COUCHDB_PASSWORD: this.options.couchdbPassword, + VERSION: version, + NODE_ENV: 'studio', + PROJECT_PATH: this.options.projectPath || process.cwd() + }; + + return new Promise((resolve, reject) => { + this.process = spawn('node', [expressMainPath], { + env, + stdio: ['pipe', 'pipe', 'pipe'], + cwd: expressAssetsPath + }); + + if (!this.process) { + reject(new Error('Failed to start Express process')); + return; + } + + let started = false; + + // Handle stdout + this.process.stdout?.on('data', (data: Buffer) => { + const message = data.toString().trim(); + if (message.includes('listening on port') && !started) { + started = true; + this.options.port = availablePort; // Update with actual port + resolve(); + } + this.callbacks.onLog?.(message); + }); + + // Handle stderr + this.process.stderr?.on('data', (data: Buffer) => { + const error = data.toString().trim(); + this.callbacks.onError?.(error); + }); + + // Handle process exit + this.process.on('exit', (code) => { + this.process = null; + if (code !== 0 && code !== null) { + this.callbacks.onError?.(`Express process exited with code ${code}`); + } + }); + + // Handle process errors + this.process.on('error', (error) => { + this.process = null; + if (!started) { + reject(error); + } else { + this.callbacks.onError?.(`Express process error: ${error.message}`); + } + }); + + // Timeout if server doesn't start within 10 seconds + setTimeout(() => { + if (!started) { + this.stop(); + reject(new Error('Express server failed to start within timeout')); + } + }, 10000); + }); + } + + async stop(): Promise { + if (!this.process) { + return; + } + + return new Promise((resolve) => { + if (!this.process) { + resolve(); + return; + } + + this.process.on('exit', () => { + this.process = null; + resolve(); + }); + + // Try graceful shutdown first + this.process.kill('SIGTERM'); + + // Force kill after 5 seconds if still running + setTimeout(() => { + if (this.process) { + this.process.kill('SIGKILL'); + this.process = null; + resolve(); + } + }, 5000); + }); + } + + getConnectionDetails() { + return { + url: `http://localhost:${this.options.port}`, + port: this.options.port + }; + } + + private async findAvailablePort(startPort: number): Promise { + const net = await import('net'); + + for (let port = startPort; port < startPort + 100; port++) { + if (await this.isPortAvailable(port, net)) { + return port; + } + } + + throw new Error(`No available port found starting from ${startPort}`); + } + + private isPortAvailable(port: number, net: any): Promise { + return new Promise((resolve) => { + const server = net.createServer(); + + server.listen(port, '127.0.0.1', () => { + server.close(() => resolve(true)); + }); + + server.on('error', () => resolve(false)); + }); + } + + private extractServerFromUrl(url: string): string { + // Extract hostname:port from URL like "http://localhost:5984" + const match = url.match(/https?:\/\/([^\/]+)/); + return match ? match[1] : 'localhost:5984'; + } + + private extractProtocolFromUrl(url: string): string { + // Extract protocol from URL like "http://localhost:5984" + return url.startsWith('https') ? 'https' : 'http'; + } +} \ No newline at end of file diff --git a/packages/cli/src/utils/NodeFileSystemAdapter.ts b/packages/cli/src/utils/NodeFileSystemAdapter.ts index 4c42825fb..b9ff929fe 100644 --- a/packages/cli/src/utils/NodeFileSystemAdapter.ts +++ b/packages/cli/src/utils/NodeFileSystemAdapter.ts @@ -1,6 +1,7 @@ // packages/cli/src/utils/NodeFileSystemAdapter.ts import fs from 'fs'; +import fse from 'fs-extra'; import path from 'path'; import { FileSystemAdapter, FileStats, FileSystemError } from '@vue-skuilder/db'; @@ -62,10 +63,53 @@ export class NodeFileSystemAdapter implements FileSystemAdapter { } } + async writeFile(filePath: string, data: string | Buffer): Promise { + try { + await fse.writeFile(filePath, data); + } catch (error) { + throw new FileSystemError( + `Failed to write file: ${error instanceof Error ? error.message : String(error)}`, + 'writeFile', + filePath, + error instanceof Error ? error : undefined + ); + } + } + + async writeJson(filePath: string, data: any, options?: { spaces?: number }): Promise { + try { + await fse.writeJson(filePath, data, options); + } catch (error) { + throw new FileSystemError( + `Failed to write JSON file: ${error instanceof Error ? error.message : String(error)}`, + 'writeJson', + filePath, + error instanceof Error ? error : undefined + ); + } + } + + async ensureDir(dirPath: string): Promise { + try { + await fse.ensureDir(dirPath); + } catch (error) { + throw new FileSystemError( + `Failed to ensure directory: ${error instanceof Error ? error.message : String(error)}`, + 'ensureDir', + dirPath, + error instanceof Error ? error : undefined + ); + } + } + joinPath(...segments: string[]): string { return path.join(...segments); } + dirname(filePath: string): string { + return path.dirname(filePath); + } + isAbsolute(filePath: string): boolean { return path.isAbsolute(filePath); } diff --git a/packages/common/src/wire-format.ts b/packages/common/src/wire-format.ts index bd8201387..877ee20b7 100644 --- a/packages/common/src/wire-format.ts +++ b/packages/common/src/wire-format.ts @@ -139,6 +139,21 @@ export interface AddCourseData extends IServerRequest { }; } +export interface PackCourse extends IServerRequest { + type: ServerRequestType.PACK_COURSE; + courseId: string; + outputPath?: string; + response: { + status: Status; + ok: boolean; + errorText?: string; + packedFiles?: string[]; + outputPath?: string; + totalFiles?: number; + duration?: number; + } | null; +} + export type ServerRequest = | CreateClassroom | DeleteClassroom @@ -146,7 +161,8 @@ export type ServerRequest = | LeaveClassroom | CreateCourse | DeleteCourse - | AddCourseData; + | AddCourseData + | PackCourse; export enum ServerRequestType { CREATE_CLASSROOM = 'CREATE_CLASSROOM', @@ -156,4 +172,5 @@ export enum ServerRequestType { CREATE_COURSE = 'CREATE_COURSE', DELETE_COURSE = 'DELETE_COURSE', ADD_COURSE_DATA = 'ADD_COURSE_DATA', + PACK_COURSE = 'PACK_COURSE', } diff --git a/packages/db/src/util/migrator/FileSystemAdapter.ts b/packages/db/src/util/migrator/FileSystemAdapter.ts index 6c6e6b5ca..749428f79 100644 --- a/packages/db/src/util/migrator/FileSystemAdapter.ts +++ b/packages/db/src/util/migrator/FileSystemAdapter.ts @@ -26,11 +26,31 @@ export interface FileSystemAdapter { */ stat(filePath: string): Promise; + /** + * Write text data to a file + */ + writeFile(filePath: string, data: string | Buffer): Promise; + + /** + * Write JSON data to a file with formatting + */ + writeJson(filePath: string, data: any, options?: { spaces?: number }): Promise; + + /** + * Ensure a directory exists, creating it and parent directories if needed + */ + ensureDir(dirPath: string): Promise; + /** * Join path segments into a complete path */ joinPath(...segments: string[]): string; + /** + * Get the directory name of a path + */ + dirname(filePath: string): string; + /** * Check if a path is absolute */ diff --git a/packages/db/src/util/packer/CouchDBToStaticPacker.ts b/packages/db/src/util/packer/CouchDBToStaticPacker.ts index 410afa348..3c12d1fc7 100644 --- a/packages/db/src/util/packer/CouchDBToStaticPacker.ts +++ b/packages/db/src/util/packer/CouchDBToStaticPacker.ts @@ -14,6 +14,7 @@ import { StaticCourseManifest, AttachmentData, } from './types'; +import { FileSystemAdapter } from '../migrator/FileSystemAdapter'; export class CouchDBToStaticPacker { private config: PackerConfig; @@ -86,6 +87,94 @@ export class CouchDBToStaticPacker { }; } + /** + * Pack a CouchDB course database and write the static files to disk + */ + async packCourseToFiles( + sourceDB: PouchDB.Database, + courseId: string, + outputDir: string, + fsAdapter: FileSystemAdapter + ): Promise<{ + manifest: StaticCourseManifest; + filesWritten: number; + attachmentsFound: number; + }> { + logger.info(`Packing course ${courseId} to files in ${outputDir}`); + + // First, pack the course data + const packedData = await this.packCourse(sourceDB, courseId); + + // Write the files using the FileSystemAdapter + const filesWritten = await this.writePackedDataToFiles(packedData, outputDir, fsAdapter); + + return { + manifest: packedData.manifest, + filesWritten, + attachmentsFound: packedData.attachments ? packedData.attachments.size : 0, + }; + } + + /** + * Write packed course data to files using FileSystemAdapter + */ + private async writePackedDataToFiles( + packedData: PackedCourseData, + outputDir: string, + fsAdapter: FileSystemAdapter + ): Promise { + let totalFiles = 0; + + // Ensure output directory exists + await fsAdapter.ensureDir(outputDir); + + // Write manifest + const manifestPath = fsAdapter.joinPath(outputDir, 'manifest.json'); + await fsAdapter.writeJson(manifestPath, packedData.manifest, { spaces: 2 }); + totalFiles++; + logger.info(`Wrote manifest: ${manifestPath}`); + + // Create subdirectories + const chunksDir = fsAdapter.joinPath(outputDir, 'chunks'); + const indicesDir = fsAdapter.joinPath(outputDir, 'indices'); + await fsAdapter.ensureDir(chunksDir); + await fsAdapter.ensureDir(indicesDir); + + // Write chunks + for (const [chunkId, chunkData] of packedData.chunks) { + const chunkPath = fsAdapter.joinPath(chunksDir, `${chunkId}.json`); + await fsAdapter.writeJson(chunkPath, chunkData); + totalFiles++; + } + logger.info(`Wrote ${packedData.chunks.size} chunk files`); + + // Write indices + for (const [indexName, indexData] of packedData.indices) { + const indexPath = fsAdapter.joinPath(indicesDir, `${indexName}.json`); + await fsAdapter.writeJson(indexPath, indexData, { spaces: 2 }); + totalFiles++; + } + logger.info(`Wrote ${packedData.indices.size} index files`); + + // Write attachments + if (packedData.attachments && packedData.attachments.size > 0) { + for (const [attachmentPath, attachmentData] of packedData.attachments) { + const fullAttachmentPath = fsAdapter.joinPath(outputDir, attachmentPath); + + // Ensure attachment directory exists + const attachmentDir = fsAdapter.dirname(fullAttachmentPath); + await fsAdapter.ensureDir(attachmentDir); + + // Write binary file + await fsAdapter.writeFile(fullAttachmentPath, attachmentData.buffer); + totalFiles++; + } + logger.info(`Wrote ${packedData.attachments.size} attachment files`); + } + + return totalFiles; + } + private async extractCourseConfig(db: PouchDB.Database): Promise { try { return await db.get('CourseConfig'); @@ -322,11 +411,12 @@ export class CouchDBToStaticPacker { try { const designDocId = designDoc._id; // e.g., "_design/elo" - const viewPath = `${designDocId}/${viewName}`; + const designDocName = designDocId.replace('_design/', ''); // Extract just "elo" + const viewPath = `${designDocName}/${viewName}`; logger.info(`Querying CouchDB view: ${viewPath}`); - // Query the view directly from CouchDB + // Query the view directly from CouchDB using PouchDB format: "designDocName/viewName" const viewResults = await this.sourceDB.query(viewPath, { include_docs: false, }); diff --git a/packages/express/package.json b/packages/express/package.json index b2402adee..19b06c34b 100644 --- a/packages/express/package.json +++ b/packages/express/package.json @@ -32,6 +32,7 @@ "express": "^4.21.2", "express-rate-limit": "^7.5.0", "ffmpeg-static": "^5.2.0", + "fs-extra": "^11.2.0", "hashids": "^2.3.0", "morgan": "^1.10.0", "nano": "^ 9.0.5", diff --git a/packages/express/src/app.ts b/packages/express/src/app.ts index 6ff483257..e0c670a7f 100644 --- a/packages/express/src/app.ts +++ b/packages/express/src/app.ts @@ -20,6 +20,7 @@ import { CourseCreationQueue, initCourseDBDesignDocInsert, } from './client-requests/course-requests.js'; +import { packCourse } from './client-requests/pack-requests.js'; import { requestIsAuthenticated } from './couchdb/authentication.js'; import CouchDB, { useOrCreateCourseDB, @@ -169,6 +170,22 @@ async function postHandler( logger.info(`\t\t\tCouchDB insert error: ${JSON.stringify(e)}`); res.json(e); }); + } else if (body.type === RequestEnum.PACK_COURSE) { + if (process.env.NODE_ENV !== 'studio') { + logger.info( + `\tPACK_COURSE request received in production mode, but this is not supported!` + ); + res.status(400); + res.statusMessage = 'Packing courses is not supported in production mode.'; + res.send(); + return; + } + + body.response = await packCourse({ + courseId: body.courseId, + outputPath: body.outputPath + }); + res.json(body.response); } } else { logger.info(`\tREQUEST UNAUTHORIZED!`); diff --git a/packages/express/src/attachment-preprocessing/index.ts b/packages/express/src/attachment-preprocessing/index.ts index 70393e73f..50e2271d8 100644 --- a/packages/express/src/attachment-preprocessing/index.ts +++ b/packages/express/src/attachment-preprocessing/index.ts @@ -4,6 +4,7 @@ import { normalize } from './normalize.js'; import AsyncProcessQueue, { Result } from '../utils/processQueue.js'; import logger from '../logger.js'; import { CourseLookup } from '@vue-skuilder/db'; +import ENV from '../utils/env.js'; // @ts-expect-error [todo] const Q = new AsyncProcessQueue( @@ -65,16 +66,51 @@ export default async function postProcess(): Promise { try { logger.info(`Following all course databases for changes...`); + // Existing behavior for platform-ui courses const courses = await CourseLookup.allCourses(); + const processedCourseIds = new Set(); for (const course of courses) { try { postProcessCourse(course._id); + processedCourseIds.add(`coursedb-${course._id}`); } catch (e) { logger.error(`Error processing course ${course._id}: ${e}`); throw e; } } + + // Studio mode: discover additional databases not in coursedb-lookup + if (ENV.NODE_ENV === 'studio') { + logger.info('Studio mode detected: scanning for additional course databases...'); + + try { + const allDbs = await CouchDB.db.list(); + const studioDbs = allDbs.filter(db => + db.startsWith('coursedb-') && + !processedCourseIds.has(db) + ); + + logger.info(`Found ${studioDbs.length} potential studio databases`); + + for (const studioDb of studioDbs) { + const courseId = studioDb.replace('coursedb-', ''); + + try { + if (await hasCourseConfig(studioDb)) { + logger.info(`Starting postprocessing for studio database: ${studioDb}`); + postProcessCourse(courseId); + } else { + logger.debug(`Skipping ${studioDb}: no course config found`); + } + } catch (e) { + logger.error(`Error processing studio database ${studioDb}: ${e}`); + } + } + } catch (e) { + logger.error(`Error discovering studio databases: ${e}`); + } + } } catch (e) { logger.error(`Error in postProcess: ${e}`); } @@ -202,3 +238,31 @@ interface ProcessingField { mimetype: string; returnData?: string; } + +/** + * Check if a database contains course configuration (indicating it's a valid course database) + */ +async function hasCourseConfig(databaseName: string): Promise { + try { + const db = CouchDB.use(databaseName); + + // Try to find a course configuration document + // Course databases should have documents with course metadata + const result = await db.find({ + selector: { + $or: [ + { 'type': 'course' }, + { 'shape': { $exists: true } }, + { 'course_id': { $exists: true } }, + { 'courseID': { $exists: true } } + ] + }, + limit: 1 + }); + + return result.docs && result.docs.length > 0; + } catch (e) { + logger.debug(`Error checking course config for ${databaseName}: ${e}`); + return false; + } +} diff --git a/packages/express/src/client-requests/pack-requests.ts b/packages/express/src/client-requests/pack-requests.ts new file mode 100644 index 000000000..42ac247d8 --- /dev/null +++ b/packages/express/src/client-requests/pack-requests.ts @@ -0,0 +1,166 @@ +import { Status } from '@vue-skuilder/common'; +import logger from '../logger.js'; +import ENV from '../utils/env.js'; +import PouchDb from 'pouchdb'; + +interface PackCourseData { + courseId: string; + outputPath?: string; +} + +interface PackCourseResponse { + status: Status; + ok: boolean; + packedFiles?: never; // No longer relevant since we're writing to files + outputPath?: string; + attachmentsFound?: number; + filesWritten?: number; + totalFiles?: number; + duration?: number; + errorText?: string; +} + +export async function packCourse(data: PackCourseData): Promise { + logger.info(`Starting PACK_COURSE for ${data.courseId}...`); + + try { + const startTime = Date.now(); + + // Use CouchDBToStaticPacker directly from db package + const { CouchDBToStaticPacker } = await import('@vue-skuilder/db'); + + // Create database connection URL + const dbUrl = `${ENV.COUCHDB_PROTOCOL}://${ENV.COUCHDB_ADMIN}:${ENV.COUCHDB_PASSWORD}@${ENV.COUCHDB_SERVER}`; + const dbName = `coursedb-${data.courseId}`; + + // Determine output path based on environment and provided path + let outputPath: string; + + if (data.outputPath) { + // If output path is provided, check if it's absolute or relative + const pathModule = await import('path'); + const path = pathModule.default || pathModule; + + if (path.isAbsolute(data.outputPath)) { + // Use absolute path as-is + outputPath = data.outputPath; + } else { + // Relative path - combine with project path in studio mode + const projectPath = process.env.PROJECT_PATH || process.cwd(); + outputPath = path.join(projectPath, data.outputPath); + } + } else { + // No output path provided - use default + outputPath = ENV.NODE_ENV === 'studio' ? + '/tmp/skuilder-studio-output' : + process.cwd(); + } + + logger.info(`Packing course ${data.courseId} from ${dbName} to ${outputPath}`); + + // Clean up existing output directory for replace-in-place functionality + const fsExtra = await import('fs-extra'); + const fs = fsExtra.default || fsExtra; + + try { + if (await fs.pathExists(outputPath)) { + logger.info(`Removing existing directory: ${outputPath}`); + await fs.remove(outputPath); + } + } catch (cleanupError) { + logger.warn(`Warning: Could not clean up existing directory ${outputPath}:`, cleanupError); + // Continue anyway - the write operation might still succeed + } + + // Create course database connection + const courseDbUrl = `${dbUrl}/${dbName}`; + + // Initialize packer and perform pack operation with file writing + const packer = new CouchDBToStaticPacker(); + + // For Express, we create a simple FileSystemAdapter using dynamic imports + const createFsAdapter = async () => { + const fsExtra = await import('fs-extra'); + const pathModule = await import('path'); + + // Access the default export for fs-extra + const fs = fsExtra.default || fsExtra; + const path = pathModule.default || pathModule; + + return { + async readFile(filePath: string): Promise { + return await fs.readFile(filePath, 'utf8'); + }, + async readBinary(filePath: string): Promise { + return await fs.readFile(filePath); + }, + async exists(filePath: string): Promise { + try { + await fs.access(filePath); + return true; + } catch { + return false; + } + }, + async stat(filePath: string) { + const stats = await fs.stat(filePath); + return { + isDirectory: () => stats.isDirectory(), + isFile: () => stats.isFile(), + size: stats.size + }; + }, + async writeFile(filePath: string, data: string | Buffer): Promise { + await fs.writeFile(filePath, data); + }, + async writeJson(filePath: string, data: any, options?: { spaces?: number }): Promise { + await fs.writeJson(filePath, data, options); + }, + async ensureDir(dirPath: string): Promise { + await fs.ensureDir(dirPath); + }, + joinPath(...segments: string[]): string { + return path.join(...segments); + }, + dirname(filePath: string): string { + return path.dirname(filePath); + }, + isAbsolute(filePath: string): boolean { + return path.isAbsolute(filePath); + } + }; + }; + + const fsAdapter = await createFsAdapter(); + const packResult = await packer.packCourseToFiles(new PouchDb(courseDbUrl), data.courseId, outputPath, fsAdapter); + + const duration = Date.now() - startTime; + + const response: PackCourseResponse = { + status: Status.ok, + ok: true, + outputPath: outputPath, + attachmentsFound: packResult.attachmentsFound, + filesWritten: packResult.filesWritten, + totalFiles: packResult.filesWritten, // Updated to reflect actual files written + duration: duration + }; + + logger.info(`Pack completed in ${duration}ms. Attachments: ${response.attachmentsFound}, Files written: ${response.filesWritten}`); + + return response; + } catch (error) { + logger.error('Pack operation failed:', error); + + const response: PackCourseResponse = { + status: Status.error, + ok: false, + errorText: error instanceof Error ? error.message : 'Pack operation failed' + }; + + return response; + } +} + +// Export types for use in app.ts +export type { PackCourseData, PackCourseResponse }; \ No newline at end of file diff --git a/packages/express/src/couchdb/authentication.ts b/packages/express/src/couchdb/authentication.ts index 04e6e4db5..99c4ad620 100644 --- a/packages/express/src/couchdb/authentication.ts +++ b/packages/express/src/couchdb/authentication.ts @@ -49,6 +49,12 @@ function logRequest(req: VueClientRequest) { export async function requestIsAuthenticated(req: VueClientRequest) { logRequest(req); + // Studio mode bypass: skip authentication for local development + if (process.env.NODE_ENV === 'studio') { + logger.info('Studio mode: bypassing authentication for local development'); + return true; + } + if (req.headers.authorization) { const auth = Buffer.from(req.headers.authorization.split(' ')[1], 'base64') .toString('ascii') diff --git a/packages/express/src/utils/env.ts b/packages/express/src/utils/env.ts index 6d5691522..fb39d2081 100644 --- a/packages/express/src/utils/env.ts +++ b/packages/express/src/utils/env.ts @@ -17,6 +17,7 @@ type Env = { COUCHDB_ADMIN: string; COUCHDB_PASSWORD: string; VERSION: string; + NODE_ENV: string; }; function getVar(name: string): string { @@ -33,6 +34,7 @@ const env: Env = { COUCHDB_ADMIN: getVar('COUCHDB_ADMIN'), COUCHDB_PASSWORD: getVar('COUCHDB_PASSWORD'), VERSION: getVar('VERSION'), + NODE_ENV: getVar('NODE_ENV'), }; initializeDataLayer({ diff --git a/packages/studio-ui/eslint.config.mjs b/packages/studio-ui/eslint.config.mjs new file mode 100644 index 000000000..e01472af2 --- /dev/null +++ b/packages/studio-ui/eslint.config.mjs @@ -0,0 +1,32 @@ +import eslint from '@eslint/js'; +import eslintConfigPrettier from 'eslint-config-prettier'; +import eslintPluginVue from 'eslint-plugin-vue'; +import eslintPlugiVuetify from 'eslint-plugin-vuetify'; +import globals from 'globals'; +import typescriptEslint from 'typescript-eslint'; + +export default typescriptEslint.config( + { ignores: ['*.d.ts', '**/coverage', '**/dist', '**/src/courses/chess/chessground'] }, + { + extends: [ + eslint.configs.recommended, + ...typescriptEslint.configs.recommended, + ...eslintPluginVue.configs['flat/recommended'], + ...eslintPlugiVuetify.configs['flat/recommended'], + ], + files: ['**/*.{ts,vue}'], + languageOptions: { + ecmaVersion: 'latest', + sourceType: 'module', + globals: globals.browser, + parserOptions: { + parser: typescriptEslint.parser, + }, + }, + rules: { + // your rules + '@typescript-eslint/no-explicit-any': 'warn', + }, + }, + eslintConfigPrettier +); diff --git a/packages/studio-ui/package.json b/packages/studio-ui/package.json index 504fb4d7d..a9b1c6531 100644 --- a/packages/studio-ui/package.json +++ b/packages/studio-ui/package.json @@ -8,7 +8,9 @@ "scripts": { "dev": "vite", "build": "vite build", - "preview": "vite preview" + "preview": "vite preview", + "lint": "eslint . --fix", + "lint:check": "eslint ." }, "dependencies": { "@mdi/font": "^7.3.67", @@ -24,6 +26,7 @@ }, "devDependencies": { "@vitejs/plugin-vue": "^5.2.1", + "eslint": "^9.30.1", "typescript": "^5.7.2", "vite": "^6.0.9", "vue-tsc": "^1.8.0" diff --git a/packages/studio-ui/src/App.vue b/packages/studio-ui/src/App.vue index 740d936f2..dbe784f00 100644 --- a/packages/studio-ui/src/App.vue +++ b/packages/studio-ui/src/App.vue @@ -1,8 +1,8 @@