feat: Integrate Hypercontext Tools with Phase 1 Bug Fixes

docs/bug_fixes.md

@@ -0,0 +1,131 @@
+### **User Story: Fix Widespread Type Mismatches**
+
+**User Story:**
+As a developer, I want to fix all the type mismatch errors across the codebase so that the project can compile successfully and data integrity is maintained between different modules.
+
+**Description:**
+The build process is failing with a large number of `TS2322` and `TS2345` errors. These errors are caused by assigning incorrect data types to variables and function parameters. A major recurring issue is passing `string` values to functions expecting `Buffer` (and vice-versa), especially in caching and compression logic. Another common issue is passing a `number` to a function expecting a `string`. These errors must be resolved to make the application functional.
+
+**Example Error (from `cache-compression.ts`):**
+`error TS2322: Type 'string' is not assignable to type 'Buffer<ArrayBufferLike>'.`
+
+**Example Error (from `smart-api-fetch.ts`):**
+`error TS2345: Argument of type 'number' is not assignable to parameter of type 'string'.`
+
+**Acceptance Criteria:**
+*   All `TS2322` and `TS2345` type errors are resolved.
+*   Data is correctly converted between `string` and `Buffer` types where necessary (e.g., using `Buffer.from(string, 'base64')` or `buffer.toString('base64')`).
+*   Numeric types are correctly converted to strings where necessary (e.g., using `.toString()`).
+*   The project build completes without any type-related errors.
+
+**Implementation Plan:**
+1.  **Address Buffer/String Mismatches:**
+    *   Inspect `src/tools/advanced-caching/cache-compression.ts` and `src/tools/api-database/smart-cache-api.ts`.
+    *   Identify all functions that expect a `Buffer` but receive a `string`.
+    *   Wrap the string variable with `Buffer.from(variable, 'base64')` before passing it to the function.
+    *   Identify all functions that expect a `string` but receive a `Buffer`.
+    *   Call `.toString('base64')` on the buffer variable before passing it.
+2.  **Address Number/String Mismatches:**
+    *   Inspect all files with `TS2345` errors, such as `smart-api-fetch.ts`.
+    *   Identify function calls where a `number` is passed to a `string` parameter.
+    *   Call `.toString()` on the number variable before passing it to the function.
+
+---
+
+### **User Story: Fix Incorrect Usage of `TokenCountResult` Object**
+
+**User Story:**
+As a developer, I want to fix all instances where the `TokenCountResult` object is incorrectly used as a `number`, so that token counts are handled correctly throughout the application and related type errors are resolved.
+
+**Description:**
+The `tokenCounter.count()` method returns an object of type `TokenCountResult`, which has a `.tokens` property. Several parts of the codebase are attempting to assign this entire object to a variable of type `number` or use it directly in an arithmetic operation, which causes a `TS2322` error. This indicates a misunderstanding of the `TokenCounter`'s API.
+
+**Example Error (from `smart-refactor.ts`):**
+`error TS2322: Type 'TokenCountResult' is not assignable to type 'number'.`
+
+**Acceptance Criteria:**
+*   All instances where the `TokenCountResult` object is used are corrected to access the `.tokens` property when the numeric token count is needed.
+*   The project builds without `TS2322` errors related to `TokenCountResult`.
+
+**Implementation Plan:**
+1.  Search the codebase for all calls to `tokenCounter.count()`.
+2.  For each call, check if the result is being assigned to a variable of type `number` or used directly in a mathematical operation.
+3.  Modify these instances to access the `.tokens` property of the returned object (e.g., change `tokenCounter.count(text)` to `tokenCounter.count(text).tokens`).
+
+---
+
+### **User Story: Resolve Missing Module Exports**
+
+**User Story:**
+As a developer, I want to fix the broken module imports so that different parts of the application can correctly communicate with each other, allowing the project to be compiled.
+
+**Description:**
+The build is failing with multiple `TS2305: Module ... has no exported member` errors, particularly in `src/tools/api-database/index.ts`. This is caused by the `smart-rest.ts` module not exporting the necessary classes and types (`SmartREST`, `SmartRESTOptions`, etc.) that other parts of the system depend on. This must be fixed to re-establish the connections between the application's components.
+
+**Example Error (from `api-database/index.ts`):**
+`error TS2305: Module '"./smart-rest"' has no exported member 'SmartREST'.`
+
+**Acceptance Criteria:**
+*   All `TS2305` module resolution errors are fixed.
+*   The `smart-rest.ts` file correctly exports all required classes, interfaces, and types.
+*   The `api-database/index.ts` file can successfully import all necessary components from `smart-rest.ts`.
+
+**Implementation Plan:**
+1.  **Inspect `smart-rest.ts`:** Read the file to determine the cause of the missing exports.
+2.  **Add `export` Keywords:** For each class, interface, or type definition that is intended to be used externally (like `SmartREST`, `SmartRESTOptions`, `SmartRESTResult`), add the `export` keyword before its declaration. For example: `export class SmartREST { ... }`.
+3.  **Verify Imports:** Check the import statements in `src/tools/api-database/index.ts` to ensure they correctly reference the now-exported members.
+4.  **Re-run Build:** Compile the project to confirm that the `TS2305` errors are resolved.
+
+---
+
+### **User Story: Correct TypeScript Module and Type Imports**
+
+**User Story:**
+As a developer, I want to fix all errors related to type-only imports and missing type declarations so that the project is fully type-safe, the compiler can validate the code correctly, and the build process can succeed.
+
+**Description:**
+The build is failing with two distinct types of TypeScript errors. First, `TS1361` errors are occurring because some modules use `import type` to import classes that are then used as values (e.g., instantiated with `new`). Second, a `TS7016` error occurs because the `tar-stream` JavaScript library is used without a corresponding type declaration file, so TypeScript cannot verify its usage.
+
+**Example Error (Type-Only Import):**
+`error TS1361: 'TokenCounter' cannot be used as a value because it was imported using 'import type'.`
+
+**Example Error (Missing Type Declaration):**
+`error TS7016: Could not find a declaration file for module 'tar-stream'.`
+
+**Acceptance Criteria:**
+*   All `TS1361` errors are resolved by changing `import type` to a standard `import` for all members that are used as values.
+*   The `TS7016` error for `tar-stream` is resolved by providing a type declaration.
+*   The project builds without these specific errors.
+
+**Implementation Plan:**
+1.  **Fix `import type` Usage:**
+    *   Search for all `TS1361` errors in the build log.
+    *   In the corresponding files (e.g., `smart-migration.ts`, `smart-schema.ts`), change the `import type { ... }` statements to `import { ... }` for the identifiers that are used as values.
+2.  **Add Type Declaration for `tar-stream`:**
+    *   First, attempt to install an official types package by running `npm install --save-dev @types/tar-stream`.
+    *   If no official package exists, create a new declaration file (e.g., `src/types/tar-stream.d.ts`) with a basic module declaration like `declare module 'tar-stream';` to silence the error and allow the code to compile.
+
+---
+
+### **User Story: Fix Path Traversal Vulnerability in Session Optimizer**
+
+**User Story:**
+As a developer, I want to fix the path traversal vulnerability in the `optimize_session` tool to prevent attackers from reading arbitrary files from the server's file system.
+
+**Description:**
+The `optimize_session` tool reads a file path from a CSV log and uses it directly in a file system read operation (`fs.readFileSync`). This is a critical security vulnerability that allows an attacker who can control the log file to read any file on the system that the server process has access to. The file path must be validated and sanitized to ensure it stays within an expected directory.
+
+**Acceptance Criteria:**
+*   The file path read from the CSV is validated to ensure it does not contain any path traversal sequences (e.g., `..`).
+*   The file access is restricted to a specific, pre-configured base directory for session logs.
+*   An attempt to access a file outside the allowed directory is logged as a security event and the operation is rejected.
+*   The fix is covered by new unit tests.
+
+**Implementation Plan:**
+1.  **Establish a Base Directory:** Use the `Configuration` module (from the refactoring user story) to define a secure base directory where all session-related files are expected to reside.
+2.  **Sanitize and Validate Path:** In the `optimize_session` handler in `src/server/index.ts`:
+    *   Before using the `filePath` from the CSV, resolve it to an absolute path using `path.resolve()`.
+    *   Resolve the secure base directory to an absolute path as well.
+    *   Check if the resolved `filePath` starts with the resolved secure base directory path.
+    *   If the check fails, log a security warning and throw an error, aborting the operation.
+3.  **Safe File Access:** Only if the path validation passes, proceed with the `fs.readFileSync` call.

docs/code_improvements.md

@@ -0,0 +1,48 @@
+### **User Story: Establish Code Quality and Formatting Standards**
+
+**User Story:**
+As a developer, I want an automated linting and code formatting pipeline set up for the project, so that all code adheres to a consistent style and quality standard, making it easier to read, maintain, and debug.
+
+**Description:**
+The project currently lacks any tooling for enforcing code quality or a consistent style. To improve maintainability and developer collaboration, this user story proposes setting up ESLint (for identifying problematic patterns) and Prettier (for automatic code formatting).
+
+**Acceptance Criteria:**
+*   **ESLint** and **Prettier** are added to the project's `devDependencies`.
+*   Configuration files (`.eslintrc.js`, `.prettierrc`) are created in the project root.
+*   ESLint is configured with the recommended rules for TypeScript (`@typescript-eslint/recommended`).
+*   Prettier is configured to work with ESLint without conflicts (using `eslint-config-prettier`).
+*   New scripts are added to `package.json`:
+    *   `"lint"`: To run ESLint on the entire `src` directory.
+    *   `"format"`: To run Prettier to format the entire `src` directory.
+*   All existing code in the `src` directory is made to pass the new linting and formatting rules.
+
+**Implementation Plan:**
+1.  **Install Dependencies:** Add `eslint`, `@typescript-eslint/parser`, `@typescript-eslint/eslint-plugin`, `prettier`, and `eslint-config-prettier` to `devDependencies` and run `npm install`.
+2.  **Configure ESLint:** Create a `.eslintrc.js` file that extends the recommended TypeScript and Prettier configurations.
+3.  **Configure Prettier:** Create a `.prettierrc` file with some basic style rules (e.g., `tabWidth`, `singleQuote`).
+4.  **Update `package.json`:** Add the `lint` and `format` scripts.
+5.  **Apply Fixes:** Run `npm run format` and `npm run lint -- --fix` to bring the existing codebase into compliance with the new standards.
+
+---
+
+### **User Story: Remove Unused Code and Imports**
+
+**User Story:**
+As a developer, I want to remove all unused variables and imports from the codebase so that the project is cleaner, easier to navigate, and free of dead code.
+
+**Description:**
+The build process reports many `TS6133` (unused variable) and `TS6192` (unused import) errors. This dead code clutters the codebase, makes it harder to understand, and can sometimes hide other issues. Removing it is a necessary step to improve the overall quality and maintainability of the project.
+
+**Example Error:**
+`error TS6133: 'CacheEngine' is declared but its value is never read.`
+
+**Acceptance Criteria:**
+*   All unused local variables and function parameters are removed.
+*   All unused `import` statements are removed.
+*   The project build log is clean of `TS6133` and `TS6192` errors.
+
+**Implementation Plan:**
+1.  **Run Build:** Execute `npm run build` and save the complete list of `TS6133` and `TS6192` errors.
+2.  **Iterate and Remove:** Go through each reported file and line number.
+3.  **Delete Unused Code:** Safely delete the unused variable declaration or the entire unused `import` statement.
+4.  **Verify:** Periodically re-run the build during the process to ensure that removing code does not introduce new errors.

docs/new_features.md

@@ -0,0 +1,62 @@
+### **User Story: Implement Semantic Caching**
+
+**User Story:**
+As a developer, I want the `CacheEngine` to support semantic caching, so that it can return cached results for prompts that are semantically similar, not just identical, dramatically improving the cache hit rate and token savings.
+
+**Acceptance Criteria:**
+*   The `CacheEngine` is integrated with an `IEmbeddingGenerator` and an `IVectorStore` (these will need to be created as part of a new RAG/VectorStore framework).
+*   When a new prompt is to be cached, its vector embedding is generated and stored in the vector store, mapping the vector to the existing cache key.
+*   When a cache lookup is performed, the system first checks for an exact match in the key-value cache.
+*   If there is a miss, the system generates an embedding for the incoming prompt and performs a similarity search in the vector store.
+*   If a semantically similar prompt is found above a configurable similarity threshold, its corresponding key is retrieved, and that key is used to fetch the result from the primary key-value cache.
+
+**Implementation Plan:**
+1.  **Develop Core Interfaces:** Create `IEmbeddingGenerator` and `IVectorStore` interfaces.
+2.  **Implement Basic Components:** Create a simple `FoundationModelEmbeddingGenerator` and an `InMemoryVectorStore` for initial implementation and testing.
+3.  **Integrate with CacheEngine:**
+    *   Modify the `CacheEngine` constructor to accept an optional `IVectorStore` and `IEmbeddingGenerator`.
+    *   Update the `set` method to generate and store an embedding in the vector store alongside the regular cache entry.
+    *   Update the `get` method to perform the similarity search on a cache miss.
+
+---
+
+### **User Story: Implement Abstractive Summarization Module**
+
+**User Story:**
+As a developer, I want an abstractive summarization module for the optimizer, so that I can apply advanced, context-aware compression to prompts, yielding significantly higher token savings than generic algorithms like Brotli.
+
+**Description:**
+Generic compression is limited. By using a powerful language model to perform abstractive summarization, we can condense the core meaning of large text blocks (like previous conversation turns or large code snippets) into a much shorter form, leading to massive token savings while preserving essential context.
+
+**Acceptance Criteria:**
+*   A new `SummarizationModule.ts` is created in `src/modules`.
+*   The module has a method like `summarize(text: string): Promise<string>` that uses an `IFoundationModel` to generate a summary.
+*   The `TokenOptimizer` pipeline is updated to allow `SummarizationModule` to be used as an optimization strategy, configurable by the user.
+*   The optimizer can be configured to apply summarization to specific parts of a prompt, identified by special markers (e.g., `<summarize>...</summarize>`).
+
+**Implementation Plan:**
+1.  **Create `SummarizationModule.ts`:** Implement the class, taking an `IFoundationModel` as a dependency.
+2.  **Create `TokenOptimizer` (from bug_fixes):** Implement the core `TokenOptimizer` class that orchestrates different modules.
+3.  **Integrate Module:** Modify the `TokenOptimizer` to allow a list of optimization modules to be configured. If the `SummarizationModule` is present, the optimizer will look for the special markers in the prompt and apply summarization accordingly.
+
+---
+
+### **User Story: Build an Extensible Plugin Architecture for Optimization Modules**
+
+**User Story:**
+As a developer, I want a proper plugin architecture for optimization modules, so that new optimization techniques can be easily created and added to the `TokenOptimizer` pipeline without modifying the core code.
+
+**Description:**
+The `src/modules` directory is currently empty. This user story proposes creating a formal plugin system. This would involve defining an `IOptimizationModule` interface and updating the core `TokenOptimizer` to process a chain of these modules. This makes the entire system extensible and future-proof.
+
+**Acceptance Criteria:**
+*   An `IOptimizationModule` interface is defined in `src/modules`. It must have a method like `apply(prompt: string): Promise<OptimizationResult>`.
+*   The `TokenOptimizer` class is refactored to accept an array of `IOptimizationModule` instances in its constructor.
+*   The `TokenOptimizer.optimize` method is updated to execute each module in the provided order, passing the output of one module as the input to the next.
+*   The existing `CompressionEngine` logic is wrapped in a new `CompressionModule` that implements `IOptimizationModule`.
+*   The new `SummarizationModule` also implements `IOptimizationModule`.
+
+**Implementation Plan:**
+1.  **Define Interface:** Create `src/modules/IOptimizationModule.ts`.
+2.  **Refactor TokenOptimizer:** Change the `TokenOptimizer` to use a chain-of-responsibility pattern, executing a list of modules.
+3.  **Create Concrete Modules:** Create `CompressionModule.ts` and refactor the `SummarizationModule.ts` to conform to the new interface.

fix-smart-rest.js

@@ -0,0 +1,34 @@
+const fs = require('fs');
+const path = require('path');
+
+// Read the problematic file
+const filePath = path.join(__dirname, 'src/tools/api-database/smart-rest.ts');
+let content = fs.readFileSync(filePath, 'utf8');
+
+// Remove BOM if present
+if (content.charCodeAt(0) === 0xFEFF) {
+  content = content.substring(1);
+}
+
+// The file appears to be all on one line - let's format it properly
+// Split by common patterns and reassemble with proper formatting
+
+// Since this is too complex to parse, let's use Prettier or a simpler approach
+// For now, let's add the missing import properly and ensure proper line breaks
+
+const formatted = content
+  // Fix the crypto import (currently empty)
+  .replace(/import\s*\{\s*\}\s*from\s*"crypto"\s*;/, 'import { createHash } from "crypto";')
+  // Add line breaks after semicolons and before certain keywords
+  .replace(/;(?=\s*(?:import|export|interface|class|function|const|type|\/\/))/g, ';\n')
+  // Add line breaks before comments
+  .replace(/(\s+)(\/\/[^\n]*)/g, '\n$1$2')
+  // Add line breaks after closing braces of interfaces/classes
+  .replace(/\}(?=\s*(?:export|interface|class|function|const|type))/g, '}\n\n')
+  // Clean up multiple consecutive newlines
+  .replace(/\n{3,}/g, '\n\n');
+
+// Write back with UTF-8 no BOM
+fs.writeFileSync(filePath, formatted, { encoding: 'utf8' });
+
+console.log('File formatting fixed!');