Skip to content

Fix token limit overflow with includeMetadata (fixes #24) #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 16, 2025
Merged

Fix token limit overflow with includeMetadata (fixes #24) #25

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a2b110c
Fix token limit overflow with includeMetadata (fixes #24)
mkreyman Sep 16, 2025
2e1f1ea
Fix unhandled JSON.parse in checkTokenLimit function
mkreyman Sep 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 41 additions & 17 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.10.1] - 2025-01-11
## [0.10.2] - 2025-09-16

### Fixed

- **Critical Token Limit Issue (#24)** - Fixed token overflow with includeMetadata
- Implemented dynamic token limit calculation based on actual content size
- Automatically adjusts item limits based on average item size in session
- More accurate token estimation (3.5 chars/token vs 4)
- Configurable via environment variables (MCP_MAX_TOKENS, MCP_TOKEN_SAFETY_BUFFER)
- Added tokenInfo to response metadata for transparency
- Resolves "MCP tool context_get response exceeds maximum allowed tokens" errors

### Added

- **Token Limit Management Module** (`utils/token-limits.ts`)
- Dynamic calculation of safe item limits
- Response overhead estimation
- Configurable token limits via environment
- Better visibility into token usage
- Proper TypeScript interfaces for context items
- Environment variable validation with bounds checking
- Safe JSON parsing with error handling
- Well-documented constants replacing magic numbers

## [0.10.1] - 2025-07-11

### Fixed

Expand Down Expand Up @@ -63,7 +87,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added recipes for common patterns in RECIPES.md
- Added troubleshooting tips for new features

## [0.10.0] - 2025-01-26
## [0.10.0] - 2025-06-26

### Added

Expand Down Expand Up @@ -102,7 +126,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Enhanced validation for channel names
- Backward compatible - existing items default to 'default' channel

## [0.9.0] - 2025-01-20
## [0.9.0] - 2025-06-21

### Changed (BREAKING)

Expand All @@ -125,7 +149,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `context_get_shared` tool (use `context_get` instead)
- Complex sharing mechanism that was causing inconsistencies

## [0.8.4] - 2025-01-19
## [0.8.4] - 2025-06-19

### Fixed

Expand All @@ -139,7 +163,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Feature flags system (planned)
- Database migration system (planned)

## [0.8.3] - 2025-01-19
## [0.8.3] - 2025-06-19

### Added

Expand All @@ -160,7 +184,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- Automatic schema migration for existing databases to add the `working_directory` column

## [0.8.0] - 2024-01-18
## [0.8.0] - 2025-06-18

### Added

Expand Down Expand Up @@ -200,7 +224,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- New tables: `journal_entries`, `compressed_context`, `tool_events`
- All 255 tests passing

## [0.7.0] - 2024-01-18
## [0.7.0] - 2025-06-18

### Added

Expand All @@ -224,7 +248,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added comprehensive test coverage (30 new tests)
- All 236 tests passing

## [0.6.0] - 2024-01-17
## [0.6.0] - 2025-06-17

### Added

Expand All @@ -247,7 +271,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Comprehensive test coverage for semantic search
- All 206 tests passing

## [0.5.0] - 2024-01-17
## [0.5.0] - 2025-06-17

### Added

Expand All @@ -270,7 +294,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added `knowledge-graph.ts` utility module
- Comprehensive test coverage for graph operations

## [0.4.2] - 2024-01-16
## [0.4.2] - 2025-06-17

### Added

Expand All @@ -284,7 +308,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Git integration error handling
- Session list date filtering

## [0.4.1] - 2024-01-16
## [0.4.1] - 2025-06-17

### Fixed

Expand All @@ -297,7 +321,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Improved error messages for better debugging
- Enhanced validation for file paths

## [0.4.0] - 2024-01-15
## [0.4.0] - 2025-06-17

### Added

Expand All @@ -318,7 +342,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Created `git.ts` utility module
- 97% test coverage maintained

## [0.3.0] - 2024-01-14
## [0.3.0] - 2025-06-17

### Added

Expand Down Expand Up @@ -347,7 +371,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Implemented streaming for large exports
- Transaction support for atomic operations

## [0.2.0] - 2024-01-13
## [0.2.0] - 2025-06-17

### Added

Expand Down Expand Up @@ -377,7 +401,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Memory leak in file cache operations
- Session switching race condition

## [0.1.0] - 2024-01-12
## [0.1.0] - 2025-06-17

### Added

Expand Down Expand Up @@ -405,12 +429,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## Development Releases

### [0.1.0-beta.2] - 2024-01-11
### [0.1.0-beta.2] - 2025-06-17

- Fixed Windows path handling
- Added Node.js 18+ compatibility

### [0.1.0-beta.1] - 2024-01-10
### [0.1.0-beta.1] - 2025-06-17

- Initial beta release
- Basic functionality testing
Expand Down
26 changes: 26 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,10 +183,36 @@ claude mcp add memory-keeper node /absolute/path/to/mcp-memory-keeper/dist/index

### Environment Variables

#### Storage and Installation

- `DATA_DIR` - Directory for database storage (default: `~/mcp-data/memory-keeper/`)
- `MEMORY_KEEPER_INSTALL_DIR` - Installation directory (default: `~/.local/mcp-servers/memory-keeper/`)
- `MEMORY_KEEPER_AUTO_UPDATE` - Set to `1` to enable auto-updates

#### Token Limit Configuration

- `MCP_MAX_TOKENS` - Maximum tokens allowed in responses (default: `25000`, range: `1000-100000`)
- Adjust this if your MCP client has different limits
- `MCP_TOKEN_SAFETY_BUFFER` - Safety buffer percentage (default: `0.8`, range: `0.1-1.0`)
- Uses only this fraction of the max tokens to prevent overflows
- `MCP_MIN_ITEMS` - Minimum items to return even if exceeding limits (default: `1`, range: `1-100`)
- Ensures at least some results are returned
- `MCP_MAX_ITEMS` - Maximum items allowed per response (default: `100`, range: `10-1000`)
- Upper bound for result sets regardless of token limits
- `MCP_CHARS_PER_TOKEN` - Characters per token ratio (default: `3.5`, range: `2.5-5.0`) **[Advanced]**
- Adjusts token estimation accuracy for different content types
- Lower values = more conservative (safer but returns fewer items)
- Higher values = more aggressive (returns more items but risks overflow)

Example configuration for stricter token limits:

```bash
export MCP_MAX_TOKENS=20000 # Lower max tokens
export MCP_TOKEN_SAFETY_BUFFER=0.7 # More conservative buffer
export MCP_MAX_ITEMS=50 # Fewer items per response
export MCP_CHARS_PER_TOKEN=3.0 # More conservative estimation (optional)
```

### Claude Code (CLI)

#### Configuration Scopes
Expand Down
10 changes: 8 additions & 2 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "mcp-memory-keeper",
"version": "0.10.1",
"version": "0.10.2",
"description": "MCP server for persistent context management in AI coding assistants",
"main": "dist/index.js",
"bin": {
Expand Down
Loading