fix: resolve ENOBUFS error during GitHub indexing #125
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add 50MB maxBuffer to all execSync calls in GitHub fetcher to handle large repositories with many issues/PRs. Previously, the default 1MB buffer would overflow on repositories with extensive GitHub data. Changes: - Set maxBuffer to 50MB for issue/PR list commands - Set maxBuffer to 50MB for individual issue/PR view commands - Set maxBuffer to 10MB for repo name fetch (smaller payload) - Add helpful error messages suggesting --gh-limit flag Testing: - 56 tests passing (2 test files) - Existing functionality unchanged - Error handling improved Issue: Related to ENOBUFS error during repository indexing
Reduce default GitHub fetch limit from 1000 to 500 to prevent buffer overflow on repositories with many issues/PRs. Add --gh-limit CLI flag to allow users to customize the limit based on their repository size. Changes: - Lower default limit from 1000 to 500 in fetchIssues() - Lower default limit from 1000 to 500 in fetchPullRequests() - Add --gh-limit flag to 'dev index' command - Pass limit option through to GitHubIndexer.index() - Add inline comments explaining the limit reduction CLI Usage: - dev index --gh-limit 100 # Fetch only 100 issues/PRs - dev index --gh-limit 1000 # Fetch up to 1000 (old default) - dev gh index --limit 200 # Alternative command Testing: - 62 tests passing (3 test files) - CLI command integration tests pass - GitHub indexer tests pass Issue: Related to ENOBUFS error during repository indexing
Add comprehensive unit tests for GitHub CLI fetcher functions to verify: - Default limit is now 500 (reduced from 1000) - Custom limits work correctly via options - maxBuffer is set appropriately (50MB for lists, 10MB for repo name) - Helpful error messages on ENOBUFS and maxBuffer exceeded - All required JSON fields are included in gh CLI commands - Buffer size management across different operation types Test Coverage: - 23 new tests for fetcher utilities - Total: 79 tests passing (3 test files) - Tests gh CLI installation/authentication detection - Tests default and custom limit behavior - Tests improved error handling with user guidance - Tests buffer management strategy Testing: - All 79 GitHub-related tests passing - New tests use proper mocking of execSync - Mock returns match actual execSync behavior (strings with encoding) Issue: Related to ENOBUFS error during repository indexing
Update documentation to reflect the fix for ENOBUFS errors during GitHub indexing and the new default limit of 500 items. Changes: - TROUBLESHOOTING.md: Add comprehensive ENOBUFS troubleshooting section - Explain the error cause (buffer overflow) - Provide solutions with --gh-limit flag examples - Give repository size-based recommendations - Document batch indexing strategy - packages/cli/README.md: Document new index command flags - Add --gh-limit flag documentation - Provide guidance for different repository sizes - Update index command options section - packages/subagents/src/github/README.md: Update GitHub agent docs - Change default limit from 100 to 500 in configuration docs - Add limit recommendations based on repository size - Document new buffer management (50MB/10MB) - Add ENOBUFS error to error handling section - Update performance considerations with new batch sizes - Add ENOBUFS troubleshooting section with solutions - Update test coverage stats (79 total tests) User Guidance: - Default 500 limit works for most repos - Large repos (200+): Use --gh-limit 100-200 - Very active repos (500+): Use --gh-limit 50-100 - Batch indexing available for very large repositories Testing: - All 1100+ tests passing - Documentation accuracy verified Issue: Related to ENOBUFS error during repository indexing
Update The Drill to reflect concise, focused PR descriptions rather than verbose ones. Context is important but excessive noise makes PRs harder to parse. Principles: - Focus on essential information - Include the 'why' and key decisions - Provide usage examples and verification details - Exclude verbose change logs and obvious information Based on PR #125 as a good example.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes ENOBUFS error when indexing repositories with many GitHub issues/PRs.
Problem
dev indexwould fail withENOBUFSon repositories with extensive GitHub activity due to buffer overflow (default 1MB buffer, fetching 1000+ items with large bodies).Solution
--gh-limit <number>CLI flag for customizationUsage
Testing
Changes
@lytics/dev-agent,@lytics/dev-agent-cli,@lytics/dev-agent-subagents