Skip to content

feat: enhance core block picking algo in data synthesis pipeline - #116

Merged
fishmingyu merged 1 commit into
mainfrom
enhance-context-loader
Apr 16, 2026
Merged

feat: enhance core block picking algo in data synthesis pipeline#116
fishmingyu merged 1 commit into
mainfrom
enhance-context-loader

Conversation

@boqiny

@boqiny boqiny commented Apr 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

This pull request enhances the core-block selection logic in the context_loader.py module to improve the quality and relevance of code blocks chosen for behavioral query synthesis. The main changes introduce new heuristics to filter out repetitive, overly large, or trivial code blocks, and use a more sophisticated scoring system that considers block size, connectivity, and type.

Core-block selection improvements:

  • Added a heuristic method _is_repetitive_block to detect and filter out blocks that are mostly repetitive mappings, lookup tables, or enum listings, based on the proportion of lines sharing the same leading token.
  • Introduced _score_core_block to score candidate blocks by balancing size (char_count), graph connectivity (degree), and node type, with configurable thresholds for minimum and maximum block size.
  • Modified pick_core_block to use the new scoring and filtering logic, falling back to size-based selection only if all blocks are filtered out, and logging a warning in that case.

General codebase update:

  • Imported the math module to support logarithmic scoring in the new selection logic.

Changes

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Refactoring
  • Performance improvement
  • Tests

Testing

  • Tests pass locally
  • Added new tests for the changes

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

@boqiny boqiny changed the title update core block picking algo feat: enhance core block picking algo in data synthesis pipeline Apr 15, 2026
@fishmingyu
fishmingyu merged commit 93b96b6 into main Apr 16, 2026
9 checks passed
@fishmingyu fishmingyu added scope/eval Evaluation, baselines, experiment scripts and removed dataset labels Apr 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

scope/eval Evaluation, baselines, experiment scripts

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants