ComboPrompt

This repository contains the official implementation of the paper "Example Quality Matters: Multi-Aspects Example Augmentation for Private Library Programming".

Overview

Private library programming is challenging for Retrieval-Augmented Generation (RAG) systems because these codebases usually lack publicly available training corpora. In practice, developers often rely on API documentation and its accompanying examples as the main source of retrieval knowledge. However, existing documentation examples are frequently too simple, insufficiently commented, and sometimes even erroneous and unexecutable, which limits their value for downstream code generation.

To address this problem, we propose ComboPrompt, a training-free method that augments documentation examples from multiple aspects before they are used in RAG-based private library programming. The method improves example quality through three complementary stages: generating more compositional multi-API examples, verifying their correctness and complexity, and refining their readability and code style. In this way, ComboPrompt turns weak documentation examples into stronger supervision signals for code generation.

Problem

Private libraries lack publicly available training corpora, but typically have API documentation with examples. However, these examples often fail to meet quality standards:

• Insufficient complexity: Most examples are too simple (single API calls)
• Poor readability: Lack of detailed comments and clear structure
• Correctness issues: May contain semantic errors that mislead LLMs

Core Design

ComboPrompt systematically enhances examples through three integrated modules:

1. API Combination Search

• Functional Grouping: Organize APIs by functionality
• Cross-Functional Sampling: Randomly select examples from different functional groups
• LLM-Based Combination: Generate complex multi-API examples via strategic prompting

2. Quality Verification

• Execution Verification: Validate code correctness through runtime testing
• Complexity Verification: Ensure examples meet minimum API call thresholds
• Simple Combination Detection: Filter out trivial concatenations

3. Code Style Refinement

• Readability Enhancement: Add detailed comments and improve code structure
• Semantic Variable Naming: Optimize variable names for clarity

Key Features

• Decoupled Design: Data loading is independent of specific codebase implementations
• Modular Architecture: Each module is self-contained and easily extensible
• Pure Random Sampling: No heuristic tricks, ensuring reproducibility
• Automated Validation: Multi-stage verification ensures example quality

Results

ComboPrompt achieves up to 22% accuracy improvement over baseline approaches across five private library benchmarks and different LLMs.

Code Structure

comboprompt_method.py
├── Data Loading Module
│   ├── ExampleLoader
│   └── APIDocumentLoader
├── Module 1: API Combination Search
│   ├── FunctionalGrouper
│   ├── CrossFunctionalSampler
│   ├── PromptBuilder
│   └── APICombinationSearch
├── Module 2: Quality Verification
│   ├── ExecutionVerifier
│   ├── ComplexityVerifier
│   ├── SimpleCombineDetector
│   └── QualityVerifier
└── Module 3: Code Style Refinement
    └── CodeStyleRefiner

Quick Start

from comboprompt_method import (
    ExampleLoader, organize_examples_by_api,
    create_comboprompt_from_config
)

# Load examples
loader = ExampleLoader(doc_examples_path="path/to/examples.json")
all_examples = loader.load_all_examples()
examples_by_api = organize_examples_by_api(all_examples)

# Configure functional groups
function_groups = {
    'Data Creation': ['DataFrame', 'Series'],
    'Data Manipulation': ['rename', 'drop', 'apply'],
    # ... more groups
}

# Create ComboPrompt instance
comboprompt = create_comboprompt_from_config(
    function_groups=function_groups,
    group_descriptions=group_descriptions,
    examples_by_api=examples_by_api,
    lib_name="your_library",
    min_api_calls=3
)

# Enhance examples
enhanced = comboprompt.enhance_example(
    target_api, api_doc, target_examples, llm_client
)

Additional Script

python_code_generation_evaluation.py is a separate utility script for RAG-based Python code generation and evaluation on private code libraries. It is not part of the core ComboPrompt method itself; the main method description in this repository remains centered on comboprompt_method.py.