Skip to content

Conversation

@konard
Copy link
Member

@konard konard commented Dec 28, 2025

Summary

Fixes #17 - Benchmark is executing too long

The CI benchmark was timing out after 6+ hours because the number of links being created was too high for PostgreSQL operations.

Root Cause Analysis

  1. CI Configuration (main branch):

    • BENCHMARK_LINKS: 1000 links per operation
    • BENCHMARK_BACKGROUND_LINKS: 100,000 background links per iteration
  2. Criterion Warning: "Unable to complete 100 samples in 5.0s. You may wish to increase target time to 15547.2s"

    • Each single sample was taking ~155 seconds for PostgreSQL tests
    • With 48 benchmarks (8 groups × 6 backends), the total time exceeded 6 hours
  3. Why PostgreSQL is slow: Network I/O + disk I/O for each operation, compared to in-memory Doublets operations

Changes Made

Parameter Before After Reduction
BENCHMARK_LINKS (main) 1000 100 10x
BENCHMARK_BACKGROUND_LINKS (main) 100000 1000 100x
Default in Rust code 3000/1000 1000/100 ~10x

Expected Impact

  • Benchmark should complete within 30-60 minutes instead of timing out
  • Results still meaningful for comparing PostgreSQL vs Doublets performance
  • PR checks remain fast (using 10/100 links)

Files Changed

  • .github/workflows/rust.yml - Reduced CI benchmark parameters
  • rust/src/lib.rs - Updated default values and documentation

🤖 Generated with Claude Code

Adding CLAUDE.md with task information for AI processing.
This file will be removed when the task is complete.

Issue: #17
@konard konard self-assigned this Dec 28, 2025
- BENCHMARK_LINKS: 1000 → 100 for main branch (10x reduction)
- BENCHMARK_BACKGROUND_LINKS: 100000 → 1000 for main branch (100x reduction)
- Updated default values in Rust code to match

The benchmark was taking 6+ hours and timing out because with
100,000 background links per iteration, PostgreSQL operations
were too slow. Criterion warned: "Unable to complete 100 samples
in 5.0s. You may wish to increase target time to 15547.2s"

Fixes #17

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@konard konard changed the title [WIP] Benchmark is executing too long, may be we should reduce number of links per operation type Reduce benchmark link counts to prevent CI timeout Dec 28, 2025
@konard konard marked this pull request as ready for review December 28, 2025 22:32
@konard
Copy link
Member Author

konard commented Dec 28, 2025

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

  • Public pricing estimate: $5.841294 USD
  • Calculated by Anthropic: $3.251763 USD
  • Difference: $-2.589531 (-44.33%)
    📎 Log file uploaded as GitHub Gist (478KB)
    🔗 View complete solution draft log

Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard konard merged commit 707b41f into main Dec 31, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark is executing too long, may be we should reduce number of links per operation type

2 participants