Skip to content

Conversation

@sergical
Copy link
Member

@sergical sergical commented Oct 29, 2025

always said slower

From Claude Code

⏺ Looking at the benchmark script, the calculation is mathematically correct:

  diff=$(echo "scale=2; $agent_avg - $direct_avg" | bc)
  percent=$(echo "scale=1; ($agent_avg - $direct_avg) / $direct_avg * 100" | bc)

  However, there's a presentation issue - the output assumes agent mode is always slower:

  echo "  +${diff}s (${percent}% slower)"

  Problem: If direct mode is actually slower than agent mode, you'd get confusing output like:
  - +-1.5s (-10% slower)

  The fix should handle both cases:

  # Calculate difference
  diff=$(echo "scale=2; $agent_avg - $direct_avg" | bc)
  percent=$(echo "scale=1; ($agent_avg - $direct_avg) / $direct_avg * 100" | bc)

  # Determine if slower or faster
  if (( $(echo "$diff > 0" | bc -l) )); then
    echo "Difference:"
    echo "  +${diff}s (${percent}% slower)"
  else
    abs_diff=$(echo "scale=2; -1 * $diff" | bc)
    abs_percent=$(echo "scale=1; -1 * $percent" | bc)
    echo "Difference:"
    echo "  -${abs_diff}s (${abs_percent}% faster)"
  fi

  Would you like me to fix this in the script?

@dcramer dcramer merged commit a5f0a78 into main Oct 29, 2025
14 checks passed
@dcramer dcramer deleted the sergical/fix-benchmark-test branch October 29, 2025 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants