Improve benchmark with statistical analysis and fair comparison #15
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Complete project structure overview - Common commands and workflows - Release process (always use ./scripts/release.sh) - Architecture details and optimizations - Linting, testing, and CI/CD guides - Troubleshooting tips This file provides context for Claude Code to work more effectively with the project without repeating instructions.
- Use generic sans-serif font for better cross-platform compatibility - Add preserveAspectRatio for proper scaling across devices - Adjust viewBox and text positioning for better layout - Update README description: msgspec is a validation and serialization library The logo now uses a simple sans font with proper responsive attributes, ensuring consistent and readable rendering on all devices and screen sizes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Unified text in one line: "msgspec-ext" - Use golden color (#a67c00) for all text - Remove line break for simpler layout Note: Text rendering still varies across devices. Consider using PNG or icon-only approach for truly consistent rendering. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Move assets/ to docs/assets/ for better organization - Generate high-quality PNG logos (600 DPI) using Inkscape - Update README to reference PNG logos in new location - Update README description: msgspec is a validation and serialization library PNG logos ensure pixel-perfect identical rendering across all browsers, operating systems, and devices, eliminating font-dependent variations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
The update command was failing because the checkout action wasn't specifying which repository to checkout from. This caused it to only fetch the default branches and not the PR branch. Added 'repository' parameter to the checkout action to explicitly specify the head repository, fixing the branch checkout issue. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Includes CLAUDE.md documentation from PR msgflux#13
vilsonrodrigues
force-pushed
the
fix/improve-benchmark
branch
2 times, most recently
from
88b5c5f to
41293e2
Compare
Contributor
Author
|
/update |
|
✅ Branch updated successfully by @vilsonrodrigues! Merged latest changes from |
Changes: - Move benchmark files to dedicated /benchmark directory - Add run_benchmark.sh script for easy execution with uvx - Improve benchmark.py with: - Statistical analysis: 10 runs with mean, median, std dev, min, max - Fair comparison: Both libraries test equivalent configuration complexity - System information: Display Python version, platform, library versions - Better warmup: 50 iterations before timing - Raw data output: All timing data for verification - Comprehensive documentation Benchmark improvements: - Test realistic configuration with 18 fields across multiple categories - msgspec-ext: Flat structure with __ delimiter (optimized for .env) - pydantic-settings: Nested structure with BaseSettings classes - Both load from identical .env file Updated README with accurate benchmark results: - 2.7x faster than pydantic-settings (down from 3.8x claim) - Based on 10 runs × 1000 iterations with statistical validation - More honest and reproducible results Run: ./benchmark/run_benchmark.sh to reproduce results 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
vilsonrodrigues
force-pushed
the
fix/improve-benchmark
branch
from
5eb17f0 to
69c6f36
Compare
Contributor
Author
|
/merge |
|
✅ PR merged successfully by @vilsonrodrigues! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Significantly improves the benchmark suite to ensure accurate and reproducible performance claims:
/benchmarkdirectory./benchmark/run_benchmark.shscript using uvxChanges
Benchmark Improvements
README Updates
File Organization
Benchmark Results
Current (10 runs × 1000 iterations, Python 3.12):
More honest, reproducible, and statistically sound results.
Test Plan
./benchmark/run_benchmark.sh)🤖 Generated with Claude Code