diff --git a/.gitignore b/.gitignore index 92cb619..55a456c 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,6 @@ coverage.out # Local worktrees .worktrees/ + +# Local OMO planning/state artifacts +.sisyphus/ diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..92dabf7 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 go-modkit authors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/MAINTAINERS.md b/MAINTAINERS.md new file mode 100644 index 0000000..bd623b5 --- /dev/null +++ b/MAINTAINERS.md @@ -0,0 +1,35 @@ +# Maintainers + +This document outlines the maintainership roles, triage expectations, and response windows for the `benchmarks` repository. + +## Roles + +- **Lead Maintainers**: Responsible for the overall direction, architecture, and final approval of major changes. +- **Maintainers**: Responsible for reviewing pull requests, triaging issues, and maintaining specific components (e.g., parity runner, benchmark scripts). +- **Triage**: Responsible for initial issue screening, labeling, and routing questions to [Discussions](https://github.com/go-modkit/benchmarks/discussions). + +## Triage Expectations + +We aim to provide timely feedback on community contributions while balancing other project priorities. + +| Activity | Initial Response Window | Goal | +|----------|-------------------------|------| +| **New Issues** | 2-3 business days | Acknowledge, label, and request missing info. | +| **Pull Requests** | 3-5 business days | Provide initial review or feedback. | +| **Discussions** | Best effort | Community-driven; maintainers participate as time permits. | + +*Note: Response windows are targets, not guarantees. We appreciate your patience.* + +## Escalation Paths + +### Security Vulnerabilities +Do not report security vulnerabilities via public issues. Follow the [Security Policy](SECURITY.md) for private disclosure. + +### Urgent Issues +For critical regressions or build failures impacting the main branch, please mention a maintainer in the relevant issue or PR. + +## Governance + +Decisions are made through consensus among maintainers. For major architectural changes, a design document (RFC) should be submitted to `docs/design/` for review. + +See [CONTRIBUTING.md](CONTRIBUTING.md) for the technical contribution process. diff --git a/README.md b/README.md index b50d7c6..b9d4673 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,11 @@ # benchmarks +[![CI](https://github.com/go-modkit/benchmarks/actions/workflows/ci.yml/badge.svg)](https://github.com/go-modkit/benchmarks/actions/workflows/ci.yml) +[![Codecov](https://codecov.io/gh/go-modkit/benchmarks/branch/main/graph/badge.svg)](https://codecov.io/gh/go-modkit/benchmarks) +[![CodeQL](https://github.com/go-modkit/benchmarks/actions/workflows/codeql.yml/badge.svg)](https://github.com/go-modkit/benchmarks/actions/workflows/codeql.yml) +[![Go Report Card](https://goreportcard.com/badge/github.com/go-modkit/benchmarks)](https://goreportcard.com/report/github.com/go-modkit/benchmarks) +[![License](https://img.shields.io/github/license/go-modkit/benchmarks)](LICENSE) + Benchmark harness for framework parity and performance comparisons. ## What this repository does @@ -8,6 +14,23 @@ Benchmark harness for framework parity and performance comparisons. - stores declarative parity fixtures and seed data - provides benchmark orchestration scripts and report generation +## How this complements modkit + +This repository serves as the companion performance laboratory for [go-modkit/modkit](https://github.com/go-modkit/modkit). While modkit focuses on developer ergonomics and modular architecture, this harness ensures that those abstractions do not come at the cost of performance or correctness. + +By maintaining a strict parity-gate, we ensure that every framework implementation compared here—including modkit—adheres to the same API contract before a single request is timed. + +## Why trust these benchmarks + +We prioritize correctness and reproducibility over "hero numbers": + +- **Parity-Gated**: Benchmarks are automatically skipped if a target fails the API behavior contract. We only measure what is correct. +- **Reproducible**: All runs use Docker-based orchestration with pinned resource limits and standardized load profiles. +- **Transparent**: Raw metrics, environment fingerprints, and statistical variance are preserved for every run. +- **Policy-Driven**: Quality gates enforce statistical significance and schema validation for all artifacts. + +For detailed execution details and fairness principles, see our [Methodology](METHODOLOGY.md) and [Benchmark Workflow](docs/guides/benchmark-workflow.md). + ## Quickstart Run parity against a local target: @@ -76,11 +99,21 @@ benchmarks/ ## Documentation +**Design & Architecture:** - `docs/design/002-api-parity-contract.md` - parity contract rationale - `docs/architecture.md` - repository architecture and execution flow + +**Guides:** - `docs/guides/parity-contract.md` - fixture and matcher semantics - `docs/guides/adding-scenarios.md` - how to add parity scenarios - `docs/guides/benchmark-workflow.md` - benchmark and reporting flow +- `docs/guides/benchmark-publication-policy.md` - minimum disclosure for publishable results +- `docs/guides/repository-metadata.md` - repository metadata and positioning + +**Governance:** +- `MAINTAINERS.md` - maintainership roles and triage expectations +- `CONTRIBUTING.md` - contribution process and validation commands +- `SECURITY.md` - security policy and vulnerability reporting ## Contributing diff --git a/ROLLOUT_CHECKLIST.md b/ROLLOUT_CHECKLIST.md new file mode 100644 index 0000000..2c9e47b --- /dev/null +++ b/ROLLOUT_CHECKLIST.md @@ -0,0 +1,36 @@ +# OSS Hardening and OMO Enablement - Rollout Checklist + +## Summary +All 16 tasks completed successfully. Work is ready for PR. + +## Branch +- **Branch**: feat/oss-hardening-and-omo-enablement +- **Worktree**: .worktrees/feat-oss-hardening/ +- **Commits**: 7 commits ahead of main + +## Files Changed +1. LICENSE (new) - MIT License +2. MAINTAINERS.md (new) - Governance doc +3. README.md - Badge row, modkit narrative, trust section +4. docs/guides/AGENTS.md - Guide index updates +5. docs/guides/benchmark-publication-policy.md (new) - Publication policy +6. docs/guides/repository-metadata.md (new) - Metadata profile + +## Suggested PR Sequence + +### PR 1: Legal and Baseline Trust + + +### PR 2: Governance and Positioning + + +### PR 3: OMO Setup Guide + + +## Post-Merge Actions +1. Apply GitHub metadata via commands in docs/guides/repository-metadata.md +2. Verify badges render correctly on README +3. Confirm docs/guides navigation works + +## Evidence +See .sisyphus/evidence/task-8-final-gate.txt for verification output. diff --git a/docs/guides/AGENTS.md b/docs/guides/AGENTS.md index 8811c99..aa32981 100644 --- a/docs/guides/AGENTS.md +++ b/docs/guides/AGENTS.md @@ -14,6 +14,9 @@ Operator-facing guides for parity contract usage, scenario authoring, and benchm | Understand fixture semantics | `docs/guides/parity-contract.md` | Request/response schema and matcher tokens | | Add or modify scenarios safely | `docs/guides/adding-scenarios.md` | Endpoint-grouped workflow + checklist | | Run benchmark pipeline | `docs/guides/benchmark-workflow.md` | Standard, per-target, and artifact expectations | +| Publish benchmark results | `docs/guides/benchmark-publication-policy.md` | Minimum disclosure for publishable results | +| Manage repo metadata | `docs/guides/repository-metadata.md` | Description, topics, and gh commands | +| View maintainership info | `MAINTAINERS.md` | Roles, triage SLA, and escalation | ## CONVENTIONS diff --git a/docs/guides/benchmark-publication-policy.md b/docs/guides/benchmark-publication-policy.md new file mode 100644 index 0000000..76223be --- /dev/null +++ b/docs/guides/benchmark-publication-policy.md @@ -0,0 +1,68 @@ +# Benchmark Publication and Reproducibility Policy + +This document defines the minimum disclosure requirements and operational bounds for publishable benchmark results generated by this harness. Adherence to this policy ensures that published claims are reproducible, statistically sound, and fair. + +## Minimum Disclosure Requirements + +Any public claim or report referencing benchmarks from this repository must include or link to the following artifacts from the `results/latest/` directory: + +### 1. Environment Manifest and Fingerprint +- **Artifacts**: `environment.manifest.json`, `environment.fingerprint.json` +- **Requirement**: Disclosure of the runner metadata, toolchain versions (Go, Python, Docker), and host environment characteristics. +- **Purpose**: Ensures results can be contextualized against the hardware and software stack used during the run. + +### 2. Schema Validation +- **Requirement**: All artifacts must pass `make benchmark-schema-validate`. +- **Purpose**: Guarantees that the raw metrics and summaries adhere to the versioned data contract. + +### 3. Quality Policy Compliance +- **Requirement**: The run must pass `make ci-benchmark-quality-check` as defined in `stats-policy.yaml`. +- **Metrics**: Must include at least RPS (median) and Latency (P50, P95, P99). +- **Variance**: Coefficient of Variation (CV) must stay within the thresholds defined in `stats-policy.yaml` (e.g., <10% for RPS). + +### 4. Parity and Skip Transparency +- **Requirement**: Any target implementation that failed the parity gate must be disclosed as "Skipped (Parity Failure)". +- **Purpose**: Prevents "cherry-picking" results from implementations that do not yet meet the functional contract. + +## Operational Bounds + +To maintain comparability and prevent resource exhaustion in CI/CD environments, publishable runs must stay within the following bounds (aligned with `benchmark-manual` workflow): + +| Parameter | Range | Default | +|-----------|-------|---------| +| **Runs per target** | 1 .. 10 | 3 | +| **Requests per run** | 50 .. 1000 | 300 | +| **Frameworks** | Subset of `modkit,nestjs,baseline,wire,fx,do` | - | + +Runs exceeding these bounds are considered "Experimental" and should not be used for official framework comparisons or public performance claims. + +## Reproducibility Workflow + +To generate a publishable report, follow the standard workflow: + +1. **Clean Environment**: Ensure a clean working tree and stable runtime versions. +2. **Execution**: + ```bash + make benchmark + ``` +3. **Validation**: + ```bash + make benchmark-schema-validate + make ci-benchmark-quality-check + ``` +4. **Reporting**: + ```bash + make report + ``` + +The resulting `results/latest/report.md` and `results/latest/summary.json` are the authoritative sources for publication. + +## Fairness and Interpretation + +- **Fairness**: All runs must adhere to the principles in [METHODOLOGY.md](../../METHODOLOGY.md). +- **Disclaimer**: Every published report must include the fairness disclaimer regarding cross-language comparisons and runtime effects. +- **Correctness First**: Performance data from a run with parity failures is invalid for comparative claims. + +## Policy Enforcement + +The `make publication-sync-check` command (and its CI equivalent) verifies that published artifacts align with the current methodology and versioned policy. diff --git a/docs/guides/repository-metadata.md b/docs/guides/repository-metadata.md new file mode 100644 index 0000000..f896ef8 --- /dev/null +++ b/docs/guides/repository-metadata.md @@ -0,0 +1,55 @@ +# Repository Metadata Profile + +This document defines the GitHub repository metadata for the `benchmarks` project. Maintaining accurate metadata ensures discoverability and clear positioning within the `go-modkit` ecosystem. + +## Metadata Profile + +| Field | Value | +|-------|-------| +| **Description** | Parity-gated benchmark harness for API framework performance comparison. Ensures correctness via declarative contract fixtures before measuring throughput and latency. | +| **Topics** | `go`, `benchmark`, `api-parity`, `performance-testing`, `reproducibility`, `modkit`, `framework-comparison`, `api-contract`, `docker-orchestration`, `quality-gates` | +| **Homepage** | `https://github.com/go-modkit/benchmarks` | + +## Application Checklist + +### Automated (via GitHub CLI) + +If you have the `gh` CLI installed and authenticated, run the following command to apply the profile: + +```bash +gh repo edit go-modkit/benchmarks \ + --description "Parity-gated benchmark harness for API framework performance comparison. Ensures correctness via declarative contract fixtures before measuring throughput and latency." \ + --add-topic "go,benchmark,api-parity,performance-testing,reproducibility,modkit,framework-comparison,api-contract,docker-orchestration,quality-gates" \ + --homepage "https://github.com/go-modkit/benchmarks" +``` + +### Manual Fallback + +If the GitHub CLI is unavailable, follow these steps: + +1. Navigate to the repository on GitHub: [go-modkit/benchmarks](https://github.com/go-modkit/benchmarks) +2. Click on the **Settings** gear icon (or the "About" section edit icon on the main page). +3. In the **Description** field, paste: + > Parity-gated benchmark harness for API framework performance comparison. Ensures correctness via declarative contract fixtures before measuring throughput and latency. +4. In the **Website** field, paste: + > https://github.com/go-modkit/benchmarks +5. In the **Topics** section, add the following tags one by one: + - `go` + - `benchmark` + - `api-parity` + - `performance-testing` + - `reproducibility` + - `modkit` + - `framework-comparison` + - `api-contract` + - `docker-orchestration` + - `quality-gates` +6. Click **Save changes**. + +## Verification + +To verify the current metadata, run: + +```bash +gh repo view go-modkit/benchmarks --json description,repositoryTopics,homepage +```