Only stewardship committee members have rights to modify the repository, although others are welcome to send pull requests. Modifying the repository should not be done except as outlined in the process below.
Each issue (conceptually, a specific performance pain point) has its own issue in the issue tracker. Any logged-in user may create an issue--they need not be a committee member. Issues may be merged into other issues if they are deemed to be similar.
Remember the guidelines for what makes a good issue. Issues are included in new versions of the benchmark by default unless a compelling case can be made by a committee member that the issue violates one of the requirements. This means that issues that list real-world examples of this being a performance problem have a higher likelihood of being included.
Proposed benchmarks that target an issue should be sent as a pull request that clearly identifies which performance issue number they target. Issues don’t need to have a benchmark attached to be considered for inclusion in the suite, but the barrier for inclusion is lower (and thus they’re more likely to be included) if they do. Any community member may propose a benchmark as a pull request, and each issue may have multiple proposed benchmarks in the form of patches--however, the community is encouraged to coordinate on patches to keep the number small.
Issues that have not been included in any version of the benchmark and have not yet met the requirements to be Up For Consideration (i.e. they do not have 25 stars from the community and they also do not have 5 stars + a proposed benchmark). The default status for new issues.
Issues that have not been included in any version of the benchmark but are up for consideration in the next iteration because they have met the minimum requirements (i.e. they have either 25 stars or 5 stars + a proposed benchmark). All issues in this status will be included in the voting procedure at the beginning of the next refresh cycle.
Issues that were vetoed by the committee as not meeting the standards for inclusion in the suite. The committee will leave a comment detailing why the issue was vetoed. Issues may be set to Up For Consideration again if the community believes they have addressed the reason the issue was vetoed.
Issues that are up for consideration in this update of the suite but do not have a proposed benchmark yet that meets the minimum quality bar. If they do not have a proposed benchmark by the time issue inclusion closes, the issue will not make it into this release of the benchmark and the status will be moved back to Up For Consideration.
Issues that are up for consideration and have a proposed benchmark that meets the quality bar. They will be included in the next version of the suite.
Issues that are currently included in the suite.
Issues that are currently in the suite but have proposals to change them (for example because a browser has special-cased the benchmark as it exists today).
Issues that are currently in the suite but are being proposed to be removed in the next update of the benchmark.
Issues that were once in the benchmark but have been removed by the committee. Issues in this status should have a comment from the committee about why they were removed. If the community believes they have addressed the reason it was removed, the issue can be moved back to Up For Consideration.
Issues that have been deemed too similar to existing issues.
Status transitions that are not automatic based on certain criteria (e.g. New --> Up For Consideration is an automatic transition, but Included --> Proposed for modification is not) can only have their status changed once a movement to change the status has been raised by a member of the community in the comments, and a separate member of the community has seconded the movement. Once these conditions have been met, it is up to a member of the stewardship committee to modify the actual status of the issue in the issue tracker.
Last edited by jkomoros,