Report the results of Polymer/tachometer in a comment for Pull Requests.

name: Pull Request Test

on: [pull_request]

jobs:
  pr_test:
    runs-on: ubuntu-latest

    steps:
      # Setup repo to run benchmarks
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v1
      - run: npm ci

      # Run benchmarks
      - name: Run tachometer and generate results file
      - run: npm run tach --config benchmarks.json --json-file results.json

      # Read results and post comment
      - name: Report Tachometer Result
        uses: andrewiggins/tachometer-reporter-action@v2
        with:
          path: results.json
          report-id: benchmark1

name: PR Setup Job Flow

on: [pull_request]

# Demo how to use the setup job flow
#
# In this flow, a workflow must create a separate "setup" job that initializes
# the comment by passing initialize: true. Then each other job that actually
# reports benchmark results must declare the "setup" job as a dependency in its
# "needs" array.

jobs:
  setup:
    name: Setup Tachometer Reporting
    runs-on: ubuntu-latest
    steps:
      - name: Initialize tachometer comment
        uses: andrewiggins/tachometer-reporter-action@v2
        with:
          initialize: true

  bench_1:
    name: First Bench Job
    # Wait for setup job to complete before running this job
    needs: [setup]
    runs-on: ubuntu-latest
    steps:
      # Setup repo to run benchmarks
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v1
      - run: npm ci

      # Run benchmarks
      - name: Run tachometer and generate results file
      - run: npm run tach --config benchmarks.json --json-file results.json

      # Read results and post comment
      - name: Report Tachometer Result
        uses: andrewiggins/tachometer-reporter-action@v2
        with:
          path: results.json
          report-id: benchmark1

  bench_2:
    name: Second Bench Job
    # Wait for setup job to complete before running this job
    needs: [setup]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      # Add other tasks similar to bench_1 job to run the second benchmark

      # Report second benchmark results
      - name: Report Other Tachometer Result
        uses: andrewiggins/tachometer-reporter-action@v2
        with:
          path: other-results.json

• Pick a summary of comparisons to highlight at the top
• See what benchmarks are currently running

tachometer-reporter-action will generate a summary of results for you if you provide a pr-bench-name and base-bench-name. This summary will take the results of comparing the results identified by the pr-bench-name vs. the results identified by the base-bench-name and put them at the top of the of the comment.

By default, if pr-bench-name or base-bench-name are not provided, then the first two benchmarks in the tachometer results will be compared.

If the report-id option is provided, then tachometer-reporter-action will add a stopwatch icon (⏱) next to any benchmark that is currently running. If the action can determine the current job id, then the icon is a link to the action job that is running the benchmark (see #7).

Tachometer supports defining multiple measurements per benchmark configuration. In the summary section, we group summaries by measurements. In the result details for each benchmark, we build tables for each measurement in that benchmark.

The path to the results JSON file of the tachometer run.

Give this report (i.e. results file) an ID to uniquely identify it against other instances of tachometer-reporter action. If you have multiple jobs or steps all running tachometer-reporter action for the same PR, use this field so that the actions don't collide with each other

Use this option with report-id.

Determines whether this action instance should initialize the comment to report results. Useful if multiple jobs are sharing the same comment. Pass in true if this job should always create the comment, false if this job should never create the comment, or leave empty if the default behavior is desired (wait a random time before creating comment if it doesn't exist. Due to race conditions, this could lead to duplicate comments and is not recommended if your workflow has multiple jobs using this action).

Generally this option is only necessary if you are running multiple jobs in the same workflow with this action. If so, create a "setup" job to that initializes the comment by passing initialize: true. Then each other job that actually reports benchmark results must declare the "setup" job as a dependency in its "needs" array. See the Multiple benchmark jobs usage sample for an example.

Pass true to this option to automatically open this actions benchmark results in the PR comment. By default, all results are collapsed.

The benchmark or version (identified in Tachometer results by either the "name" field or "version" field) which represents the changes made in this PR.

The benchmark or version (identified in Tachometer results by either the "name" field or "version" field) which serves as the base this PR is to be compared against (e.g. the latest published version of your library/website).

This action will create one comment per workflow that uses it. So if you have two workflows that each run two jobs that use this action (a total of 4 instances of this action), you should have two comments in your PR.

By default, tachometer-reporter-action relies on a timing heuristic so that multiple jobs don't try to create multiple comments at the same time. However this timing heuristic doesn't work for all workflow configurations. To workaround this use the initialize option to instruct only one tachometer-reporter-action instance in a workflow to initialize the comment all other tachometer-reporter-actions will share. See the Multiple benchmark jobs usage sample for an example.

The results are inserted into the comment based on the title of the benchmark that produced the results. So a workflow that has multiple jobs reporting results will show the results in the lexical order of their report-id inputs. If no report-id is provided, a title is determined based on the name and version fields of the tachometer benchmark results.

If you quickly push to a PR, two different workflow runs could be triggered at close to the same time. Depending on how long the benchmarks take to run, the earlier workflow run running not the latest code could complete after the later workflow run running the latest code.

To prevent this situation, where older out-of-date results could override the latest results, only results that come from a workflow run with a run number equal or higher to the current run number in the comment will be written.

If you action has multiple benchmarks running in different jobs, it is possible that the reporter-action will try to overwrite each other's results if both jobs read the comment before either has updated it with their results. In this situation, both jobs get a view of the comment with no results and only adds their results to the comment. The last job whose update is written would overwrite the update of the other job.

To mitigate this situation, this action implements a basic "locking" mechanism on the comment. Before any job can make significant changes to the comment, it must first write an empty span to the comment with an ID and wait a short time to ensure all other jobs have seen it's span, claiming the comment as locked. If another job sees that this span exists, it will wait a random amount of time before trying to acquire the comment lock for itself. This protocol isn't perfect, but it is likely good enough for our purposes.