Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a way to bless minicrater tests #546

Open
jyn514 opened this issue Sep 10, 2020 · 0 comments
Open

Add a way to bless minicrater tests #546

jyn514 opened this issue Sep 10, 2020 · 0 comments

Comments

@jyn514
Copy link
Member

jyn514 commented Sep 10, 2020

Right now, failed tests have output like this:

Output
test minicrater::resource_exhaustion ... FAILED

failures:

---- minicrater::resource_exhaustion stdout ----
Difference between expected and actual reports:
{
  "categories": [],
  "comparison_colors": {},
  "categories": [
    [
      "spurious-fixed",
      {
        "Plain": [
          {
            "name": "memory-hungry (local)",
            "res": "spurious-fixed",
            "runs": [
              {
                "log": "stable/local/memory-hungry",
                "res": 0
              },
              {
                "log": "beta/local/memory-hungry",
                "res": 1
              }
            ],
            "url": "https://github.com/rust-lang/crater/tree/master/local-crates/memory-hungry"
          }
        ]
      }
    ]
  ],
  "comparison_colors": {
    "spurious-fixed": {
      "Striped": [
        "#5630db",
        "#5d3dcf"
      ]
    }
  },
  "crates_count": 2,
  "full": false,
  "info": {
    "test-pass": 2
    "spurious-fixed": 1,
    "test-pass": 1
  },
  "nav": [
    {
      "active": true,
      "label": "Summary",
      "url": "index.html"
    },
    {
      "active": false,
      "label": "Full report",
      "url": "full.html"
    },
    {
      "active": false,
      "label": "Downloads",
      "url": "downloads.html"
    }
  ],
  "result_colors": [],
  "result_names": []
  "result_colors": [
    {
      "Single": "#db3026"
    },
    {
      "Single": "#65461e"
    }
  ],
  "result_names": [
    "build OOM",
    "test OOM"
  ]
}


To expect the new report in the future run:
$ cp tests/minicrater/resource-exhaustion/index.html.context.actual.json tests/minicrater/resource-exhaustion/index.html.context.expected.json

Difference between expected and actual reports:
{
  "available_archives": [
    {
      "name": "All the crates",
      "path": "logs-archives/all.tar.gz"
    },
    {
      "name": "test-pass crates",
      "path": "logs-archives/test-pass.tar.gz"
    },
    {
      "name": "spurious-fixed crates",
      "path": "logs-archives/spurious-fixed.tar.gz"
    }
  ],
  "crates_count": 2,
  "nav": [
    {
      "active": false,
      "label": "Summary",
      "url": "index.html"
    },
    {
      "active": false,
      "label": "Full report",
      "url": "full.html"
    },
    {
      "active": true,
      "label": "Downloads",
      "url": "downloads.html"
    }
  ]
}


To expect the new report in the future run:
$ cp tests/minicrater/resource-exhaustion/downloads.html.context.actual.json tests/minicrater/resource-exhaustion/downloads.html.context.expected.json

Difference between expected and actual reports:
{
  "categories": [
    [
      "test-pass",
      {
        "Plain": [
          {
            "name": "build-pass (local)",
            "res": "test-pass",
            "runs": [
              {
                "log": "stable/local/build-pass",
                "res": 0
              },
              {
                "log": "beta/local/build-pass",
                "res": 0
              }
            ],
            "url": "https://github.com/rust-lang/crater/tree/master/local-crates/build-pass"
          },
          }
        ]
      }
    ],
    [
      "spurious-fixed",
      {
        "Plain": [
          {
            "name": "memory-hungry (local)",
            "res": "test-pass",
            "res": "spurious-fixed",
            "runs": [
              {
                "log": "stable/local/memory-hungry",
                "res": 0
                "res": 1
              },
              {
                "log": "beta/local/memory-hungry",
                "res": 0
                "res": 2
              }
            ],
            "url": "https://github.com/rust-lang/crater/tree/master/local-crates/memory-hungry"
          }
        ]
      }
    ]
  ],
  "comparison_colors": {
    "spurious-fixed": {
      "Striped": [
        "#5630db",
        "#5d3dcf"
      ]
    },
    "test-pass": {
      "Single": "#72a156"
    }
  },
  "crates_count": 2,
  "full": true,
  "info": {
    "test-pass": 2
    "spurious-fixed": 1,
    "test-pass": 1
  },
  "nav": [
    {
      "active": false,
      "label": "Summary",
      "url": "index.html"
    },
    {
      "active": true,
      "label": "Full report",
      "url": "full.html"
    },
    {
      "active": false,
      "label": "Downloads",
      "url": "downloads.html"
    }
  ],
  "result_colors": [
    {
      "Single": "#62a156"
    },
    {
      "Single": "#db3026"
    },
    {
      "Single": "#65461e"
    }
  ],
  "result_names": [
    "test passed"
    "test passed",
    "build OOM",
    "test OOM"
  ]
}


To expect the new report in the future run:
$ cp tests/minicrater/resource-exhaustion/full.html.context.actual.json tests/minicrater/resource-exhaustion/full.html.context.expected.json

Difference between expected and actual reports:
{
  "categories": [],
  "categories": [
    [
      "spurious-fixed",
      {
        "Plain": [
          {
            "krate": {
              "Local": "memory-hungry"
            },
            "name": "memory-hungry (local)",
            "res": "spurious-fixed",
            "runs": [
              {
                "log": "stable/local/memory-hungry",
                "res": "build-fail:oom"
              },
              {
                "log": "beta/local/memory-hungry",
                "res": "test-fail:oom"
              }
            ],
            "url": "https://github.com/rust-lang/crater/tree/master/local-crates/memory-hungry"
          }
        ]
      }
    ]
  ],
  "crates_count": 2,
  "full": false,
  "info": {
    "test-pass": 2
    "spurious-fixed": 1,
    "test-pass": 1
  }
}


To expect the new report in the future run:
$ cp tests/minicrater/resource-exhaustion/markdown.md.context.actual.json tests/minicrater/resource-exhaustion/markdown.md.context.expected.json

thread 'minicrater::resource_exhaustion' panicked at 'invalid report generated by Crater', tests/minicrater/driver.rs:212:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    minicrater::resource_exhaustion

This does have instructions for blessing the failure (cp tests/minicrater/resource-exhaustion/markdown.md.context.actual.json tests/minicrater/resource-exhaustion/markdown.md.context.expected.json), but there's so much output it's easy to miss it. In particular, if more than one test fails it's very easy to miss that there are multiple commands you need to run, since they might have disappeared off your terminal screen, requiring you to run the tests again to see the output.

It would be great to have a cargo test minicrater -- --bless command that automatically updates the expected JSON files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant