Add gem version and SemVer compatibility badges to README

[greysteil]

Goal

Make it easier for new users to add bugsnag to their Gemfile.

Design

Badges that show the version and how SemVer compatible this gem is (very) give users all the information they need to pick the right gem requirement.

This PR adds a version number and SemVer compatibility badge. Data comes from Dependabot (disclosure: I built it).

Here's what the badges look like:

Tests

Discussion

Alternative Approaches

Outstanding Questions

Linked issues

Review

For the submitter, initial self-review:

• Commented on code changes inline explain the reasoning behind the approach
• Reviewed the test cases added for completeness and possible points for discussion
• A changelog entry was added for the goal of this pull request
• Check the scope of the changeset - is everything in the diff required for the pull request?
• This pull request is ready for:
  • Initial review of the intended approach, not yet feature complete
  • Structural review of the classes, functions, and properties modified
  • Final review

For the pull request reviewer(s), this changeset has been reviewed for:

• Consistency across platforms for structures or concepts added or modified
• Consistency between the changeset and the goal stated above
• Internal consistency with the rest of the library - is there any overlap between existing interfaces and any which have been added?
• Usage friction - is the proposed change in usage cumbersome or complicated?
• Performance and complexity - are there any cases of unexpected O(n^3) when iterating, recursing, flat mapping, etc?
• Concurrency concerns - if components are accessed asynchronously, what issues will arise
• Thoroughness of added tests and any missing edge cases
• Idiomatic use of the language

[snmaynard]

So dependabot upgrades other people's packages automatically and then if their CI fails it decreases our semver score? Even if the failure has nothing to do with us (just flakey tests or whatever)?

[greysteil]

Yep - the idea here is to test SemVer by checking whether updates to bugsnag cause test failures for your users. The "score" is just the percentage of CI runs that pass when updating between SemVer compatible versions, but it still gives a pretty clear indicator that bugsnag updates won't require code changes for your users if/when the upgrade it.

There are some difficulties with flaky tests that I haven't 100% ironed out yet (I should be able to filter out flaky test suites based on all the other updates Dependabot is doing for that repo) but in practice it seems flaky tests only affect the score very slightly. Of the 163 updates Dependabot has done of bugsnag only 3 have ever "caused" test failures.

[snmaynard]

Yeah - I just don't think that a 98% score is particularly clear to people who don't know what it is and how it works. I think that we'll likely get confused users if we merge this. Maybe you could give a grading system instead, but as it is I dont think it's clearer than us saying we use semver versioning.

[greysteil]

👍, although sorry to hear that. I've renamed the badge to be "semver stability" to make it clearer what it does, and I'm going to try to filter out the flaky specs later today / tomorrow.

Keep up the great work with Bugsnag - great product! ❤️

[snmaynard]

You could do something like 95%+ is given an excellent rating?

[greysteil]

The feedback I've had from others is that they like the percentage (generally because it's honest about release processes not being perfect, or because they think it's easily understandable if you click through). It's colour-coded based on the % so there's some "this is good" conditioning in there.

I'll definitely take your feedback on board, though - if others tell me the same I'll look at making it less granular.

[snmaynard]

Yeah fair - do you have examples of other projects that use it?

[greysteil]

I only created it a few days, but jest, sentry, capybara, oj, skylight and rubocop (in prep for going 1.0.0 in the next few weeks) are all using it. Think there's a couple of others, too.

Edit: now also fog, rack, sinatra and mocha.

[snmaynard]

I'll reopen it and see what @kattrali thinks when she is back from pto next week - thanks for all the context!

[greysteil]

@kattrali - any thoughts? The folks at Webpack suggested a version that only displayed updates to the latest version, so there's this option now, too:

[kattrali]

This is pretty cool! The context discussion helps too, so thanks @greysteil, @snmaynard.