Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: command-line option to run test on matrix of Electron versions #672

Open
ckerr opened this issue Apr 26, 2021 · 0 comments
Open

feat: command-line option to run test on matrix of Electron versions #672

ckerr opened this issue Apr 26, 2021 · 0 comments
Assignees

Comments

@ckerr
Copy link
Member

ckerr commented Apr 26, 2021

This is analogous to the autobisect feature but will also be useful for other issues.

Specifically what I have in mind is running the test against

[ last N stable branches ] x [ first nightly release, first stable release, latest stable release ]

Use cases:

  1. It's sometimes unclear branches will need a fix backported. By testing against each supported branch, the answer to this can be automated and the baton can be handed to trop via labels. Human maintainers will not have to guess for the answer or spend time confirming the answer.

  2. This operation could be re-run each time a new major branch goes stable s.t. all pre-existing issues could be updated (via a bugbot comment + labels) to reflect how the pre-existing issues fare under the new version.

  3. bisect and autobisect assume that Electron's code flows in a linear path from older to newer releases, but this is not the case. Since we backport code to previous versions, it's possible that we could also backport a regression. Thus, a bug might not be present in 10.0.0, but exist in 10.2.0, not exist in 11.0.0, and exist in 11.1.0... if we ever hit this situation, bisection is going to bite us. Including the first version of a branch in the results will flag this situation if it arises.

TBD what N should be here. At a minimum it should be supported versions. Arguably it should be a few majors more than that to see when a regression was introduced. If we really want to get fancy we could even take the age of the bug into consideration, e.g. what version was the bug originally reported against. That policy outside of Fiddle's domain & would be decided in bugbot, so maybe N should be a command-line option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant