[Feature]: Add more modes to testOptions.video & testOptions.trace

[Az8th]

🚀 Feature Request

Hello,

We currently have 4 video & trace modes : on, off, on-first-retry, retain-on-failure.
They are nice but they are suboptimal (or even irrelevant) when doing more than one retry :

Given we have a flaky test, retries are set to 2 and we do not want to keep successes.

• Case 1 : Test passes on first try

  Mode	Run ✅	Retry 1 ⏭️	Retry 2 ⏭️	Comment
  on	💾	/	/	Suboptimal (space wise)
  off	❌	/	/	Optimal
  on-first-retry	❌	/	/	Optimal
  retain-on-failure	❌	/	/	Optimal

• Case 2 : Test fails but passes on first retry

  Mode	Run ❌	Retry 1 ✅	Retry 2 ⏭️	Comment
  on	💾	💾	/	Suboptimal (space wise)
  off	❌	❌	/	Irrelevant (does not cover failure)
  on-first-retry	❌	💾	/	Irrelevant (does not cover failure)
  retain-on-failure	💾	❌	/	Optimal

• Case 3 : Test fails twice but passes on second retry

  Mode	Run ❌	Retry 1 ❌	Retry 2 ✅	Comment
  on	💾	💾	💾	Suboptimal (space wise)
  off	❌	❌	❌	Irrelevant
  on-first-retry	❌	💾	💾	Suboptimal (space wise & if different fail causes)
  retain-on-failure	💾	💾	❌	Suboptimal (space wise) or optimal (if different fail causes)

• Case 4 : Test keeps failing

  Mode	Run ❌	Retry 1 ❌	Retry 2 ❌	Comment
  on	💾	💾	💾	Suboptimal (space wise)
  off	❌	❌	❌	Irrelevant
  on-first-retry	❌	💾	💾	Suboptimal (space wise & if different fail causes)
  retain-on-failure	💾	💾	💾	Suboptimal (space wise) or optimal (if different fail causes)

  Simply add a mode that retain only the first/last fail would be optimal in any of those cases.

Example

use: {
    video: 'retain-first-failure',
    trace: 'retain-last-failure'
}

Motivation

It would cover more usecases and reduce reports size in the best scenario (especially when performing multiple retries), while avoiding to lack clues on why a test failed in the worst one.

Thanks for reading, do not hesitate to upvote or give your insights !

[Az8th]

In order to illustrate the behaviour for added modes :

• Case 1 : Test passes on first try

  Mode	Run ✅	Retry 1 ⏭️	Retry 2 ⏭️	Comment
  retain-on-first-failure	❌	/	/	Optimal
  retain-on-last-failure	❌	/	/	Optimal

• Case 2 : Test fails but passes on first retry

  Mode	Run ❌	Retry 1 ✅	Retry 2 ⏭️	Comment
  retain-on-first-failure	💾	❌	/	Optimal
  retain-on-last-failure	💾	❌	/	Optimal

• Case 3 : Test fails twice but passes on second retry

  Mode	Run ❌	Retry 1 ❌	Retry 2 ✅	Comment
  retain-on-first-failure	💾	❌	/	Optimal (space wise) or suboptimal (if different fail causes)
  retain-on-last-failure	❌	💾	/	Optimal (space wise) or suboptimal (if different fail causes)

• Case 4 : Test keeps failing

  Mode	Run ❌	Retry 1 ❌	Retry 2 ❌	Comment
  retain-on-first-failure	💾	❌	❌	Optimal (space wise) or suboptimal (if different fail causes)
  retain-on-last-failure	❌	❌	💾	Optimal (space wise) or suboptimal (if different fail causes)

Maybe those modes could also be added to make testOptions.screenshot ? (Besides, I suggest to make it match same modes names, as it seems 'only-on-failure' serves the same purpose as 'retain-on-failure')

[Az8th]

Was partially introcuded in 1.43 (traces and retain-on-first-failure only)
Will see if I can produce a PR regarding the other cases !

[pavelfeldman]

Why was this issue closed?

Thank you for your involvement. This issue was closed due to limited engagement (upvotes/activity), lack of recent activity, and insufficient actionability. To maintain a manageable database, we prioritize issues based on these factors.

If you disagree with this closure, please open a new issue and reference this one. More support or clarity on its necessity may prompt a review. Your understanding and cooperation are appreciated.