feat(ci): do not run expensive E2E tests on draft PRs

[hainenber]

feat(ci): do not run expensive E2E tests on draft PRs

SUMMARY

Reduce CI runner usage for the project for PRs in draft mode.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[bito-code-review]

Bito Automatic Review Skipped - Files Excluded

Bito didn't auto-review this change because all changed files are in the exclusion list for automatic reviews. No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the excluded files settings here, or contact your Bito workspace admin at evan@preset.io.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.07%. Comparing base (cb2a56d) to head (483c180).
⚠️ Report is 16 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #40720   +/-   ##
=======================================
  Coverage   64.07%   64.07%           
=======================================
  Files        2663     2663           
  Lines      143505   143505           
  Branches    32993    32993           
=======================================
  Hits        91951    91951           
  Misses      49952    49952           
  Partials     1602     1602           

Flag	Coverage Δ
javascript	67.57% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sadpandajoe]

@hainenber why would not run e2e until post merge. The question is post-merge, say it fails e2e, who is reverting or responsible for fixing this? I think this will cause a lot more issues instead of just doing the e2e tests now. A good compromise would be to not run e2e in draft and make that the manual run step, that way you can still run all your tests in draft and only pull it out when you are ready for an actual review.

[rusackas]

I would hold off on this, and consider it a last rather than first resort. We want E2E tests to grow, be reliable, and be relied on more rather than letting problems through and only find them on master (if anyone is even looking)

[hainenber]

@sadpandajoe @rusackas yep, this is an attempt at most to eliminate the issue with CI time that is predominant by E2E. This was done rather radically but other PRs from Evans seemed to help reducing E2E's CI time hugely so I do agree with preventing E2E on draft PRs. My bad for overreacting there!

[rusackas]

I appreciate you jumping in! It's the nuclear option, but nice to have the button there if needed.

Now we just need to make sure to rebase any PRs we're working on so the open PRs use less CI.

[sadpandajoe]

@hainenber not an overreaction at all and thanks for the help. I think I'm mostly concerned with the after merge state. I don't think anyone is verifying master isn't red post-merge and that's the concern. If we had a mechanism to retry then auto-revert on master I would be all for this.