ci: skipping tests in areas of the code that haven't changed

[gunar]

Continuing work started on #1149, #1150, #1151, and #1154 (with regards to this card, another way we could speed up CI times is by skipping tests in areas of the code that haven't changed. For example, if a PR contains only changes to editor.planx.uk/*, then no need to run API tests (although you do need to run E2E tests because they interface with the front-end). Another example is skipping Pulumi Preview if there's only been changes to the "code" code, and no change to the Infrastructure as Code code.

What do you think?

[jessicamcinchak]

Thanks for brainstorming so many ideas around this! When you suggest "skipping", at what level are you thinking? So, for individual commits/pushes in a PR, when merging to main (staging deploy), and/or merging to production (prod deploy)?

I imagine it's the nicest time-saver at the first level, but in my opinion, it's always helpful to have all tests & Pulumi commands run from staging deploy forwards independent of which files have changed - as that feels like the best re-assurance that we'd catch bugs or misconfigurations before they hit prod!

[gunar]

Yeah I was thinking about skipping them in Pull Request CI jobs only. If the PR contains changes to API, run API tests, and if it doesn't, don't — however many commits are there (i.e. diff origin/main...HEAD).

Assuming you'd be up for it then! Would love to get @DafyddLlyr's feedback too before moving forward..

[DafyddLlyr]

This sounds like a sensible approach to me - I can't think of any examples where we would need to run these tests if no related code has been touched 👍

[gunar]

Implemented in #1159