clang-tidy CI job consistently times out at 6h

[wdconinc]

Environment: (where does this bug occur, have you tried other environments)

• Which branch (often main for latest released): main
• Which version (or HEAD for the most recent on git): HEAD
• Any specific OS or system where the issue occurs? CI

Steps to reproduce: (give a step by step account of how to trigger the bug)

First main pipeline where this happened: https://github.com/eic/EICrecon/actions/runs/7048345481/job/19185731729 and all since.

Expected Result: (what do you expect when you execute the steps above)

Clang-tidy should be able to complete in 6 hours.

Actual Result: (what do you get when you execute the steps above)

Times out and is canceled at 6 hours, marking the entire pipeline as failed.

This then causes all capybara reference artifacts to be stale since it downloads from the latest successful(!) pipeline on main.

[wdconinc]

Started with f7d793d

[wdconinc]

Overview of main pipelines affected: https://github.com/eic/EICrecon/actions/workflows/linux-eic-shell.yml?query=branch%3Amain

[wdconinc]

Strangely manually submitted run was fine: https://github.com/eic/EICrecon/actions/runs/7067204244

[wdconinc]

Strangely manually submitted run was fine

It didn't run clang-tidy since neither push not pull request event which is what clang-tidy requires.

[wdconinc]

Correlated in time with clang-tidy 16 upgrade.

Jobs are at different points in the output at timeout.

But seems like it hangs for a long time when enabling time stamps.

Thu, 30 Nov 2023 16:06:59 GMT
  if(!filter_surface)
Thu, 30 Nov 2023 16:06:59 GMT
                     ^
Thu, 30 Nov 2023 16:06:59 GMT
                      {
Thu, 30 Nov 2023 21:39:55 GMT
Error: The operation was canceled.

[veprbl]

We could check with --enable-check-profile, but it looks like "run-clang-tidy" doesn't support passing that.

[wdconinc]

Might be llvm/llvm-project#55530.