CI flakes under "Ensure cleanup crons are working"

[chadwhitacre]

Seeing integration tests fail quite frequently, rerun sometimes works. Noticing a few failure modes, here is one (cf. #1170):

https://github.com/getsentry/self-hosted/runs/4317568879?check_suite_focus=true

Ensure cleanup crons are working ...
  + grep -q -E -e '\-cleanup\s+running\s+' -e '\-cleanup[_-].+\s+Up\s+'
  + docker compose --ansi never --env-file /home/runner/work/self-hosted/self-hosted/.env ps
  ++ teardown ERR 121
  ++ '[' 0 -eq 1 ']'
  ++ DID_TEAR_DOWN=1
  ++ '[' ERR '!=' EXIT ']'
  ++ echo 'An error occurred, caught SIGERR on line 121'
  An error occurred, caught SIGERR on line 121
  ++ echo 'Tearing down ...'
  Tearing down ...
  ++ rm /tmp/tmp.9Qws9UthXM
  ++ echo Done.
  Done.
  + teardown EXIT 1
  + '[' 1 -eq 1 ']'
  + return 0
  Error: Process completed with exit code 255.
  ##[debug]Finishing: Integration Test

Not consistent between 1.28 and 2.0.1.

Line:

self-hosted/_integration-test/run.sh

Line 123 in da8f490

$dc ps | grep -q -E -e '\-cleanup\s+running\s+' -e '\-cleanup[_-].+\s+Up\s+'

[chadwhitacre]

https://github.com/getsentry/self-hosted/runs/4372607126?check_suite_focus=true#step:4:3336

[github-actions]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[aminvakil]

I was trying to understand what has happened in times prior to regression starting to happen yesterday and could it be because of #1119?

I couldn't understand why this should fail sometimes, maybe others have some idea?

[chadwhitacre]

Another on #1289:

https://github.com/getsentry/self-hosted/runs/5058157966?check_suite_focus=true#step:4:3617

My hunch is that cleanup runs but the test hits either before or after.

[chadwhitacre]

https://github.com/getsentry/self-hosted/runs/5116818258?check_suite_focus=true#step:4:3641

[chadwhitacre]

https://github.com/getsentry/self-hosted/runs/5116818258?check_suite_focus=true#step:4:3624

[aminvakil]

https://github.com/getsentry/self-hosted/runs/5163497601?check_suite_focus=true#step:4:1304

On e0a8ae commit.

[chadwhitacre]

https://github.com/getsentry/self-hosted/runs/5517601056?check_suite_focus=true#step:4:1513

[aminvakil]

https://github.com/getsentry/self-hosted/runs/5541434168?check_suite_focus=true#step:4:3034

[chadwhitacre]

https://github.com/getsentry/self-hosted/runs/5561229447?check_suite_focus=true#step:4:3562

This one during the release getsentry/publish#916 (comment)

Fortunately we have granular reruns now :)

[aminvakil]

https://github.com/getsentry/self-hosted/runs/5669181355?check_suite_focus=true