Webhooks to Github (maybe other hooks) stops reporting back at around 2600 tests

[KyleThen]

Summary

When using the webhook to github, it seems to stop around ~2600 some tests. We have ~3700 tests that run so the webhook never reports that the run is complete.

On github:

On the dashboard:

How to reproduce

Environment

• sorry-cypress version: 1.1.1
• cypress version: 8.2.0
• platform: aws
• service: api | director | dashboard | all

[ImanMahmoudinasab]

@KyleThenTR It may be because of the timeout setting. check the timeout setting of the project.

[KyleThen]

@KyleThenTR It may be because of timeout setting. check the timeout setting of the project.

I thought that was the case, initially it was only at 5 minutes but I changed it to 20 minutes and it still didn't finish.

[agoldis]

@KyleThenTR probably a rate limit issue from GH. It's quite tricky to troubleshot without any access to the data / logs. If you can share / provide either I'd be happy to help!
,

[KyleThen]

@agoldis here is the error that starts popping up in the director logs when the webhooks start to not report for a run.

[agoldis]

oh that's a nasty error!

We're storing too much information in the DB and the document exceeds 16MB size. Will issue a fix ASAP!

[agoldis]

@KyleThenTR check out #427

[KyleThen]

@agoldis appears to be working on v2, thanks!