ci: handle RemoteDisconnected in snapshot tests

[emmettbutler]

This change handles errors like this in the snapshot test harness by retrying requests. This makes sense because we always expect such errors to be transient, and if they happen not to be the test will eventually time out.

This change also marks a few recently observed unreliable failures (one, two) and adds an exception condition to a Django test that recently exhibited it.

Checklist

• Change(s) are motivated and described in the PR description
• Testing strategy is described if automated tests are not included in the PR
• Risks are described (performance impact, potential for breakage, maintainability)
• Change is maintainable (easy to change, telemetry, documentation)
• Library release note guidelines are followed or label changelog/no-changelog is set
• Documentation is included (in-code, generated user docs, public corp docs)
• Backport labels are set (if applicable)
• If this PR changes the public interface, I've notified @DataDog/apm-tees.

Reviewer Checklist

• Title is accurate
• All changes are related to the pull request's stated goal
• Description motivates each change
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Change is maintainable (easy to change, telemetry, documentation)
• Release note makes sense to a user of the library
• Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: emmett.butler/flakes-june
Commit report: 91c8c80
Test service: dd-trace-py

✅ 0 Failed, 119390 Passed, 58867 Skipped, 4h 9m 40.39s Total duration (3h 19m 32.86s time saved)

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 12 lines in your changes missing coverage. Please review.

Project coverage is 10.30%. Comparing base (c035b91) to head (51e385c).
Report is 2 commits behind head on main.

Files	Patch %	Lines
tests/utils.py	0.00%	10 Missing ⚠️
tests/debugging/test_debugger.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #9464       +/-   ##
===========================================
- Coverage   75.91%   10.30%   -65.61%     
===========================================
  Files        1314     1284       -30     
  Lines      124462   122626     -1836     
===========================================
- Hits        94482    12642    -81840     
- Misses      29980   109984    +80004     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[erikayasuda]

Just one nit question

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-06-10 19:40:25

Comparing candidate commit 1e5f1da in PR branch emmett.butler/flakes-june with baseline commit bcf0939 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 221 metrics, 9 unstable metrics.

[brettlangdon]

tiny suggestion, otherwise lgtm