chore(dsm): processor retries now fail as debug, unless retry limit i…

…s exceeded [backport 2.6] (#8750)

Backport e95e097 from #8739 to 2.6.

Prior to this change DSM Processor failures to contact the agent were
logged with `log.error(...)`, even though they might be retried and be
successful.

After this change, those messages are now _debug level_. To make sure
the message still appears, the enclosing try/catch statement has been
updated to include the stack trace in the error message.

## Checklist

- [x] Change(s) are motivated and described in the PR description
- [x] Testing strategy is described if automated tests are not included
in the PR
- [x] Risks are described (performance impact, potential for breakage,
maintainability)
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
are followed or label `changelog/no-changelog` is set
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/))
- [x] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))
- [x] If this PR changes the public interface, I've notified
`@DataDog/apm-tees`.
- [x] If change touches code that signs or publishes builds or packages,
or handles credentials of any kind, I've requested a review from
`@DataDog/security-design-and-guidance`.

## Reviewer Checklist

- [x] Title is accurate
- [x] All changes are related to the pull request's stated goal
- [x] Description motivates each change
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- [x] Testing strategy adequately addresses listed risks
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] Release note makes sense to a user of the library
- [x] Author has acknowledged and discussed the performance implications
of this PR as reported in the benchmarks PR comment
- [x] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

Co-authored-by: Teague Bick <teague.bick@datadoghq.com>

ddtrace/internal/datastreams/processor.py

@@ -247,7 +247,7 @@ def _flush_stats(self, payload):
             conn.request("POST", self._endpoint, payload, self._headers)
             resp = get_connection_response(conn)
         except Exception:
-            log.error("failed to submit pathway stats to the Datadog agent at %s", self._agent_endpoint, exc_info=True)
+            log.debug("failed to submit pathway stats to the Datadog agent at %s", self._agent_endpoint, exc_info=True)
             raise
         else:
             if resp.status == 404:
@@ -290,7 +290,11 @@ def periodic(self):
         try:
             self._flush_stats_with_backoff(compressed)
         except Exception:
-            log.error("retry limit exceeded submitting pathway stats to the Datadog agent at %s", self._agent_endpoint)
+            log.error(
+                "retry limit exceeded submitting pathway stats to the Datadog agent at %s",
+                self._agent_endpoint,
+                exc_info=True,
+            )
 
     def shutdown(self, timeout):
         # type: (Optional[float]) -> None

releasenotes/notes/dsm-error-logs-to-debug-1b3322c4a8790b72.yaml

@@ -0,0 +1,4 @@
+---
+fixes:
+  - |
+    datastreams: Changed DSM processor error logs to debug logs for a statement which is retried.  If all retries fail, the stack trace is included