[exporter/loki] Do not retry on 4xx status code (excluding 429)

[wiardvanrij]

Description:
Do not retry on 4xx status code (excluding 429), as these are permanent errors (i.e. retrying will never change the state). Therefore causing an unnecessary increase in retries (and its queue).

Link to tracking Issue:
#18059

Testing:
Tested locally with a full setup:

• Logs that do not receive a 4xx status code (excluding 429) continue to get pushed into Loki
• Logs that do get a 4xx status code (excluding 429) get dropped:

2023-01-28T14:20:18.105+0100	error	exporterhelper/queued_retry.go:394	Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "data_type": "logs", "name": "loki", "error": "Permanent error: HTTP 400 \"Bad Request\": entry for stream '{exporter=\"OTLP\"}' has timestamp too old: 2023-01-01T16:06:26Z, oldest acceptable timestamp is: 2023-01-21T13:20:18Z", "dropped_items": 19}

I was considering making tests for it, but actually seems somewhat challenging to do properly. Might require a new issue to improve test coverage in general.

Documentation:
Change-log added

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: wiardvanrij / name: Wiard van Rij (c8af42d, 0c0120d)

[mar4uk]

look good to me!

[runforesight]

Foresight Summary

    
Major Impacts

TestMySqlIntegration ❌ failed 1 times in 2 runs (50% fail rate).

TestMySqlIntegration/Running_mysql_version_8.0 ❌ failed 1 times in 2 runs (50% fail rate).

build-and-test-windows duration(5 seconds) has decreased 40 minutes 22 seconds compared to main branch avg(40 minutes 27 seconds).

View More Details

⭕  build-and-test-windows workflow has finished in 5 seconds (40 minutes 25 seconds less than main branch avg.) and finished at 30th Jan, 2023.

---

Job	Failed Steps	Tests
windows-unittest-matrix	-     🔗	N/A	See Details
windows-unittest	-     🔗	N/A	See Details

---

✅  check-links workflow has finished in 1 minute 49 seconds and finished at 30th Jan, 2023.

---

Job	Failed Steps	Tests
changed files	-     🔗	N/A	See Details
check-links	-     🔗	N/A	See Details

---

✅  telemetrygen workflow has finished in 2 minutes 23 seconds and finished at 30th Jan, 2023.

---

Job	Failed Steps	Tests
build-dev	-     🔗	N/A	See Details
publish-latest	-     🔗	N/A	See Details
publish-stable	-     🔗	N/A	See Details

---

✅  prometheus-compliance-tests workflow has finished in 3 minutes 30 seconds (4 minutes 2 seconds less than main branch avg.) and finished at 30th Jan, 2023.

---

Job	Failed Steps	Tests
prometheus-compliance-tests	-     🔗	✅ 21  ❌ 0  ⏭ 0    🔗	See Details

---

✅  changelog workflow has finished in 4 minutes 1 second (⚠️ 1 minute 55 seconds more than main branch avg.) and finished at 30th Jan, 2023.

---

Job	Failed Steps	Tests
changelog	-     🔗	N/A	See Details

---

✅  load-tests workflow has finished in 21 minutes 4 seconds (⚠️ 6 minutes 51 seconds more than main branch avg.) and finished at 30th Jan, 2023.

---

Job	Failed Steps	Tests
loadtest (TestTraceAttributesProcessor)	-     🔗	✅ 3  ❌ 0  ⏭ 0    🔗	See Details
loadtest (TestIdleMode)	-     🔗	✅ 1  ❌ 0  ⏭ 0    🔗	See Details
loadtest (TestMetric10kDPS|TestMetricsFromFile)	-     🔗	✅ 6  ❌ 0  ⏭ 0    🔗	See Details
loadtest (TestTraceBallast1kSPSWithAttrs|TestTraceBallast1kSPSAddAttrs)	-     🔗	✅ 10  ❌ 0  ⏭ 0    🔗	See Details
loadtest (TestMetricResourceProcessor|TestTrace10kSPS)	-     🔗	✅ 12  ❌ 0  ⏭ 0    🔗	See Details
loadtest (TestTraceNoBackend10kSPS|TestTrace1kSPSWithAttrs)	-     🔗	✅ 8  ❌ 0  ⏭ 0    🔗	See Details
loadtest (TestBallastMemory|TestLog10kDPS)	-     🔗	✅ 19  ❌ 0  ⏭ 0    🔗	See Details
setup-environment	-     🔗	N/A	See Details

---

❌  build-and-test workflow has finished in 35 minutes 11 seconds (17 minutes 28 seconds less than main branch avg.) and finished at 30th Jan, 2023. 1 job failed.

---

Job	Failed Steps	Tests
integration-tests	-     🔗	✅ 55  ❌ 0  ⏭ 0    🔗	See Details
cross-compile (darwin, amd64)	-     🔗	N/A	See Details
cross-compile (darwin, arm64)	-     🔗	N/A	See Details
cross-compile (linux, 386)	-     🔗	N/A	See Details
cross-compile (linux, amd64)	-     🔗	N/A	See Details
cross-compile (linux, arm)	-     🔗	N/A	See Details
cross-compile (linux, arm64)	-     🔗	N/A	See Details
cross-compile (linux, ppc64le)	-     🔗	N/A	See Details
cross-compile (windows, 386)	-     🔗	N/A	See Details
cross-compile (windows, amd64)	-     🔗	N/A	See Details
build-package (deb)	Test deb package     🔗	N/A	See Details
build-package (rpm)	-     🔗	N/A	See Details
windows-msi	-     🔗	N/A	See Details
publish-check	-     🔗	N/A	See Details
publish-dev	-     🔗	N/A	See Details
publish-stable	-     🔗	N/A	See Details

---

🔎 See details on Foresight

^{*You can configure Foresight comments in your organization settings page.}