charm tracing fails over tls

[michaeldmitry]

Bug Description

When trying to integrate loki with tempo to send its charm traces over tls, it fails to do with an exception "Exception while exporting Span batch."

To Reproduce

1. juju deploy cos-lite --trust
2. juju refresh loki --channel latest/edge
3. juju deploy tempo-k8s tempo --channel latest/edge
4. juju deploy self-signed-certificates
5. juju integrate self-signed-certificates:certificates loki:certificates
6. juju integrate self-signed-certificates:certificates tempo:certificates
7. juju integrate tempo:tracing loki:tracing
8. wait for an event on loki (e.g: update-status)
9. look for error in juju logs

Environment

juju version: 3.4.2-genericlinux-amd64
microk8s version: MicroK8s v1.28.7 revision 6532
loki rev: 144

Relevant log output

unit-loki-0: 11:50:22 ERROR unit.loki/0.juju-log Exception while exporting Span batch.
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/urllib3/response.py", line 737, in _error_catcher
    yield
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/urllib3/response.py", line 862, in _raw_read
    data = self._fp_read(amt, read1=read1) if not fp_closed else b""
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/urllib3/response.py", line 845, in _fp_read
    return self._fp.read(amt) if amt is not None else self._fp.read()
  File "/usr/lib/python3.8/http/client.py", line 459, in read
    n = self.readinto(b)
  File "/usr/lib/python3.8/http/client.py", line 503, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/urllib3/response.py", line 1043, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/urllib3/response.py", line 935, in read
    data = self._raw_read(amt)
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/urllib3/response.py", line 891, in _raw_read
    self._fp.close()
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/urllib3/response.py", line 764, in _error_catcher
    raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/opentelemetry/sdk/trace/export/__init__.py", line 368, in _export_batch
    self.span_exporter.export(self.spans_list[:idx])  # type: ignore
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 145, in export
    resp = self._export(serialized_data)
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 114, in _export
    return self._session.post(
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/requests/sessions.py", line 747, in send
    r.content
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/requests/models.py", line 899, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
  File "/var/lib/juju/agents/unit-loki-0/charm/venv/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

Additional context

No response