loki-docker-driver does not support reconstructing docker logs split by docker in 16kB chunks

[andoks]

Describe the bug
Log entries are being split into multiple lines by docker, and as the loki docker driver (aka. Docker loki log plugin) does not reconstruct entries that are split into multiple lines/chunks , these chunks/lines are treated as individual log entries in Loki and Grafana, which breaks any JSON formatted structured logs

To Reproduce
Steps to reproduce the behavior:

1. Installed loki-docker-driver (grafana/loki-docker-driver:2.9.2)
2. Configured docker-compose service to use the loki-docker-driver docker plugin for logs
3. Configured plugin to put logs into loki
4. Started Loki (FROM grafana/loki:2.9.5@sha256:55561cf48b19e03d982e1f959efc0d8842b7be141eb5eaaa4b8a8eeb5a2dfc58)
5. Started service using docker-compose
6. Made service produce a huge log entry (this happens outside of my control, as this is not actually an internal service)
7. Observed "broken" log entries split over multiple lines in Loki using Grafana.
   By broken I mean that log entries that were supposed to be a single entry are being parsed and treated as multiple entries, many of which are missing valuable fields like level and original timestamp, as those fields are only detected on the single entry/line which contains those fields, and not for the lines that are being displayed as different entries.

Expected behavior
The loki-docker-driver recombines the split chunks/lines into a single entry/line, and treats them as such, even before the pipeline-configuration in the loki-docker-driver enters the picture.

Environment:

• Infrastructure: Docker compose running on desktop computer
• Deployment tool: Docker compose

Screenshots, Promtail config, or terminal output
I can share screenshots and logs privately on request

Links

• Equivalent issue for promtail: Reconstruct split docker log lines in Promtail #2281
• Related loki/promtail issue: Consistently corrupt log data for long (>16kB) log lines #12197
• Possibly the origin PR over at docker/moby that introduced line split on 16kB: Improve logging of long log lines moby/moby#22982
• Interesting related issue with some information about the splitting of log lines by docker daemon: Allow log line max size to be configurable moby/moby#32923
• Possibly related code in the docker daemon: https://github.com/moby/moby/blob/cd14846d0cde098bb83037d99104db6fadfef039/daemon/logger/copier.go#L139-L152
• Comment ref docker log consumers are supposed to reconstruct partial log lines: Allow log line max size to be configurable moby/moby#32923 (comment)