promtail: add metrics on sent and dropped log entries

[pracucci]

What this PR does / why we need it:
If the ingester is unavailable, log entries batches are dropped (discarded) once all maxretries are exausted, but there's no metric - exported by promtail - to alert on it.

In this PR I've added the following metrics exported by promtail:

• promtail_dropped_bytes_total: Number of bytes of log lines dropped because failed to be sent to the ingester after all retries
• promtail_log_entries_sent_total: Total number of log entries sent to the ingester
• promtail_log_entries_dropped_total: Total number of log entries dropped because failed to be sent to the ingester after all retries

Which issue(s) this PR fixes:
Fixes #925

Checklist

• Documentation added
• Tests updated

[slim-bean]

:) will merge this tomorrow, I was just looking at adding these exact metrics today

[slim-bean]

LGTM!

[slim-bean]

Interesting, a conversation around #963 just came up which points out the accuracy of these metrics as implemented in this PR will be off a little. That is, the pushes to Loki are not atomic, some of the entries may have succeeded while some didn't.

I'm tempted to merge this as is until improvements are made to return the actual count of success vs failure in the error response. (I think @wardbekker is looking at adding this)

WDYT @pracucci ?

[pracucci]

@slim-bean As you noted, the metrics added in this PR are specular to the already existing ones, which also suffers the issue described in the PR #963. These metrics are inaccurate in case the same push request only a subset (but not all) of them fail to be ingested.

I would personally merge this PR given it doesn't introduce a new issue (the metrics we already have suffer it too), but at the same time start working to fix #963 (I will follow up there).

Up to you picking the last decision 🙏

[slim-bean]

I agree this is still valuable as is, merged!