Robust log agent for loki

[toshipp]

What

Promtail, grafana official log agent for loki which currently we used, does not have a backpressure feature, so that it may lose some logs when loki server is temporarily available.
To overcome such a situation, we should improve promtail or replace it with any other log agent.

How

• Improve promtail
  In its design, promtail reads logs regardless of the upstream clogged. So it is not an easy way.
• Use fluent-bit
  we face some issues with fluent-bit.
  • Pausing sending logs because loki does not accept out-of-order logs.
    [Question] Loki plugin out of order error behaviour? fluent/fluent-bit#3082
  • invalid memory access
    Loki: Labels with specifying only key causes invalid memory access fluent/fluent-bit#3279
  • many warning
    out_loki: Allow specifiling possibly empty record accessor for labels fluent/fluent-bit#3269
  • mishandling connection timeout. it seems fixed in upstream but not released.
    Flood of connection timed out message and accordingly side effect fluent/fluent-bit#3192
• Use logstash
  it is described in loki upstream document. we have not evaluated it yet.
  https://grafana.com/docs/loki/latest/clients/logstash/
• Use vector
  it supports loki, not evaluated yet.
  https://vector.dev/

Checklist

• Finish implementation of the issue
• Test all functions
• Have enough logs to trace activities
• Notify developers of necessary actions