journal reading: Timestamp too old errors

[rfratto]

Because the journal reader starts from the very beginning of the journal (i.e., not even the current boot, but the very first entry in the journal), promtail may end up sending extremely old logs to ingesters and getting errors back:

level=error ts=2019-08-19T18:59:05.693794932Z caller=client.go:185 component=client host=localhost:3100 msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): sample for 'logs' has timestamp too old: 1562712027507"

These errors will go away once the positions file is updated, but will still appear when a user uses journal reading in promtail for the first time.

To work around this, the journal config should be updated to support a new field called from that takes a relative timestamp:

- job_name: journal
  journal:
    # read entries no older than 6h 
    from: 6h 
    path: /var/log/journal
    labels:
      job: systemd-journal
  relabel_configs:
    - source_labels: ['__journal__systemd_unit']
      target_label: 'unit'

Implementation

With the from field present, the journal reader has to do the following:

1. If a positions file is present:
2. Get journal entry at cursor.
3. If entry's date is older than the allowed time specified by from, use the from date to initialize the journal entry. Entries will be lost.
4. If entry's date is newer than the allowed time specified by from, use the cursor as the start point. No entries will be lost.
5. If no position file is present, just use the from date to initialize the journal reader. Old entries will not be imported into Loki.