Object client scrape configs

[cyriltovena]

Is your feature request related to a problem? Please describe.
I think it could be interesting to add to Promtail scraping capability from object storage such as s3, gcs, etc..

We already have this abstraction :

// ObjectClient is used to store arbitrary data in Object Store (S3/GCS/Azure/Etc)
type ObjectClient interface {
	PutObject(ctx context.Context, objectKey string, object io.ReadSeeker) error
	GetObject(ctx context.Context, objectKey string) (io.ReadCloser, error)
	List(ctx context.Context, prefix string) ([]StorageObject, []StorageCommonPrefix, error)
	DeleteObject(ctx context.Context, objectKey string) error
	Stop()
}

And it seems we have everything to list files and store a positions file. Although we could still use the position file to be local ?

Describe the solution you'd like

The configuration could be like this:

scrape_configs:
  - job_name: gcs
    object_storage:
         bucket_name: GCS_BUCKET_NAME
    labels:
      __path__: /foo/**.log

  # Additional labels to assign to the logs
  [ <labelname>: <labelvalue> ... ]

Describe alternatives you've considered

Fluentd but it seems to have struggle with out of order entries.

[Punkoivan]

Hello,
it's great initiative. As a user I would like to add smth:

1. Position file should be kept on that storage or at least this ability should be provided. Reading from S3 or GCS is a kind of stateless process for promtail. If promtail is running in a docker or on some spot instances, we should be able to start exactly on the same step where we finished.
   For sure, it'll require additional permission PutObject while just reading will not.

2. It would be great to "teach" promtail to read compressed data.

About alternative way (fluentd):
We're using this approach now and it works fine, the only caveat is really out of order entries - fluentd reads new objects from S3 via SQS and SQS doesn't use FIFO. So some time we hit out of order (I guess due to SQS not fluentd itself)

[chancez]

Hello,
it's great initiative. As a user I would like to add smth:

1. Position file should be kept on that storage or at least this ability should be provided. Reading from S3 or GCS is a kind of stateless process for promtail. If promtail is running in a docker or on some spot instances, we should be able to start exactly on the same step where we finished.

You gotta be very careful with this due to the data consistency model in object storage systems. It might also be an option to use something like etcd/consul similar to how Loki uses it for it's ring, you could store cursors in there as well. You could also do locking this way, which would allow multiple promtails to coordinate scraping from the same, or different object storage systems.

[leon-seagate]

Hey @adityacs,
Are there any updates about scraping logs from s3 bucket using promtail ?

[AnthonyWC]

Would be interested in this as well.

Design doc: object store scrape #2107

This was merged #2270 (has anyone have success story with it)?

[jeschkies]

@AnthonyWC #2270 was dropped not merged.

However, it came to my attention that Lambda Promtail can scrape logs from S3. That feature is not well documented. It's also specific to load balancer logs.

[bt909]

Anyone use the Lambda Promtail possibilty? Or someone even tried Vector for this use-case?
I don't have the use-case yet, but maybe in the future and if Loki/Promtail doesn't support this, I would maybe give Vector a try.

[neerajgk]

Hey @adityacs, Are there any updates about scraping logs from s3 bucket using promtail ?

yes even i am interested, anyone please.

[cstyan]

In general, pulling logs from s3 buckets using lambda-promtail is currently possible. Some tweaks may be needed to have it work for all cases though.

As far as promtail and object storage scraping, we likely won't support it within promtail. We're currently reevaluating promtails position as a project within Grafana Labs. Internally we're actually using the Agent for both metrics and logs collection at this point.

While we haven't made a formal decision yet, we expect in the near future that all new feature work will be done in the Agent's log collection pipelines rather than in Promtail.