Question: Amazon Kinesis

[MartinLoeper]

I searched for a way to get my vpc flow log data into elasticsearch and just stumbled across cloudwatch-logs-subscription-consumer.

You write in the description that cloudwatchlogsbeat's "operation is subject to AWS limitations and throttling policies".

Out of pure curiosity: Would this throttling policies also apply when using Amazon Kinesis - as cloudwatch-logs-subscription-consumer does?

[kkentzo]

I am not familiar with the particular project you are referring to but kinesis is certainly subject to usage limits.

[MartinLoeper]

Yes, of course there are usage limits.

However, I think the Amazon Kinesis approach might suit better for log streaming than using the CloudWatch logs apis directly.
That is because the CloudWatch Logs Limits look quite restrictive to me, compared to the Kinesis usage limits.

[kkentzo]

OK, now I see you point. Yes, I guess what you describe is a valid approach as well although I am not sure how the pricing aspect would play out in this case. Another approach could be to bypass the beat alltogether and work with logstash input plugins.