s3 input with cloudtrail codec

[cdenneen]

Currently what input does the cloudtrail codec work with? Documentation isn't clear but using with s3 input ingests raw json into message.

If you specify a filter { json { source => 'message' } } it will convert it but I believe that was supposed to be replaced with cloudtrail codec instead of just using json codec?

[DanielRedOak]

I'm using the s3 input with codec => cloudtrail. Events are output as json, so your output would need to use the codec => json to correctly receive the message I believe. My process is to grab them with s3 + cloudtrail codec then output using lumberjack + json codec to be later ingested by my main logstash setup.

[tedder]

The problem with the JSON codec is that it doesn't expect there to be an array of JSON objects per event, only one JSON object.

For the fun of it, this is my cloudtrail-input.conf that does a few extra things.

input {
  s3 {
    bucket => "bucketname"
    delete => false
    interval => 60 # seconds
    prefix => "bucket_prefix_to_cloudtrail/"
    type => "cloudtrail"
    codec => "cloudtrail"
    credentials => "/etc/logstash/s3_credentials.ini"
    sincedb_path => "/opt/logstash_cloudtrail/sincedb"
  }
}

filter {
  if [type] == "cloudtrail" {
    mutate {
      gsub => [ "eventSource", "\.amazonaws\.com$", "" ]
    }
    ruby {  
      code => "event['ingest_time'] = Time.now.utc.strftime '%FT%TZ'"
    }  

    if [eventSource] == "elasticloadbalancing" and [eventName] == "describeInstanceHealth" and [userIdentity.userName] == "a_specific_user" {
      drop {}
    }
  }
}

[cdenneen]

Having issue with s3 input with cloudtrail codec keeps crashing

{:timestamp=>"2015-06-11T16:12:27.808000-0400", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n  Plugin: <LogStash::Inputs::S3 bucket=>\"our.cloudtrail.NNNNNNNN\", credentials=>[\"XXXXXXXXXXXXX\", \"YYYYYYYYYYYY\"], region_endpoint=>\"us-east-1\", sincedb_path=>\"/var/log/logstash/cloudtrail.NNNNNNN.sincedb\", type=>\"cloudtrail\">\n  Error: unexpected token at '0c22b8e6593b3eabfb00cf5f1ed73cba1a1200fdf19aeb0646b9da1d01522010 produsaphotoevent [07/Jun/2015:23:40:37 +0000] XXX.XXX.XXX.10 arn:aws:iam::NNNNNNNN:user/Prod_User 5B8075E1A6D42855 REST.GET.BUCKET - \"GET /?prefix=HeartBeat%2FHeartBeat.txt HTTP/1.1\" 200 - 605 - 59 58 \"-\" \"aws-sdk-dotnet/1.5.18.0 .NET Runtime/4.0 .NET Framework/4.0 OS/6.1.7601.65536 S3Sync\" -\n'", :level=>:error}