Fluentd output plugin to add Amazon EC2 metadata into messages
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
lib/fluent/plugin
test
.gitignore
.travis.yml
Gemfile
LICENSE.txt
README.md
Rakefile
fluent-plugin-ec2-metadata.gemspec

README.md

fluent-plugin-ec2-metadata

Gem Version Build Status Dependency Status Test Coverage Code Climate Codacy Badge

Fluentd plugin to add Amazon EC2 metadata fields to a event record

Requirements

fluent-plugin-ec2-metadata fluentd ruby
>= 0.1.0 v0.14.x >= 2.1
0.0.15 <= v0.12.x >= 1.9

Installation

Use RubyGems:

gem install fluent-plugin-ec2-metadata

Configuration

Example:

<match foo.**>
  @type ec2_metadata

  aws_key_id  YOUR_AWS_KEY_ID
  aws_sec_key YOUR_AWS_SECRET/KEY

  metadata_refresh_seconds 300 # Optional, default 300 seconds

  output_tag ${instance_id}.${tag}
  <record>
    hostname      ${tagset_name}
    instance_id   ${instance_id}
    instance_type ${instance_type}
    az            ${availability_zone}
    private_ip    ${private_ip}
    vpc_id        ${vpc_id}
    ami_id        ${image_id}
    account_id    ${account_id}
  </record>
</match>

Assume following input is coming:

foo.bar {"message":"hello ec2!"}

then output becomes as below (indented):

i-28b5ee77.foo.bar {
  "hostname"      : "web0001",
  "instance_id"   : "i-28b5ee77",
  "instance_type" : "m1.large",
  "az"            : "us-west-1b",
  "private_ip     : "10.21.34.200",
  "vpc_id"        : "vpc-25dab194",
  "account_id"    : "123456789",
  "image_id"      : "ami-123456",
  "message"       : "hello ec2!"
}

Or you can use filter version:

<filter foo.**>
  @type ec2_metadata

  aws_key_id  YOUR_AWS_KEY_ID      
  aws_sec_key YOUR_AWS_SECRET/KEY

  metadata_refresh_seconds 300 # Optional, default 300 seconds

  <record>
    hostname      ${tagset_name}
    instance_id   ${instance_id}
    instance_type ${instance_type}
    private_ip    ${private_ip}
    az            ${availability_zone}
    vpc_id        ${vpc_id}
    ami_id        ${image_id}
    account_id    ${account_id}
  </record>
</filter>

Placeholders

The following placeholders are always available:

  • ${tag} input tag
  • ${tag_parts} input tag splitted by '.'. you can use it like ${tag_parts[0]} or ${tag_parts[-1]}
  • ${instance_id} instance id
  • ${instance_type} instance type
  • ${availability_zone} availability zone
  • ${region} region
  • ${private_ip} private ip
  • ${mac} MAC address
  • ${vpc_id} vpc id
  • ${subnet_id} subnet id
  • ${account_id} account id
  • ${image_id} ami image id

The followings are available when you define aws_key_id and aws_sec_key(or define IAM Policy):

  • ${tagset_xxx} EC2 tag (e.g. tagset_name is replaced by the value of Key = Name)

The following is an example for a minimal IAM policy needed to ReadOnlyAccess to EC2.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:Describe*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "elasticloadbalancing:Describe*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:Describe*"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "autoscaling:Describe*",
      "Resource": "*"
    }
  ]
}

Refer to the AWS documentation for example policies. Using IAM roles with a properly configured IAM policy are preferred over embedding access keys on EC2 instances.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request