feat(new transform): Initial aws_ec2_metadata transform imple…

[LucioFranco]

Opening this for initial feedback, most of the internals are complete, what is left is to build out the integration tests and figure out how we want to handle the subnet_id and vpc_id since those require us to supply a mac address of the network interface we want to get the id's from.

As for internal implementation, we are using an evmap that allows us to fill a map and then switch the pointers which should mean we don't have to acquire a lock for each event. This should perform just about the same as a regular transform. I think this method improves upon the previous design I came up with while being simpler in the critical path.

This PR also introduces ways we can start to use async/await without our codebase.

Closes #736

Ref fluent/fluent-bit#1780

Signed-off-by: Lucio Franco luciofranco14@gmail.com

[LucioFranco]

Update on this: I am not going to block this PR on this issue #1338 but it will mean that we will ship this initially without proper correlation logging. I don't think this will be a big deal since it should be quite obvious where the logs are coming from but this would be a good enhancment.

[LucioFranco]

This PR should be ready for review, I plan on adding docs tomorrow morning.

@binarylogic just FYI this does not add iam role name as I couldn't find a way to fetch that, the api you provided expects the user to pass the role-name and the iam/info api call doesn't provide the role name. If we find a better solution we probably want to add that in a follow up.

[binarylogic]

just FYI this does not add iam role name

That's ok. Thanks for letting me know.

[jszwedko]

You should be able to fetch the IAM role via:

$ curl http://169.254.169.254/latest/meta-data/iam/info
{
  "Code" : "Success",
  "LastUpdated" : "2019-12-10T16:43:08Z",
  "InstanceProfileArn" : "arn:aws:iam::754402436383:instance-profile/odin_client_logs_hive_ingestion/production/odin_client_logs_hive_ingestion-production-ec2-role",
  "InstanceProfileId" : "AIPAJFMF2ZTI22JBHEJ54"
}

The InstanceProfileArn there is the role. Realized that that is the instance profile and may not match.

You can curl /iam/security-credentials and get a "directory listing":

$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials
odin_client_logs_hive_ingestion-production-ec2-role

[LucioFranco]

@jszwedko ah good point, the docs are not very clear about it being a directory listing but instead suggests that you need to have the name prior to invoking that request. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-categories.html

I will add this now then. Looks like a decent alternative.

[lukesteensen]

This is nice! I was going to say we should chat before bringing in the new futures stuff, but I definitely do not want to review the version of this that doesn't use async/await 😄

[lukesteensen]

😱

[binarylogic]

I added the output fields to the docs. Everything else is good.