out_es: Add experimental support for Amazon ElasticSearch for 1.4

[PettitWesley]

Addresses #1804 and #1807

[PettitWesley]

AWS credentials must be specified with environment variables, but it works (by 1.5, we'll have full support for all credentials providers).

Config:

[SERVICE]
    Log_Level debug

[INPUT]
    Name  cpu
    Tag   cpu

[OUTPUT]
    Name  es
    Match *
    Host  vpc-test-domain-ke7thhzoo7jawsrhmm6mb7ite7y.us-west-2.es.amazonaws.com
    Port  443
    Index my_index
    Type  my_type
    Aws_Auth On
    Aws_Region us-west-2
    tls     On
    tls.verify On

Output:

[2020/01/29 00:43:01] [debug] [task] created task=0x267aba0 id=0 OK
[2020/01/29 00:43:02] [debug] [out_es] Signing request with AWS Sigv4
[2020/01/29 00:43:02] [debug] [out_es] HTTP Status=200 URI=/_bulk
[2020/01/29 00:43:02] [debug] [out_es] Elasticsearch response
{"took":71,"errors":false,"items":[{"index":{"_index":"my_index","_type":"my_type","_id":"TijA7m8Bspwi5JFlXs4u","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":177,"_primary_term":1,"status":201}},{"index":{"_index":"my_index","_type":"my_type","_id":"TyjA7m8Bspwi5JFlXs4u","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":173,"_primary_term":1,"status":201}},{"index":{"_index":"my_index","_type":"my_type","_id":"UCjA7m8Bspwi5JFlXs4u","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":174,"_primary_term":1,"status":201}},{"index":{"_index":"my_index","_type":"my_type","_id":"USjA7m8Bspwi5JFlXs4u","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":171,"_primary_term":1,"status":201}}]}
[2020/01/29 00:43:02] [debug] [task] destroy task=0x267aba0 (task_id=0)

[PettitWesley]

I also built Fluent Bit with -DFLB_SIGNV4=Off and out_es works fine (and as expected AWS options in the config are then rejected)

[PettitWesley]

Doc PR: fluent/fluent-bit-docs#260

[edsiper]

thanks for this.

Would you please split the commits per components ? in two like :

1. out_es: ....
2. http_client: ...

[PettitWesley]

@edsiper updated

[edsiper]

thanks!

[jujugrrr]

(by 1.5, we'll have full support for all credentials providers).

Thanks a lot for the work @edsiper !
Because of short lived credentials(STS session will be 1h), using environment variable is not really an option in a secured context.

Once you support credentials provider could you include the Web identity provider? It's supported by all the latest AWS SDK versions, and is the go to solution, for a secure IAM/EKS role mapping.

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

[PettitWesley]

@jujugrrr

include the Web identity provider?

We will. I'm working on a full set of providers including EKS:

• AWS Credentials Providers PR 1: STS, EKS, and Env #1852
• AWS Credentials Providers PR 2: ECS #1854
• AWS Credentials Providers PR 3: EC2 #1855
• AWS Credentials Providers PR 4: AWS Profile #1857

using environment variable is not really an option in a secured context.

Agreed. The Fluent Bit docs for this feature note that it's experimental and that it might not be suitable for production workloads. I probably won't enable this feature in AWS for Fluent Bit either; instead we'll wait for the full set of providers to be completed.

[jujugrrr]

@PettitWesley Thanks! it looks like it's all coming. It seems much better than using proxy for the IAM signing.

I'll follow all of this closely. Let us know if we can helm with testing

[PettitWesley]

Let us know if we can help with testing

@jujugrrr You can for sure; that would be awesome. The ECR image link noted below should be accessible from any AWS account. It includes all of the credential providers except for EC2 instance role.

I'd be especially interested to know if you're an app mesh user- that's one test case I haven't covered yet- making sure the envoy proxy does not affect any of the calls Fluent Bit makes. (We've had some issues in the past where on startup calls will fail- though that was in ECS).

• 714124127858.dkr.ecr.us-west-2.amazonaws.com/fluent-bit-elastic-search-bug-bash:latest

[jujugrrr]

No app-mesh here, probably Istio in the future though. I tried #1852, but when merging your branch to fluent-bit master we were getting conflicts and some build issues during the docker build. Does the ECS image include the ENV changes only or all your other providers?

…

On Thu, 13 Feb 2020 at 21:54, Wesley Pettit ***@***.***> wrote: Let us know if we can help with testing @jujugrrr <https://github.com/jujugrrr> You can for sure; that would be awesome. The ECR image link noted below should be accessible from any AWS account. It includes all of the credential provider except for EC2 instance role. I'd be especially interested to know if you're an app mesh user- that's one test case I haven't covered yet- making sure the envoy proxy does not affect any of the calls Fluent Bit makes. (We've had some issues in the past where on startup calls will fail- though that was in ECS). - 714124127858.dkr.ecr.us-west-2.amazonaws.com/fluent-bit-elastic-search-bug-bash:latest — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1920?email_source=notifications&email_token=AA6ICL2EI22U4KQ32TKDBKLRCW6RRA5CNFSM4KM5TZP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELWYDBQ#issuecomment-585990534>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6ICLZABFBMWD4MMK2FYLDRCW6RRANCNFSM4KM5TZPQ> .

[PettitWesley]

Does the ECR image include the ENV changes only or all your other providers?

Yup, as I noted, it has all of the credential providers besides EC2 instance role.