Cannot use environment variable for aws_sec_key or aws_key_id

[woodcockjosh]

I have the following configmap which gets mounted to /fluentd/etc/conf.d/s3.conf

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-extra-config
data:
  s3.conf: |-
    <match **>
      @type copy
      copy_mode deep
      <store>
        @type s3
        aws_key_id "#{ENV['AWS_ACCESS_KEY_ID']}"
        aws_sec_key "#{ENV['AWS_SECRET_ACCESS_KEY']}"
        s3_bucket "#{ENV['AWS_S3_BUCKET']}"
        s3_region "#{ENV['AWS_REGION']}"
        buffer_path /var/log/fluent/s3
        s3_object_key_format %{path}%{time_slice}/cluster-log-%{index}.%{file_extension}
        time_slice_format %Y%m%d-%H
        time_slice_wait 10m
        flush_interval 60s
        buffer_chunk_limit 256m
      </store>
    </match>

I get the following error in the fluentd container:

2020-07-09 18:06:04 +0000 [error]: #0 unexpected error error_class=RuntimeError error="can't call S3 API. Please check your credentials or s3_region configuration. error = #<ArgumentError: header authorization has field value \"AWS4-HMAC-SHA256 Credential=AKIAT75DXZTUAGQVGRLQ\\n/20200709/us-east-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=*************************redacted*********************\", this cannot include CR/LF>"

I have verified that the environment variable is properly set. If I change the config to hard code the aws_sec_key and aws_key_id into the configmap then everything works fine but that is obviously not ideal.

Could this have something to do with there being a + symbol in the secret access key?

[stevenarvar]

I use https://github.com/helm/charts/tree/master/stable/fluentd helm chart and using env var is fine just like your config. However, when I use google-fluentd, I ran into the same problem you have.

service google-fluentd start
google-fluentde-fluentd 1.7.0:                             [  OK  ]
2020-09-10 06:55:19 +0000 [error]: fluent/log.rb:362:error: unexpected error error_class=RuntimeError error="can't call S3 API. Please check your credentials or s3_region configuration. error = #<Aws::S3::Errors::AccessDenied: Access Denied>"
  2020-09-10 06:55:19 +0000 [error]: fluent/supervisor.rb:740:main_process: /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-s3-1.1.10/lib/fluent/plugin/out_s3.rb:409:in `rescue in check_apikeys'
  2020-09-10 06:55:19 +0000 [error]: fluent/supervisor.rb:740:main_process: /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-s3-1.1.10/lib/fluent/plugin/out_s3.rb:404:in `check_apikeys'
  2020-09-10 06:55:19 +0000 [error]: fluent/supervisor.rb:740:main_process: /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluent-plugin-s3-1.1.10/lib/fluent/plugin/out_s3.rb:218:in `start'
  2020-09-10 06:55:19 +0000 [error]: fluent/supervisor.rb:740:main_process: /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluentd-1.7.4/lib/fluent/root_agent.rb:203:in `block in start'
  2020-09-10 06:55:19 +0000 [error]: fluent/supervisor.rb:740:main_process: /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluentd-1.7.4/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'

My config looks like this with the keys hard coded:

<match secure>
  @type s3
  @log_level debug
  aws_key_id "<MY KEY>"
  aws_sec_key "<MY KEY>"
  s3_bucket my-bucket
  s3_region us-west-2
</match>

I tried using IAM with S3 permission, that did not work. I also try AWS access and secret key. That did not work. Not sure why may be fluent-plugin-s3-1.1.10 does not work?

[stevenarvar]

I figured out what is wrong with my config. My bucket name is wrong, once I corrected it. The error went away. I did not have to put "" for the keys.

[woodcockjosh]

@repeatedly why is this closed?

[repeatedly]

Ah, sorry, this is my mistake. I didn't notice last comment author and issue author are different....

[michael-zucchetta]

I am also observing this issue using the bitnami chart https://artifacthub.io/packages/helm/bitnami/fluentd

For OP: even with alphanumeric characters I get the same error

For me it looked like a problem of how the base64 string was generated (and I guess encoding),

{{ "<my secret>" | b64enc | quote }}

by doing this (in Helm), it started working

[willemveerman]

In my case I found that this was caused by improperly encoding the secret value as base64 before loading it into a Secret object

if you do

echo "access_key_here" | base64

The base 64 encode will be wrong because by default echo appends a newline character at the end of strings (hence the error cannot include CR/LF - i.e. carriage-return, line-feed)

Instead you need to do

echo -n "access_key_here" | base64

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[woodcockjosh]

/reopen

[kenhys]

@woodcockjosh #340 (comment) work for you?

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[woodcockjosh]

My issue was also including the newline character in the base64 encoding. Very tricky bug 🐛. Thank you.