Question: Does Camel looks for IAM roles from Env variables #282

saranyaeu2987 · 2020-06-17T23:58:46Z

This is a followup of #267.

We configured EKS k8s pod with serviceaccount with IAM role for S3 authentication (https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html) which will create 2 ENV variables inside pod

AWS_ROLE_ARN=arn:aws:iam::AWS_ACCOUNT_ID:role/IAM_ROLE_NAME
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

When I try to run Camel Sink connector, I get Access Denied issue for S3

QUESTION: Does camel.component.aws-s3.useIAMCredentials: true looks for IAM role arn from those env variables in Pod?

The text was updated successfully, but these errors were encountered:

oscerd · 2020-06-18T05:11:08Z

The useiamcredentials option just looks for using the ec2 IAM credentials and use them.

saranyaeu2987 · 2020-06-18T13:47:21Z

@oscerd
Where do you look for IAM credentials in a EC2 instance? Are Env variables in credentials provider chain?

oscerd · 2020-06-18T13:57:03Z

That's related to the underline instance. It's the IAM role with you created the ec2 instance. No the env variables are not taken into account. With the Aws2 components you can inject your own client instance and it will look to env variables if you don't provide a credential provider

valdar · 2020-06-18T14:04:13Z

@saranyaeu2987 there are more info here: aws/aws-sdk-java-v2#1470 but you need sdk version >= 2.10.11, so you need to use camel-aws2-*-kafka-connector versions...

saranyaeu2987 · 2020-06-18T14:20:18Z

@valdar
Thanks for prompt response. I am currently using camel-aws-s3-kafka-connector-0.2.0.jar
https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/ has only 0.2.0 version.
Where can I find camel-aws2-2.10.11-kafka-connector jar/binaries ?

geota · 2020-06-18T14:24:18Z

@valdar is correct. This should be as easy as Camel using the correct SDK version that has support for the AWS_WEB_IDENTITY_TOKEN_FILE env var. This should just be crawled in the DefaultCredentialProvider chain that the SDK executes to find credentials.

oscerd · 2020-06-18T14:28:00Z

For the env variables for secret and access key, the standard client instance should check them out of the box

geota · 2020-06-18T14:34:38Z

@oscerd we are trying to get EKS role based credentials working. To be clear, this is the official AWS supported way of providing role base credentials to Kubernetes pods. Our company does not allow access key/secret keys to avoid credential management overhead (securing, cycling, invalidating etc) and enforces all AWS access to use roles.

AWS added a new way to source credentials in their SDKs to support K8s pod scoped IAM roles. @valdar linked the correct issue discussing this need in the java-sdk. It was fxed via aws/aws-sdk-java-v2#1501 and released as part of AWS Java SDK 2.10.11

See:
https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

oscerd · 2020-06-18T14:39:47Z

Anyway, it should possible when instancing your own client. Using the correct version

saranyaeu2987 · 2020-06-18T14:43:07Z

@oscerd @valdar
Camel repo has only camel-aws2-0.2.0-kafka-connector.jar , Does it support aws-sdk >= 2.10.11 ?

saranyaeu2987 · 2020-06-18T19:10:10Z

@valdar
I am not very successful in resolving access issue.

I tried both camel-aws2-s3-kafka-connector-0.2.0.jar and camel-aws-s3-kafka-connector-0.2.0.jar both gave me access issue.

Below are list of plugins used.

I am unsure if camel-aws2-s3-kafka-connector-0.2.0.jar and camel-aws-s3-kafka-connector-0.2.0.jar uses aws-sdk >= 2.10.11

oscerd · 2020-06-18T19:56:57Z

Only the Aws2 connectors are based on sdk v2. The version used on 0.2.0 kafka connector of sdk v2 is 2.11.5

saranyaeu2987 · 2020-06-18T20:15:36Z

@oscerd @valdar
I tried 2 images having camel aws2 jars and both gave the same following error
Image 1 with following plugins obtained from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws-s3-kafka-connector/0.2.0/ and replaced 2 aws jars ( camel-aws-s3-3.2.0.jar, camel-aws-s3-kafka-connector-0.2.0.jar) with aws2 jar (camel-aws2-s3-3.3.0.jar, camel-aws2-s3-kafka-connector-0.2.0.jar)

Image 2 with following plugins obtained from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/

Caused by: java.lang.IllegalArgumentException: Error configuring property: camel.component.aws-s3.configuration.autocloseBody because cannot find component with name aws-s3. Make sure you have the component on the classpath
	at org.apache.camel.main.BaseMainSupport.lambda$autoConfigurationFromProperties$15(BaseMainSupport.java:989)
	at org.apache.camel.main.BaseMainSupport.computeProperties(BaseMainSupport.java:1249)
	at org.apache.camel.main.BaseMainSupport.autoConfigurationFromProperties(BaseMainSupport.java:985)
	at org.apache.camel.main.BaseMainSupport.autoconfigure(BaseMainSupport.java:562)
	at org.apache.camel.main.BaseMainSupport.postProcessCamelContext(BaseMainSupport.java:607)
	at org.apache.camel.main.BaseMainSupport.initCamelContext(BaseMainSupport.java:440)
	at org.apache.camel.main.Main.doInit(Main.java:109)
	at org.apache.camel.support.service.BaseService.init(BaseService.java:83)
	at org.apache.camel.support.service.BaseService.start(BaseService.java:111)
	at org.apache.camel.main.MainSupport.run(MainSupport.java:83)
	at org.apache.camel.kafkaconnector.utils.CamelMainSupport$CamelContextStarter.run(CamelMainSupport.java:220)
	... 3 more
2020-06-18 18:43:23,384 ERROR WorkerSinkTask{id=heb-s3-s123456-connector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO Stopping CamelSinkTask connector task (org.apache.camel.kafkaconnector.CamelSinkTask) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO Stopping CamelContext (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO CamelContext stopped (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO CamelSinkTask connector task stopped (org.apache.camel.kafkaconnector.CamelSinkTask) [task-thread-heb-s3-s123456-connector-0]

I feel camel-aws2-s3-.jar is giving me this error, because if I have camel-aws-s3-.jar doesnt through above error.

Please guide me which jar Am I missing !!

oscerd · 2020-06-18T20:27:13Z

The scheme for aws2-s3 is aws2-s3, so it is camel.component.aws2-s3...

oscerd · 2020-06-18T21:35:31Z

You have all the information and download links here
https://camel.apache.org/camel-kafka-connector/latest/connectors.html

saranyaeu2987 · 2020-06-18T21:43:08Z

@oscerd @valdar

You have all the information and download links here
https://camel.apache.org/camel-kafka-connector/latest/connectors.html

Yes, thats where I am getting all jars from

I still getting access denied issues after using aws2-s3. TBH I am clueless

oscerd · 2020-06-18T21:53:58Z

Please report your configuration, what you're setting, how is your client instance?

oscerd · 2020-06-18T21:55:52Z

And you don't have to replace any jar in your plugin. Simply download the Aws2 s3 zipped connector and unzip it in your plugin path. Don't mix up dependencies.

saranyaeu2987 · 2020-06-18T22:27:23Z

And you don't have to replace any jar in your plugin. Simply download the Aws2 s3 zipped connector and unzip it in your plugin path. Don't mix up dependencies.

Thats what I did in last run.

Configurations

K8S on EKS cluster. Pod instance config
Yaml file

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
  name: s3-abcd-connector
  labels:
    strimzi.io/cluster: emd-kafka-cluster
spec:
  class: org.apache.camel.kafkaconnector.CamelSinkConnector
  tasksMax: 1
  config:
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.kafka.connect.storage.StringConverter
    topics: DB.EMD.COMPANY.0
    camel.sink.url: aws2-s3://<bucketname>?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
    camel.sink.maxPollDuration: 1000
    camel.sink.maxBatchPollSize: 10
    camel.component.aws2-s3.configuration.autocloseBody: false
    camel.component.aws2-s3.useIAMCredentials: true

Env variables property set in pod

[kafka@emd-kafka-cluster-connect-5fb8cdf44d-jf6s2 kafka]$ printenv |grep aws
AWS_ROLE_ARN=arn:aws:iam::<arn>:role/emd-kafka-cluster-connect
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

(K8s Pod Configuration)
Service Account which has access to bucket

ML-C02Z605SLVDQ:selumalai-k8s-s3-connect e192270$ aws-vault exec heb-dsol-foundational-projects-bld-staff -- kubectl  get serviceaccounts emd-kafka-cluster-connect -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<xxxxxxxxx>:role/emd-kafka-cluster-connect
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{"eks.amazonaws.com/role-arn":"arn:aws:iam::<xxxxxxxx>:role/emd-kafka-cluster-connect"},"creationTimestamp":"2020-06-17T21:48:25Z","labels":{"app.kubernetes.io/instance":"emd-kafka-cluster","app.kubernetes.io/managed-by":"strimzi-cluster-operator","app.kubernetes.io/name":"kafka-connect","app.kubernetes.io/part-of":"strimzi-emd-kafka-cluster","strimzi.io/cluster":"emd-kafka-cluster","strimzi.io/kind":"KafkaConnect","strimzi.io/name":"strimzi"},"name":"emd-kafka-cluster-connect","namespace":"default","resourceVersion":"855936","selfLink":"/api/v1/namespaces/default/serviceaccounts/emd-kafka-cluster-connect","uid":"24bdb5d3-1d43-4bf6-b883-b275b0f5397e"},"secrets":[{"name":"emd-kafka-cluster-connect-token-glprh"}]}
  creationTimestamp: "2020-06-17T22:19:54Z"
  labels:
    app.kubernetes.io/instance: emd-kafka-cluster
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: kafka-connect
    app.kubernetes.io/part-of: strimzi-emd-kafka-cluster
    strimzi.io/cluster: emd-kafka-cluster
    strimzi.io/kind: KafkaConnect
    strimzi.io/name: strimzi
  name: emd-kafka-cluster-connect
  namespace: default
  resourceVersion: "859885"
  selfLink: /api/v1/namespaces/default/serviceaccounts/emd-kafka-cluster-connect
  uid: 528c4309-4581-4fcb-a66f-f64741299709
secrets:
- name: emd-kafka-cluster-connect-token-glprh
- name: emd-kafka-cluster-connect-token-bvq9n

oscerd · 2020-06-19T05:56:46Z

If you use the useIAMCredential parameter, the client won't check for env variables. The documentation is clear on this: "Set whether the S3 client should expect to load credentials on an EC2 instance or to expect static credentials to be passed in."

You need to provide your own client instance, like explained here: https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/credentials.html

Region region = Region.US_WEST_2; S3Client s3 = S3Client.builder() .region(region) .credentialsProvider(EnvironmentVariableCredentialsProvider.create()) .build();

And then reference this as camel.sink.endpoint.amazonS3Client

saranyaeu2987 · 2020-06-19T06:32:51Z

@oscerd

If I dont provide
camel.component.aws2-s3.useIAMCredentials: true
Camel expects static for value for camel.component.aws-s3.accessKey AND camel.component.aws-s3.secretKey right ? What value should I need to provide them?
Also, camel.sink.endpoint.amazonS3Client is a reference to a com.amazonaws.services.s3.AmazonS3 in the registry.
I didnt understand why we need to set this config ? and what value goes into it ?

oscerd · 2020-06-19T06:41:27Z

useIamCredentials works by using the underline EC2 credentials used to create the ec2 instance, if you don't use that you need to provide static credentials.
You need to create an instance of the client and pass it to the configuration like we did for example here: https://github.com/apache/camel-kafka-connector/blob/master/examples/CamelAmqpSourceConnector.properties#L27 with the #class notation.

I'll try to create an example or something when I'll have time.

saranyaeu2987 · 2020-06-19T15:18:43Z

@oscerd

Do you mean something like below ? Please let me know if I am wrong

public class CustomS3Client implements S3Client{
    private static Logger log = LoggerFactory.getLogger(customS3Client.class);
    Region region = Region.US_EAST_1;
    S3Client s3;
    public CustomS3Client() {
        this.s3 = S3Client.builder() .region(region) .credentialsProvider(EnvironmentVariableCredentialsProvider.create()).build();;
    }

    @Override
    public String serviceName() {
        return "S3 Client with Env variable Auth";
    }

    @Override
    public void close() {

    }
    public static void main(String args[]){
        CustomS3Client customs3Client = new CustomS3Client();
        log.info("customs3Client created!");
    }
}

and in kafkaconnector.yaml

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
  name: s3-abcd-connector
  labels:
    strimzi.io/cluster: emd-kafka-cluster
spec:
  class: org.apache.camel.kafkaconnector.CamelSinkConnector
  tasksMax: 1
  config:
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.kafka.connect.storage.StringConverter
    topics: DB.EMD.COMPANY.0
    camel.sink.url: aws2-s3://test1232?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
    camel.component.aws2-s3.configuration.autocloseBody: false
    camel.sink.endpoint.amazonS3Client =#class:org.apache.camel.kafkaconnector.CustomS3Client. <-- class will be added in org.apache.camel.kafkaconnector

oscerd · 2020-06-19T15:25:01Z

yes, something like that

saranyaeu2987 · 2020-06-19T15:31:04Z

@oscerd
I get this error now. If my S3Client class and yaml file right ?

org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: aws2-s3://heb-stg-emd-store-raw-ora.pemd?keyName=s3-connect/20200619/20200619-152832018 due to: useIAMCredentials is set to false, AmazonS3Client or accessKey and secretKey must be specified

Here is the kafkaconnector.yaml file

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
  name: s3-abcde-connector
  labels:
    strimzi.io/cluster: emd-kafka-cluster
spec:
  class: org.apache.camel.kafkaconnector.CamelSinkConnector
  tasksMax: 1
  config:
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.kafka.connect.storage.StringConverter
    topics: DB.EMD.COMPANY.0
    camel.sink.url: aws-s3://testse1232?keyName=s3-connect/${date:now:yyyyMMdd}/${date:now:yyyyMMdd-HHmmssSSS}
    camel.sink.maxPollDuration: 1000
    camel.sink.maxBatchPollSize: 10
    camel.component.aws2-s3.configuration.autocloseBody: false
    camel.sink.endpoint.amazonS3Client:org.apache.camel.kafkaconnector.CustomS3Client

oscerd · 2020-06-19T15:36:19Z

I do have to try. I'll have a look next week, maybe.

saranyaeu2987 · 2020-06-19T21:54:36Z

@oscerd @valdar

Is the above error is because camel-aws2-s3 uses only StaticCredentialsProvider (accesskey and secret) and not WebIdentityTokenFileCredentialsProvider in case of no useIAMCredentials ?

https://github.com/apache/camel/blob/39cbd7f6866cb041d77f6fddc3c7c0721e4d0926/components/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/client/AWS2S3ClientFactory.java#L40

Is this a bug at camel-aws2-s3 because WebIdentityTokenFileCredentialsProvider is the one which looks for credentials in Env variable: AWS_WEB_IDENTITY_TOKEN_FILE?

oscerd · 2020-06-19T21:58:13Z

It's not a bug, not all the credential provider can be supported in the component out of the box. We have to choose and btw it's something else. You need to instantiate the client and from the error it's clear that the client is not in the registry.

oscerd · 2020-06-19T21:59:09Z

Any client instantiated should work. So it's definitely something in the way you are defining the client and configuring it.

saranyaeu2987 · 2020-06-19T22:06:26Z

You need to instantiate the client and from the error it's clear that the client is not in the registry.
Any client instantiated should work. So it's definitely something in the way you are defining the client and configuring it.

--> ok, How to check if that my custom S3 client is in registry or not and what is this registry?

This is how I have added - I have created a separate jar with one class HEBS3Client with following definition

public class HEBS3Client implements S3Client{
    private static Logger log = LoggerFactory.getLogger(HEBS3Client.class);
    Region region = Region.US_EAST_1;
    S3Client s3;

    public static void main(String args[]){
        HEBS3Client hebs3Client = new HEBS3Client();
        hebs3Client.s3 = S3Client.builder() .region(hebs3Client.region) .credentialsProvider(WebIdentityTokenFileCredentialsProvider.create()).build();

    public String serviceName() {
        return "";
    }
    public void close() {
    }

with config
camel.sink.endpoint.amazonS3Client: heb.dsol.fp.HEBS3Client

and added jar in docker container which have other camel files.

Any other config or setting needed to add S3client class in registry ?

saranyaeu2987 mentioned this issue Jun 18, 2020

[Question] Can I have a new serviceAccount in kafkaConnect spec ? strimzi/strimzi-kafka-operator#3179

Closed

oscerd closed this as completed Oct 2, 2020

This was referenced Jun 22, 2021

AWSS2-S3-Kafka connector sink returns 403 with bucket and object not public #1216

Closed

Java bean error when using aws2-s3-source connector #1217

Closed

pthalasta mentioned this issue Jan 6, 2022

UseDefaultCredentials Provider doesn't recognize web identity token file #1263

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Question: Does Camel looks for IAM roles from Env variables #282

Question: Does Camel looks for IAM roles from Env variables #282

saranyaeu2987 commented Jun 17, 2020 •

edited

oscerd commented Jun 18, 2020 •

edited

saranyaeu2987 commented Jun 18, 2020

oscerd commented Jun 18, 2020

valdar commented Jun 18, 2020

saranyaeu2987 commented Jun 18, 2020 •

edited

geota commented Jun 18, 2020

oscerd commented Jun 18, 2020

geota commented Jun 18, 2020

oscerd commented Jun 18, 2020

saranyaeu2987 commented Jun 18, 2020 •

edited

saranyaeu2987 commented Jun 18, 2020

oscerd commented Jun 18, 2020

saranyaeu2987 commented Jun 18, 2020 •

edited

oscerd commented Jun 18, 2020

oscerd commented Jun 18, 2020

saranyaeu2987 commented Jun 18, 2020 •

edited

oscerd commented Jun 18, 2020

oscerd commented Jun 18, 2020

saranyaeu2987 commented Jun 18, 2020

oscerd commented Jun 19, 2020

saranyaeu2987 commented Jun 19, 2020 •

edited

oscerd commented Jun 19, 2020

saranyaeu2987 commented Jun 19, 2020 •

edited

oscerd commented Jun 19, 2020

saranyaeu2987 commented Jun 19, 2020 •

edited

oscerd commented Jun 19, 2020

saranyaeu2987 commented Jun 19, 2020

oscerd commented Jun 19, 2020

oscerd commented Jun 19, 2020

saranyaeu2987 commented Jun 19, 2020 •

edited

Question: Does Camel looks for IAM roles from Env variables #282

Question: Does Camel looks for IAM roles from Env variables #282

Comments

saranyaeu2987 commented Jun 17, 2020 • edited

oscerd commented Jun 18, 2020 • edited

saranyaeu2987 commented Jun 18, 2020

oscerd commented Jun 18, 2020

valdar commented Jun 18, 2020

saranyaeu2987 commented Jun 18, 2020 • edited

geota commented Jun 18, 2020

oscerd commented Jun 18, 2020

geota commented Jun 18, 2020

oscerd commented Jun 18, 2020

saranyaeu2987 commented Jun 18, 2020 • edited

saranyaeu2987 commented Jun 18, 2020

oscerd commented Jun 18, 2020

saranyaeu2987 commented Jun 18, 2020 • edited

oscerd commented Jun 18, 2020

oscerd commented Jun 18, 2020

saranyaeu2987 commented Jun 18, 2020 • edited

oscerd commented Jun 18, 2020

oscerd commented Jun 18, 2020

saranyaeu2987 commented Jun 18, 2020

oscerd commented Jun 19, 2020

saranyaeu2987 commented Jun 19, 2020 • edited

oscerd commented Jun 19, 2020

saranyaeu2987 commented Jun 19, 2020 • edited

oscerd commented Jun 19, 2020

saranyaeu2987 commented Jun 19, 2020 • edited

oscerd commented Jun 19, 2020

saranyaeu2987 commented Jun 19, 2020

oscerd commented Jun 19, 2020

oscerd commented Jun 19, 2020

saranyaeu2987 commented Jun 19, 2020 • edited

saranyaeu2987 commented Jun 17, 2020 •

edited

oscerd commented Jun 18, 2020 •

edited

saranyaeu2987 commented Jun 18, 2020 •

edited

saranyaeu2987 commented Jun 18, 2020 •

edited

saranyaeu2987 commented Jun 18, 2020 •

edited

saranyaeu2987 commented Jun 18, 2020 •

edited

saranyaeu2987 commented Jun 19, 2020 •

edited

saranyaeu2987 commented Jun 19, 2020 •

edited

saranyaeu2987 commented Jun 19, 2020 •

edited

saranyaeu2987 commented Jun 19, 2020 •

edited