fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey properties must be present

[Aayushpatel007]

Even after having IAM role access to S3 and specifying AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY. I am having this error in server logs.
As it uses hadoop-aws to read files, how do we authenticate and pass credentials in server?

Before running the below command, where should I specify fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey properties?
bin/delta-sharing-server -- --config conf/delta-sharing-server.yaml

[gauravbrills]

@Aayushpatel007 how did you solv ethis .. I am getting this when using s3:// .. but a different error when I use s3a:/// . In my case also IAm role access is already there not sure why this error .

[skaplan81]

getting the same error, did you solve your problem? Where should I place these keys, in the .yaml file or .share file?

[zsxwing]

getting the same error, did you solve your problem? Where should I place these keys, in the .yaml file or .share file?

@skaplan81 could you try to use the s3a:// path?

[gauravbrills]

I think it worked for me finally remember adding hadoop core-site.xml with aws config to get it running on a container .Just on docker the path was diff /opt/docker/conf which @zsxwing helped me point at

[skaplan81]

getting the same error, did you solve your problem? Where should I place these keys, in the .yaml file or .share file?

@skaplan81 could you try to use the s3a:// path?

I am copying the address from S3 page and it starts 's3://', but everywhere I see links starting with s3a, doesn't understand

[skaplan81]

I have yaml file as below :

version: 1
shares:

• name: "share1"
  schemas:
  • name: "schema1"
    tables:
    • name: "table1"
      location: "s3://sharingcentertrial1/deneme"
      host: "localhost"
      port: 8080
      endpoint: "/delta-sharing"
      preSignedUrlTimeoutSeconds: 3600
      deltaTableCacheSize: 10
      stalenessAcceptable: false
      evaluatePredicateHints: false

share file as this :

{
"shareCredentialsVersion": 1,
"endpoint": "http://localhost:8080/delta-sharing",
"bearerToken": ""
}

I started the server with

bin/delta-sharing-server -- --config conf/delta-sharing-server.yaml

And when I try to execute the Python code

data = delta_sharing.load_as_pandas(table_url, limit=10)

I got this error about awsAccessKeyId etc

I created IAM user with a role and privacy settings. But I don't know where should I place these lines?

export AWS_ACCESS_KEY_ID=my.aws.key
export AWS_SECRET_ACCESS_KEY=my.secret.key

thanks for all in advance

[zsxwing]

Can you run the following commands?

export AWS_ACCESS_KEY_ID=my.aws.key
export AWS_SECRET_ACCESS_KEY=my.secret.key
bin/delta-sharing-server -- --config conf/delta-sharing-server.yaml

[zsxwing]

I am copying the address from S3 page and it starts 's3://', but everywhere I see links starting with s3a, doesn't understand

s3:// paths usually are mapped to EMR's own S3 file system implementation (Delta Sharing cannot use it since it's not open sourced), and s3a:// paths are mapped to open source Hadoop S3AFileSystem which is the one used by Delta Sharing.

[skaplan81]

Can you run the following commands?

export AWS_ACCESS_KEY_ID=my.aws.key
export AWS_SECRET_ACCESS_KEY=my.secret.key
bin/delta-sharing-server -- --config conf/delta-sharing-server.yaml

tried this but got the same error

[zsxwing]

@skaplan81 Is the exception thrown from S3FileSystem or S3*A*FileSystem?

[skaplan81]

@skaplan81 Is the exception thrown from S3FileSystem or S3*A*FileSystem?

I am using S3 URI in the yaml file, is it what are you asking?

as you mentioned above, it is not possible to reach via the s3 link

[skaplan81]

I am trying to follow the steps mentioned in this link, but I think skipping some steps

https://github.com/delta-io/delta-sharing#s3

[zsxwing]

I am using S3 URI in the yaml file, is it what are you asking?

Yep. Please use s3a://. Will highlight this in the doc as this is not obvious for people not familiar with multiple S3 file systems story.

[skaplan81]

I am using S3 URI in the yaml file, is it what are you asking?

Yep. Please use s3a://. Will highlight this in the doc as this is not obvious for people not familiar with multiple S3 file systems story.

yes it worked with s3a:... link, there are other errors, I will try to fix them, thanks a lot

[zsxwing]

FYI I submitted #179 to improve the doc