Databricks permission issue to access S3

[sunank200]

Describe the bug
Currently, the CI is failing for Databricks. More details can be found at: https://astronomer.astronomer.run/dhb539ck/log?dag_id=example_load_file&task_id=load_file__23&execution_date=2023-02-08T00%3A00%3A00%2B00%3A00&map_index=-1

[dimberman]

Hey @sunank200 what is this airflow instance? this doesn't appear to be a part of our CI/CD? Do we need to upload creds separately to this perhaps?

[sunank200]

@dimberman This URL is of astro-cloud as part of running example DAGs on astro-cloud. On astro cloud it's @rajaths010494 's credential. Can you confirm @rajaths010494 ?

But this happens on CI/CD as well. For example https://github.com/astronomer/astro-sdk/actions/runs/4132642982/jobs/7141598599 This seems flaky.

[rajaths010494]

Yeah, the credentials are mine used in Astro cloud.

[sunank200]

@dimberman will be giving KT on databricks this week and cover this as well

[dimberman]

When I run this DAG locally with personal creds it runs fine, but fails in CI #1585. My best guess is that the credentials are expired for S3. I'm gonna let this run one more time and if it fails I'm going to cycle the creds.

[tatiana]

Do we have any alternatives to using personal credentials to run Databricks on the CI?
@dimberman did we manage to advance on this?

[dimberman]

@tatiana not yet, but we can discuss in our 1:1

[phanikumv]

any update on this?

[phanikumv]

@tatiana @dimberman have we explored any alternatives on the solution? For example using IAM passthrough this may be an option

[tatiana]

@phanikumv I'd be happy with that! The sooner we can close this ticket and #1585, the better.
@utkarsharma2 @dimberman ATM, both of you are assigned to this ticket, do you still have bandwidth to work on it?

[pankajkoti]

I looked up for this here https://kb.databricks.com/en_US/security/forbidden-access-to-s3-data

They are suggesting to use IAM roles instead of IAM keys. We might want to try this.