Allow authentication by environment IAM credentials

[FaridNeshat-TomTom]

Driver version

v2.0.872

Redshift version

PostgreSQL 8.0.2 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.4.2 20041017 (Red Hat 3.4.2-6.fc3), Redshift 1.0.22169

Client Operating System

Amazon Linux 2 for Python 3.8 Lambda

Python version

Python 3.8

Problem description

I would like the driver to authenticate via the environment IAM credentials(for examples in AWS Lambda environment with IAM Role with sufficient permissions for get_cluster_credentials).

I should be able to execute the following code in an environment with IAM credentials with sufficient permissions:

import redshift_connector
conn = redshift_connector.connect(
    cluster_identifier='examplecluster,
    database='dev',
    user='awsuser',
    iam=True,
    credentials_provider='IAMProvider', # proposed.
 )

1. Expected behaviour: Should work
2. Actual behaviour: Doesn't work.
3. Error message/stack trace: Invalid connection property setting. password must be specified
4. Any other details that can be helpful:

The driver can already authenticate via get_cluster_credentials method:

amazon-redshift-python-driver/redshift_connector/iam_helper.py

Lines 246 to 265 in 77a9c1d

	client = boto3.client(
	"redshift",
	region_name=info.region,
	aws_access_key_id=credentials.get_aws_access_key_id(),
	aws_secret_access_key=credentials.get_aws_secret_key(),
	aws_session_token=credentials.get_session_token(),
	)
	info.host, info.port = client.describe_clusters(ClusterIdentifier=info.cluster_identifier)["Clusters"][0][
	"Endpoint"
	].values()
	cred: dict = client.get_cluster_credentials(
	DbUser=info.db_user,
	DbName=info.db_name,
	DbGroups=info.db_groups,
	ClusterIdentifier=info.cluster_identifier,
	AutoCreate=info.auto_create,
	)
	info.user_name = cred["DbUser"]
	info.password = cred["DbPassword"]

The code just needs the following some small adjustments to achieve this:

• The connection validation logic should allow password not being specified in the case iam == True and credentials_provider == 'IAMProvider'
• an IAMProvider as a credential provider is created, which would return None for access key, secret and session token.

I would be happy to make a pull request to do above, in case someone from the project agrees with the changes.

Our use case

We always use get_cluster_credentials so we don't have to care about rotating secrets, since our code is already running in AWS managed compute environment(Lambda, Fargate, Glue Python Shell), the IAM credentials are already present in the environment. Since this driver already contains the code for get_cluster_credentials it would be shame that we would have to duplicate that code everywhere.

Reproduction code

import redshift_connector
conn = redshift_connector.connect(
    cluster_identifier='examplecluster,
    database='dev',
    user='awsuser',
    iam=True,
    provider='iam'
 )

[Brooke-white]

Hey @FaridNeshat-TomTom ,

Thank you for reaching out regarding IAM authentication. My team and I are currently working to add support for this functionality. We plan to support IAM authentication via AWS profile or directly passing IAM credentials to the redshift_connector.connect(...) method.

Please see here for information on how our JDBC and ODBC drivers support IAM authentication. We plan to do something similar for redshift_connector.

This would allow you to authenticate using environment variable like:

import os
import redshift_connector

conn = redshift_connector.connect(
  ...
  access_key_id=os.environ.get('my_access_key_id'),
  secret_access_key=os.environ.get('my_secret_access_key'),
  session_token=os.environ.get('my_session_token') # can be included optionally
)

Would this satisfy your use case?

[FaridNeshat-TomTom]

Yes, that'd be great. I assume if they are not provided, boto3 default logic for acquiring the credentials will kick in(from environment, or ~/.aws/credentials or IAM roles: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#guide-credentials), which would make it as seamless as using boto3 or AWS CLI.

[Brooke-white]

Hey @FaridNeshat-TomTom ,

Thanks for your patience, we've added support for this in our latest release, v2.0.873.

Yes, if credentials are not provided the default boto3 logic kicks in. A region must be provided either in AWS config file (which will be picked up by boto3) or be passed as an argument to redshift_connector.connect(...)

[FaridNeshat-TomTom]

Hi @Brooke-white

Thanks for taking care of this. I guess this can be closed now.

One thing caught my eye looking at the documentation, is that the usage with the IAM credentials is a little bit awkward because user and password has to be passed but their value is ignored and then one has to care db_user has to be passed as well.

[Brooke-white]

Thank you for the feedback -- I'l pass it along to my team and see what we can do there :)

[rectalogic]

Yes, if credentials are not provided the default boto3 logic kicks in. A region must be provided either in AWS config file (which will be picked up by boto3) or be passed as an argument to redshift_connector.connect(...)

@Brooke-white this doesn't seem to be the case with 2.0.900 - I have to explicitly pass access_key_id, secret_access_key and session_token - boto doesn't pick them up from the environment:

>>> conn = redshift_connector.connect(
...     iam=True,
...     database='xxx',
...     db_user='xxx',
...     cluster_identifier='myredshift',
...     region="us-west-2",
...  )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/venv/lib/python3.8/site-packages/redshift_connector/__init__.py", line 327, in connect
    IamHelper.set_iam_properties(info)
  File "/tmp/venv/lib/python3.8/site-packages/redshift_connector/iam_helper.py", line 123, in set_iam_properties
    raise InterfaceError(
redshift_connector.error.InterfaceError: Invalid connection property setting. Credentials provider, AWS credentials, Redshift auth profile or AWS profile must be provided when IAM is enabled

[Brooke-white]

@rectalogic -- thank you for the heads up, this will be resolved in our next release.