The Amazon Web Services connection type enables the :ref:`AWS Integrations <AWS>`.
Authentication may be performed using any of the boto3 options. Alternatively, one can pass credentials in as a Connection initialisation parameter.
To use IAM instance profile, create an "empty" connection (i.e. one with no Login or Password specified, or
aws://
).
The default connection ID is aws_default
.
Note
Previously, the aws_default
connection had the "extras" field set to {"region_name": "us-east-1"}
on install. This means that by default the aws_default
connection used the us-east-1
region. This is no longer the case and the region needs to be set manually, either in the connection screens in Airflow, or via the AWS_DEFAULT_REGION
environment variable.
- Login (optional)
- Specify the AWS access key ID.
- Password (optional)
- Specify the AWS secret access key.
- Extra (optional)
Specify the extra parameters (as json dictionary) that can be used in AWS connection. The following parameters are all optional:
aws_session_token
: AWS session token used for the initial connection if you use external credentials. You are responsible for renewing these.aws_account_id
: AWS account ID for the connectionaws_iam_role
: AWS IAM role for the connectionexternal_id
: AWS external ID for the connectionhost
: Endpoint URL for the connectionregion_name
: AWS region for the connectionrole_arn
: AWS role ARN for the connectionaws_session_token
: AWS session token if you use external credentials. You are responsible for renewing these.host
: Endpoint URL for the connection.region_name
: AWS region for the connection.external_id
: AWS external ID for the connection (deprecated, rather useassume_role_kwargs
).config_kwargs
: Additionalkwargs
used to construct abotocore.config.Config
passed to boto3.client and boto3.resource.session_kwargs
: Additionalkwargs
passed to boto3.session.Session.
If you are configuing the connection via a URI, ensure that all components of the URI are URL-encoded.
- Using instance profile:
export AIRFLOW_CONN_AWS_DEFAULT=aws://
This will use boto's default credential look-up chain (the profile named "default" from the ~/.boto/ config files, and instance profile when running inside AWS)
- With a AWS IAM key pair:
export AIRFLOW_CONN_AWS_DEFAULT=aws://AKIAIOSFODNN7EXAMPLE:wJalrXUtnFEMI%2FK7MDENG%2FbPxRfiCYEXAMPLEKEY@
Note here, that the secret access key has been URL-encoded (changing
/
to%2F
), and also the trailing@
(without which, it is treated as<host>:<port>
and will not work)
- Using ~/.aws/credentials and ~/.aws/config file, with a profile.
This assumes all other Connection fields eg Login are empty.
{
"session_kwargs": {
"profile_name": "my_profile"
}
}
Specifying a role_arn to assume and a region_name
{ "aws_iam_role": "aws_iam_role_name", "region_name": "ap-southeast-2" }