Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 25 additions & 6 deletions docs/apache-airflow-providers-amazon/connections/aws.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,21 @@
Amazon Web Services Connection
==============================

The Amazon Web Services connection type enables the :ref:`AWS Integrations
<AWS>`.
The Amazon Web Services connection type enables the :ref:`AWS Integrations <AWS>`.

.. important:: The Amazon Web Services Connection can be tested in the UI/API or by calling
:meth:`~airflow.providers.amazon.aws.hooks.base_aws.AwsGenericHook.test_connection`,
it is **important** to correctly interpret the result of this test.
During this test components of Amazon Provider invoke AWS Security Token Service API
`GetCallerIdentity <https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html>`__.
This service can **only** check if your credentials are valid.
Unfortunately it is not possible to validate if credentials have access to specific AWS service or not.

If you use the Amazon Provider to communicate with AWS API compatible services (MinIO, LocalStack, etc.)
test connection failure **doesn't mean** that your connection has wrong credentials.
Many compatible services provide only a limited number of AWS API services,
and most of them do not implement the AWS STS GetCallerIdentity method.


Authenticating to AWS
---------------------
Expand All @@ -32,19 +45,26 @@ Alternatively, one can pass credentials in as a Connection initialisation parame
To use IAM instance profile, create an "empty" connection (i.e. one with no AWS Access Key ID or AWS Secret Access Key
specified, or ``aws://``).


Default Connection IDs
-----------------------

The default connection ID is ``aws_default``. If the environment/machine where you are running Airflow has the
file credentials in ``/home/.aws/``, and the default connection has user and pass fields empty, it will take
automatically the credentials from there.

.. note:: Previously, the ``aws_default`` connection had the "extras" field set to ``{"region_name": "us-east-1"}``
.. important:: Previously, the ``aws_default`` connection had the "extras" field set to ``{"region_name": "us-east-1"}``
on install. This means that by default the ``aws_default`` connection used the ``us-east-1`` region.
This is no longer the case and the region needs to be set manually, either in the connection screens in Airflow,
or via the ``AWS_DEFAULT_REGION`` environment variable.

.. caution:: If you do not set ``[database] load_default_connections`` to ``True``
most probably you do not have ``aws_default``. For historical reasons, the Amazon Provider
components (Hooks, Operators, Sensors, etc.) fallback to the default boto3 credentials strategy
in case of a missing Connection ID. This behaviour is deprecated and will be removed in a future releases.

If you need to use the default boto3 credential strategy (credentials in environment variables, IAM Profile, etc.)
please provide ``None`` instead of a connection ID.

.. _howto/connection:aws:configuring-the-connection:

Configuring the Connection
Expand Down Expand Up @@ -144,9 +164,8 @@ Snippet to create Connection and convert to URI
print(f"{env_key}={conn_uri}")
# AIRFLOW_CONN_SAMPLE_AWS_CONNECTION=aws://AKIAIOSFODNN7EXAMPLE:wJalrXUtnFEMI%2FK7MDENG%2FbPxRfiCYEXAMPLEKEY@/?region_name=eu-central-1

# Test connection
os.environ[env_key] = conn_uri
print(conn.test_connection())
print(conn.test_connection()) # Validate connection credentials.


.. warning:: When using the Airflow CLI, a ``@`` may need to be added when:
Expand Down