Using multiple AWS profiles

[whiskey]

I've configured multiple AWS credential profiles on my system and usually select the right one via named profiles or environment variables such das export AWS_DEFAULT_PROFILE=foo.

However, my created User Model will be fetched from my default configuration.
Model definition:

class AWSUser(Model):
    class Meta:
        table_name = 'Users'
        region = 'eu-central-1'

    id = UnicodeAttribute(hash_key=True)
    display_name = UnicodeAttribute(attr_name='displayName')
    # ...

    def __str__(self):
        return self.display_name

How can I select the correct AWS profile?

[ma-rp]

Did you find a way to do this? Im facing this issue right now too.

[brandond]

pynamodb uses botocore under the hood, which should honor environment variables like AWS_DEFAULT_PROFILE. Have you tried something like:

AWS_DEFAULT_PROFILE=alt-profile python my-pynamodb-code.py

[alastairmccormack]

It would be better if a pre-configured boto session could be passed somewhere rather than relying on the user environment.

[brandond]

@use-sparingly yeah that would be nice. The model Meta has a connection_cls property, but this (counterintuitively) is used to look up the requests session class, NOT the botocore session. It might be nice to expose this, but as far as I can tell you'd have to plumb it all the way through model.MetaModel, connection.base.MetaTable, connection.table.TableConnection, and connection.base.Connection.

If you have an immediate need and don't want to set the environment variable, you could always monkeypatch connection.base.Connection.session (globally) or set self._get_connection().connection._session within your model's __init__ (for a specific model).

[bedge]

I ran into this too. This is contorted around some child classes, but here's the gist of it:

    SESSION = Session(profile='CrossDynamodbAccess')

    MyClass._connection = MyClass._get_connection()
    MyClass._connection.connection._client = SESSION.create_client(
        'dynamodb', region_name='us-west-2')

A cleaner option would be appreciated.

[kevgliss]

More control would also be preferable in the case where you have to assume role before using pynamodb, as it stands @bedge solution works okay for the time being.

[alexanderfanz]

@brandond can you explain your comment a little more and/or write an example code? I'm lost.

If you have an immediate need and don't want to set the environment variable, you could always monkeypatch connection.base.Connection.session (globally) or set self._get_connection().connection._session within your model's __init__ (for a specific model).

[brandond]

@alexanderfanz since _session is now thread-local, I don't think it's safe to set within the model any longer. Monkeypatching is probably your best option. Something like this - either in the module that holds your model definitions as shown in this example, or somewhere else that's called once during app startup - should do the trick.

from pynamodb.models import Model
from pynamodb.attributes import UnicodeAttribute, NumberAttribute


class MyModel(Model):
    class Meta:
        table_name = 'mytable'

    group = UnicodeAttribute(hash_key=True)
    id = NumberAttribute(range_key=True)
    name = UnicodeAttribute()


def monkeypatch_connection(profile=None):
    from botocore.session import Session
    from pynamodb.connection.base import Connection

    @property
    def session(self):
        if getattr(self._local, 'session', None) is None:
            self._local.session = Session(profile=profile)
        return self._local.session

    Connection.session = session


monkeypatch_connection(profile='my-aws-profile')

[mikegrima]

In general, we would love a good way of performing a role assumption and caching credentials. I would be happy to submit a PR for this if I knew where to look.

Profiles aren't great for programmatic and dynamic credential needs.

[twkiiim]

Any updates for this issue?

[ikonst]

@garrettheel, any thoughts on that?

I would rather have an API where models aren't dealing with auth or host. For that we'd need a public API for setting it globally. A context manager which overrides things like that in thread local-storage might be a good idea too. What do you think?

[garrettheel]

I'm not sure that I fully understand the use case here, so I'd want to gather a little more information. We intentionally expose as little of the boto/botocore interface as possible so that it's possible to optimize/change this under the hood without breaking users.

Is is that we want to configure a specific AWS_DEFAULT_PROFILE in code, rather than as an environment variable? Is there a desire to apply different profiles to different model classes/instances?

[bedge]

"Is there a desire to apply different profiles to different model classes/instances?"

• yes, exactly.
  See earlier post: Using multiple AWS profiles #204 (comment)

[garrettheel]

Gotcha. We already allow configuring region, host, etc. in Connection (and via the Meta class on models), would exposing aws_profile there be helpful?

[alastairmccormack]

From my point of view, the most flexible approach is to be able to pass a whole Boto session object.

It would also be beneficial to be able to connect tables to multiple DynamoDB instances from different accounts. Therefore, it would be great to be able to bind the Boto session and tablename at runtime. Perhaps on the method (E.g. query), or perhaps by instantiating the table with something like my_table = myTable(tablename, session).

[ikonst]

PynamoDB models are also effectively "tables", i.e. they deal with the details of how tables and their indices are defined. Perhaps instead of trying to create a new "table" class (do we duplicate the get/query methods to be on this new class? should model subclass table?) it should be encouraged to define new model subclasses in runtime with their own metas?

[atyshka]

+1 on this, want to use dynamic credentials via IAM roles assigned to cognito users. Without this feature I'm going to have to use the regular boto3 api

[jl-DaDar]

really need this for code that interact with dynamodb tables in multiple accounts

[rnag]

not sure if it helps but the solution in code is actually really simple from what I could tell. The only issue is that it doesn't handle multiple profiles well (which was also mentioned). I just add this line before I'm retrieving any items or saving anything to table using the PynamoDB model.

os.environ['AWS_PROFILE'] = 'my-profile'

For multiple profiles though, maybe there's actually a simple workaround. Like creating a context manager, which sets the AWS_PROFILE environment variable (temporarily) and then calls the _get_connection class method on a model which sets up in the initial boto connection. Not entirely sure if that'll work, but just a thought I had to resolve it for now.

[weegolo]

One use case for being able to set the session is when you're using the AWS Multi-Account Strategy. This leads to the pattern of having one account that contains your (possibly serverless) application, and multiple single-tenancy client accounts (for security, data sovereignty issues). Combine that with assumed roles with minimum IAM privileges and separate dev, test and prod accounts, and the botocore session you wish to use can change from one lambda invocation to the next.

In a case like this, setting the session becomes more complex that simply exporting environment variables or setting profile names, and so it makes more sense to abstract the session setting to a different module so that you don't need to deal with it every time you invoke a Model.

This should also save you (the developers) a lot of effort - why reimplement all the code to use profiles, session tokens, credentials, etc when it's already implemented in botocore.session