Pickle and authentication information

[mrocklin]

So I create an S3FileSystem locally with my authentication. I then use this within some object that I pickle up and send to another machine. Some questions:

1. Does the remote object have the same authentication? Or perhaps a token?
2. Did I ever send my credentials in the clear?

[martindurant]

Yes, key and secret are saved as attributes and pickled. I believe the same auth should work from any machine. If you didn't explicitly supply credentials, but boto3 found them in environment variables or config files, these are not stored as attributes (in fact, I don't think there is a way to find out what they are)

You can always create a token for temporary access with get_delegated_s3pars().

[mrocklin]

I guess I'm curious if we should endeavor to use get_delegated_s3pars by default in order to avoid sending credentials around.

[martindurant]

Personally, I expect that people will distribute their credential conf and call S3FileSystem without parameters. In some cases, passing the original creds around is perfectly fine, and saves the time for the creation of a token and worrying about its expiry.

[mrocklin]

The former case is an argument to not pickle the parameters up with
S3FileSystem. We do now though. I personally find this convenient when
spinning up clusters on EC2. I don't normally distribute my credentials to
all machines for ephemeral clusters. I don't have enough experience to
know what is done in the wild.

On Thu, Apr 28, 2016 at 3:50 PM, Martin Durant notifications@github.com
wrote:

Personally, I expect that people will distribute their credential conf and
call S3FileSystem without parameters. In some cases, passing the original
creds around is perfectly fine, and saves the time for the creation of a
token and worrying about its expiry.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#35 (comment)

[martindurant]

Fixed in #51 ?