Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding support for AWS creds via instance metadata with session token #42

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

seanhall
Copy link

@seanhall seanhall commented Mar 9, 2016

It looks like Fog is able to fetch credentials via IAM/instance profile. Adding a case to new_fog_s3_from() to check for use_iam_profile => true to prep for Snowplow EmrEtlRunner IAM/EC2 instance metadata support.

@seanhall
Copy link
Author

seanhall commented Mar 9, 2016

I can't tell if build is failing because Coveralls is down or if there's a problem with my change. Either way, thanks!

@alexanderdean
Copy link
Contributor

Thanks! Can you sign our CLA please: https://github.com/snowplow/snowplow/wiki/CLA

@seanhall
Copy link
Author

seanhall commented Mar 9, 2016

Sent, thanks again.

@alexanderdean
Copy link
Contributor

Confirmed! Thanks so much

@seanhall
Copy link
Author

Any luck in rerunning tests? We've had only minor success compiling from our fork, running into issues with Fog version compatibility. Fog 1.24 returns a 'net-ssh missing' error, 1.36 an AWS request header region mismatch issue (config.yml passes us-west-2, Fog is defaulting to us-east-1).

@seanhall
Copy link
Author

So interesting collision...it seems the Snowplow Storage Loader IAM handler and this PR are redundant. The Storage Loader handler actually rewrites the creds before Sluice receives them, so sluice is getting the actual instance profile creds instead of 'iam'. EMR-ETL-Runner is working great, but the double translation is failing Storage Loader. I still think this is best handled by Sluice, which leverages fog-aws's credentials fetcher naturally, but am not sure if there are other downstream dependencies or user applications outside of Snowplow.

@alexanderdean
Copy link
Contributor

Right - this problem should go away when we do: snowplow/snowplow#2548

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants