EC2 Snapshots gets lots of stuff, takes very long

[danielsaporo]

I'm trying Cartography on an AWS account that is practically empty (just a few users), and noticed that it takes a while to run. What really stands out is that it takes over 9 minutes to get EBS snapshot data. And this account does not have (and never had) any EBS volumes in any regions.

Yet, when I debug get_snapshots() in snapshots.py, I see that client.get_paginator('describe_snapshots') returns many thousands of records despite the fact that there is no snapshot data in any region visible in the AWS Console.

So, what is describe_snapshots returning exactly, and do we need all that?

[ryan-lane]

Ah. It looks like that function returns all available snapshots. I think what we probably want here is to filter to snapshots owned by the current account. @achantavy @ramonpetgrave64, thoughts?

[danielsaporo]

Ah. It looks like that function returns all available snapshots. I think what we probably want here is to filter to snapshots owned by the current account.

Why does describe_snapshots return a lot of snapshots that don't belong to the current account? From the description I see:

Describes the specified EBS snapshots available to you or all of the EBS snapshots available to you.

The snapshots available to you include public snapshots, private snapshots that you own, and private snapshots owned by other Amazon Web Services accounts for which you have explicit create volume permissions.

However I don't really understand this. I'm using a new AWS account which hasn't been granted access to any other account. I just hope AWS isn't returning a lot of stuff from other accounts that it shouldn't.

[ryan-lane]

@danielsaporo There's public EBS snapshots for public datasets, and there's a lot of them. This is similar to public AMIs. See, for example, how I changed that to only pull AMIs from owned accounts, but then introspected the graph to find other images that should be fetched: https://github.com/lyft/cartography/blob/master/cartography/intel/aws/ec2/images.py#L17-L59

[ramonpetgrave64]

Fixed in #799

[danielsaporo]

Thanks for fixing this. What is the process to upgrade to a newer version of Cartography that includes this fix? Do I need to wait for some kind of release to happen first?