efs_facts needs a way to gather a subset of facts due to API limits

[ryansb]

ISSUE TYPE

• Feature Idea

COMPONENT NAME

efs_facts

ANSIBLE VERSION

Ansible 2.5

CONFIGURATION

N/A

OS / ENVIRONMENT

N/A - only affects AWS module

SUMMARY

The AWS EFS API is heavily rate limited, leading to frequent throttle/retry cycles. Previously, these caused the efs_facts module to error out (per #36040). Now the module does take a long time to execute over many EFS shares, but does complete. It would be nice to have a set of arguments to reduce the number of API calls by excluding some data, such as security groups, that is less frequently used to make the normal usage of efs_facts faster.

Per comment on the more acute fix for the API limits: #36266 (comment)

Related: #36040

cc/ @ julienvey

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

[ansibot]

Files identified in the description:

• lib/ansible/modules/cloud/amazon/efs_facts.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibot]

cc @ryansydnor @willthames
click here for bot help

[julienvey]

Looking more closely at the code I now understand why it can sometimes take a lot of time.

The code looks like this

file_systems_info = connection.get_file_systems(fs_id, name)

if tags:
    file_systems_info = [item for item in file_systems_info if has_tags(item['tags'], tags)]

if targets:
    targets = [(item, prefix_to_attr(item)) for item in targets]
    file_systems_info = [item for item in file_systems_info if has_targets(item['mount_targets'], targets)]

What happen is:

• List all EFS in the region and collect all information (including mount targets and security groups)
• Then only, do the filtering if tags or targets is set

In my case, I wanted the facts about only 3 EFS (filtered by tag), but we have more than 100 in the same region, this is why it tooks so long to complete.

I'll try to propose a PR to improve this behaviour and limit the number of API calls

[ryansb]

Thanks - I've left a review on the PR

[ryansb]

Closed in 0f612d1