Add boto3 and boto3_wait modules

[ericcitaire]

ISSUE TYPE

• New Module Pull Request

COMPONENT NAME

boto3 and boto3_wait

ANSIBLE VERSION

ansible 2.0.0.2
  config file = 
  configured module search path = Default w/o overrides

SUMMARY

These two modules make AWS API calls using boto3. The general idea is to avoid using AWS CLI when Ansible does not provide a dedicated Ansible module, especiallly for new services (ie. Lambda, Kinesis, etc.).

EXAMPLE

- name: Describe running EMR Clusters
  boto3:
    name: emr
    region: us-east-1
    operation: list_clusters
    parameters:
      ClusterStates:
        - STARTING
        - BOOTSTRAPPING
        - RUNNING
        - WAITING
  changed_when: False
  register: emr

- set_fact:
    emr_cluster_exists: "{{ emr.response.Clusters | selectattr('Name', 'equalto', 'my-jobflow') | list | length > 0 }}"

- set_fact:
    emr_cluster_id: "{{ emr.response.Clusters | selectattr('Name', 'equalto', 'my-jobflow') | map(attribute='Id') | first }}"
  when: emr_cluster_exists

# if 'my-jobflow' cluster does not exist, create it.
- name: Create EMR Cluster
  boto3:
    name: emr
    region: us-east-1
    operation: run_job_flow
    parameters:
      Name: my-jobflow
      Applications:
        - Name: Hive
      Instances:
        KeepJobFlowAliveWhenNoSteps: True
        Ec2KeyName: my-key
        Ec2SubnetId: subnet-xxxxxxx
        EmrManagedSlaveSecurityGroup: sg-xxxxxxx
        EmrManagedMasterSecurityGroup: sg-xxxxxxx
        InstanceGroups:
          - InstanceCount: 1
            InstanceRole: MASTER
            InstanceType: m1.medium
            Market: ON_DEMAND
          - InstanceCount: 1
            InstanceRole: CORE
            InstanceType: m1.medium
            Name: Core Instance Group
            Market: ON_DEMAND
      JobFlowRole: EMR_EC2_DefaultRole
      ServiceRole: EMR_DefaultRole
      ReleaseLabel: emr-4.6.0
      VisibleToAllUsers: True
  when: not emr_cluster_exists
  register: emr_create

- set_fact:
    emr_cluster_id: "{{ emr_create.response.JobFlowId }}"
    emr_cluster_exists: True
  when: not emr_cluster_exists

- name: Wait for EMR Cluster to be running
  boto3_wait:
    name: emr
    region: us-east-1
    until: cluster_running
    parameters:
      ClusterId: "{{ emr_cluster_id }}"
    delay: 30
    max_attempts: 50

[ericcitaire]

Migrated from ansible/ansible-modules-extras#2205

[ansibot]

@mmochan @michaeljs1990 @wimnat @willthames @jarv @steynovich @ryansydnor @simplesteph @Java1Guy @rmorlok @pwnall @naslanidis @pjodouin @RickMendes @amir343 @linuxdynasty @timmahoney @tedder @jsdalton @jmenga @tastychutney @scottanderson42 @mjschultz @bpennypacker @brandond @joelthompson @alachaum @TomBamford @ @j-carl @fiunchinho @Etherdaemon @bekelchik @minichate @MichaelBaydoun @loia @akazakov @Zeekin @silviud @whiter

As a maintainer of a module in the same namespace this new module has been submitted to, your vote counts for shipits. Please review this module and add shipit if you would like to see it merged.

click here for bot help

[willthames]

There seems to be a lot missing here compared to standard modules - you're not making use of aws_argument_spec or boto3_conn, or indeed the aws documentation fragment for example.

The reason we have those is so that all of our AWS modules work correctly in unusual environments without additional further effort on the part of module authors (particularly profile and security token handling)

[ericcitaire]

@willthames Thank you for your feedback. I updated the code accordingly.

Cheers,
Eric.

[s-hertel]

Hey @ericcitaire. It's clear you've spent time and energy on this, but I don’t think it will be able to be merged. There’s a lot of error handling that needs to happen and Ansible is not a programming language. I imagine this would be impossible or a nightmare to support.

[ericcitaire]

Hi @s-hertel. Thank you for your comment.

I don't see how error handling would be worst with boto3 and boto3_wait than hacking with shell or command with an extensive use of AWS CLI and jq to parse and transform JSON.

Most of AWS functionality is not covered by official Ansible AWS modules and will never be (but I could be wrong). Developing a module for every AWS API call is virtually impossible.

I think that making use of the AWS API via boto3 is an elegant alternative to plain AWS CLI. Boto3 is well documented and has pretty good support.

Just try to create an EMR cluster with command or shell modules, and you'll see the nightmare. And of course this cluster creation must be idempotent... ;)

[s-hertel]

@ericcitaire Thanks for your thoughts. I can see why this is helpful and would be a more manageable alternative to using shell/command.

That being said, getting an error like

fatal: [localhost]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_secret_key": null,
            "delay": 30,
            "ec2_url": null,
            "max_attempts": 50,
            "name": "emr",
            "parameters": {
                "ClusterId": "j-3K087TW22MDEA"
            },
            "profile": null,
            "region": "us-east-1",
            "security_token": null,
            "service": "emr",
            "until": "cluster_running",
            "validate_certs": true
        }
    },
    "msg": "Waiter ClusterRunning failed: Waiter encountered a terminal failure state"
}
	to retry, use: --limit @/Users/shertel/Workspace/ansible/my_playbooks/boto3-module/test.retry

PLAY RECAP **************************************************************************************************************
localhost                  : ok=5    changed=1    unreachable=0    failed=1

doesn't provide anything helpful and it means I would end up debugging/trying to tell what is an issue with boto3 and what is user error without a lot to go on.

What is your process for determining everything required for a task to run?

This also requires the user to add their own logic. Any time they run the boto3 module changed will be True. What would your process be for determining if the task should run? If the user wants a resource to be present they can just check if it exists yet, but what if they are updating an attribute (or many)?

How would you validate parameters for this? I am able to literally set the subnet id of your example above as 'subnet-xxxxxxx', and the task completes successfully (although there is no matching subnet).

[ghost]

@s-hertel I am faced with such errors on a daily basis from AWS boto-based modules and have no problem finding the issues, most being caused by updated AWS services with no equivalent support in boto.

This, in my opinion, is sufficient reason to justify the need for a "generic" API module which can bridge the gap between a broken module and its re-write (in the case it was written in boto) or provide functionality to new services such as ELBv2 long before a module is developed.

For example, if you want to use Target Groups with the autoscaling module (ec2_asg), forget it!

I've written my own API style Ansible module (https://github.com/pjodouin/ansible-boto3) such as this one and have been using it in many production playbooks, directly or in roles and it works as expected. With Ansible's conditional statements, it's not hard to create roles that are idempotent and handle CRUD properly.

So, I definitely support the inclusion of an API-style module such as this one from @ericcitaire.

Despite the downsides raised by @s-hertel, I still think such a module would be a useful addition to the Ansible toolkit. The speed at which AWS releases new services and updates existing ones, it's impossible to have custom modules that give complete coverage. (For example, what is the timeline for updating all boto-based modules to boto3?)

[s-hertel]

@pjodouin Good points. Random side note- you should now be able to use Target Groups with ec2_asg since it has been ported to Boto3. :)

[ghost]

@s-hertel That's music to my ears! Thanks!!

Understanding the need for standardizing module behaviour, might it be possible to 'classify' this module under say Utilities? Currently there's Helper and Logic subcategories. Could you not add API and include the boto3 modules there? This would allow you to maintain your standards for custom cloud modules while providing low level API access via a Utilities module.

[simplesteph]

@s-hertel this module would definitely help bridging so many gaps. I think ansible should focus on creating high value modules for AWS, those who can return important information and have good error handling. But many AWS API calls are so straightforward they do not need a module for it. This would bridge the gap. People who know boto3 will know the errors encountered and just won't be afraid of them

[tedder]

This module will get (over) used like lineinfile and command. Not enthused by that.

[ryansb]

The problem with a boto3 module vs. custom modules is that Ansible isn't a programming language, and most operations are going to take multiple branches/failure handling paths/awaits to get the work done. In the case of AWS CLI + command: that's handled in awscli and then parsed with --query or jsonpath:. If we had a straight boto3 call module, you'd have to handle all that in a play rather than being able to use either custom Python or awscli.

[mixja]

This doesn't appear to have support for handling paginated responses, which is a big show stopper

[ericcitaire]

@mixja : You can handle pagination. For example, to list S3 bucket objects with boto3, you can use Marker and MaxKeys of list_objects operation, just as you would with s3 module in list mode.

[mixja]

@ericcitaire - you can, but what if you want a number of results that requires pagination on the boto3 side? A problem if you attempted to solve this is that boto3 is not consistent on pagination between different services.

I agree with @ryansb that using the AWS CLI is actually not that bad of an option, the JMESPath queries are very powerful (generally eliminates the requirement for jq), JSON is a first class citizen from an output perspective and all of these edge cases are dealt with for you.

I don't see any net gain of using the AWS CLI vs this module, rather the AWS CLI is more functional, well maintained and well tested.

[ansibot]

@ericcitaire this PR contains more than one new module.

Please submit only one new module per pullrequest. For further explanation, please read grouped module documentation

click here for bot help

[ericcitaire]

These 2 modules are closely related. It doesn't make sense to have them in separate PRs.

[ericcitaire]

@mattclay Can you please comment on the bot broken tag removal please? Do I really need to split this PR in 2?

This PR was submitted in May 2016, then was migrated from ansible-modules-extras in February (4 participants were left behind in the process). Now it needs to be split in 2... What's next?
Contributing new modules to Ansible seems almost impossible!

[ryansb]

Per earlier discussion, the official stance here is that AWSCLI + json_path is recommended, and handles more cases (like pagination and retries) than a boto API wrapper can.

[mattclay]

@ericcitaire The use of that tag is for cases where the bot is malfunctioning and needs to be stopped to avoid causing problems, like comment spamming, etc. Setting it in other scenarios only complicates the review and merge process, often causing PRs to be overlooked since we rely heavily on the tag management the bot provides.

[jonatcorus]

+1 for this, too bad it wasn't accepted...