Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lightsail - Use AnsibleAWSModule #65275

Merged
merged 4 commits into from Dec 2, 2019
Merged

Conversation

prasadkatti
Copy link
Contributor

@prasadkatti prasadkatti commented Nov 26, 2019

SUMMARY
  • Use AnsibleAWSModule
  • Refactor the logic for wait into a separate function (Fixes aws lightsail does not support wait: yes on create #63869). Fix related tests.
  • Use the default keypair from the region when creating instances if one is not provided
  • Handle exceptions in find_instance_info and add a fail_if_not_found parameter
  • Add a new state rebooted as an alias for restarted. AWS calls the action Reboot.
  • Add required_if clause for when state is present
  • Add a comment that the module does not support tagging instances at this time.

Fixes #63869 #36313

ISSUE TYPE
  • Bugfix Pull Request
  • Feature Pull Request
COMPONENT NAME

lightsail

@ansibot
Copy link
Contributor

ansibot commented Nov 26, 2019

@ansibot
Copy link
Contributor

ansibot commented Nov 26, 2019

@prasadkatti, just so you are aware we have a dedicated Working Group for aws.
You can find other people interested in this in #ansible-aws on Freenode IRC
For more information about communities, meetings and agendas see https://github.com/ansible/community

click here for bot help

@ansibot ansibot added affects_2.10 This issue/PR affects Ansible v2.10 aws bug This issue/PR relates to a bug. cloud community_review In order to be merged, this PR must follow the community review workflow. module This issue/PR relates to a module. needs_triage Needs a first human triage before being processed. support:community This issue/PR relates to code supported by the Ansible community. labels Nov 26, 2019
- Use AnsibleAWSModule
- Refactor the logic for wait into a separate function (Fixes ansible#63869)
- Handle exceptions in find_instance_info and add a fail_if_not_found parameter
- Add a new state `rebooted` as an alias for `restarted`. AWS calls the action Reboot.
- Add required_if clause for when state is present
@ansibot
Copy link
Contributor

ansibot commented Nov 26, 2019

The test ansible-test sanity --test validate-modules [explain] failed with 23 errors:

lib/ansible/modules/cloud/amazon/lightsail.py:0:0: doc-choices-do-not-match-spec: Argument 'state' in argument_spec defines choices as (['present', 'absent', 'stopped', 'running', 'restarted', 'rebooted']) but documentation defines choices as ([])
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: doc-default-does-not-match-spec: Argument 'state' in argument_spec defines default as ('present') but documentation defines default as (None)
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: doc-default-does-not-match-spec: Argument 'validate_certs' in argument_spec defines default as (True) but documentation defines default as (False)
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: doc-default-does-not-match-spec: Argument 'wait' in argument_spec defines default as (True) but documentation defines default as (False)
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: doc-default-does-not-match-spec: Argument 'wait_timeout' in argument_spec defines default as (300) but documentation defines default as (None)
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: doc-missing-type: Argument 'aws_access_key' in argument_spec uses default type ('str') but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: doc-missing-type: Argument 'aws_secret_key' in argument_spec uses default type ('str') but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: doc-missing-type: Argument 'ec2_url' in argument_spec uses default type ('str') but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: doc-missing-type: Argument 'profile' in argument_spec uses default type ('str') but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: doc-missing-type: Argument 'region' in argument_spec uses default type ('str') but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: doc-missing-type: Argument 'security_token' in argument_spec uses default type ('str') but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: parameter-type-not-in-doc: Argument 'blueprint_id' in argument_spec defines type as 'str' but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: parameter-type-not-in-doc: Argument 'bundle_id' in argument_spec defines type as 'str' but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: parameter-type-not-in-doc: Argument 'debug_botocore_endpoint_logs' in argument_spec defines type as 'bool' but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: parameter-type-not-in-doc: Argument 'key_pair_name' in argument_spec defines type as 'str' but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: parameter-type-not-in-doc: Argument 'name' in argument_spec defines type as 'str' but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: parameter-type-not-in-doc: Argument 'state' in argument_spec defines type as 'str' but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: parameter-type-not-in-doc: Argument 'user_data' in argument_spec defines type as 'str' but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: parameter-type-not-in-doc: Argument 'validate_certs' in argument_spec defines type as 'bool' but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: parameter-type-not-in-doc: Argument 'wait' in argument_spec defines type as 'bool' but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: parameter-type-not-in-doc: Argument 'wait_timeout' in argument_spec defines type as 'int' but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:0:0: parameter-type-not-in-doc: Argument 'zone' in argument_spec defines type as 'str' but documentation doesn't define type
lib/ansible/modules/cloud/amazon/lightsail.py:29:20: documentation-syntax-error: DOCUMENTATION is not valid YAML

The test ansible-test sanity --test docs-build [explain] failed with the error:

Command "/usr/bin/python3.6 /root/ansible/test/sanity/code-smell/docs-build.py" returned exit status 1.
>>> Standard Error
Command 'make singlehtmldocs' failed with status code: 2
--> Standard Output
PYTHONPATH=../../lib ../../hacking/build-ansible.py collection-meta --template-file=../templates/collections_galaxy_meta.rst.j2 --output-dir=rst/dev_guide/ ../../lib/ansible/galaxy/data/collections_galaxy_meta.yml
PYTHONPATH=../../lib ../../hacking/build-ansible.py document-config --template-file=../templates/config.rst.j2 --output-dir=rst/reference_appendices/ ../../lib/ansible/config/base.yml
mkdir -p rst/cli
PYTHONPATH=../../lib ../../hacking/build-ansible.py generate-man --template-file=../templates/cli_rst.j2 --output-dir=rst/cli/ --output-format rst ../../lib/ansible/cli/*.py
PYTHONPATH=../../lib ../../hacking/build-ansible.py document-keywords --template-dir=../templates --output-dir=rst/reference_appendices/ ./keyword_desc.yml
PYTHONPATH=../../lib ../../hacking/build-ansible.py document-plugins -t rst --template-dir=../templates --module-dir=../../lib/ansible/modules -o rst/modules/ 
Evaluating module files...
Makefile:99: recipe for target 'modules' failed
--> Standard Error
Traceback (most recent call last):
  File "../../hacking/build-ansible.py", line 92, in <module>
    main()
  File "../../hacking/build-ansible.py", line 81, in main
    retval = command.main(args)
  File "/root/ansible/hacking/build_library/build_ansible/command_plugins/plugin_formatter.py", line 721, in main
    plugin_info, categories = get_plugin_info(args.module_dir, limit_to=args.limit_to, verbose=(args.verbosity > 0))
  File "/root/ansible/hacking/build_library/build_ansible/command_plugins/plugin_formatter.py", line 225, in get_plugin_info
    doc, examples, returndocs, metadata = plugin_docs.get_docstring(module_path, fragment_loader, verbose=verbose)
  File "/root/ansible/lib/ansible/utils/plugin_docs.py", line 124, in get_docstring
    data = read_docstring(filename, verbose=verbose, ignore_errors=ignore_errors)
  File "/root/ansible/lib/ansible/parsing/plugin_docs.py", line 59, in read_docstring
    data[varkey] = AnsibleLoader(child.value.s, file_name=filename).get_single_data()
  File "/usr/local/lib/python3.6/dist-packages/yaml/constructor.py", line 41, in get_single_data
    node = self.get_single_node()
  File "ext/_yaml.pyx", line 707, in _yaml.CParser.get_single_node
  File "ext/_yaml.pyx", line 725, in _yaml.CParser._compose_document
  File "ext/_yaml.pyx", line 776, in _yaml.CParser._compose_node
  File "ext/_yaml.pyx", line 890, in _yaml.CParser._compose_mapping_node
  File "ext/_yaml.pyx", line 776, in _yaml.CParser._compose_node
  File "ext/_yaml.pyx", line 890, in _yaml.CParser._compose_mapping_node
  File "ext/_yaml.pyx", line 776, in _yaml.CParser._compose_node
  File "ext/_yaml.pyx", line 890, in _yaml.CParser._compose_mapping_node
  File "ext/_yaml.pyx", line 774, in _yaml.CParser._compose_node
  File "ext/_yaml.pyx", line 853, in _yaml.CParser._compose_sequence_node
  File "ext/_yaml.pyx", line 905, in _yaml.CParser._parse_next_event
yaml.parser.ParserError: while parsing a block collection
  in "<unicode string>", line 15, column 7
did not find expected '-' indicator
  in "<unicode string>", line 16, column 20
make: *** [modules] Error 1

The test ansible-test sanity --test package-data [explain] failed with the error:

Command "/usr/bin/python3.6 /root/ansible/test/sanity/code-smell/package-data.py" returned exit status 1.
>>> Standard Error
Traceback (most recent call last):
  File "/root/ansible/test/sanity/code-smell/package-data.py", line 383, in <module>
    main()
  File "/root/ansible/test/sanity/code-smell/package-data.py", line 360, in main
    sdist_path = create_sdist(tmp_dir)
  File "/root/ansible/test/sanity/code-smell/package-data.py", line 174, in create_sdist
    raise Exception('make snapshot failed:\n%s' % stderr)
Exception: make snapshot failed:
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
docs/man/man1/ansible-galaxy.1.rst:51: (WARNING/2) Definition list ends without a blank line; unexpected unindent.
docs/man/man1/ansible-galaxy.1.rst:57: (WARNING/2) Definition list ends without a blank line; unexpected unindent.
ERROR! module lightsail at /tmp/tmpij_3egrv/lib/ansible/modules/cloud/amazon/lightsail.py has a documentation error formatting or is missing documentation.
Traceback (most recent call last):
  File "packaging/release/changelogs/changelog.py", line 835, in <module>
    main()
  File "packaging/release/changelogs/changelog.py", line 102, in main
    args.func(args)
  File "packaging/release/changelogs/changelog.py", line 132, in command_release
    plugins = load_plugins(version=version, force_reload=reload_plugins)
  File "packaging/release/changelogs/changelog.py", line 184, in load_plugins
    '--json', '--metadata-dump', '-t', plugin_type])
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/tmp/tmpij_3egrv/bin/ansible-doc', '--json', '--metadata-dump', '-t', 'module']' returned non-zero exit status 1.
make: *** [changelog] Error 1

The test ansible-test sanity --test ansible-doc [explain] failed with the error:

Command "ansible-doc -t module lightsail" returned exit status 1.
>>> Standard Error
ERROR! module lightsail missing documentation (or could not parse documentation): while parsing a block collection
  in "<unicode string>", line 15, column 7
did not find expected '-' indicator
  in "<unicode string>", line 16, column 20

The test ansible-test sanity --test yamllint [explain] failed with 1 error:

lib/ansible/modules/cloud/amazon/lightsail.py:29:20: error: DOCUMENTATION: syntax error: expected <block end>, but found '<scalar>'

click here for bot help

@ansibot ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. community_review In order to be merged, this PR must follow the community review workflow. and removed community_review In order to be merged, this PR must follow the community review workflow. needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels Nov 26, 2019
lib/ansible/modules/cloud/amazon/lightsail.py Outdated Show resolved Hide resolved
if not HAS_BOTO3:
module.fail_json(msg='Python module "boto3" is missing, please install it')
module = AnsibleAWSModule(argument_spec=argument_spec,
required_if=[['state', 'present', ('zone', 'blueprint_id', 'bundle_id', 'key_pair_name')]])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personal niggle: boto3 doesn't list keyPairName as a 'required' argument.

I have some use cases where I bake the keys into the AMI and don't want to pass an additional key to the instances...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think Lightsail allows custom AMIs, at least from what I've seen in the docs and prodding at the console a bit. There is a question of if we should have a default to use the region default key when state=present instead of requiring a user specification, generally the way lightsail does keys is different from the way ec2 does keys though and I'm still familiarizing myself with the differences.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going off my knowledge of EC2, I've not worked with Lightsail.

Fair enough if you can't use custom AMIs (today). At the same time, adding additional 'requirements' on top of boto3's only risks breaking use cases you might not have thought of.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What concerns me is the implication (not directly stated in the docs, at least that I can find) that you can't add/modify a Lightsail instance's keypair after creation (short of ssh'ing to the instance with an existing key and editing auth keys by hand). If this is correct, I feel like we should anticipate users having a problem here and make keypair a required option for create actions. It may not be boto-y, but it's still something we can easily anticipate and make nicer for our users. If that's not correct and I'm just missing something about Lightsail key management, then maybe keypair management or a lightsail_key module would be good future improvements (but outside the scope of this PR).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tremble - The create_instances method takes in a blueprintId parameter that is analogous to an EC2 AMI. The method had a customImageName parameter but that is now deprecated.

It does not look like it is possible to change the keypair used by an existing instance. Boto3 does not require the keyPairName because when it is absent lightsail uses the default key for the region. Currently, the module does not support using that key AFAICT. Updating the module to add the capability of using the default keypair is quite easy. In fact, now that I think about it, we should absolutely do that.

@jillr - I didn't quite get the connection between being able to update an instance's keypair and making the key_pair_name a required option. (required option..lol). Are you saying we don't want users to unknowingly use the default key and then not be able to change it? The private key from the default keypair can always be downloaded at a later time unlike keypairs created explicitly.

I did a bare bones implementation of the lightsail_keypair module that is being used to create a keypair for every run of the lightsail integration testsuite. I would be happy to flesh it out into a fully featured module.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can set key_pair_name to "LightsailDefaultKeyPair" ("key_pair_name": "LightsailDefaultKeyPair") to use the default lightsail keypair for that region.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@prasadkatti : @jillr is mostly worried about protecting folks from making stupid mistakes. If not setting the Key Pair meant it was never possible to get into a host, then there's a case to be made for making it 'required' on the Ansible side (Amazon for example gives you a nice big warning on the console when you do this). If there was a mechanism for injecting a new key somewhere down the line, then this is less of a worry.

Given that AWS will inject a default key when you don't supply one, I think it's reasonable to err towards not putting an additional 'requirement' in front of boto.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I agree. We should not make it a requirement.

@ansibot ansibot removed the needs_triage Needs a first human triage before being processed. label Nov 26, 2019
@prasadkatti
Copy link
Contributor Author

Added changes to allow usage of the default keypair from the region when creating instances.

@ansibot ansibot added core_review In order to be merged, this PR must follow the core review workflow. support:core This issue/PR relates to code supported by the Ansible Engineering Team. and removed community_review In order to be merged, this PR must follow the community review workflow. labels Nov 27, 2019
Copy link
Contributor

@tremble tremble left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see anything that should block this PR moving forward.

@@ -1,79 +0,0 @@
#!/usr/bin/python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If Lightsail Key Pairs aren't the same as EC2 Key Pairs it would be nice to get this added as an additional module. At which point it would make sense to adjust these integration tests to test two creation modes:

  1. create a lightsail instance without specifying a key
  2. create a lightsail instance with a key

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I can work on a module to manage lightsail keypairs. Although, there are some other higher priority lightsail modules needed IMO. The current module only manages lightsail instances. Lightsail also has databases, networks, storage that will need separate modules. In fact, the current lightsail module should be renamed to lightsail_instance.

lib/ansible/modules/cloud/amazon/lightsail.py Outdated Show resolved Hide resolved
Copy link
Contributor

@jillr jillr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarifications on how keypairs work in lightsail - this LGTM now.

@jillr jillr merged commit 37ce55f into ansible:devel Dec 2, 2019
@prasadkatti
Copy link
Contributor Author

Thanks for the reviews and the feedback @jillr @tremble

@prasadkatti prasadkatti deleted the lightsail_wait branch December 3, 2019 22:03
@prasadkatti
Copy link
Contributor Author

Fixes #36313 .

D3DeFi pushed a commit to D3DeFi/ansible that referenced this pull request Dec 8, 2019
* lightsail - Use AnsibleAWSModule

- Use AnsibleAWSModule
- Refactor the logic for wait into a separate function (Fixes ansible#63869)
- Handle exceptions in find_instance_info and add a fail_if_not_found parameter
- Add a new state `rebooted` as an alias for `restarted`. AWS calls the action Reboot.
- Add required_if clause for when state is present

* lightsail - Use the default keypair if one is not provided

* lightsail - add a required_if for when state=present

* Update short description for lightsail module
anshulbehl pushed a commit to anshulbehl/ansible that referenced this pull request Dec 10, 2019
* lightsail - Use AnsibleAWSModule

- Use AnsibleAWSModule
- Refactor the logic for wait into a separate function (Fixes ansible#63869)
- Handle exceptions in find_instance_info and add a fail_if_not_found parameter
- Add a new state `rebooted` as an alias for `restarted`. AWS calls the action Reboot.
- Add required_if clause for when state is present

* lightsail - Use the default keypair if one is not provided

* lightsail - add a required_if for when state=present

* Update short description for lightsail module
@ansible ansible locked and limited conversation to collaborators Jan 2, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
affects_2.10 This issue/PR affects Ansible v2.10 aws bug This issue/PR relates to a bug. cloud core_review In order to be merged, this PR must follow the core review workflow. module This issue/PR relates to a module. support:community This issue/PR relates to code supported by the Ansible community. support:core This issue/PR relates to code supported by the Ansible Engineering Team.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

aws lightsail does not support wait: yes on create
4 participants