"ansible-doc --json aws_s3_bucket_info" crashes

[felixfontein]

SUMMARY

When running ansible-doc --json aws_s3_bucket_info in stable-2.9:

ansible-doc 2.9.7.post0
  config file = None
  configured module search path = ['/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /ansible/lib/ansible
  executable location = /ansible/bin/ansible-doc
  python version = 3.8.2 (default, Apr  8 2020, 14:31:25) [GCC 9.3.0]
No config file found; using defaults
/ansible/lib/ansible/cli/doc.py:219: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  doc_data['return'] = yaml.load(doc_data['return'])
ERROR! Unexpected Exception, this is probably a bug: Object of type datetime is not JSON serializable
the full traceback was:

Traceback (most recent call last):
  File "/ansible/bin/ansible-doc", line 123, in <module>
    exit_code = cli.run()
  File "/ansible/lib/ansible/cli/doc.py", line 223, in run
    jdump(plugin_docs)
  File "/ansible/lib/ansible/cli/doc.py", line 36, in jdump
    display.display(json.dumps(text, sort_keys=True, indent=4))
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/lib/python3.8/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.8/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.8/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.8/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  [Previous line repeated 1 more time]
  File "/usr/lib/python3.8/json/encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "/usr/lib/python3.8/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.8/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type datetime is not JSON serializable

I guess the problem is that the docs YAML contains something that is interpreted as a datetime by the YAML parser.

IMO ansible-doc should not crash, but instead convert everything that is not JSON-serializable to a string (or omit it).

This is important for everything that uses ansible-doc --json to extract docs (AFAIK Galaxy plans to do that).

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible-doc

ANSIBLE VERSION

2.9
(probably also devel)

[felixfontein]

ec2_vpc_vpn also causes a crash: ERROR! Unexpected Exception, this is probably a bug: '<' not supported between instances of 'int' and 'str'. This is caused by the return value docs for vgw_telemetry, whose example contains something that is parsed as the following (items of the inner dict):

[('outside_ip_address', 'string'), ('status', 'up'), ('last_status_change', 'datetime(2015'), (1, None), ('1)', None), ('status_message', 'string'), ('accepted_route_count', 123)]

[felixfontein]

The datetime bug is also triggered by more modules, see ansible-collections/community.aws#55 and ansible-collections/azure#101.

[bcoca]

fyi, running the provided reproducer in 2.9 i get:

{
    "aws_s3_bucket_info": {
        "doc": {
            "author": "Gerben Geijteman (@hyperized)",
            "description": [
                "Lists S3 buckets in AWS",
                "This module was called C(aws_s3_bucket_facts) before Ansible 2.9, returning C(ansible_facts). Note that the M(aws_s3_bucket_info) module no longer returns C(ansible_facts)!"
            ],
            "filename": "/home/bcoca/work/ansible/lib/ansible/modules/cloud/amazon/aws_s3_bucket_info.py",
            "module": "aws_s3_bucket_info",
            "notes": [
                "If parameters are not set within the module, the following environment variables can be used in decreasing order of precedence C(AWS_URL) or C(EC2_URL), C(AWS_ACCESS_KEY_ID) or
C(AWS_ACCESS_KEY) or C(EC2_ACCESS_KEY), C(AWS_SECRET_ACCESS_KEY) or C(AWS_SECRET_KEY) or C(EC2_SECRET_KEY), C(AWS_SECURITY_TOKEN) or C(EC2_SECURITY_TOKEN), C(AWS_REGION) or C(EC2_REGION)",
                "Ansible uses the boto configuration file (typically ~/.boto) if no credentials are provided. See https://boto.readthedocs.io/en/latest/boto_config_tut.html",
                "C(AWS_REGION) or C(EC2_REGION) can be typically be used to specify the AWS region, when required, but this can also be configured in the boto config file"
            ],
            "options": {
                "aws_access_key": {
                    "aliases": [
                        "ec2_access_key",
                        "access_key"
                    ],
                    "description": [
                        "AWS access key. If not set then the value of the AWS_ACCESS_KEY_ID, AWS_ACCESS_KEY or EC2_ACCESS_KEY environment variable is used."
                    ],
                    "type": "str"
                },
                "aws_secret_key": {
                    "aliases": [
                        "ec2_secret_key",
                        "secret_key"
                    ],
                    "description": [
                        "AWS secret key. If not set then the value of the AWS_SECRET_ACCESS_KEY, AWS_SECRET_KEY, or EC2_SECRET_KEY environment variable is used."
                    ],
                    "type": "str"
                },
                "debug_botocore_endpoint_logs": {
                    "default": "no",
                    "description": [
                        "Use a botocore.endpoint logger to parse the unique (rather than total) \"resource:action\" API calls made during a task, outputing the set to the resource_actions key in
 the task results. Use the aws_resource_action callback to output to total list made during a playbook. The ANSIBLE_DEBUG_BOTOCORE_LOGS environment variable may also be used."
                    ],
                    "type": "bool",
                    "version_added": "2.8"
                },
                "ec2_url": {
                    "description": [
                        "Url to use to connect to EC2 or your Eucalyptus cloud (by default the module will use EC2 endpoints). Ignored for modules where region is required. Must be specified for
 all other modules if region is not used. If not set then the value of the EC2_URL environment variable, if any, is used."
                    ],
                    "type": "str"
                },
                "profile": {
                    "description": [
                        "Uses a boto profile. Only works with boto >= 2.24.0."
                    ],
                    "type": "str",
                    "version_added": "1.6"
                },
                "region": {
                    "aliases": [
                        "aws_region",
                        "ec2_region"
                    ],
                    "description": [
                        "The AWS region to use. If not specified then the value of the AWS_REGION or EC2_REGION environment variable, if any, is used. See U(http://docs.aws.amazon.com/general/la
test/gr/rande.html#ec2_region)"
                    ],
                    "type": "str"
                },
                "security_token": {
                    "aliases": [
                        "access_token"
                    ],
                    "description": [
                        "AWS STS security token. If not set then the value of the AWS_SECURITY_TOKEN or EC2_SECURITY_TOKEN environment variable is used."
                    ],
                    "type": "str",
                    "version_added": "1.6"
                },
                "validate_certs": {
                    "default": true,
                    "description": [
                        "When set to \"no\", SSL certificates will not be validated for boto versions >= 2.6.0."
                    ],
                    "type": "bool",
                    "version_added": "1.5"
                }
            },
            "requirements": [
                "boto",
                "boto3 >= 1.4.4",
                "python >= 2.6"
            ],
            "short_description": "Lists S3 buckets in AWS",
            "version_added": "2.4"
        },
        "examples": "\n# Note: These examples do not set authentication details, see the AWS Guide for details.\n\n# Note: Only AWS S3 is currently supported\n\n# Lists all s3 buckets\n- aws_s3_
bucket_info:\n  register: result\n\n- name: List buckets\n  debug:\n    msg: \"{{ result['buckets'] }}\"\n",
        "metadata": {
            "status": [
                "preview"
            ],
            "supported_by": "community"
        },
        "return": "\nbuckets:\n  description: \"List of buckets\"\n  returned: always\n  sample:\n    - creation_date: 2017-07-06 15:05:12 +00:00\n      name: my_bucket\n  type: list\n"
    }
}

[felixfontein]

@bcoca that's the reproducer for the bug you fixed. try ec2_vpc_vpn. There the problem is in sample for return value vgw_telemetry.

[bcoca]

@felixfontein that still works for me, I get full json output and no errors

[felixfontein]

When I copy the code of that module from stable-2.9 into devel (and comment out the doc fragments), and then run ansible-doc --json ec2_vpc_vpn, I get:

[WARNING]: You are running the development version of Ansible. You should only run Ansible from "devel" if you are modifying the Ansible engine, or trying out features under development. This is a rapidly changing source of code and can become unstable at any point.
ERROR! We could not convert all the documentation into JSON as there was a conversion issue: '<' not supported between instances of 'int' and 'str'