ec2.py dynamic inventory fails

[rob42]

When i use the ec2.py dynamic inventory script I get a failure:

robert@cbr:~/gitrep/cp_ansible/inventory$ ansible-cmdb -d -i ec2.py out/ > overview.html
data_dir = /home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/data
tpl_dir = /home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/data/tpl
static_dir = /home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/data/static
inventory files = ['ec2.py']
template params = {'exclude_columns': None, 'data_dir': '/home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/data', 'log': <logging.RootLogger object at 0x7fbe017ca610>, 'lib_dir': '/home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/data', 'version': '1.30', 'columns': None}
Parsing fact dir: out/
Determining type of inventory_path ec2.py
ec2.py is a executable. Handle as dynamic inventory script
Reading dynamic inventory ec2.py
Exception while executing dynamic inventory script 'ec2.py':

[Errno 2] No such file or directory
Parsing host vars (dir): host_vars
No such dir host_vars
Parsing group vars (dir): group_vars
No such dir group_vars
Whoops, it looks like something went wrong while rendering the template.

The reported error was: TypeError: 'Undefined' object is not iterable

The full error was:
    
    Traceback (most recent call last):
      File "/usr/local/bin/../lib/ansiblecmdb/ansible-cmdb.py", line 177, in <module>
        output = renderer.render(ansible.hosts, params)
      File "/home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/render.py", line 42, in render
        return self._render_mako(hosts, vars)
      File "/home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/render.py", line 59, in _render_mako
        return template.render(hosts=hosts, **vars)
      File "/home/robert/.local/lib/python2.7/site-packages/mako/template.py", line 462, in render
        return runtime._render(self, self.callable_, args, data)
      File "/home/robert/.local/lib/python2.7/site-packages/mako/runtime.py", line 838, in _render
        **_kwargs_for_callable(callable_, data))
      File "/home/robert/.local/lib/python2.7/site-packages/mako/runtime.py", line 873, in _render_context
        _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
      File "/home/robert/.local/lib/python2.7/site-packages/mako/runtime.py", line 899, in _exec_template
        callable_(context, *args, **kwargs)
      File "/home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/data/tpl/html_fancy.tpl", line 6, in render_body
        <%
    TypeError: 'Undefined' object is not iterable
    
The output is probably not correct.
....

The same ec2.py script (from https://docs.ansible.com/ansible/2.4/intro_dynamic_inventory.html#example-aws-ec2-external-inventory-script) runs fine on the commandline, I piped the output to ec2.json and its fine.

So I tried using the ec2.json file, much the same problem:

robert@cbr:~/gitrep/cp_ansible/inventory$ ansible-cmdb -d -i ec2.json out/ > overview.html
...lots of these...
Unsupported vars syntax. Skipping line: "vpc_id_vpc_0dce4068": [
Unsupported vars syntax. Skipping line: "vpc_id_vpc_2da09149": [
Unsupported vars syntax. Skipping line: "vpc_id_vpc_a64ee4c3": [
Parsing host vars (dir): host_vars
No such dir host_vars
Parsing group vars (dir): group_vars
No such dir group_vars
Whoops, it looks like something went wrong while rendering the template.

The reported error was: TypeError: 'Undefined' object is not iterable

The full error was:
    
    Traceback (most recent call last):
      File "/usr/local/bin/../lib/ansiblecmdb/ansible-cmdb.py", line 177, in <module>
        output = renderer.render(ansible.hosts, params)
      File "/home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/render.py", line 42, in render
        return self._render_mako(hosts, vars)
      File "/home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/render.py", line 59, in _render_mako
        return template.render(hosts=hosts, **vars)
      File "/home/robert/.local/lib/python2.7/site-packages/mako/template.py", line 462, in render
        return runtime._render(self, self.callable_, args, data)
      File "/home/robert/.local/lib/python2.7/site-packages/mako/runtime.py", line 838, in _render
        **_kwargs_for_callable(callable_, data))
      File "/home/robert/.local/lib/python2.7/site-packages/mako/runtime.py", line 873, in _render_context
        _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
      File "/home/robert/.local/lib/python2.7/site-packages/mako/runtime.py", line 899, in _exec_template
        callable_(context, *args, **kwargs)
      File "/home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/data/tpl/html_fancy.tpl", line 6, in render_body
        <%
    TypeError: 'Undefined' object is not iterable
    
The output is probably not correct.
....

[fboender]

Hey Rob!

I think the problem is related to this part of the output

Exception while executing dynamic inventory script 'ec2.py':
[Errno 2] No such file or directory

It look like ansible-cmdb can't find the script. Can you try prepending the script with "./", like so:

 ansible-cmdb -d -i ./ec2.py out/ > overview.html

So I tried using the ec2.json file, much the same problem:

Ansible-cmdb currently only understands json output if it came from the output of a script, but not as input to the -i parameter. I guess this could be considered the same bug as #154.

[fboender]

Could also be possible that you need to specify the full path to the script. Something like:

ansible-cmdb -d -i /home/fboender/ansible/ec2.py out/ > overview.html

[rob42]

Tried ./ec2.py and full path, both result in:

robert@cbr:~/gitrep/cp_ansible/inventory$ ansible-cmdb -d -i ./ec2.py out/ > overview.html
data_dir = /home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/data
tpl_dir = /home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/data/tpl
static_dir = /home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/data/static
inventory files = ['./ec2.py']
template params = {'exclude_columns': None, 'data_dir': '/home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/data', 'log': <logging.RootLogger object at 0x7f14f1b99610>, 'lib_dir': '/home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/data', 'version': '1.30', 'columns': None}
Parsing fact dir: out/
Determining type of inventory_path ./ec2.py
./ec2.py is a executable. Handle as dynamic inventory script
Reading dynamic inventory ./ec2.py
Parsing host vars (dir): ./host_vars
No such dir ./host_vars
Parsing group vars (dir): ./group_vars
No such dir ./group_vars
Whoops, it looks like something went wrong while rendering the template.

The reported error was: TypeError: 'Undefined' object is not iterable

The full error was:
    
    Traceback (most recent call last):
      File "/usr/local/bin/../lib/ansiblecmdb/ansible-cmdb.py", line 177, in <module>
        output = renderer.render(ansible.hosts, params)
      File "/home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/render.py", line 42, in render
        return self._render_mako(hosts, vars)
      File "/home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/render.py", line 59, in _render_mako
        return template.render(hosts=hosts, **vars)
      File "/home/robert/.local/lib/python2.7/site-packages/mako/template.py", line 462, in render
        return runtime._render(self, self.callable_, args, data)
      File "/home/robert/.local/lib/python2.7/site-packages/mako/runtime.py", line 838, in _render
        **_kwargs_for_callable(callable_, data))
      File "/home/robert/.local/lib/python2.7/site-packages/mako/runtime.py", line 873, in _render_context
        _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
      File "/home/robert/.local/lib/python2.7/site-packages/mako/runtime.py", line 899, in _exec_template
        callable_(context, *args, **kwargs)
      File "/home/robert/.local/lib/python2.7/site-packages/ansiblecmdb/data/tpl/html_fancy.tpl", line 6, in render_body
        <%
    TypeError: 'Undefined' object is not iterable
    
The output is probably not correct.

[rob42]

Tried the markdown output and it runs but just outputs each host with 'No info collected'
The ec2.py output doesnt have any ansible_facts object, but does have hostvars.

I added a debug and the data is a long list of:

u '51.90.19.150': {
		'hostvars': {
			u 'ec2_account_id': u '2929299292',
			u 'ec2_kernel': u '',
			u 'ec2_state': u 'running',
 ....
		},
		'name': u '51.90.19.150',
		'groups': set([u 'tag_Type_tomcat',..])
	},
	....

[fboender]

Okay, so it looks like it's running the script now at least:

Determining type of inventory_path ./ec2.py
./ec2.py is a executable. Handle as dynamic inventory script
Reading dynamic inventory ./ec2.py
Parsing host vars (dir): ./host_vars

Could you send the raw output (just redirect it to a file) of the script to ferry.boender@gmail.com so I can analyse it? I don't have access to EC2, so I can't generate any myself. The output will be treated 100% confidentially and I'll remove all traces of it when I'm done debugging the issue. If you want, you can anonymize it before sending.

[rob42]

Looks like ec2.py gets ec2_facts rather than ansible_facts. To get ansible_facts I have to ssh to each host (via ansible) which fails as there are many different ssh keys for different groups of hosts.
I guess the fix is an ec2_facts centered template?

[rob42]

FYI my use case is to automate a scan of the AWS account and produce a reliable list of all hosts and groups for staff reference and for inventory.

[rob42]

See https://github.com/lorin/ansible-quickref/blob/master/ec2.rst#hostvars-from-ec2py-dynamic-inventory-script