added template verification playbook

[networkop]

Hey Jason,
i hope this doesn't feel like intrusion but i've allowed myself the liberty of adding a template testing functionality to your project. There's a few changes I've made

1. Renamed template and raw files to have .template or .raw extension
2. created a ./tests directory to store .raw files, .parse files and .yml files for each template
3. .yml file will set all variables necessary to run ntc_ansible module
4. .parse will contain a parsed result of .raw file
5. verify.yml playbook will ask for a template file, run it through textfsm and compare the result to .parse. The idea is to prevent stuffing things up in the future when someone modifies any of the old templates.
   and finally, i've added a template (along with the tests) for 'show ip int brief' command.
   I hope I didn't come off rude or anything. This is my first pull request and I don't have much experience in collaborative development so apologies if i did anything wrong.

[jedelman8]

No problem. I need more time to review this, but on the surface most of it makes sense. Can you clarify anything on your .yml files and roles? Are these meant more for testing or when users will actually use the ntc_show_command module? Again, I only took a real quick look and will try and dive deeper within the next 1-2 days.

Thanks!
Jason

[networkop]

Thanks Jason.
The .yml file is supposed to be a test definition file - the file that would setup all variables needed to run the ntc module on test input (e.g. device type, command).
The roles - these are just ansible's best practices. I just find the code to be a lot cleaner with roles. However the main goal is code reuse. For example the task that runs your custom module can only be setup once as a role and then reused by a testing playbook and by the main site.yml. Debug role is for code re-use. And the other two are specifically for verify.yml playbook - one to read all the input files and second to run the custom compare_dict module.
It's not perfect, I realise, so I'm open for suggestions if you think things need to be done otherwise. For example, I'm thinking it's possible to 'dry-out' the test definition file by assuming a strict naming convention and therefore avoid having to define individual names for .raw, .template and .parse files.
Anyhow, that's for consideration and let me know if there's any questions.
Cheers,
Michael

[jedelman8]

Yes, understand roles, just wanted more context :)

Slowly going through this...can you remove cisco_show_ip_int_brief.raw from ntc_templates? It should only be in the tests dir. Would you mind also creating a separate directory for each template in tests, i.e. a dir called cisco_show_ip_int_brief, cisco_show_vlan, etc.? That should clean things up.

[networkop]

Ok, so i've cleaned up all the redundant files. so ntp_temlpates will only have templates and tests will have a vendor directory and command directory within it, each with two files - raw and parsed. Also I got rid of test-specific definitions in favour of vendor-specific definitions. All files now assume a strict naming convention of vendor_command_to_be_executed.{extension}, where if extension==template - the file goes to ntp_temlpates, if extension == parsed | raw the file goes under `./tests/vendor/command_to_be_executed/

[jedelman8]

Please verify this is accurate - should this be vlans.response or something more abstract?

[jedelman8]

FYI- Just commented inline on the repo in a few locations.

[networkop]

yep, cool, agreed. It did feel like a bit of over-engineering when i was doing it. 4 roles for testing and none for the actual module. so cool, i'll collapse all testing in a single role. that "vlan.response" is actually returned by your module and i was planning to fix it in a separate branch but can fix it here, no issues. I'll also update contributing.md to clarify instructions on how to add test cases for templates. should get it pushed later today.

[jedelman8]

Would you mind adding .pyc files to the .gitignore and then removing yours from this request?

Is from_dev still required or can you remove that too?

After this, I think we'll be good...just lots of files to review :)

BTW, based on your testing platform, can you update the template names to cisco_nxos_command or cisco_ios, etc. See other PR coming in...outputs are different, and we are going to try and go with template name to match the netmiko vendor type and then _command.

Just update your template name for now though and other changes will soon follow

nice work!!!

[networkop]

All done. I've updated tests as well as template files (including index). I haven't touched that main site.yml.

[jedelman8]

Think it's good...I'll try and test more over the coming days and may update as necessary! Thank you very much!!

[networkop]

Cool. Thank you too, mate. In the future, if this project takes off, you would be able to completely automate the addition of new templates with a CI server and a test playbook.