Initial commit for fortios_webfilter

[migumun]

SUMMARY

Fortinet is adding Ansible support for FortiOS and FortiGate products. There is a previous work which was left on hold while we discussed a new approach more compatible with Ansible guidelines:
#33591 (comment)

In this new approach we will be creating a number of modules for FortiGate: system, wireless-controller, firewall, webfilter, ips, web-proxy, wanopt, application, dlp spamfilter, log, vpn, certificate, user, dnsfilter, antivirus, report, waf, authentication, switch controller, endpoint-control and router

This initial commit is intended for webfilter functionality and contains a part of the things that can be done with webfilter feature in FortiGate.

More features and module will be added later. We prefer to submit code in small chunks to ensure next modules are developed according to guidelines.

ISSUE TYPE

• New Module Pull Request

COMPONENT NAME

fortios_webfilter

ANSIBLE VERSION

ansible 2.6.0 (devel 9d4ead1edb) last updated 2018/03/08 16:47:18 (GMT +200)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/magonzalez/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/magonzalez/ansible/lib/ansible
  executable location = /home/magonzalez/ansible/bin/ansible
  python version = 2.7.14 (default, Sep 23 2017, 22:06:14) [GCC 7.2.0]

[ansibot]

cc @bjolivot
click here for bot help

[ansibot]

The test ansible-test sanity --test pep8 [explain] failed with 25 errors:

lib/ansible/modules/network/fortios/fortios_webfilter.py:71:21: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:97:30: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:105:30: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:113:24: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:123:26: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:131:39: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:132:28: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:142:28: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:150:28: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:152:86: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:153:82: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:174:85: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:175:80: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:180:30: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:183:70: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:208:26: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:221:28: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:229:26: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:244:27: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:249:28: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:260:85: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:261:80: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:266:30: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:339:25: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:348:13: W291 trailing whitespace

The test ansible-test sanity --test validate-modules [explain] failed with 20 errors:

lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.host.type: not a valid value for dictionary value @ data['options']['host']['type']. Got 'string'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.password.type: not a valid value for dictionary value @ data['options']['password']['type']. Got 'string'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.username.type: not a valid value for dictionary value @ data['options']['username']['type']. Got 'string'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.vdom.type: not a valid value for dictionary value @ data['options']['vdom']['type']. Got 'string'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_content.suboptions.comment.type: not a valid value for dictionary value @ data['options']['webfilter_content']['suboptions']['comment']['type']. Got 'string'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_content.suboptions.entries.suboptions: extra keys not allowed @ data['options']['webfilter_content']['suboptions']['entries']['suboptions']. Got {'name': {'description': ['Banned word.'], 'required': True, 'type': 'string'}, 'pattern-type': {'description': [{'Banned word pattern type': 'wildcard pattern or Perl regular expression.'}], 'required': True, 'type': 'string', 'choices': ['wildcard', 'regexp']}, 'status': {'description': ['Enable/disable banned word.'], 'required': True, 'type': 'string', 'choices': ['enable', 'disable']}, 'lang': {'description': ['Language of banned word.'], 'required': True, 'type': 'string', 'choices': ['...
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_content.suboptions.entries.type: not a valid value for dictionary value @ data['options']['webfilter_content']['suboptions']['entries']['type']. Got 'list'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_content.suboptions.id.type: not a valid value for dictionary value @ data['options']['webfilter_content']['suboptions']['id']['type']. Got 'string'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_content.suboptions.name.type: not a valid value for dictionary value @ data['options']['webfilter_content']['suboptions']['name']['type']. Got 'string'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_content.type: not a valid value for dictionary value @ data['options']['webfilter_content']['type']. Got 'dict'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_url.suboptions.comment.type: not a valid value for dictionary value @ data['options']['webfilter_url']['suboptions']['comment']['type']. Got 'string'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_url.suboptions.entries.suboptions: extra keys not allowed @ data['options']['webfilter_url']['suboptions']['entries']['suboptions']. Got {'id': {'description': ['Id of URL.'], 'required': True, 'type': 'integer'}, 'url': {'description': ['URL to be filtered.'], 'required': True, 'type': 'string'}, 'type': {'description': ['Filter type (simple, regex, or wildcard).'], 'required': True, 'type': 'string', 'choices': ['simple', 'regex', 'wildcard']}, 'action': {'description': ['Action to take for URL filter matches.'], 'required': True, 'type': 'string', 'choices': ['exempt', 'block', 'allow', 'monitor']}, 'status': {'description'...
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_url.suboptions.entries.type: not a valid value for dictionary value @ data['options']['webfilter_url']['suboptions']['entries']['type']. Got 'list'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_url.suboptions.id.type: not a valid value for dictionary value @ data['options']['webfilter_url']['suboptions']['id']['type']. Got 'string'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_url.suboptions.ip-addr-block.type: not a valid value for dictionary value @ data['options']['webfilter_url']['suboptions']['ip-addr-block']['type']. Got 'string'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_url.suboptions.name.type: not a valid value for dictionary value @ data['options']['webfilter_url']['suboptions']['name']['type']. Got 'string'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_url.suboptions.one-arm-ips-url-filter.type: not a valid value for dictionary value @ data['options']['webfilter_url']['suboptions']['one-arm-ips-url-filter']['type']. Got 'string'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_url.type: not a valid value for dictionary value @ data['options']['webfilter_url']['type']. Got 'dict'
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E307 version_added should be 2.6. Currently 2.5
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E319 RETURN.http_status.type: not a valid value for dictionary value @ data['type']. Got 'integer'

click here for bot help

[ansibot]

The test ansible-test sanity --test validate-modules [explain] failed with 3 errors:

lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_content.suboptions.entries.suboptions: extra keys not allowed @ data['options']['webfilter_content']['suboptions']['entries']['suboptions']. Got {'name': {'description': ['Banned word.'], 'required': True}, 'pattern-type': {'description': [{'Banned word pattern type': 'wildcard pattern or Perl regular expression.'}], 'required': True, 'choices': ['wildcard', 'regexp']}, 'status': {'description': ['Enable/disable banned word.'], 'required': True, 'choices': ['enable', 'disable']}, 'lang': {'description': ['Language of banned word.'], 'required': True, 'choices': ['western', 'simch', 'trach', 'japanese', 'korean', 'french', 'thai', 'spa...
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E305 DOCUMENTATION.options.webfilter_url.suboptions.entries.suboptions: extra keys not allowed @ data['options']['webfilter_url']['suboptions']['entries']['suboptions']. Got {'id': {'description': ['Id of URL.'], 'required': True, 'type': 'integer'}, 'url': {'description': ['URL to be filtered.'], 'required': True}, 'type': {'description': ['Filter type (simple, regex, or wildcard).'], 'required': True, 'choices': ['simple', 'regex', 'wildcard']}, 'action': {'description': ['Action to take for URL filter matches.'], 'required': True, 'choices': ['exempt', 'block', 'allow', 'monitor']}, 'status': {'description': ['Enable/disable this URL filter.'], 'required': Tru...
lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E307 version_added should be 2.6. Currently 2.5

click here for bot help

[sivel]

Since you are using suboptions in your DOCUMENTATION you also need to add options to your arguments that support this.

[sivel]

This should not raise an exception. Instead you should use module.fail_json

[sivel]

Please don't use a global. Pass it to your methods that require this object.

[ansibot]

The test ansible-test sanity --test ansible-doc --python 2.7 [explain] failed with 1 error:

lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 2.6 [explain] failed with 1 error:

lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 3.5 [explain] failed with 1 error:

lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 3.6 [explain] failed with 1 error:

lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 3.7 [explain] failed with 1 error:

lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test pep8 [explain] failed with 2 errors:

lib/ansible/modules/network/fortios/fortios_webfilter.py:156:37: W291 trailing whitespace
lib/ansible/modules/network/fortios/fortios_webfilter.py:513:72: E231 missing whitespace after ':'

The test ansible-test sanity --test validate-modules [explain] failed with 2 errors:

lib/ansible/modules/network/fortios/fortios_webfilter.py:0:0: E324 Value for "default" from the argument_spec ('root') for "vdom" does not match the documentation (None)
lib/ansible/modules/network/fortios/fortios_webfilter.py:153:36: E302 DOCUMENTATION is not valid YAML

The test ansible-test sanity --test yamllint [explain] failed with 1 error:

lib/ansible/modules/network/fortios/fortios_webfilter.py:153:36: error DOCUMENTATION: syntax error: could not find expected ':'

click here for bot help

[migumun]

HI @sivel,
I have modified the code according to your comments but I would like to call your attention on the argument_spec cause I see an strange behavior:

• argument_spec works fine for arguments on the first level, e.g., host, username, password, vdom, etc.. If I make a typo in the playbook, Ansible detects it and prevents me from going on. This is ok.
• However, when I make a typo in second level arguments, e.g. id, name, comment, one-arm-ips-url-filter they are detected only when I am debugging, that is, when I pass as argument a json containing the yml content ("ANSIBLE_MODULE_ARGS": {....}). If I run it with CLI ansible-playbook.... they are not detected.
• And finally if I make a typo in third level arguments such as status, web-proxy-profile, referrer-host, etc they are not detected in any case, either CLI or as json argument directly into the module.

I suspect this may be wrong behavior or maybe I have coded something wrong in the argument spec (though it is still strange that 2nd level arguments work fine when I am debugging and using a json file as input, and not working ok when I run it CLI with ansible-playbook). Your validation here would be of great help.