Plugins from collection not shown in ansible-doc

[AlanCoding]

SUMMARY

The CLI command ansible-doc is not showing plugins which are active, if those plugins come from a collection.

ISSUE TYPE

• Bug Report

COMPONENT NAME

lib/ansible/cli/doc.py

ANSIBLE VERSION

current devel branch

 ansible --version
ansible 2.9.0.dev0
  config file = None
  configured module search path = ['/Users/alancoding/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/alancoding/Documents/repos/ansible/lib/ansible
  executable location = /Users/alancoding/.virtualenvs/ansible3/bin/ansible
  python version = 3.6.5 (default, Apr 25 2018, 14:23:58) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)]

CONFIGURATION

ansible-config dump --only-changed
ANSIBLE_NOCOWS(env: ANSIBLE_NOCOWS) = True

OS / ENVIRONMENT

N/A, control machine, Mac OS

STEPS TO REPRODUCE

File structure used is found in source control at:

https://github.com/chrismeyersfsu/ansible-examples/tree/master/collections_hello_world

You need to be in that exact folder to reproduce these steps.

$ tree
.
├── ansible_collections
│   └── redhat
│       └── cmeyers
│           └── roles
│               └── hello_world_role
│                   └── tasks
│                       └── main.yml
├── collections
│   └── ansible_collections
│       └── redhat
│           └── cmeyers
│               ├── plugins
│               │   ├── action
│               │   │   └── my_action.py
│               │   ├── inventory
│               │   │   └── my_inventory.py
│               │   ├── lookup
│               │   │   └── my_lookup.py
│               │   ├── module_utils
│               │   │   └── my_shared_lib.py
│               │   └── modules
│               │       └── my_module.py
│               └── roles
│                   └── my_role
│                       └── tasks
│                           └── main.yml
├── main.yml
└── my_inventory.yml

19 directories, 9 files

the inventory my_inventory.yml has contents:

plugin: redhat.cmeyers.my_inventory
hostname: 'my_host'

run command:

ansible-doc -t inventory -l

EXPECTED RESULTS

Verify functionality of the playbook via ansible-playbook -i my_inventory.yml main.yml. In output, note:

TASK [debug] **************************************************************************************************************************************************************************
ok: [my_host] => {
    "msg": "I am host 'my_host'"
}

This demonstrates that the plugin is being used. Because this inventory plugin is active inside of this folder, I expect to obtain an entry corresponding to the plugin. Like:

$ ansible-doc -t inventory -l
...
redhat.cmeyers.my_inventory     Add a single host
...

The text "Add a single host" is in the documentation

$ head collections/ansible_collections/redhat/cmeyers/plugins/inventory/my_inventory.py 
# Copyright (c) 2018 Ansible Project
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

DOCUMENTATION = '''
    inventory: my_inventory
    short_description: Add a single host
    options:

ACTUAL RESULTS

There is no mention of this plugin in the output.

$ ansible-doc -t inventory -l
advanced_host_list  Parses a 'host list' with ranges                                                                                                                              
auto                Loads and executes an inventory plugin specified in a YAML config                                                                                             
aws_ec2             EC2 inventory source                                                                                                                                          
aws_rds             rds instance source                                                                                                                                           
azure_rm            Azure Resource Manager inventory plugin                                                                                                                       
cloudscale          cloudscale.ch inventory source                                                                                                                                
constructed         Uses Jinja2 to construct vars and groups based on existing inventory                                                                                          
docker_swarm        Ansible dynamic inventory plugin for Docker swarm nodes                                                                                                       
foreman             foreman inventory source                                                                                                                                      
gcp_compute         Google Cloud Compute Engine inventory source                                                                                                                  
generator           Uses Jinja2 to construct hosts and groups from patterns                                                                                                       
gitlab_runners      Ansible dynamic inventory plugin for Gitlab runners                                                                                                           
hcloud              Ansible dynamic inventory plugin for the Hetzner Cloud                                                                                                        
host_list           Parses a 'host list' string                                                                                                                                   
ini                 Uses an Ansible INI file as inventory source                                                                                                                  
k8s                 Kubernetes (K8s) inventory source                                                                                                                             
kubevirt            KubeVirt inventory source                                                                                                                                     
linode              Ansible dynamic inventory plugin for Linode                                                                                                                   
netbox              NetBox inventory source                                                                                                                                       
nmap                Uses nmap to find hosts to target                                                                                                                             
online              Online inventory source                                                                                                                                       
openshift           OpenShift inventory source                                                                                                                                    
openstack           OpenStack inventory source                                                                                                                                    
scaleway            Scaleway inventory source                                                                                                                                     
script              Executes an inventory script that returns JSON                                                                                                                
toml                Uses a specific TOML file as an inventory source                                                                                                              
tower               Ansible dynamic inventory plugin for Ansible Tower                                                                                                            
virtualbox          virtualbox inventory source                                                                                                                                   
vmware_vm_inventory VMware Guest inventory source                                                                                                                                 
vultr               Vultr inventory source                                                                                                                                        
yaml                Uses a specific YAML file as an inventory source 

[bcoca]

you can use -M option to point to the plugin dir as a workaround:

ansible-doc -M collections/ansible_collections/redhat/cmeyers/plugins/inventory/ -t inventory -l
results in

advanced_host_list  Parses a 'host list' with ranges
auto                Loads and executes an inventory plugin specified in a YAML config
aws_ec2             EC2 inventory source
aws_rds             rds instance source
azure_rm            Azure Resource Manager inventory plugin
cloudscale          cloudscale.ch inventory source
constructed         Uses Jinja2 to construct vars and groups based on existing inventory
docker_swarm        Ansible dynamic inventory plugin for Docker swarm nodes
foreman             foreman inventory source
gcp_compute         Google Cloud Compute Engine inventory source
generator           Uses Jinja2 to construct hosts and groups from patterns
gitlab_runners      Ansible dynamic inventory plugin for Gitlab runners
hcloud              Ansible dynamic inventory plugin for the Hetzner Cloud
host_list           Parses a 'host list' string
ini                 Uses an Ansible INI file as inventory source
k8s                 Kubernetes (K8s) inventory source
kubevirt            KubeVirt inventory source
linode              Ansible dynamic inventory plugin for Linode
my_inventory        Add a single host
netbox              NetBox inventory source
nmap                Uses nmap to find hosts to target
online              Online inventory source
openshift           OpenShift inventory source
openstack           OpenStack inventory source
scaleway            Scaleway inventory source
script              Executes an inventory script that returns JSON
toml                Uses a specific TOML file as an inventory source
tower               Ansible dynamic inventory plugin for Ansible Tower
virtualbox          virtualbox inventory source
vmware_vm_inventory VMware Guest inventory source
vultr               Vultr inventory source
yaml                Uses a specific YAML file as an inventory source

[AlanCoding]

^ since that comment was left, the folder structure changed. Even those paths, I don't think, would be representative of how collections are today. So start with that modification.

This method you gave with -M alone does not work for me. Tree structure:

tree alan
alan
└── ansible_collections
    └── chrismeyersfsu
        └── tower_modules
            ├── FILES.json
            ├── MANIFEST.json
            ├── README.md
            ├── galaxy.yml
            └── plugins
                ├── doc_fragments
                │   └── tower.py
                ├── module_utils
                │   └── ansible_tower.py
                └── modules
                    ├── tower_credential.py
                    ├── tower_credential_type.py
                    ├── tower_group.py
                    ├── tower_host.py
                    ├── tower_inventory.py
                    ├── tower_inventory_source.py
                    ├── tower_job_cancel.py
                    ├── tower_job_launch.py
                    ├── tower_job_list.py
                    ├── tower_job_template.py
                    ├── tower_job_wait.py
                    ├── tower_label.py
                    ├── tower_notification.py
                    ├── tower_organization.py
                    ├── tower_project.py
                    ├── tower_receive.py
                    ├── tower_role.py
                    ├── tower_send.py
                    ├── tower_settings.py
                    ├── tower_team.py
                    ├── tower_user.py
                    ├── tower_workflow_launch.py
                    └── tower_workflow_template.py

7 directories, 29 files

Output:

ansible-doc -M alan/ansible_collections/chrismeyersfsu/tower_modules/plugins/modules/ -t module tower_organization
ERROR! module tower_organization missing documentation (or could not parse documentation): unknown doc_fragment(s) in file /Users/alancoding/Documents/repos/utility-playbooks/alan/ansible_collections/chrismeyersfsu/tower_modules/plugins/modules/tower_organization.py: chrismeyersfsu.tower_modules.tower

After much experimentation I found this:
(note: everything works the same if I rename the alan/ folder to collections/)

ANSIBLE_COLLECTIONS_PATHS=alan ansible-doc -M alan/ansible_collections/chrismeyersfsu/tower_modules/plugins/modules/ -t module tower_organization

This works. This is very important syntax, because otherwise you can't test if you are afflicted by the bug #59226 or not. You can't even know because you can't use it in the first place.

To test that the above command works, I modified the description field in alan/ansible_collections/chrismeyersfsu/tower_modules/plugins/modules/tower_organization.py and then modified tower_host in alan/ansible_collections/chrismeyersfsu/tower_modules/plugins/doc_fragments/tower.py. Both changes successfully took effect. That was true if I used the fully qualified namespace for the docs fragment. If I use the unqualified namespace only the description changes, this command seems to show I am verifying the findings in 59226.

This seems to be a valid workaround for present-day devel, and the only such workaround I can find on the internet right now.

---

Let me fully clarify the details of that structure for anyone looking for instructions on how to do this.

The contents of tower.py in the doc_fragments is exactly the content of https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/doc_fragments/tower.py

The file tower_organization.py has the line in DOCUMENTATION

extends_documentation_fragment: chrismeyersfsu.tower_modules.tower

This is all specific to the tree structure I posted, and constitutes an example of using docs & doc fragments with a collection.

[AlanCoding]

Let me directly state expectations here:

1. If the above folder structure is moved from alan/ to collections/, then ansible-doc -t module tower_organization by itself should work
2. Even with the custom folder name (alan/) the command ANSIBLE_COLLECTIONS_PATHS=alan ansible-doc -t module tower_organization should work
3. Number 2 should work via the -M flag without specifying that environment variable as well.

[bcoca]

The issue is 'listing' (-l option) plugins that are part of a collection, individual plugins when correctly referenced and the collection is in expected paths, have been working for a while now.

The expectation of using shorthand on configured collection paths should not work as there is no way to disambiguate conflicts, i.e tower_org module in 'alan' collection vs the one in 'chris' collection.

-M on a collection will not load collection context, it 'works' if the docs don't require collection specific fragments or the collection is already loaded some other way (env var does this). I don't expect this behaviour to change as referencing the full collection name already does show docs appropriately, this option is mostly kept for 'adjacent' playbook/role X_plugin dirs.

[AlanCoding]

the collection list command was merged, as a spiritual successor to this.

Otherwise I'm still completely unaware of how to use the doc command to list content in a collection. Didn't think that was intended to be supported.