cisco.catalystcenter (successor to the deprecated cisco.dnac)

[bvargasre]

New collection inclusion request: cisco.catalystcenter

Description

cisco.catalystcenter provides Ansible modules, roles, and Cisco Validated
Playbooks (CVP) to automate Cisco Catalyst Center (formerly DNA Center). It
covers device inventory, provisioning, SDA, network settings, assurance,
templates, and day-N operations through the Catalyst Center REST API.

This collection is the direct successor to cisco.dnac, which is deprecated
following Cisco renamed DNA Center to Catalyst Center. This request follows
up on cisco-en-programmability/dnacenter-ansible#327, where the Ansible
community package release managers asked us to either follow the standard
renaming process or submit cisco.catalystcenter for inclusion review and
remove cisco.dnac in Ansible 14.

Backward compatibility with cisco.dnac

To smooth migration for existing cisco.dnac users, we have restored full
backward-compatible aliases (released in 2.10.0):

• All renamed connection options accept their legacy dnac_* names as aliases:
  dnac_host, dnac_port, dnac_username, dnac_password, dnac_verify,
  dnac_version, dnac_debug, dnac_log, dnac_log_level,
  dnac_log_file_path, dnac_log_append, dnac_api_task_timeout,
  dnac_task_poll_interval, plus the user alias on the username option.
• catalystcenter_api_port is kept as an alias of catalystcenter_port.
• The dnac_response return key is still emitted alongside
  catalystcenter_response.

These aliases are present in the shared argument spec, in the inline argument
specs of all workflow_manager and playbook_config_generator modules, and are
documented in the doc fragments. ansible-test sanity (validate-modules)
passes for all the affected modules.

With backward compatibility in place, cisco.dnac can now be removed in favor
of cisco.catalystcenter.

Links

• Ansible Galaxy: https://galaxy.ansible.com/ui/repo/published/cisco/catalystcenter/
• GitHub repository: https://github.com/cisco-en-programmability/catalystcenter-ansible
• Issue tracker: https://github.com/cisco-en-programmability/catalystcenter-ansible/issues
• Documentation: https://cisco-en-programmability.github.io/catalystcenter-ansible/
• Changelog: https://github.com/cisco-en-programmability/catalystcenter-ansible/blob/main/changelogs/changelog.yaml

Maintainers

• @bvargasre
• @madhansansel
• @rishitachowdhary
• @fmunozmiranda
• @DNACENSolutions

Automation Hub:

Yes

Collection Requirements

We meet the Collection Requirements: Yes

Note

Backward compatibility for the cisco.dnac -> cisco.catalystcenter rename is
in place. If anything else is required to proceed with the inclusion review, we
are actively watching this discussion and will address it promptly.

[Andersson007]

@bvargasre hello, thanks for the inclusion request! As soon as someone from @ansible-collections/steering-committee has time, they'll review it

[Andersson007]

Ansible Collections Checklist (short version)

For details about the following points, refer to the Collection Requirements.

Every comment should say whether the reviewer expects it to be addressed, or whether it's optional.

Note for reviewers: If you don't know how to check any of the points below, please ask maintainers of the collection you're reviewing or a Steering Committee member for clarifications in comments of corresponding inclusion discussion.

Public availability and communication:

• published on Ansible Galaxy with version 1.0.0 or later
• have a Code of Conduct (CoC) compatible with the Ansible Code of Conduct (CoC)
• have a publicly available issue tracker that does not require a paid level of service to create an account or view issues
• have a public git repository
• releases of the collection are tagged in its repository

Standards and documentation:

• adheres to semantic versioning
• follows licensing rules
• follows the Ansible documentation standards and the style guide
  MUST FIX: version_added in modules vs collection version mismatch, e.g. in plugins/modules/activities_info.py version_added is 6.18.0 wheres the latest collection version is 2.10.1. Make sure those irrelevant versions are removed/updated all over the collection
• follows development conventions; as well as these other requirements:
  • modules satisfy the concept of idempotency
  • modules that only gather information are named <something>_info
  • modules that return ansible_facts are named <something>_facts and do not return non-facts
  • other modules must not allow querying information using specific state option values, or similar mechanisms (like state=get or state=query). These features should be moved to <something>_info or <something>_fact modules.
  • check_mode is supported in all _info and _facts modules
• supports all Python versions supported by all ansible-core versions its supports. If it does not, read the full guidelines to see if you qualify for an exception and document the unsupported Python versions in the collection README.md and in every module and plugin (or in doc fragments)
  • MUST FIX: It's stated in README that the collection supports Python 3.12 but ansible-core 2.16 supports 3.10 - 3.12. I know that it's an AH certification requirement to support Python 2.12+. In this case, according to the inclusion requirements, you also need to document that the collection supports Python 2.12+ in every module/plugin (can be done via a doc fragment under plugins/doc_fragments)
• only uses the allowed plugin types in the plugins/ directory
• has README.md
• FQCNs are used for all plugins and modules including ansible.builtin. for builtin ones from ansible-core in all their appearances in the documentation, examples, and return sections, and in extends_documentation_fragment:s.
• documentation and return sections use version_added: containing the collection version for which an option, module or plugin was added (except cases when they were added in the very first release of the collection)
• public plugins, roles and playbooks do not use files outside of meta/, plugins/, roles/, and playbooks/

Collection management:

• galaxy.yml having tags field set
• collection dependencies must have a lower bound on the version which is at least 1.0.0, and are all part of the ansible package
• meta/runtime.yml defines the minimal version of Ansible which the collection works with
• has changelog, preferably with changelogs/changelog.yaml
  • Question: is there any reason why you don't generate CHANGELOG.rst with antsibull-changelog while releasing as explained in https://docs.ansible.com/projects/ansible/devel/community/collection_contributors/collection_release_without_branches.html#generating-the-changelog ?
• collection repository should not contain any large objects (binaries) comparatively to the current Galaxy tarball size limit of 20 MB. For example, package installers for testing purposes shouldn't be added.
• collection repository should not contain any unnecessary files like, for example, temporary files. Temporary files should be added to .gitignore.
  • QUESTION: is the catalystcenter.log necessary there? it's empty

Tests:

Note for reviewers: If you don't know how to check the points below, please ask maintainers of the collection you're reviewing how you can do it.

• In most cases, CI is set up via GitHub Actions.
• Check .yml files in the .github/workflows directory. There must be at least the sanity section under jobs containing the ansible-test sanity command running against all supported ansible-core versions that must be also listed there, for example, - stable-2.15, - stable-2.16, - stable-2.17.
• Check workflow runs by clicking the Actions tab in the repository's page - you're interested in Scheduled runs, runs against release commits and runs against pull requests.
• If there are no workflows in the Actions tab, ask the collection maintainers how CI is implemented.

• passed ansible-test sanity
• if test/sanity/ignore*.txt exists, it MUST not contain error codes listed in the list of errors that must not be ignored
• has CI tests up and running against each of the "major versions" of ansible-base/ansible-core that the collection supports
  • MUST FIX: Add 2.17, 2.18 to the .github/workflows/sanity_tests.yml because the collection supports >= 2.16.0; I see you run 2.16.0 itself in certification.yml
• all CI tests MUST run against every pull request
• all CI tests MUST run regularly (nightly, or at least once per week)
• sanity tests MUST run against a commit that releases the collection; if they don't pass, the collection won't be released

[Andersson007]

@bvargasre hello, here's ^ the first review. Please let us know if you have any questions or, if not, please ping us when it's done. Thank you

[bvargasre]

Thanks a lot for the detailed explanation! I'll take some time to go through everything carefully. If I have any questions while reviewing it, I'll be sure to let you know. Thanks again for your help!

[bvargasre]

Hi @Andersson007

I'm rolling out antsibull-nox for CI on collection and I'm about to add support for a new Catalyst Center API/lab version, which will push that modules count even higher. That's surfacing a real problem I wanted your take on.

With the nox matrix in place, sanity against devel was failing, so I dug in with py-spy while it was running. It wasn't hung, it was actively burning CPU and climbing past 3GB of memory in a couple of minutes, stuck inside ansible.constants/PyYAML repeatedly re-parsing ansible-core's own config YAML. That lines up with ansible-test's importer unloading ansible./ansible_collections. modules between each file it checks, so the cost seems to scale with how many files get imported. Separately, sanity for ansible-core 2.20 and 2.21 in the same matrix takes over an hour and failing me, while 2.16-2.18 finish under 10 minutes on the same PR.

I've disabled include_devel for now and I'm turning on change-detection-in-prs, but that doesn't fix full/scheduled runs, and the module count is only going up with the new API version, so I want to actually bring the number down.

My plan: a lot of our modules are pure read-only (_info) siblings of the same resource that only differ in which "view" of the data they return (foo_info, foo_count_info, foo_query_info, foo_summary_analytics_info, foo_trend_analytics_info, foo_top_n_analytics_info, etc.). I want to merge each of those families into one _info module with a new view parameter, and retire the absorbed ones. In our case that's roughly 250 families with 2+ sibling GET modules (a bit over 500 modules total), collapsing down to those ~250, so it cuts somewhere around 300 modules plus their matching action files.

One thing I'm not sure about: would removing the old module names outright be a problem, given the functionality itself isn't actually going away (it's still there, just behind the view parameter on the surviving module), or do we have to go through a full deprecation cycle regardless? My concern is that if a deprecation period is required, the module-count problem would still be sitting there for as long as that cycle lasts, since the old modules would keep existing until the window closes. Part of what I'm trying to do here is actually test whether cutting the count down gets nox's sanity matrix to pass reliably in the first place, or whether more changes would still be needed on top of this either way.

This is a PR where I had the aforementioned problem
cisco-en-programmability/catalystcenter-ansible#59

What would you recommend here?

[Andersson007]

@bvargasre hello, thanks for your questions!

1. On the antsibull-nox, I don't use it much and I'm not its developer. Maybe @felixfontein could give some advice here.

2. On the following:

One thing I'm not sure about: would removing the old module names outright be a problem, given the functionality itself isn't actually going away (it's still there, just behind the view parameter on the surviving module), or do we have to go through a full deprecation cycle regardless?

My opinion w/o going much to details in your particular case: if the change you're going to make is braking for your collection users, i.e. if they have to make any changes to their playbooks too, then we must treat them as breaking and go through a proper deprecation cycle.

[bvargasre]

Hi @Andersson007,

Following up, I'd like your opinion from the Ansible side on a CI problem we hit.

Our sanity, nox and certification jobs all failed the same way, in the import sanity test. ansible-test purges the ansible/ansible_collections namespaces from sys.modules after each file, so every one of our ~1475 action plugins re-executed the whole ansible-core import graph. Memory grew monotonically until the jobs became extremely slow or were killed — the certification Sanity (Ⓐstable-2.20) leg ran 4h22m before being SIGTERMed.

As a workaround I changed the generated action plugin template to build ActionModule on first access instead of importing ansible-core at file import time:

def _build_action_module():
    from ansible.plugins.action import ActionBase
    ...
    class ActionModule(ActionBase):
        ...
    return ActionModule


def __getattr__(name):  # PEP 562
    if name == "ActionModule":
        cls = _build_action_module()
        globals()["ActionModule"] = cls
        return cls
    raise AttributeError(name)

The action loader resolves plugins with getattr(module, "ActionModule"), so runtime behaviour is unchanged and only --test import stops paying the cost. That leg now finishes in ~9 minutes and sanity, nox and certification are all green. The coverage --test import no longer gives us for plugins/action/ moved to a unit test that imports every plugin, builds every class and instantiates it.

Aside from that, most of your July 22 review is also addressed on develop:

• version_added now uses real collection versions everywhere (activities_info.py is 2.2.0).
• The empty catalystcenter.log is gone.
• changelogs/CHANGELOG.rst is generated with antsibull-changelog.
• CI runs sanity on 2.16 through 2.21. Note it's driven by antsibull-nox now, so the versions are derived from requires_ansible rather than listed in sanity_tests.yml.

All of this is on the develop branch and in PR #60, along with the commits and test runs.

I'll probably drop the standalone sanity workflows later and keep only nox and certification, but I'm keeping them for now to validate the change.

I haven't had feedback from the nox side yet, so I'd really value your view as a reviewer: does this lazy-loading approach look acceptable, or would you consider it bad practice for a collection? Everything so far points to a scalability issue caused by the size of the collection.

I'll be out this week — @fmunozmiranda will be watching the discussion and can follow up in the meantime.

Thanks!

[felixfontein]

1. On the antsibull-nox, I don't use it much and I'm not its developer. Maybe @felixfontein could give some advice here.

For the problematic sessions, antsibull-nox simply runs ansible-test, and ansible-test chokes on this pretty huge collection. I would suggest creating an issue in https://github.com/ansible/ansible, or see if @mattclay has some ideas. I don't think antsibull-nox (or any regular collection testing tool) is well suited for such a huge collection (compared to it, community.general is tiny).

[Andersson007]

@bvargasre thanks for fixing the points

1. On the ansible-core related question, I would also suggest contacting core team by opening an issue
2. There are still a few files that have incorrect version added number. Please update

plugins/modules/discovery_workflow_manager.py

• Line 320: version_added: 6.12.0

plugins/modules/network_settings_workflow_manager.py (9 occurrences)

• Line 209: version_added: 6.14.0
• Line 241: version_added: 4.0.0
• Line 287: version_added: 6.14.0
• Line 399: version_added: 6.14.0
• Line 406: version_added: 6.14.0
• Line 413: version_added: 6.14.0
• Line 443: version_added: 6.14.0
• Line 450: version_added: 6.14.0
• Line 457: version_added: 6.14.0

plugins/modules/inventory_workflow_manager.py (3 occurrences)

• Line 537: version_added: 6.12.0
• Line 557: version_added: 6.12.0
• Line 568: version_added: 6.12.0

plugins/modules/swim_workflow_manager.py (4 occurrences)

• Line 457: version_added: 3.1.3.0
• Line 493: version_added: 6.12.0
• Line 572: version_added: 3.1.3.0
• Line 589: version_added: 6.12.0

plugins/modules/template_workflow_manager.py

• Line 673: version_added: 6.17.0

[Andersson007]

There's also a forum discussion whether to include this collection or not https://forum.ansible.com/t/including-large-specialized-collections-in-ansible-community-package/46222 , FYI