Update README to include jmespath and add test for 3rd party libs#396
Merged
Update README to include jmespath and add test for 3rd party libs#396
Conversation
danischm
approved these changes
Nov 7, 2025
aitestino
added a commit
that referenced
this pull request
Jan 20, 2026
* Bump pluggy from 1.5.0 to 1.6.0 (#365) Bumps [pluggy](https://github.com/pytest-dev/pluggy) from 1.5.0 to 1.6.0. - [Changelog](https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pluggy@1.5.0...1.6.0) --- updated-dependencies: - dependency-name: pluggy dependency-version: 1.6.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump click from 8.2.0 to 8.2.1 (#368) Bumps [click](https://github.com/pallets/click) from 8.2.0 to 8.2.1. - [Release notes](https://github.com/pallets/click/releases) - [Changelog](https://github.com/pallets/click/blob/main/CHANGES.rst) - [Commits](pallets/click@8.2.0...8.2.1) --- updated-dependencies: - dependency-name: click dependency-version: 8.2.1 dependency-type: indirect update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump typer from 0.15.3 to 0.16.0 (#371) Bumps [typer](https://github.com/fastapi/typer) from 0.15.3 to 0.16.0. - [Release notes](https://github.com/fastapi/typer/releases) - [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md) - [Commits](fastapi/typer@0.15.3...0.16.0) --- updated-dependencies: - dependency-name: typer dependency-version: 0.16.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump requests from 2.32.3 to 2.32.4 (#372) Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.3...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump urllib3 from 2.4.0 to 2.5.0 (#374) Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.4.0...2.5.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.5.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump rpds-py from 0.24.0 to 0.26.0 (#375) Bumps [rpds-py](https://github.com/crate-py/rpds) from 0.24.0 to 0.26.0. - [Release notes](https://github.com/crate-py/rpds/releases) - [Commits](crate-py/rpds@v0.24.0...v0.26.0) --- updated-dependencies: - dependency-name: rpds-py dependency-version: 0.26.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cryptography from 44.0.3 to 45.0.5 (#376) Bumps [cryptography](https://github.com/pyca/cryptography) from 44.0.3 to 45.0.5. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@44.0.3...45.0.5) --- updated-dependencies: - dependency-name: cryptography dependency-version: 45.0.5 dependency-type: indirect update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Migrate to uv * Refactor error handling * Various linter updates * Dependency updates * Update readme * Update dependencies and pyproject * Fix dependency * Fix robot dependency * Update versions * Bump astral-sh/setup-uv from 6 to 7 (#388) Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 6 to 7. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@v6...v7) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * License header updates * Update changelog * Update dependabot config for uv * Bump nac-yaml version * 1.1.0 version bump * Bump actions/upload-artifact from 4 to 5 (#391) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 5. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4...v5) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix lint workflow for fork prs * Expose pabot's process option to nac-test cli (#393) * Increase robot loglevel to debug when verbosity=DEBUG is used (#394) * Add robotframework-jmespath dependency (#395) * Update README to include jmespath and add test for 3rd party libs (#396) * Add support for test case parallelization for capable suites (#390) * Switch from pabot.main to pabot.main_program and handle exit codes (#400) Change nac-test to use pabot's main_program() instead of main() to avoid sys.exit() calls and properly handle exit codes. This allows nac-test to control the exit behavior and return appropriate exit codes to the caller. Key changes: - Update run_pabot() to call pabot.pabot.main_program() instead of main() - Change run_pabot() return type from None to int - Return exit code from pabot execution - Update CLI main() to capture and use exit code - Remove try/except wrapper in CLI main() - let exceptions propagate - Disable typer pretty exceptions for cleaner error output Benefits: - Proper exit code handling (test failures, errors, etc.) - No unexpected sys.exit() calls - Cleaner error messages without verbose typer tracebacks - Better integration with CI/CD systems 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * Add support for passing Robot Framework options to nac-test (#399) * Add support for passing Robot Framework options to nac-test Users can now pass additional Robot Framework options directly to nac-test, which are validated and appended to the pabot invocation. This enables advanced use cases like custom variables, listeners, and logging configuration without modifying nac-test's core options. Key changes: - Enable extra args in CLI via typer context settings - Add parse_and_validate_extra_args() to validate Robot Framework options - Reject pabot-specific options (--processes, --testlevelsplit, etc.) - Reject datasources/test files in extra arguments - Return exit code 252 for invalid extra arguments - Separate pabot_args and robot_args for proper argument ordering - Add comprehensive unit tests (14 tests covering all scenarios) - Add type hints and fix all mypy errors - Update CLI help text to document extra args feature Implementation details: - Uses pabot's parse_args() to validate options against Robot Framework - Filters out datasources and pabot-specific options - Appends validated args to robot_args before pabot invocation - Proper error handling with logging and exit codes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Remove pabot.main_program changes from robot-args branch This commit removes the pabot-specific changes (main_program, exit codes, exception handling) that belong in the separate pabot-call branch. The robot-args branch now focuses solely on the extra arguments feature. Changes: - Revert run_pabot() return type from int to None - Use pabot.pabot.main() instead of main_program() - Remove exit code handling (no return 252) - Keep try/except in CLI main() for proper error handling - Update tests to use pabot.pabot.main mock - Update tests to expect exceptions instead of error codes The extra args validation and appending logic remains unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Revert "Remove pabot.main_program changes from robot-args branch" This reverts commit df21b98. --------- Co-authored-by: Claude <noreply@anthropic.com> * Add support for chunking templates (#398) * Update changelog and readms * Update lock file * Expand test coverage for chunk split test (#401) * Fix typo in readme link * Remove library version numbers from readme * 1.2.0 version bump * Support rendering on windows (#404) * Bump actions/checkout from 5 to 6 (#402) Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update changelog * 1.2.1 version bump * Bump actions/upload-artifact from 5 to 6 (#418) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v5...v6) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: resolve all ruff linting errors and warnings Fix code quality issues identified by ruff to ensure compliance with project style guidelines and best practices. Changes: - Fix syntax error in pabot.py (space in != operator) - Add missing filecmp import in test_integration.py - Add proper exception chaining (from e) for B904 violations - Rename unused loop variables to prefix with underscore (B007) - Remove trailing whitespace from blank lines (W293) - Modernize Union type annotations to use | operator (UP007) - Remove unused Union import after migration to | syntax - Exclude test fixture directories from ruff checks in pyproject.toml All ruff checks now pass successfully. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix pabot module import in tests/unit/test_pabot_args.py * no longer test for RESTinstance robot lib * add fixture to set bogus ACI_xxx environment for nac-test to pass rendering tests * add/fix --processes feature * fix/add ordering_file logic to robot orchestrator * fix/add passing extra cli args to robot orchestrator * add integration test for extra args * update README.md * fix ruff config changes broken during merge conflict resolution * fix: resolve all mypy type checking errors This commit addresses 11 mypy type errors across 4 files: - connection_utils.py: Add type annotations for dict parameters and update function signature to accept None values that are validated - subprocess_runner.py: Remove unreachable code branch in output processing that was flagged due to redundant None check - connection_broker.py: Add type annotations for testbed attribute, AsyncIterator return type, and assertion for loader result - robot_writer.py: Remove duplicate method definition and add explicit type annotation to prevent incorrect type narrowing All files now pass mypy strict type checking without errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: restore custom_data handling in render_template for chunked templates The refactoring that moved data conversion to initialization (for performance) inadvertently broke the chunked template functionality by not using the custom_data parameter when provided. The render_template method now correctly checks if custom_data is provided (used for chunked templates) and converts it on-demand, otherwise uses the pre-converted self.template_data. This fixes the test_nac_test_list_chunked test which was failing after merging main into the feature branch. Fixes functionality added in commits 18da546 and ced5983. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * ruff formatting fixes * fix: add type stub dependencies to pre-commit mypy hook The pre-commit mypy hook runs in an isolated environment and needs type stubs explicitly specified via additional_dependencies. Without these, mypy complains about missing type information for yaml, aiofiles, and markdown modules. Added: - types-PyYAML>=6.0.12.20250822 - types-aiofiles>=24.1.0.20250822 - types-markdown>=3.8.0.20250809 This aligns the pre-commit environment with the project dependencies and allows mypy to properly type-check imports from these libraries. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: add type annotations to unit test functions Add proper type annotations to all test functions in tests/unit to resolve mypy strict type checking errors. Changes include: - Add -> None return type annotations to all test methods - Add type annotations for mock parameters (mock_logger, mock_connection_class, etc.) using typing.Any - Add explicit type annotation for connections list variable - Add type: ignore comment for intentional type mismatch in test This ensures tests pass mypy strict type checking while maintaining test clarity and avoiding over-specification of mock types. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller tests from shell environment variables Add autouse fixture to clear all controller-related environment variables (ACI_*, SDWAN_*, CC_*, MERAKI_*, FMC_*, ISE_*) before each test in TestCombinedOrchestratorController. This ensures tests run in a clean environment and results are not influenced by the caller's shell environment, preventing false failures when developer has controller credentials set in their shell. The fixture uses pytest's monkeypatch to safely remove environment variables only for the duration of each test. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: migrate integration test fixture to use monkeypatch Replace manual environment variable setup/teardown with pytest's monkeypatch fixture for safer and more robust test isolation. Benefits of monkeypatch approach: - Automatic cleanup guaranteed even if test fails - Safe deletion with raising=False (no KeyError if var missing) - Preserves and restores original environment variable values - Less boilerplate code (no yield/try-finally needed) - Thread-safe state management handled by pytest The fixture now uses monkeypatch.setenv() which automatically saves the original value (if any) and restores it after the test, ensuring the caller's shell environment is never permanently modified. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller detection integration tests from shell environment Add autouse fixture to clear all controller-related environment variables before each test in TestControllerDetectionIntegration. This prevents test failures when developers have controller credentials set in their shell environment. Without this fix, tests would fail with "Multiple controller credentials detected" errors when shell variables conflicted with test-specific environment setup. The fixture uses pytest's monkeypatch.delenv() with raising=False for safe deletion that won't fail if a variable doesn't exist. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * add license headers * add missing test fixture template for extra args test * security: replace hardcoded /tmp with tempfile.gettempdir() in ConnectionBroker Fix Bandit B108 vulnerability by using tempfile.gettempdir() instead of hardcoded /tmp path for output_dir default value. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: add usedforsecurity=False to MD5 hash in AuthCache Fix Bandit B324 vulnerability by explicitly marking MD5 usage as not for security purposes. The hash is only used to generate cache filenames from URLs, not for cryptographic security. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp references in emergency dump Fix Bandit B108 vulnerabilities by: - Using tempfile.gettempdir() instead of hardcoded /tmp path - Dynamically checking if file is in temp directory instead of string matching "/tmp/" - Updated docstrings and log messages to reflect system temp dir usage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in AUTH_CACHE_DIR constant Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for AUTH_CACHE_DIR. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in batching reporter overflow dir Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for default overflow directory. This ensures cross-platform compatibility while still allowing override via NAC_TEST_OVERFLOW_DIR environment variable. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: enable Jinja2 autoescape to prevent XSS vulnerabilities Fix Bandit B701 vulnerability by enabling autoescape for HTML, XML, and .j2 files using select_autoescape(). This prevents potential XSS attacks when rendering user-controlled data in HTML reports. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * gitignore bandit-security-report.json created during pipeline run * Update readme cli help output * rename robot/pabot integration test file * Add netconf related packages and update dependencies * 1.2.2 version bump * refactor: centralize DEBUG_MODE constant for progressive disclosure Consolidates the NAC_TEST_DEBUG environment variable check into a single constant in core/constants.py, eliminating repeated inline checks. Changes: - Add DEBUG_MODE constant to nac_test/core/constants.py - Update cli/main.py to use DEBUG_MODE for pretty_exceptions_enable - Update combined_orchestrator.py to use DEBUG_MODE for exception handling - Update batching_reporter.py to use DEBUG_MODE (2 locations) This enables progressive disclosure: customers get clean error output by default, while developers can set NAC_TEST_DEBUG=true for full exception context and verbose tracebacks when debugging. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: danischm <danischm@cisco.com> Co-authored-by: Oliver Boehmer <oli@spine.de> Co-authored-by: Oliver Boehmer <oboehmer@cisco.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Justyna Chowaniec <79261946+juchowan@users.noreply.github.com>
aitestino
added a commit
that referenced
this pull request
Jan 21, 2026
* Bump pluggy from 1.5.0 to 1.6.0 (#365) Bumps [pluggy](https://github.com/pytest-dev/pluggy) from 1.5.0 to 1.6.0. - [Changelog](https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pluggy@1.5.0...1.6.0) --- updated-dependencies: - dependency-name: pluggy dependency-version: 1.6.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump click from 8.2.0 to 8.2.1 (#368) Bumps [click](https://github.com/pallets/click) from 8.2.0 to 8.2.1. - [Release notes](https://github.com/pallets/click/releases) - [Changelog](https://github.com/pallets/click/blob/main/CHANGES.rst) - [Commits](pallets/click@8.2.0...8.2.1) --- updated-dependencies: - dependency-name: click dependency-version: 8.2.1 dependency-type: indirect update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump typer from 0.15.3 to 0.16.0 (#371) Bumps [typer](https://github.com/fastapi/typer) from 0.15.3 to 0.16.0. - [Release notes](https://github.com/fastapi/typer/releases) - [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md) - [Commits](fastapi/typer@0.15.3...0.16.0) --- updated-dependencies: - dependency-name: typer dependency-version: 0.16.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump requests from 2.32.3 to 2.32.4 (#372) Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.3...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump urllib3 from 2.4.0 to 2.5.0 (#374) Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.4.0...2.5.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.5.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump rpds-py from 0.24.0 to 0.26.0 (#375) Bumps [rpds-py](https://github.com/crate-py/rpds) from 0.24.0 to 0.26.0. - [Release notes](https://github.com/crate-py/rpds/releases) - [Commits](crate-py/rpds@v0.24.0...v0.26.0) --- updated-dependencies: - dependency-name: rpds-py dependency-version: 0.26.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cryptography from 44.0.3 to 45.0.5 (#376) Bumps [cryptography](https://github.com/pyca/cryptography) from 44.0.3 to 45.0.5. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@44.0.3...45.0.5) --- updated-dependencies: - dependency-name: cryptography dependency-version: 45.0.5 dependency-type: indirect update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Migrate to uv * Refactor error handling * Various linter updates * Dependency updates * Update readme * Update dependencies and pyproject * Fix dependency * Fix robot dependency * Update versions * Bump astral-sh/setup-uv from 6 to 7 (#388) Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 6 to 7. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@v6...v7) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * License header updates * Update changelog * Update dependabot config for uv * Bump nac-yaml version * 1.1.0 version bump * Bump actions/upload-artifact from 4 to 5 (#391) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 5. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4...v5) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix lint workflow for fork prs * Expose pabot's process option to nac-test cli (#393) * Increase robot loglevel to debug when verbosity=DEBUG is used (#394) * Add robotframework-jmespath dependency (#395) * Update README to include jmespath and add test for 3rd party libs (#396) * Add support for test case parallelization for capable suites (#390) * Switch from pabot.main to pabot.main_program and handle exit codes (#400) Change nac-test to use pabot's main_program() instead of main() to avoid sys.exit() calls and properly handle exit codes. This allows nac-test to control the exit behavior and return appropriate exit codes to the caller. Key changes: - Update run_pabot() to call pabot.pabot.main_program() instead of main() - Change run_pabot() return type from None to int - Return exit code from pabot execution - Update CLI main() to capture and use exit code - Remove try/except wrapper in CLI main() - let exceptions propagate - Disable typer pretty exceptions for cleaner error output Benefits: - Proper exit code handling (test failures, errors, etc.) - No unexpected sys.exit() calls - Cleaner error messages without verbose typer tracebacks - Better integration with CI/CD systems 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * Add support for passing Robot Framework options to nac-test (#399) * Add support for passing Robot Framework options to nac-test Users can now pass additional Robot Framework options directly to nac-test, which are validated and appended to the pabot invocation. This enables advanced use cases like custom variables, listeners, and logging configuration without modifying nac-test's core options. Key changes: - Enable extra args in CLI via typer context settings - Add parse_and_validate_extra_args() to validate Robot Framework options - Reject pabot-specific options (--processes, --testlevelsplit, etc.) - Reject datasources/test files in extra arguments - Return exit code 252 for invalid extra arguments - Separate pabot_args and robot_args for proper argument ordering - Add comprehensive unit tests (14 tests covering all scenarios) - Add type hints and fix all mypy errors - Update CLI help text to document extra args feature Implementation details: - Uses pabot's parse_args() to validate options against Robot Framework - Filters out datasources and pabot-specific options - Appends validated args to robot_args before pabot invocation - Proper error handling with logging and exit codes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Remove pabot.main_program changes from robot-args branch This commit removes the pabot-specific changes (main_program, exit codes, exception handling) that belong in the separate pabot-call branch. The robot-args branch now focuses solely on the extra arguments feature. Changes: - Revert run_pabot() return type from int to None - Use pabot.pabot.main() instead of main_program() - Remove exit code handling (no return 252) - Keep try/except in CLI main() for proper error handling - Update tests to use pabot.pabot.main mock - Update tests to expect exceptions instead of error codes The extra args validation and appending logic remains unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Revert "Remove pabot.main_program changes from robot-args branch" This reverts commit df21b98. --------- Co-authored-by: Claude <noreply@anthropic.com> * Add support for chunking templates (#398) * Update changelog and readms * Update lock file * Expand test coverage for chunk split test (#401) * Fix typo in readme link * Remove library version numbers from readme * 1.2.0 version bump * feat(path_setup): auto-discover pyats_common for test imports Add find_pyats_common_parent() to search test directory tree for pyats_common subdirectory and add its parent to PYTHONPATH. This enables test files to use simple imports like: from pyats_common.apic_base_test import APICTestBase 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * add verbose option to pyats when running at DEBUG level, log pyats output as well when in DEBUG * current state, not yet working * add simple test for pipeline - WIP * add integration tests and support for mock devices * use easypy.run to invoke script * do not assume a default CONTROLLER_TYPE * remove assumption for CONTROLLER_TYPE being ACI in base_test too * adjust test * add some debug log statements * add warning about potential issues with python3.11 on macOS * connect using log_stdout=False * fix simple test, and ruff formatting * simple test changes, WIP * Support rendering on windows (#404) * Bump actions/checkout from 5 to 6 (#402) Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update changelog * 1.2.1 version bump * update with mock flask server - WIP * remove debug logs * add mock-server to perform API testing in integration tests * add integration tests for quicksilver-generated templates * add type hints * Bump actions/upload-artifact from 5 to 6 (#418) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v5...v6) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add sdwan mock api results * adjust mock-server file, create mock-server test * fix: resolve all ruff linting errors and warnings Fix code quality issues identified by ruff to ensure compliance with project style guidelines and best practices. Changes: - Fix syntax error in pabot.py (space in != operator) - Add missing filecmp import in test_integration.py - Add proper exception chaining (from e) for B904 violations - Rename unused loop variables to prefix with underscore (B007) - Remove trailing whitespace from blank lines (W293) - Modernize Union type annotations to use | operator (UP007) - Remove unused Union import after migration to | syntax - Exclude test fixture directories from ruff checks in pyproject.toml All ruff checks now pass successfully. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix pabot module import in tests/unit/test_pabot_args.py * no longer test for RESTinstance robot lib * add fixture to set bogus ACI_xxx environment for nac-test to pass rendering tests * add/fix --processes feature * fix/add ordering_file logic to robot orchestrator * fix/add passing extra cli args to robot orchestrator * add integration test for extra args * update README.md * fix ruff config changes broken during merge conflict resolution * fix: resolve all mypy type checking errors This commit addresses 11 mypy type errors across 4 files: - connection_utils.py: Add type annotations for dict parameters and update function signature to accept None values that are validated - subprocess_runner.py: Remove unreachable code branch in output processing that was flagged due to redundant None check - connection_broker.py: Add type annotations for testbed attribute, AsyncIterator return type, and assertion for loader result - robot_writer.py: Remove duplicate method definition and add explicit type annotation to prevent incorrect type narrowing All files now pass mypy strict type checking without errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: restore custom_data handling in render_template for chunked templates The refactoring that moved data conversion to initialization (for performance) inadvertently broke the chunked template functionality by not using the custom_data parameter when provided. The render_template method now correctly checks if custom_data is provided (used for chunked templates) and converts it on-demand, otherwise uses the pre-converted self.template_data. This fixes the test_nac_test_list_chunked test which was failing after merging main into the feature branch. Fixes functionality added in commits 18da546 and ced5983. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * ruff formatting fixes * fix: add type stub dependencies to pre-commit mypy hook The pre-commit mypy hook runs in an isolated environment and needs type stubs explicitly specified via additional_dependencies. Without these, mypy complains about missing type information for yaml, aiofiles, and markdown modules. Added: - types-PyYAML>=6.0.12.20250822 - types-aiofiles>=24.1.0.20250822 - types-markdown>=3.8.0.20250809 This aligns the pre-commit environment with the project dependencies and allows mypy to properly type-check imports from these libraries. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: add type annotations to unit test functions Add proper type annotations to all test functions in tests/unit to resolve mypy strict type checking errors. Changes include: - Add -> None return type annotations to all test methods - Add type annotations for mock parameters (mock_logger, mock_connection_class, etc.) using typing.Any - Add explicit type annotation for connections list variable - Add type: ignore comment for intentional type mismatch in test This ensures tests pass mypy strict type checking while maintaining test clarity and avoiding over-specification of mock types. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller tests from shell environment variables Add autouse fixture to clear all controller-related environment variables (ACI_*, SDWAN_*, CC_*, MERAKI_*, FMC_*, ISE_*) before each test in TestCombinedOrchestratorController. This ensures tests run in a clean environment and results are not influenced by the caller's shell environment, preventing false failures when developer has controller credentials set in their shell. The fixture uses pytest's monkeypatch to safely remove environment variables only for the duration of each test. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: migrate integration test fixture to use monkeypatch Replace manual environment variable setup/teardown with pytest's monkeypatch fixture for safer and more robust test isolation. Benefits of monkeypatch approach: - Automatic cleanup guaranteed even if test fails - Safe deletion with raising=False (no KeyError if var missing) - Preserves and restores original environment variable values - Less boilerplate code (no yield/try-finally needed) - Thread-safe state management handled by pytest The fixture now uses monkeypatch.setenv() which automatically saves the original value (if any) and restores it after the test, ensuring the caller's shell environment is never permanently modified. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller detection integration tests from shell environment Add autouse fixture to clear all controller-related environment variables before each test in TestControllerDetectionIntegration. This prevents test failures when developers have controller credentials set in their shell environment. Without this fix, tests would fail with "Multiple controller credentials detected" errors when shell variables conflicted with test-specific environment setup. The fixture uses pytest's monkeypatch.delenv() with raising=False for safe deletion that won't fail if a variable doesn't exist. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * add license headers * add missing test fixture template for extra args test * security: replace hardcoded /tmp with tempfile.gettempdir() in ConnectionBroker Fix Bandit B108 vulnerability by using tempfile.gettempdir() instead of hardcoded /tmp path for output_dir default value. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: add usedforsecurity=False to MD5 hash in AuthCache Fix Bandit B324 vulnerability by explicitly marking MD5 usage as not for security purposes. The hash is only used to generate cache filenames from URLs, not for cryptographic security. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp references in emergency dump Fix Bandit B108 vulnerabilities by: - Using tempfile.gettempdir() instead of hardcoded /tmp path - Dynamically checking if file is in temp directory instead of string matching "/tmp/" - Updated docstrings and log messages to reflect system temp dir usage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in AUTH_CACHE_DIR constant Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for AUTH_CACHE_DIR. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in batching reporter overflow dir Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for default overflow directory. This ensures cross-platform compatibility while still allowing override via NAC_TEST_OVERFLOW_DIR environment variable. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: enable Jinja2 autoescape to prevent XSS vulnerabilities Fix Bandit B701 vulnerability by enabling autoescape for HTML, XML, and .j2 files using select_autoescape(). This prevents potential XSS attacks when rendering user-controlled data in HTML reports. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * gitignore bandit-security-report.json created during pipeline run * Update readme cli help output * rename robot/pabot integration test file * Add netconf related packages and update dependencies * 1.2.2 version bump * adjust and fix aci integration tests * remove environment vars after testing * fix license check * remove unfinished tests * use tmpdir * add debugging for CI troubleshooting * attempt to fix event loop issue in CI * add more debug code to troubleshoot CI failure * add missing nac-test-pyats-common to CI pipeline * sum up passed/failed for check (in case we have multiple results.json files) * re-organize test case fixture directories use a common base dir, and then add specific archs as subdirectories to avoid directory bloat in fixture directory * Fix Python 3.12 asyncio compatibility with get_or_create_event_loop Python 3.12 changed asyncio.get_event_loop() behavior to raise RuntimeError when called from the main thread without an active event loop, breaking async operations that run synchronous PyATS/Unicon calls via loop.run_in_executor(). Previous behavior (Python 3.10-3.11): - asyncio.get_event_loop() automatically created a new event loop if none existed New behavior (Python 3.12+): - asyncio.get_event_loop() raises RuntimeError if no event loop exists - Must explicitly call asyncio.new_event_loop() and set_event_loop() Solution: Created get_or_create_event_loop() helper that handles both behaviors: - Tries to get existing event loop - If closed or RuntimeError, creates and sets a new loop - Works across Python 3.10-3.12+ Replaced 9 instances of asyncio.get_event_loop() across 4 modules: - nac_test/pyats_core/broker/connection_broker.py (3 calls) - nac_test/pyats_core/common/ssh_base_test.py (3 calls) - nac_test/pyats_core/ssh/connection_manager.py (2 calls) - nac_test/pyats_core/reporting/multi_archive_generator.py (1 call) These calls are used to run blocking Unicon operations (connect, disconnect, execute) in thread pools via run_in_executor() to prevent blocking the async event loop. * add sdwan integration tests (both api and d2d) * add license header * test: remove test_mock_server.py - tests mock infrastructure, not business logic These tests only verify that the MockAPIServer returns what it was configured to return: - test_mock_api_server_endpoint_matching: configures endpoints, asserts they return configured data - test_nac_test_with_mock_api_complex_urls: same pattern with complex URLs - test_nac_test_with_mock_api_dynamic: adds endpoint with response X, asserts response is X This violates the principle: "What MUST NEVER be Tested: whether mocks return what you told them to return." The mock server is infrastructure to enable testing real business logic, not something that needs its own test coverage. * test: remove test_controller_detection_consistency - tests Python determinism This test calls detect_controller_type() three times with identical environment and asserts all results are equal. This tests whether Python functions return consistent results when called with the same inputs - which is testing Python's deterministic execution, not business logic. A pure function will always return the same result for the same inputs. Testing this adds no value unless there's caching or stateful behavior that could cause inconsistency (there isn't). Violates: "What MUST NEVER be Tested: Whether Python's standard library works" * test: remove mock assertion from test_end_to_end_controller_detection Removed the pattern that mocks EnvironmentValidator and asserts it was called with the correct argument: with patch("...EnvironmentValidator") as mock_validator: orchestrator.validate_environment() mock_validator.validate_controller_env.assert_called_once_with("SDWAN") This tests whether validate_environment() calls EnvironmentValidator with the right argument - essentially testing "whether one-line wrapper functions call the functions they wrap." The real test is already done: assert orchestrator.controller_type == "SDWAN" which validates the business logic (controller was correctly detected). Violates: "What MUST NEVER be Tested: whether one-line wrapper functions call the functions they wrap" * fix(tests): remove conflicting copyright header and fix resource leak mock_unicon.py changes: - Removed Cisco proprietary copyright header (lines 5-7), keeping only MPL-2.0 - Fixed resource leak: file opened without context manager now uses 'with' statement Before: states = yaml.safe_load(open(os.path.join(mock_data_dir, file))) or [] After: with open(os.path.join(mock_data_dir, file)) as f: states = yaml.safe_load(f) or [] * fix(tests): implement proper server startup and shutdown for MockAPIServer mock_server.py changes: - Replaced time.sleep(0.5) race condition with proper readiness polling - Added _wait_for_server_ready() that polls until server responds or timeout - Implemented proper stop() method using werkzeug.serving.make_server - Added reset_endpoints() method to clear dynamic endpoints while preserving YAML-loaded baseline (enables test isolation with session-scoped fixture) - Added SERVER_STARTUP_TIMEOUT_SECONDS and SERVER_POLL_INTERVAL_SECONDS constants This fixes flaky test failures caused by server not being ready, and prevents port conflicts from improper shutdown. * fix(tests): add endpoint reset and isolated fixture for test isolation conftest.py changes: - Updated mock_api_server fixture to call reset_endpoints() after yield - Added mock_api_server_isolated fixture (function-scoped) for tests that add dynamic endpoints, preventing cross-test pollution The session-scoped mock_api_server now resets to baseline state between tests, while mock_api_server_isolated provides per-test isolation. * fix(tests): use absolute path and fix PEP8 naming in test_pyats_standard test_pyats_standard.py changes: - Fixed hardcoded relative path that breaks when run from different directories Now uses Path(__file__).parent to construct absolute path to mock_unicon.py - Renamed classes tc_one -> TcOne, tc_two -> TcTwo (PEP8 CamelCase) - Fixed decorator spacing: '@ aetest.test' -> '@aetest.test' * fix(tests): replace debug prints with logging and use monkeypatch test_integration_pyats.py changes: - Replaced all print() debug statements with logger.debug() calls - Moved 'import json' from inside function to module level - Refactored environment variable handling to use monkeypatch.setenv() instead of manual os.environ manipulation with try/finally - Removed commented-out debug code for static output directories This improves test maintainability and ensures proper cleanup via pytest's monkeypatch fixture. * fix(tests): modernize type hints and use monkeypatch in robot_pabot tests test_integration_robot_pabot.py changes: - Changed all tmpdir: str parameters to tmp_path: Path (modern pytest fixture) - Refactored environment variable handling to use monkeypatch.setenv() instead of manual os.environ with try/finally cleanup - Updated fixture references from tmpdir to tmp_path throughout - Used Path methods (.exists(), .touch()) instead of os.path equivalents This improves type safety and ensures proper test isolation via pytest's automatic cleanup of monkeypatched environment variables. * fix(tests): replace obfuscated code and emojis in PyATS quicksilver tests templates_pyats_qs test changes (3 files): - verify_sdwanmanager_all_sd_wan_edge_configurations_are_in_sync.py - verify_iosxe_all_sd_wan_control_connections_are_up.py - verify_aci_apic_appliance_operational_status.py Changes applied to all: - Replaced chr(10) with '\n' for clarity - Replaced emoji markers with text: ❌ -> [FAIL], ✅ -> [PASS] This prevents encoding issues in CI logs and improves code readability. * fix(tests): extract f-string join to variable for Python 3.11 compatibility Python 3.11 does not allow backslash escapes inside f-string expressions. The previous commit replaced chr(10) with '\n' but left it inside the f-string expression, causing a SyntaxError. This fix extracts the join operation to a separate variable before using it in the f-string: failures_text = '\n'.join(failures) f"{failures_text}\n\n" * fix(tests): add proxy bypass for localhost in integration tests Add session-scoped autouse fixture to ensure 127.0.0.1 is included in the no_proxy/NO_PROXY environment variables. This prevents the mock API server requests from being routed through corporate proxies, which was causing 504 Gateway Timeout errors in environments with proxy configured. The fixture preserves original proxy settings and restores them after the test session completes. * refactor(tests): split robot_pabot tests into focused modules Split the 493-line test_integration_robot_pabot.py into 4 focused test modules following Single Responsibility Principle: - test_cli_basic.py: Basic CLI execution tests (6 tests) - test_cli_rendering.py: Template rendering tests (8 tests) - test_cli_ordering.py: Robot test ordering tests (4 tests) - test_cli_extra_args.py: Extra arguments tests (5 tests) Improvements: - Added Google-style docstrings to all 23 test functions - Renamed generic test names to descriptive names - Added assertion messages to all assert statements - Extracted magic number to named constant (ROBOT_ARGUMENT_ERROR_EXIT_CODE) * add management_ip_variable to test fixture data --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: danischm <danischm@cisco.com> Co-authored-by: Oliver Boehmer <oli@spine.de> Co-authored-by: Oliver Boehmer <oboehmer@cisco.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Justyna Chowaniec <79261946+juchowan@users.noreply.github.com>
oboehmer
added a commit
that referenced
this pull request
Mar 19, 2026
oboehmer
added a commit
that referenced
this pull request
Mar 19, 2026
* Bump pluggy from 1.5.0 to 1.6.0 (#365) Bumps [pluggy](https://github.com/pytest-dev/pluggy) from 1.5.0 to 1.6.0. - [Changelog](https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pluggy@1.5.0...1.6.0) --- updated-dependencies: - dependency-name: pluggy dependency-version: 1.6.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump click from 8.2.0 to 8.2.1 (#368) Bumps [click](https://github.com/pallets/click) from 8.2.0 to 8.2.1. - [Release notes](https://github.com/pallets/click/releases) - [Changelog](https://github.com/pallets/click/blob/main/CHANGES.rst) - [Commits](pallets/click@8.2.0...8.2.1) --- updated-dependencies: - dependency-name: click dependency-version: 8.2.1 dependency-type: indirect update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump typer from 0.15.3 to 0.16.0 (#371) Bumps [typer](https://github.com/fastapi/typer) from 0.15.3 to 0.16.0. - [Release notes](https://github.com/fastapi/typer/releases) - [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md) - [Commits](fastapi/typer@0.15.3...0.16.0) --- updated-dependencies: - dependency-name: typer dependency-version: 0.16.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump requests from 2.32.3 to 2.32.4 (#372) Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.3...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump urllib3 from 2.4.0 to 2.5.0 (#374) Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.4.0...2.5.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.5.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump rpds-py from 0.24.0 to 0.26.0 (#375) Bumps [rpds-py](https://github.com/crate-py/rpds) from 0.24.0 to 0.26.0. - [Release notes](https://github.com/crate-py/rpds/releases) - [Commits](crate-py/rpds@v0.24.0...v0.26.0) --- updated-dependencies: - dependency-name: rpds-py dependency-version: 0.26.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cryptography from 44.0.3 to 45.0.5 (#376) Bumps [cryptography](https://github.com/pyca/cryptography) from 44.0.3 to 45.0.5. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@44.0.3...45.0.5) --- updated-dependencies: - dependency-name: cryptography dependency-version: 45.0.5 dependency-type: indirect update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Migrate to uv * Refactor error handling * Various linter updates * Dependency updates * Update readme * Update dependencies and pyproject * Fix dependency * Fix robot dependency * Update versions * Bump astral-sh/setup-uv from 6 to 7 (#388) Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 6 to 7. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@v6...v7) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * License header updates * Update changelog * Update dependabot config for uv * Bump nac-yaml version * 1.1.0 version bump * Bump actions/upload-artifact from 4 to 5 (#391) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 5. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4...v5) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix lint workflow for fork prs * Expose pabot's process option to nac-test cli (#393) * Increase robot loglevel to debug when verbosity=DEBUG is used (#394) * Add robotframework-jmespath dependency (#395) * Update README to include jmespath and add test for 3rd party libs (#396) * Add support for test case parallelization for capable suites (#390) * Switch from pabot.main to pabot.main_program and handle exit codes (#400) Change nac-test to use pabot's main_program() instead of main() to avoid sys.exit() calls and properly handle exit codes. This allows nac-test to control the exit behavior and return appropriate exit codes to the caller. Key changes: - Update run_pabot() to call pabot.pabot.main_program() instead of main() - Change run_pabot() return type from None to int - Return exit code from pabot execution - Update CLI main() to capture and use exit code - Remove try/except wrapper in CLI main() - let exceptions propagate - Disable typer pretty exceptions for cleaner error output Benefits: - Proper exit code handling (test failures, errors, etc.) - No unexpected sys.exit() calls - Cleaner error messages without verbose typer tracebacks - Better integration with CI/CD systems 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * Add support for passing Robot Framework options to nac-test (#399) * Add support for passing Robot Framework options to nac-test Users can now pass additional Robot Framework options directly to nac-test, which are validated and appended to the pabot invocation. This enables advanced use cases like custom variables, listeners, and logging configuration without modifying nac-test's core options. Key changes: - Enable extra args in CLI via typer context settings - Add parse_and_validate_extra_args() to validate Robot Framework options - Reject pabot-specific options (--processes, --testlevelsplit, etc.) - Reject datasources/test files in extra arguments - Return exit code 252 for invalid extra arguments - Separate pabot_args and robot_args for proper argument ordering - Add comprehensive unit tests (14 tests covering all scenarios) - Add type hints and fix all mypy errors - Update CLI help text to document extra args feature Implementation details: - Uses pabot's parse_args() to validate options against Robot Framework - Filters out datasources and pabot-specific options - Appends validated args to robot_args before pabot invocation - Proper error handling with logging and exit codes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Remove pabot.main_program changes from robot-args branch This commit removes the pabot-specific changes (main_program, exit codes, exception handling) that belong in the separate pabot-call branch. The robot-args branch now focuses solely on the extra arguments feature. Changes: - Revert run_pabot() return type from int to None - Use pabot.pabot.main() instead of main_program() - Remove exit code handling (no return 252) - Keep try/except in CLI main() for proper error handling - Update tests to use pabot.pabot.main mock - Update tests to expect exceptions instead of error codes The extra args validation and appending logic remains unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Revert "Remove pabot.main_program changes from robot-args branch" This reverts commit df21b98. --------- Co-authored-by: Claude <noreply@anthropic.com> * Add support for chunking templates (#398) * Update changelog and readms * Update lock file * Expand test coverage for chunk split test (#401) * Fix typo in readme link * Remove library version numbers from readme * 1.2.0 version bump * Support rendering on windows (#404) * Bump actions/checkout from 5 to 6 (#402) Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update changelog * 1.2.1 version bump * Bump actions/upload-artifact from 5 to 6 (#418) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v5...v6) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: resolve all ruff linting errors and warnings Fix code quality issues identified by ruff to ensure compliance with project style guidelines and best practices. Changes: - Fix syntax error in pabot.py (space in != operator) - Add missing filecmp import in test_integration.py - Add proper exception chaining (from e) for B904 violations - Rename unused loop variables to prefix with underscore (B007) - Remove trailing whitespace from blank lines (W293) - Modernize Union type annotations to use | operator (UP007) - Remove unused Union import after migration to | syntax - Exclude test fixture directories from ruff checks in pyproject.toml All ruff checks now pass successfully. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix pabot module import in tests/unit/test_pabot_args.py * no longer test for RESTinstance robot lib * add fixture to set bogus ACI_xxx environment for nac-test to pass rendering tests * add/fix --processes feature * fix/add ordering_file logic to robot orchestrator * fix/add passing extra cli args to robot orchestrator * add integration test for extra args * update README.md * fix ruff config changes broken during merge conflict resolution * fix: resolve all mypy type checking errors This commit addresses 11 mypy type errors across 4 files: - connection_utils.py: Add type annotations for dict parameters and update function signature to accept None values that are validated - subprocess_runner.py: Remove unreachable code branch in output processing that was flagged due to redundant None check - connection_broker.py: Add type annotations for testbed attribute, AsyncIterator return type, and assertion for loader result - robot_writer.py: Remove duplicate method definition and add explicit type annotation to prevent incorrect type narrowing All files now pass mypy strict type checking without errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: restore custom_data handling in render_template for chunked templates The refactoring that moved data conversion to initialization (for performance) inadvertently broke the chunked template functionality by not using the custom_data parameter when provided. The render_template method now correctly checks if custom_data is provided (used for chunked templates) and converts it on-demand, otherwise uses the pre-converted self.template_data. This fixes the test_nac_test_list_chunked test which was failing after merging main into the feature branch. Fixes functionality added in commits 18da546 and ced5983. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * ruff formatting fixes * fix: add type stub dependencies to pre-commit mypy hook The pre-commit mypy hook runs in an isolated environment and needs type stubs explicitly specified via additional_dependencies. Without these, mypy complains about missing type information for yaml, aiofiles, and markdown modules. Added: - types-PyYAML>=6.0.12.20250822 - types-aiofiles>=24.1.0.20250822 - types-markdown>=3.8.0.20250809 This aligns the pre-commit environment with the project dependencies and allows mypy to properly type-check imports from these libraries. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: add type annotations to unit test functions Add proper type annotations to all test functions in tests/unit to resolve mypy strict type checking errors. Changes include: - Add -> None return type annotations to all test methods - Add type annotations for mock parameters (mock_logger, mock_connection_class, etc.) using typing.Any - Add explicit type annotation for connections list variable - Add type: ignore comment for intentional type mismatch in test This ensures tests pass mypy strict type checking while maintaining test clarity and avoiding over-specification of mock types. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller tests from shell environment variables Add autouse fixture to clear all controller-related environment variables (ACI_*, SDWAN_*, CC_*, MERAKI_*, FMC_*, ISE_*) before each test in TestCombinedOrchestratorController. This ensures tests run in a clean environment and results are not influenced by the caller's shell environment, preventing false failures when developer has controller credentials set in their shell. The fixture uses pytest's monkeypatch to safely remove environment variables only for the duration of each test. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: migrate integration test fixture to use monkeypatch Replace manual environment variable setup/teardown with pytest's monkeypatch fixture for safer and more robust test isolation. Benefits of monkeypatch approach: - Automatic cleanup guaranteed even if test fails - Safe deletion with raising=False (no KeyError if var missing) - Preserves and restores original environment variable values - Less boilerplate code (no yield/try-finally needed) - Thread-safe state management handled by pytest The fixture now uses monkeypatch.setenv() which automatically saves the original value (if any) and restores it after the test, ensuring the caller's shell environment is never permanently modified. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller detection integration tests from shell environment Add autouse fixture to clear all controller-related environment variables before each test in TestControllerDetectionIntegration. This prevents test failures when developers have controller credentials set in their shell environment. Without this fix, tests would fail with "Multiple controller credentials detected" errors when shell variables conflicted with test-specific environment setup. The fixture uses pytest's monkeypatch.delenv() with raising=False for safe deletion that won't fail if a variable doesn't exist. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * add license headers * add missing test fixture template for extra args test * security: replace hardcoded /tmp with tempfile.gettempdir() in ConnectionBroker Fix Bandit B108 vulnerability by using tempfile.gettempdir() instead of hardcoded /tmp path for output_dir default value. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: add usedforsecurity=False to MD5 hash in AuthCache Fix Bandit B324 vulnerability by explicitly marking MD5 usage as not for security purposes. The hash is only used to generate cache filenames from URLs, not for cryptographic security. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp references in emergency dump Fix Bandit B108 vulnerabilities by: - Using tempfile.gettempdir() instead of hardcoded /tmp path - Dynamically checking if file is in temp directory instead of string matching "/tmp/" - Updated docstrings and log messages to reflect system temp dir usage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in AUTH_CACHE_DIR constant Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for AUTH_CACHE_DIR. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in batching reporter overflow dir Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for default overflow directory. This ensures cross-platform compatibility while still allowing override via NAC_TEST_OVERFLOW_DIR environment variable. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: enable Jinja2 autoescape to prevent XSS vulnerabilities Fix Bandit B701 vulnerability by enabling autoescape for HTML, XML, and .j2 files using select_autoescape(). This prevents potential XSS attacks when rendering user-controlled data in HTML reports. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * gitignore bandit-security-report.json created during pipeline run * Update readme cli help output * rename robot/pabot integration test file * Add netconf related packages and update dependencies * 1.2.2 version bump * refactor: centralize DEBUG_MODE constant for progressive disclosure Consolidates the NAC_TEST_DEBUG environment variable check into a single constant in core/constants.py, eliminating repeated inline checks. Changes: - Add DEBUG_MODE constant to nac_test/core/constants.py - Update cli/main.py to use DEBUG_MODE for pretty_exceptions_enable - Update combined_orchestrator.py to use DEBUG_MODE for exception handling - Update batching_reporter.py to use DEBUG_MODE (2 locations) This enables progressive disclosure: customers get clean error output by default, while developers can set NAC_TEST_DEBUG=true for full exception context and verbose tracebacks when debugging. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: danischm <danischm@cisco.com> Co-authored-by: Oliver Boehmer <oli@spine.de> Co-authored-by: Oliver Boehmer <oboehmer@cisco.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Justyna Chowaniec <79261946+juchowan@users.noreply.github.com>
oboehmer
added a commit
that referenced
this pull request
Mar 19, 2026
* Bump pluggy from 1.5.0 to 1.6.0 (#365) Bumps [pluggy](https://github.com/pytest-dev/pluggy) from 1.5.0 to 1.6.0. - [Changelog](https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pluggy@1.5.0...1.6.0) --- updated-dependencies: - dependency-name: pluggy dependency-version: 1.6.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump click from 8.2.0 to 8.2.1 (#368) Bumps [click](https://github.com/pallets/click) from 8.2.0 to 8.2.1. - [Release notes](https://github.com/pallets/click/releases) - [Changelog](https://github.com/pallets/click/blob/main/CHANGES.rst) - [Commits](pallets/click@8.2.0...8.2.1) --- updated-dependencies: - dependency-name: click dependency-version: 8.2.1 dependency-type: indirect update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump typer from 0.15.3 to 0.16.0 (#371) Bumps [typer](https://github.com/fastapi/typer) from 0.15.3 to 0.16.0. - [Release notes](https://github.com/fastapi/typer/releases) - [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md) - [Commits](fastapi/typer@0.15.3...0.16.0) --- updated-dependencies: - dependency-name: typer dependency-version: 0.16.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump requests from 2.32.3 to 2.32.4 (#372) Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.3...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump urllib3 from 2.4.0 to 2.5.0 (#374) Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.4.0...2.5.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.5.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump rpds-py from 0.24.0 to 0.26.0 (#375) Bumps [rpds-py](https://github.com/crate-py/rpds) from 0.24.0 to 0.26.0. - [Release notes](https://github.com/crate-py/rpds/releases) - [Commits](crate-py/rpds@v0.24.0...v0.26.0) --- updated-dependencies: - dependency-name: rpds-py dependency-version: 0.26.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cryptography from 44.0.3 to 45.0.5 (#376) Bumps [cryptography](https://github.com/pyca/cryptography) from 44.0.3 to 45.0.5. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@44.0.3...45.0.5) --- updated-dependencies: - dependency-name: cryptography dependency-version: 45.0.5 dependency-type: indirect update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Migrate to uv * Refactor error handling * Various linter updates * Dependency updates * Update readme * Update dependencies and pyproject * Fix dependency * Fix robot dependency * Update versions * Bump astral-sh/setup-uv from 6 to 7 (#388) Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 6 to 7. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@v6...v7) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * License header updates * Update changelog * Update dependabot config for uv * Bump nac-yaml version * 1.1.0 version bump * Bump actions/upload-artifact from 4 to 5 (#391) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 5. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4...v5) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix lint workflow for fork prs * Expose pabot's process option to nac-test cli (#393) * Increase robot loglevel to debug when verbosity=DEBUG is used (#394) * Add robotframework-jmespath dependency (#395) * Update README to include jmespath and add test for 3rd party libs (#396) * Add support for test case parallelization for capable suites (#390) * Switch from pabot.main to pabot.main_program and handle exit codes (#400) Change nac-test to use pabot's main_program() instead of main() to avoid sys.exit() calls and properly handle exit codes. This allows nac-test to control the exit behavior and return appropriate exit codes to the caller. Key changes: - Update run_pabot() to call pabot.pabot.main_program() instead of main() - Change run_pabot() return type from None to int - Return exit code from pabot execution - Update CLI main() to capture and use exit code - Remove try/except wrapper in CLI main() - let exceptions propagate - Disable typer pretty exceptions for cleaner error output Benefits: - Proper exit code handling (test failures, errors, etc.) - No unexpected sys.exit() calls - Cleaner error messages without verbose typer tracebacks - Better integration with CI/CD systems 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * Add support for passing Robot Framework options to nac-test (#399) * Add support for passing Robot Framework options to nac-test Users can now pass additional Robot Framework options directly to nac-test, which are validated and appended to the pabot invocation. This enables advanced use cases like custom variables, listeners, and logging configuration without modifying nac-test's core options. Key changes: - Enable extra args in CLI via typer context settings - Add parse_and_validate_extra_args() to validate Robot Framework options - Reject pabot-specific options (--processes, --testlevelsplit, etc.) - Reject datasources/test files in extra arguments - Return exit code 252 for invalid extra arguments - Separate pabot_args and robot_args for proper argument ordering - Add comprehensive unit tests (14 tests covering all scenarios) - Add type hints and fix all mypy errors - Update CLI help text to document extra args feature Implementation details: - Uses pabot's parse_args() to validate options against Robot Framework - Filters out datasources and pabot-specific options - Appends validated args to robot_args before pabot invocation - Proper error handling with logging and exit codes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Remove pabot.main_program changes from robot-args branch This commit removes the pabot-specific changes (main_program, exit codes, exception handling) that belong in the separate pabot-call branch. The robot-args branch now focuses solely on the extra arguments feature. Changes: - Revert run_pabot() return type from int to None - Use pabot.pabot.main() instead of main_program() - Remove exit code handling (no return 252) - Keep try/except in CLI main() for proper error handling - Update tests to use pabot.pabot.main mock - Update tests to expect exceptions instead of error codes The extra args validation and appending logic remains unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Revert "Remove pabot.main_program changes from robot-args branch" This reverts commit df21b98. --------- Co-authored-by: Claude <noreply@anthropic.com> * Add support for chunking templates (#398) * Update changelog and readms * Update lock file * Expand test coverage for chunk split test (#401) * Fix typo in readme link * Remove library version numbers from readme * 1.2.0 version bump * feat(path_setup): auto-discover pyats_common for test imports Add find_pyats_common_parent() to search test directory tree for pyats_common subdirectory and add its parent to PYTHONPATH. This enables test files to use simple imports like: from pyats_common.apic_base_test import APICTestBase 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * add verbose option to pyats when running at DEBUG level, log pyats output as well when in DEBUG * current state, not yet working * add simple test for pipeline - WIP * add integration tests and support for mock devices * use easypy.run to invoke script * do not assume a default CONTROLLER_TYPE * remove assumption for CONTROLLER_TYPE being ACI in base_test too * adjust test * add some debug log statements * add warning about potential issues with python3.11 on macOS * connect using log_stdout=False * fix simple test, and ruff formatting * simple test changes, WIP * Support rendering on windows (#404) * Bump actions/checkout from 5 to 6 (#402) Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update changelog * 1.2.1 version bump * update with mock flask server - WIP * remove debug logs * add mock-server to perform API testing in integration tests * add integration tests for quicksilver-generated templates * add type hints * Bump actions/upload-artifact from 5 to 6 (#418) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v5...v6) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add sdwan mock api results * adjust mock-server file, create mock-server test * fix: resolve all ruff linting errors and warnings Fix code quality issues identified by ruff to ensure compliance with project style guidelines and best practices. Changes: - Fix syntax error in pabot.py (space in != operator) - Add missing filecmp import in test_integration.py - Add proper exception chaining (from e) for B904 violations - Rename unused loop variables to prefix with underscore (B007) - Remove trailing whitespace from blank lines (W293) - Modernize Union type annotations to use | operator (UP007) - Remove unused Union import after migration to | syntax - Exclude test fixture directories from ruff checks in pyproject.toml All ruff checks now pass successfully. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix pabot module import in tests/unit/test_pabot_args.py * no longer test for RESTinstance robot lib * add fixture to set bogus ACI_xxx environment for nac-test to pass rendering tests * add/fix --processes feature * fix/add ordering_file logic to robot orchestrator * fix/add passing extra cli args to robot orchestrator * add integration test for extra args * update README.md * fix ruff config changes broken during merge conflict resolution * fix: resolve all mypy type checking errors This commit addresses 11 mypy type errors across 4 files: - connection_utils.py: Add type annotations for dict parameters and update function signature to accept None values that are validated - subprocess_runner.py: Remove unreachable code branch in output processing that was flagged due to redundant None check - connection_broker.py: Add type annotations for testbed attribute, AsyncIterator return type, and assertion for loader result - robot_writer.py: Remove duplicate method definition and add explicit type annotation to prevent incorrect type narrowing All files now pass mypy strict type checking without errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: restore custom_data handling in render_template for chunked templates The refactoring that moved data conversion to initialization (for performance) inadvertently broke the chunked template functionality by not using the custom_data parameter when provided. The render_template method now correctly checks if custom_data is provided (used for chunked templates) and converts it on-demand, otherwise uses the pre-converted self.template_data. This fixes the test_nac_test_list_chunked test which was failing after merging main into the feature branch. Fixes functionality added in commits 18da546 and ced5983. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * ruff formatting fixes * fix: add type stub dependencies to pre-commit mypy hook The pre-commit mypy hook runs in an isolated environment and needs type stubs explicitly specified via additional_dependencies. Without these, mypy complains about missing type information for yaml, aiofiles, and markdown modules. Added: - types-PyYAML>=6.0.12.20250822 - types-aiofiles>=24.1.0.20250822 - types-markdown>=3.8.0.20250809 This aligns the pre-commit environment with the project dependencies and allows mypy to properly type-check imports from these libraries. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: add type annotations to unit test functions Add proper type annotations to all test functions in tests/unit to resolve mypy strict type checking errors. Changes include: - Add -> None return type annotations to all test methods - Add type annotations for mock parameters (mock_logger, mock_connection_class, etc.) using typing.Any - Add explicit type annotation for connections list variable - Add type: ignore comment for intentional type mismatch in test This ensures tests pass mypy strict type checking while maintaining test clarity and avoiding over-specification of mock types. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller tests from shell environment variables Add autouse fixture to clear all controller-related environment variables (ACI_*, SDWAN_*, CC_*, MERAKI_*, FMC_*, ISE_*) before each test in TestCombinedOrchestratorController. This ensures tests run in a clean environment and results are not influenced by the caller's shell environment, preventing false failures when developer has controller credentials set in their shell. The fixture uses pytest's monkeypatch to safely remove environment variables only for the duration of each test. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: migrate integration test fixture to use monkeypatch Replace manual environment variable setup/teardown with pytest's monkeypatch fixture for safer and more robust test isolation. Benefits of monkeypatch approach: - Automatic cleanup guaranteed even if test fails - Safe deletion with raising=False (no KeyError if var missing) - Preserves and restores original environment variable values - Less boilerplate code (no yield/try-finally needed) - Thread-safe state management handled by pytest The fixture now uses monkeypatch.setenv() which automatically saves the original value (if any) and restores it after the test, ensuring the caller's shell environment is never permanently modified. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller detection integration tests from shell environment Add autouse fixture to clear all controller-related environment variables before each test in TestControllerDetectionIntegration. This prevents test failures when developers have controller credentials set in their shell environment. Without this fix, tests would fail with "Multiple controller credentials detected" errors when shell variables conflicted with test-specific environment setup. The fixture uses pytest's monkeypatch.delenv() with raising=False for safe deletion that won't fail if a variable doesn't exist. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * add license headers * add missing test fixture template for extra args test * security: replace hardcoded /tmp with tempfile.gettempdir() in ConnectionBroker Fix Bandit B108 vulnerability by using tempfile.gettempdir() instead of hardcoded /tmp path for output_dir default value. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: add usedforsecurity=False to MD5 hash in AuthCache Fix Bandit B324 vulnerability by explicitly marking MD5 usage as not for security purposes. The hash is only used to generate cache filenames from URLs, not for cryptographic security. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp references in emergency dump Fix Bandit B108 vulnerabilities by: - Using tempfile.gettempdir() instead of hardcoded /tmp path - Dynamically checking if file is in temp directory instead of string matching "/tmp/" - Updated docstrings and log messages to reflect system temp dir usage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in AUTH_CACHE_DIR constant Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for AUTH_CACHE_DIR. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in batching reporter overflow dir Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for default overflow directory. This ensures cross-platform compatibility while still allowing override via NAC_TEST_OVERFLOW_DIR environment variable. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: enable Jinja2 autoescape to prevent XSS vulnerabilities Fix Bandit B701 vulnerability by enabling autoescape for HTML, XML, and .j2 files using select_autoescape(). This prevents potential XSS attacks when rendering user-controlled data in HTML reports. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * gitignore bandit-security-report.json created during pipeline run * Update readme cli help output * rename robot/pabot integration test file * Add netconf related packages and update dependencies * 1.2.2 version bump * adjust and fix aci integration tests * remove environment vars after testing * fix license check * remove unfinished tests * use tmpdir * add debugging for CI troubleshooting * attempt to fix event loop issue in CI * add more debug code to troubleshoot CI failure * add missing nac-test-pyats-common to CI pipeline * sum up passed/failed for check (in case we have multiple results.json files) * re-organize test case fixture directories use a common base dir, and then add specific archs as subdirectories to avoid directory bloat in fixture directory * Fix Python 3.12 asyncio compatibility with get_or_create_event_loop Python 3.12 changed asyncio.get_event_loop() behavior to raise RuntimeError when called from the main thread without an active event loop, breaking async operations that run synchronous PyATS/Unicon calls via loop.run_in_executor(). Previous behavior (Python 3.10-3.11): - asyncio.get_event_loop() automatically created a new event loop if none existed New behavior (Python 3.12+): - asyncio.get_event_loop() raises RuntimeError if no event loop exists - Must explicitly call asyncio.new_event_loop() and set_event_loop() Solution: Created get_or_create_event_loop() helper that handles both behaviors: - Tries to get existing event loop - If closed or RuntimeError, creates and sets a new loop - Works across Python 3.10-3.12+ Replaced 9 instances of asyncio.get_event_loop() across 4 modules: - nac_test/pyats_core/broker/connection_broker.py (3 calls) - nac_test/pyats_core/common/ssh_base_test.py (3 calls) - nac_test/pyats_core/ssh/connection_manager.py (2 calls) - nac_test/pyats_core/reporting/multi_archive_generator.py (1 call) These calls are used to run blocking Unicon operations (connect, disconnect, execute) in thread pools via run_in_executor() to prevent blocking the async event loop. * add sdwan integration tests (both api and d2d) * add license header * test: remove test_mock_server.py - tests mock infrastructure, not business logic These tests only verify that the MockAPIServer returns what it was configured to return: - test_mock_api_server_endpoint_matching: configures endpoints, asserts they return configured data - test_nac_test_with_mock_api_complex_urls: same pattern with complex URLs - test_nac_test_with_mock_api_dynamic: adds endpoint with response X, asserts response is X This violates the principle: "What MUST NEVER be Tested: whether mocks return what you told them to return." The mock server is infrastructure to enable testing real business logic, not something that needs its own test coverage. * test: remove test_controller_detection_consistency - tests Python determinism This test calls detect_controller_type() three times with identical environment and asserts all results are equal. This tests whether Python functions return consistent results when called with the same inputs - which is testing Python's deterministic execution, not business logic. A pure function will always return the same result for the same inputs. Testing this adds no value unless there's caching or stateful behavior that could cause inconsistency (there isn't). Violates: "What MUST NEVER be Tested: Whether Python's standard library works" * test: remove mock assertion from test_end_to_end_controller_detection Removed the pattern that mocks EnvironmentValidator and asserts it was called with the correct argument: with patch("...EnvironmentValidator") as mock_validator: orchestrator.validate_environment() mock_validator.validate_controller_env.assert_called_once_with("SDWAN") This tests whether validate_environment() calls EnvironmentValidator with the right argument - essentially testing "whether one-line wrapper functions call the functions they wrap." The real test is already done: assert orchestrator.controller_type == "SDWAN" which validates the business logic (controller was correctly detected). Violates: "What MUST NEVER be Tested: whether one-line wrapper functions call the functions they wrap" * fix(tests): remove conflicting copyright header and fix resource leak mock_unicon.py changes: - Removed Cisco proprietary copyright header (lines 5-7), keeping only MPL-2.0 - Fixed resource leak: file opened without context manager now uses 'with' statement Before: states = yaml.safe_load(open(os.path.join(mock_data_dir, file))) or [] After: with open(os.path.join(mock_data_dir, file)) as f: states = yaml.safe_load(f) or [] * fix(tests): implement proper server startup and shutdown for MockAPIServer mock_server.py changes: - Replaced time.sleep(0.5) race condition with proper readiness polling - Added _wait_for_server_ready() that polls until server responds or timeout - Implemented proper stop() method using werkzeug.serving.make_server - Added reset_endpoints() method to clear dynamic endpoints while preserving YAML-loaded baseline (enables test isolation with session-scoped fixture) - Added SERVER_STARTUP_TIMEOUT_SECONDS and SERVER_POLL_INTERVAL_SECONDS constants This fixes flaky test failures caused by server not being ready, and prevents port conflicts from improper shutdown. * fix(tests): add endpoint reset and isolated fixture for test isolation conftest.py changes: - Updated mock_api_server fixture to call reset_endpoints() after yield - Added mock_api_server_isolated fixture (function-scoped) for tests that add dynamic endpoints, preventing cross-test pollution The session-scoped mock_api_server now resets to baseline state between tests, while mock_api_server_isolated provides per-test isolation. * fix(tests): use absolute path and fix PEP8 naming in test_pyats_standard test_pyats_standard.py changes: - Fixed hardcoded relative path that breaks when run from different directories Now uses Path(__file__).parent to construct absolute path to mock_unicon.py - Renamed classes tc_one -> TcOne, tc_two -> TcTwo (PEP8 CamelCase) - Fixed decorator spacing: '@ aetest.test' -> '@aetest.test' * fix(tests): replace debug prints with logging and use monkeypatch test_integration_pyats.py changes: - Replaced all print() debug statements with logger.debug() calls - Moved 'import json' from inside function to module level - Refactored environment variable handling to use monkeypatch.setenv() instead of manual os.environ manipulation with try/finally - Removed commented-out debug code for static output directories This improves test maintainability and ensures proper cleanup via pytest's monkeypatch fixture. * fix(tests): modernize type hints and use monkeypatch in robot_pabot tests test_integration_robot_pabot.py changes: - Changed all tmpdir: str parameters to tmp_path: Path (modern pytest fixture) - Refactored environment variable handling to use monkeypatch.setenv() instead of manual os.environ with try/finally cleanup - Updated fixture references from tmpdir to tmp_path throughout - Used Path methods (.exists(), .touch()) instead of os.path equivalents This improves type safety and ensures proper test isolation via pytest's automatic cleanup of monkeypatched environment variables. * fix(tests): replace obfuscated code and emojis in PyATS quicksilver tests templates_pyats_qs test changes (3 files): - verify_sdwanmanager_all_sd_wan_edge_configurations_are_in_sync.py - verify_iosxe_all_sd_wan_control_connections_are_up.py - verify_aci_apic_appliance_operational_status.py Changes applied to all: - Replaced chr(10) with '\n' for clarity - Replaced emoji markers with text: ❌ -> [FAIL], ✅ -> [PASS] This prevents encoding issues in CI logs and improves code readability. * fix(tests): extract f-string join to variable for Python 3.11 compatibility Python 3.11 does not allow backslash escapes inside f-string expressions. The previous commit replaced chr(10) with '\n' but left it inside the f-string expression, causing a SyntaxError. This fix extracts the join operation to a separate variable before using it in the f-string: failures_text = '\n'.join(failures) f"{failures_text}\n\n" * fix(tests): add proxy bypass for localhost in integration tests Add session-scoped autouse fixture to ensure 127.0.0.1 is included in the no_proxy/NO_PROXY environment variables. This prevents the mock API server requests from being routed through corporate proxies, which was causing 504 Gateway Timeout errors in environments with proxy configured. The fixture preserves original proxy settings and restores them after the test session completes. * refactor(tests): split robot_pabot tests into focused modules Split the 493-line test_integration_robot_pabot.py into 4 focused test modules following Single Responsibility Principle: - test_cli_basic.py: Basic CLI execution tests (6 tests) - test_cli_rendering.py: Template rendering tests (8 tests) - test_cli_ordering.py: Robot test ordering tests (4 tests) - test_cli_extra_args.py: Extra arguments tests (5 tests) Improvements: - Added Google-style docstrings to all 23 test functions - Renamed generic test names to descriptive names - Added assertion messages to all assert statements - Extracted magic number to named constant (ROBOT_ARGUMENT_ERROR_EXIT_CODE) * add management_ip_variable to test fixture data --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: danischm <danischm@cisco.com> Co-authored-by: Oliver Boehmer <oli@spine.de> Co-authored-by: Oliver Boehmer <oboehmer@cisco.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Justyna Chowaniec <79261946+juchowan@users.noreply.github.com>
oboehmer
added a commit
that referenced
this pull request
Mar 19, 2026
oboehmer
added a commit
that referenced
this pull request
Mar 19, 2026
* Bump pluggy from 1.5.0 to 1.6.0 (#365) Bumps [pluggy](https://github.com/pytest-dev/pluggy) from 1.5.0 to 1.6.0. - [Changelog](https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pluggy@1.5.0...1.6.0) --- updated-dependencies: - dependency-name: pluggy dependency-version: 1.6.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump click from 8.2.0 to 8.2.1 (#368) Bumps [click](https://github.com/pallets/click) from 8.2.0 to 8.2.1. - [Release notes](https://github.com/pallets/click/releases) - [Changelog](https://github.com/pallets/click/blob/main/CHANGES.rst) - [Commits](pallets/click@8.2.0...8.2.1) --- updated-dependencies: - dependency-name: click dependency-version: 8.2.1 dependency-type: indirect update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump typer from 0.15.3 to 0.16.0 (#371) Bumps [typer](https://github.com/fastapi/typer) from 0.15.3 to 0.16.0. - [Release notes](https://github.com/fastapi/typer/releases) - [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md) - [Commits](fastapi/typer@0.15.3...0.16.0) --- updated-dependencies: - dependency-name: typer dependency-version: 0.16.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump requests from 2.32.3 to 2.32.4 (#372) Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.3...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump urllib3 from 2.4.0 to 2.5.0 (#374) Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.4.0...2.5.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.5.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump rpds-py from 0.24.0 to 0.26.0 (#375) Bumps [rpds-py](https://github.com/crate-py/rpds) from 0.24.0 to 0.26.0. - [Release notes](https://github.com/crate-py/rpds/releases) - [Commits](crate-py/rpds@v0.24.0...v0.26.0) --- updated-dependencies: - dependency-name: rpds-py dependency-version: 0.26.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cryptography from 44.0.3 to 45.0.5 (#376) Bumps [cryptography](https://github.com/pyca/cryptography) from 44.0.3 to 45.0.5. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@44.0.3...45.0.5) --- updated-dependencies: - dependency-name: cryptography dependency-version: 45.0.5 dependency-type: indirect update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Migrate to uv * Refactor error handling * Various linter updates * Dependency updates * Update readme * Update dependencies and pyproject * Fix dependency * Fix robot dependency * Update versions * Bump astral-sh/setup-uv from 6 to 7 (#388) Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 6 to 7. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@v6...v7) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * License header updates * Update changelog * Update dependabot config for uv * Bump nac-yaml version * 1.1.0 version bump * Bump actions/upload-artifact from 4 to 5 (#391) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 5. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4...v5) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix lint workflow for fork prs * Expose pabot's process option to nac-test cli (#393) * Increase robot loglevel to debug when verbosity=DEBUG is used (#394) * Add robotframework-jmespath dependency (#395) * Update README to include jmespath and add test for 3rd party libs (#396) * Add support for test case parallelization for capable suites (#390) * Switch from pabot.main to pabot.main_program and handle exit codes (#400) Change nac-test to use pabot's main_program() instead of main() to avoid sys.exit() calls and properly handle exit codes. This allows nac-test to control the exit behavior and return appropriate exit codes to the caller. Key changes: - Update run_pabot() to call pabot.pabot.main_program() instead of main() - Change run_pabot() return type from None to int - Return exit code from pabot execution - Update CLI main() to capture and use exit code - Remove try/except wrapper in CLI main() - let exceptions propagate - Disable typer pretty exceptions for cleaner error output Benefits: - Proper exit code handling (test failures, errors, etc.) - No unexpected sys.exit() calls - Cleaner error messages without verbose typer tracebacks - Better integration with CI/CD systems 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * Add support for passing Robot Framework options to nac-test (#399) * Add support for passing Robot Framework options to nac-test Users can now pass additional Robot Framework options directly to nac-test, which are validated and appended to the pabot invocation. This enables advanced use cases like custom variables, listeners, and logging configuration without modifying nac-test's core options. Key changes: - Enable extra args in CLI via typer context settings - Add parse_and_validate_extra_args() to validate Robot Framework options - Reject pabot-specific options (--processes, --testlevelsplit, etc.) - Reject datasources/test files in extra arguments - Return exit code 252 for invalid extra arguments - Separate pabot_args and robot_args for proper argument ordering - Add comprehensive unit tests (14 tests covering all scenarios) - Add type hints and fix all mypy errors - Update CLI help text to document extra args feature Implementation details: - Uses pabot's parse_args() to validate options against Robot Framework - Filters out datasources and pabot-specific options - Appends validated args to robot_args before pabot invocation - Proper error handling with logging and exit codes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Remove pabot.main_program changes from robot-args branch This commit removes the pabot-specific changes (main_program, exit codes, exception handling) that belong in the separate pabot-call branch. The robot-args branch now focuses solely on the extra arguments feature. Changes: - Revert run_pabot() return type from int to None - Use pabot.pabot.main() instead of main_program() - Remove exit code handling (no return 252) - Keep try/except in CLI main() for proper error handling - Update tests to use pabot.pabot.main mock - Update tests to expect exceptions instead of error codes The extra args validation and appending logic remains unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Revert "Remove pabot.main_program changes from robot-args branch" This reverts commit df21b98. --------- Co-authored-by: Claude <noreply@anthropic.com> * Add support for chunking templates (#398) * Update changelog and readms * Update lock file * Expand test coverage for chunk split test (#401) * Fix typo in readme link * Remove library version numbers from readme * 1.2.0 version bump * Support rendering on windows (#404) * Bump actions/checkout from 5 to 6 (#402) Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update changelog * 1.2.1 version bump * Bump actions/upload-artifact from 5 to 6 (#418) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v5...v6) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: resolve all ruff linting errors and warnings Fix code quality issues identified by ruff to ensure compliance with project style guidelines and best practices. Changes: - Fix syntax error in pabot.py (space in != operator) - Add missing filecmp import in test_integration.py - Add proper exception chaining (from e) for B904 violations - Rename unused loop variables to prefix with underscore (B007) - Remove trailing whitespace from blank lines (W293) - Modernize Union type annotations to use | operator (UP007) - Remove unused Union import after migration to | syntax - Exclude test fixture directories from ruff checks in pyproject.toml All ruff checks now pass successfully. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix pabot module import in tests/unit/test_pabot_args.py * no longer test for RESTinstance robot lib * add fixture to set bogus ACI_xxx environment for nac-test to pass rendering tests * add/fix --processes feature * fix/add ordering_file logic to robot orchestrator * fix/add passing extra cli args to robot orchestrator * add integration test for extra args * update README.md * fix ruff config changes broken during merge conflict resolution * fix: resolve all mypy type checking errors This commit addresses 11 mypy type errors across 4 files: - connection_utils.py: Add type annotations for dict parameters and update function signature to accept None values that are validated - subprocess_runner.py: Remove unreachable code branch in output processing that was flagged due to redundant None check - connection_broker.py: Add type annotations for testbed attribute, AsyncIterator return type, and assertion for loader result - robot_writer.py: Remove duplicate method definition and add explicit type annotation to prevent incorrect type narrowing All files now pass mypy strict type checking without errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: restore custom_data handling in render_template for chunked templates The refactoring that moved data conversion to initialization (for performance) inadvertently broke the chunked template functionality by not using the custom_data parameter when provided. The render_template method now correctly checks if custom_data is provided (used for chunked templates) and converts it on-demand, otherwise uses the pre-converted self.template_data. This fixes the test_nac_test_list_chunked test which was failing after merging main into the feature branch. Fixes functionality added in commits 18da546 and ced5983. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * ruff formatting fixes * fix: add type stub dependencies to pre-commit mypy hook The pre-commit mypy hook runs in an isolated environment and needs type stubs explicitly specified via additional_dependencies. Without these, mypy complains about missing type information for yaml, aiofiles, and markdown modules. Added: - types-PyYAML>=6.0.12.20250822 - types-aiofiles>=24.1.0.20250822 - types-markdown>=3.8.0.20250809 This aligns the pre-commit environment with the project dependencies and allows mypy to properly type-check imports from these libraries. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: add type annotations to unit test functions Add proper type annotations to all test functions in tests/unit to resolve mypy strict type checking errors. Changes include: - Add -> None return type annotations to all test methods - Add type annotations for mock parameters (mock_logger, mock_connection_class, etc.) using typing.Any - Add explicit type annotation for connections list variable - Add type: ignore comment for intentional type mismatch in test This ensures tests pass mypy strict type checking while maintaining test clarity and avoiding over-specification of mock types. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller tests from shell environment variables Add autouse fixture to clear all controller-related environment variables (ACI_*, SDWAN_*, CC_*, MERAKI_*, FMC_*, ISE_*) before each test in TestCombinedOrchestratorController. This ensures tests run in a clean environment and results are not influenced by the caller's shell environment, preventing false failures when developer has controller credentials set in their shell. The fixture uses pytest's monkeypatch to safely remove environment variables only for the duration of each test. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: migrate integration test fixture to use monkeypatch Replace manual environment variable setup/teardown with pytest's monkeypatch fixture for safer and more robust test isolation. Benefits of monkeypatch approach: - Automatic cleanup guaranteed even if test fails - Safe deletion with raising=False (no KeyError if var missing) - Preserves and restores original environment variable values - Less boilerplate code (no yield/try-finally needed) - Thread-safe state management handled by pytest The fixture now uses monkeypatch.setenv() which automatically saves the original value (if any) and restores it after the test, ensuring the caller's shell environment is never permanently modified. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller detection integration tests from shell environment Add autouse fixture to clear all controller-related environment variables before each test in TestControllerDetectionIntegration. This prevents test failures when developers have controller credentials set in their shell environment. Without this fix, tests would fail with "Multiple controller credentials detected" errors when shell variables conflicted with test-specific environment setup. The fixture uses pytest's monkeypatch.delenv() with raising=False for safe deletion that won't fail if a variable doesn't exist. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * add license headers * add missing test fixture template for extra args test * security: replace hardcoded /tmp with tempfile.gettempdir() in ConnectionBroker Fix Bandit B108 vulnerability by using tempfile.gettempdir() instead of hardcoded /tmp path for output_dir default value. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: add usedforsecurity=False to MD5 hash in AuthCache Fix Bandit B324 vulnerability by explicitly marking MD5 usage as not for security purposes. The hash is only used to generate cache filenames from URLs, not for cryptographic security. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp references in emergency dump Fix Bandit B108 vulnerabilities by: - Using tempfile.gettempdir() instead of hardcoded /tmp path - Dynamically checking if file is in temp directory instead of string matching "/tmp/" - Updated docstrings and log messages to reflect system temp dir usage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in AUTH_CACHE_DIR constant Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for AUTH_CACHE_DIR. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in batching reporter overflow dir Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for default overflow directory. This ensures cross-platform compatibility while still allowing override via NAC_TEST_OVERFLOW_DIR environment variable. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: enable Jinja2 autoescape to prevent XSS vulnerabilities Fix Bandit B701 vulnerability by enabling autoescape for HTML, XML, and .j2 files using select_autoescape(). This prevents potential XSS attacks when rendering user-controlled data in HTML reports. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * gitignore bandit-security-report.json created during pipeline run * Update readme cli help output * rename robot/pabot integration test file * Add netconf related packages and update dependencies * 1.2.2 version bump * refactor: centralize DEBUG_MODE constant for progressive disclosure Consolidates the NAC_TEST_DEBUG environment variable check into a single constant in core/constants.py, eliminating repeated inline checks. Changes: - Add DEBUG_MODE constant to nac_test/core/constants.py - Update cli/main.py to use DEBUG_MODE for pretty_exceptions_enable - Update combined_orchestrator.py to use DEBUG_MODE for exception handling - Update batching_reporter.py to use DEBUG_MODE (2 locations) This enables progressive disclosure: customers get clean error output by default, while developers can set NAC_TEST_DEBUG=true for full exception context and verbose tracebacks when debugging. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: danischm <danischm@cisco.com> Co-authored-by: Oliver Boehmer <oli@spine.de> Co-authored-by: Oliver Boehmer <oboehmer@cisco.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Justyna Chowaniec <79261946+juchowan@users.noreply.github.com>
oboehmer
added a commit
that referenced
this pull request
Mar 19, 2026
* Bump pluggy from 1.5.0 to 1.6.0 (#365) Bumps [pluggy](https://github.com/pytest-dev/pluggy) from 1.5.0 to 1.6.0. - [Changelog](https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pluggy@1.5.0...1.6.0) --- updated-dependencies: - dependency-name: pluggy dependency-version: 1.6.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump click from 8.2.0 to 8.2.1 (#368) Bumps [click](https://github.com/pallets/click) from 8.2.0 to 8.2.1. - [Release notes](https://github.com/pallets/click/releases) - [Changelog](https://github.com/pallets/click/blob/main/CHANGES.rst) - [Commits](pallets/click@8.2.0...8.2.1) --- updated-dependencies: - dependency-name: click dependency-version: 8.2.1 dependency-type: indirect update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump typer from 0.15.3 to 0.16.0 (#371) Bumps [typer](https://github.com/fastapi/typer) from 0.15.3 to 0.16.0. - [Release notes](https://github.com/fastapi/typer/releases) - [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md) - [Commits](fastapi/typer@0.15.3...0.16.0) --- updated-dependencies: - dependency-name: typer dependency-version: 0.16.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump requests from 2.32.3 to 2.32.4 (#372) Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.3...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump urllib3 from 2.4.0 to 2.5.0 (#374) Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.4.0...2.5.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.5.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump rpds-py from 0.24.0 to 0.26.0 (#375) Bumps [rpds-py](https://github.com/crate-py/rpds) from 0.24.0 to 0.26.0. - [Release notes](https://github.com/crate-py/rpds/releases) - [Commits](crate-py/rpds@v0.24.0...v0.26.0) --- updated-dependencies: - dependency-name: rpds-py dependency-version: 0.26.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cryptography from 44.0.3 to 45.0.5 (#376) Bumps [cryptography](https://github.com/pyca/cryptography) from 44.0.3 to 45.0.5. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@44.0.3...45.0.5) --- updated-dependencies: - dependency-name: cryptography dependency-version: 45.0.5 dependency-type: indirect update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Migrate to uv * Refactor error handling * Various linter updates * Dependency updates * Update readme * Update dependencies and pyproject * Fix dependency * Fix robot dependency * Update versions * Bump astral-sh/setup-uv from 6 to 7 (#388) Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 6 to 7. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@v6...v7) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * License header updates * Update changelog * Update dependabot config for uv * Bump nac-yaml version * 1.1.0 version bump * Bump actions/upload-artifact from 4 to 5 (#391) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 5. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4...v5) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix lint workflow for fork prs * Expose pabot's process option to nac-test cli (#393) * Increase robot loglevel to debug when verbosity=DEBUG is used (#394) * Add robotframework-jmespath dependency (#395) * Update README to include jmespath and add test for 3rd party libs (#396) * Add support for test case parallelization for capable suites (#390) * Switch from pabot.main to pabot.main_program and handle exit codes (#400) Change nac-test to use pabot's main_program() instead of main() to avoid sys.exit() calls and properly handle exit codes. This allows nac-test to control the exit behavior and return appropriate exit codes to the caller. Key changes: - Update run_pabot() to call pabot.pabot.main_program() instead of main() - Change run_pabot() return type from None to int - Return exit code from pabot execution - Update CLI main() to capture and use exit code - Remove try/except wrapper in CLI main() - let exceptions propagate - Disable typer pretty exceptions for cleaner error output Benefits: - Proper exit code handling (test failures, errors, etc.) - No unexpected sys.exit() calls - Cleaner error messages without verbose typer tracebacks - Better integration with CI/CD systems 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * Add support for passing Robot Framework options to nac-test (#399) * Add support for passing Robot Framework options to nac-test Users can now pass additional Robot Framework options directly to nac-test, which are validated and appended to the pabot invocation. This enables advanced use cases like custom variables, listeners, and logging configuration without modifying nac-test's core options. Key changes: - Enable extra args in CLI via typer context settings - Add parse_and_validate_extra_args() to validate Robot Framework options - Reject pabot-specific options (--processes, --testlevelsplit, etc.) - Reject datasources/test files in extra arguments - Return exit code 252 for invalid extra arguments - Separate pabot_args and robot_args for proper argument ordering - Add comprehensive unit tests (14 tests covering all scenarios) - Add type hints and fix all mypy errors - Update CLI help text to document extra args feature Implementation details: - Uses pabot's parse_args() to validate options against Robot Framework - Filters out datasources and pabot-specific options - Appends validated args to robot_args before pabot invocation - Proper error handling with logging and exit codes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Remove pabot.main_program changes from robot-args branch This commit removes the pabot-specific changes (main_program, exit codes, exception handling) that belong in the separate pabot-call branch. The robot-args branch now focuses solely on the extra arguments feature. Changes: - Revert run_pabot() return type from int to None - Use pabot.pabot.main() instead of main_program() - Remove exit code handling (no return 252) - Keep try/except in CLI main() for proper error handling - Update tests to use pabot.pabot.main mock - Update tests to expect exceptions instead of error codes The extra args validation and appending logic remains unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Revert "Remove pabot.main_program changes from robot-args branch" This reverts commit df21b98. --------- Co-authored-by: Claude <noreply@anthropic.com> * Add support for chunking templates (#398) * Update changelog and readms * Update lock file * Expand test coverage for chunk split test (#401) * Fix typo in readme link * Remove library version numbers from readme * 1.2.0 version bump * feat(path_setup): auto-discover pyats_common for test imports Add find_pyats_common_parent() to search test directory tree for pyats_common subdirectory and add its parent to PYTHONPATH. This enables test files to use simple imports like: from pyats_common.apic_base_test import APICTestBase 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * add verbose option to pyats when running at DEBUG level, log pyats output as well when in DEBUG * current state, not yet working * add simple test for pipeline - WIP * add integration tests and support for mock devices * use easypy.run to invoke script * do not assume a default CONTROLLER_TYPE * remove assumption for CONTROLLER_TYPE being ACI in base_test too * adjust test * add some debug log statements * add warning about potential issues with python3.11 on macOS * connect using log_stdout=False * fix simple test, and ruff formatting * simple test changes, WIP * Support rendering on windows (#404) * Bump actions/checkout from 5 to 6 (#402) Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update changelog * 1.2.1 version bump * update with mock flask server - WIP * remove debug logs * add mock-server to perform API testing in integration tests * add integration tests for quicksilver-generated templates * add type hints * Bump actions/upload-artifact from 5 to 6 (#418) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v5...v6) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add sdwan mock api results * adjust mock-server file, create mock-server test * fix: resolve all ruff linting errors and warnings Fix code quality issues identified by ruff to ensure compliance with project style guidelines and best practices. Changes: - Fix syntax error in pabot.py (space in != operator) - Add missing filecmp import in test_integration.py - Add proper exception chaining (from e) for B904 violations - Rename unused loop variables to prefix with underscore (B007) - Remove trailing whitespace from blank lines (W293) - Modernize Union type annotations to use | operator (UP007) - Remove unused Union import after migration to | syntax - Exclude test fixture directories from ruff checks in pyproject.toml All ruff checks now pass successfully. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix pabot module import in tests/unit/test_pabot_args.py * no longer test for RESTinstance robot lib * add fixture to set bogus ACI_xxx environment for nac-test to pass rendering tests * add/fix --processes feature * fix/add ordering_file logic to robot orchestrator * fix/add passing extra cli args to robot orchestrator * add integration test for extra args * update README.md * fix ruff config changes broken during merge conflict resolution * fix: resolve all mypy type checking errors This commit addresses 11 mypy type errors across 4 files: - connection_utils.py: Add type annotations for dict parameters and update function signature to accept None values that are validated - subprocess_runner.py: Remove unreachable code branch in output processing that was flagged due to redundant None check - connection_broker.py: Add type annotations for testbed attribute, AsyncIterator return type, and assertion for loader result - robot_writer.py: Remove duplicate method definition and add explicit type annotation to prevent incorrect type narrowing All files now pass mypy strict type checking without errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: restore custom_data handling in render_template for chunked templates The refactoring that moved data conversion to initialization (for performance) inadvertently broke the chunked template functionality by not using the custom_data parameter when provided. The render_template method now correctly checks if custom_data is provided (used for chunked templates) and converts it on-demand, otherwise uses the pre-converted self.template_data. This fixes the test_nac_test_list_chunked test which was failing after merging main into the feature branch. Fixes functionality added in commits 18da546 and ced5983. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * ruff formatting fixes * fix: add type stub dependencies to pre-commit mypy hook The pre-commit mypy hook runs in an isolated environment and needs type stubs explicitly specified via additional_dependencies. Without these, mypy complains about missing type information for yaml, aiofiles, and markdown modules. Added: - types-PyYAML>=6.0.12.20250822 - types-aiofiles>=24.1.0.20250822 - types-markdown>=3.8.0.20250809 This aligns the pre-commit environment with the project dependencies and allows mypy to properly type-check imports from these libraries. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: add type annotations to unit test functions Add proper type annotations to all test functions in tests/unit to resolve mypy strict type checking errors. Changes include: - Add -> None return type annotations to all test methods - Add type annotations for mock parameters (mock_logger, mock_connection_class, etc.) using typing.Any - Add explicit type annotation for connections list variable - Add type: ignore comment for intentional type mismatch in test This ensures tests pass mypy strict type checking while maintaining test clarity and avoiding over-specification of mock types. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller tests from shell environment variables Add autouse fixture to clear all controller-related environment variables (ACI_*, SDWAN_*, CC_*, MERAKI_*, FMC_*, ISE_*) before each test in TestCombinedOrchestratorController. This ensures tests run in a clean environment and results are not influenced by the caller's shell environment, preventing false failures when developer has controller credentials set in their shell. The fixture uses pytest's monkeypatch to safely remove environment variables only for the duration of each test. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: migrate integration test fixture to use monkeypatch Replace manual environment variable setup/teardown with pytest's monkeypatch fixture for safer and more robust test isolation. Benefits of monkeypatch approach: - Automatic cleanup guaranteed even if test fails - Safe deletion with raising=False (no KeyError if var missing) - Preserves and restores original environment variable values - Less boilerplate code (no yield/try-finally needed) - Thread-safe state management handled by pytest The fixture now uses monkeypatch.setenv() which automatically saves the original value (if any) and restores it after the test, ensuring the caller's shell environment is never permanently modified. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller detection integration tests from shell environment Add autouse fixture to clear all controller-related environment variables before each test in TestControllerDetectionIntegration. This prevents test failures when developers have controller credentials set in their shell environment. Without this fix, tests would fail with "Multiple controller credentials detected" errors when shell variables conflicted with test-specific environment setup. The fixture uses pytest's monkeypatch.delenv() with raising=False for safe deletion that won't fail if a variable doesn't exist. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * add license headers * add missing test fixture template for extra args test * security: replace hardcoded /tmp with tempfile.gettempdir() in ConnectionBroker Fix Bandit B108 vulnerability by using tempfile.gettempdir() instead of hardcoded /tmp path for output_dir default value. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: add usedforsecurity=False to MD5 hash in AuthCache Fix Bandit B324 vulnerability by explicitly marking MD5 usage as not for security purposes. The hash is only used to generate cache filenames from URLs, not for cryptographic security. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp references in emergency dump Fix Bandit B108 vulnerabilities by: - Using tempfile.gettempdir() instead of hardcoded /tmp path - Dynamically checking if file is in temp directory instead of string matching "/tmp/" - Updated docstrings and log messages to reflect system temp dir usage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in AUTH_CACHE_DIR constant Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for AUTH_CACHE_DIR. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in batching reporter overflow dir Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for default overflow directory. This ensures cross-platform compatibility while still allowing override via NAC_TEST_OVERFLOW_DIR environment variable. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: enable Jinja2 autoescape to prevent XSS vulnerabilities Fix Bandit B701 vulnerability by enabling autoescape for HTML, XML, and .j2 files using select_autoescape(). This prevents potential XSS attacks when rendering user-controlled data in HTML reports. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * gitignore bandit-security-report.json created during pipeline run * Update readme cli help output * rename robot/pabot integration test file * Add netconf related packages and update dependencies * 1.2.2 version bump * adjust and fix aci integration tests * remove environment vars after testing * fix license check * remove unfinished tests * use tmpdir * add debugging for CI troubleshooting * attempt to fix event loop issue in CI * add more debug code to troubleshoot CI failure * add missing nac-test-pyats-common to CI pipeline * sum up passed/failed for check (in case we have multiple results.json files) * re-organize test case fixture directories use a common base dir, and then add specific archs as subdirectories to avoid directory bloat in fixture directory * Fix Python 3.12 asyncio compatibility with get_or_create_event_loop Python 3.12 changed asyncio.get_event_loop() behavior to raise RuntimeError when called from the main thread without an active event loop, breaking async operations that run synchronous PyATS/Unicon calls via loop.run_in_executor(). Previous behavior (Python 3.10-3.11): - asyncio.get_event_loop() automatically created a new event loop if none existed New behavior (Python 3.12+): - asyncio.get_event_loop() raises RuntimeError if no event loop exists - Must explicitly call asyncio.new_event_loop() and set_event_loop() Solution: Created get_or_create_event_loop() helper that handles both behaviors: - Tries to get existing event loop - If closed or RuntimeError, creates and sets a new loop - Works across Python 3.10-3.12+ Replaced 9 instances of asyncio.get_event_loop() across 4 modules: - nac_test/pyats_core/broker/connection_broker.py (3 calls) - nac_test/pyats_core/common/ssh_base_test.py (3 calls) - nac_test/pyats_core/ssh/connection_manager.py (2 calls) - nac_test/pyats_core/reporting/multi_archive_generator.py (1 call) These calls are used to run blocking Unicon operations (connect, disconnect, execute) in thread pools via run_in_executor() to prevent blocking the async event loop. * add sdwan integration tests (both api and d2d) * add license header * test: remove test_mock_server.py - tests mock infrastructure, not business logic These tests only verify that the MockAPIServer returns what it was configured to return: - test_mock_api_server_endpoint_matching: configures endpoints, asserts they return configured data - test_nac_test_with_mock_api_complex_urls: same pattern with complex URLs - test_nac_test_with_mock_api_dynamic: adds endpoint with response X, asserts response is X This violates the principle: "What MUST NEVER be Tested: whether mocks return what you told them to return." The mock server is infrastructure to enable testing real business logic, not something that needs its own test coverage. * test: remove test_controller_detection_consistency - tests Python determinism This test calls detect_controller_type() three times with identical environment and asserts all results are equal. This tests whether Python functions return consistent results when called with the same inputs - which is testing Python's deterministic execution, not business logic. A pure function will always return the same result for the same inputs. Testing this adds no value unless there's caching or stateful behavior that could cause inconsistency (there isn't). Violates: "What MUST NEVER be Tested: Whether Python's standard library works" * test: remove mock assertion from test_end_to_end_controller_detection Removed the pattern that mocks EnvironmentValidator and asserts it was called with the correct argument: with patch("...EnvironmentValidator") as mock_validator: orchestrator.validate_environment() mock_validator.validate_controller_env.assert_called_once_with("SDWAN") This tests whether validate_environment() calls EnvironmentValidator with the right argument - essentially testing "whether one-line wrapper functions call the functions they wrap." The real test is already done: assert orchestrator.controller_type == "SDWAN" which validates the business logic (controller was correctly detected). Violates: "What MUST NEVER be Tested: whether one-line wrapper functions call the functions they wrap" * fix(tests): remove conflicting copyright header and fix resource leak mock_unicon.py changes: - Removed Cisco proprietary copyright header (lines 5-7), keeping only MPL-2.0 - Fixed resource leak: file opened without context manager now uses 'with' statement Before: states = yaml.safe_load(open(os.path.join(mock_data_dir, file))) or [] After: with open(os.path.join(mock_data_dir, file)) as f: states = yaml.safe_load(f) or [] * fix(tests): implement proper server startup and shutdown for MockAPIServer mock_server.py changes: - Replaced time.sleep(0.5) race condition with proper readiness polling - Added _wait_for_server_ready() that polls until server responds or timeout - Implemented proper stop() method using werkzeug.serving.make_server - Added reset_endpoints() method to clear dynamic endpoints while preserving YAML-loaded baseline (enables test isolation with session-scoped fixture) - Added SERVER_STARTUP_TIMEOUT_SECONDS and SERVER_POLL_INTERVAL_SECONDS constants This fixes flaky test failures caused by server not being ready, and prevents port conflicts from improper shutdown. * fix(tests): add endpoint reset and isolated fixture for test isolation conftest.py changes: - Updated mock_api_server fixture to call reset_endpoints() after yield - Added mock_api_server_isolated fixture (function-scoped) for tests that add dynamic endpoints, preventing cross-test pollution The session-scoped mock_api_server now resets to baseline state between tests, while mock_api_server_isolated provides per-test isolation. * fix(tests): use absolute path and fix PEP8 naming in test_pyats_standard test_pyats_standard.py changes: - Fixed hardcoded relative path that breaks when run from different directories Now uses Path(__file__).parent to construct absolute path to mock_unicon.py - Renamed classes tc_one -> TcOne, tc_two -> TcTwo (PEP8 CamelCase) - Fixed decorator spacing: '@ aetest.test' -> '@aetest.test' * fix(tests): replace debug prints with logging and use monkeypatch test_integration_pyats.py changes: - Replaced all print() debug statements with logger.debug() calls - Moved 'import json' from inside function to module level - Refactored environment variable handling to use monkeypatch.setenv() instead of manual os.environ manipulation with try/finally - Removed commented-out debug code for static output directories This improves test maintainability and ensures proper cleanup via pytest's monkeypatch fixture. * fix(tests): modernize type hints and use monkeypatch in robot_pabot tests test_integration_robot_pabot.py changes: - Changed all tmpdir: str parameters to tmp_path: Path (modern pytest fixture) - Refactored environment variable handling to use monkeypatch.setenv() instead of manual os.environ with try/finally cleanup - Updated fixture references from tmpdir to tmp_path throughout - Used Path methods (.exists(), .touch()) instead of os.path equivalents This improves type safety and ensures proper test isolation via pytest's automatic cleanup of monkeypatched environment variables. * fix(tests): replace obfuscated code and emojis in PyATS quicksilver tests templates_pyats_qs test changes (3 files): - verify_sdwanmanager_all_sd_wan_edge_configurations_are_in_sync.py - verify_iosxe_all_sd_wan_control_connections_are_up.py - verify_aci_apic_appliance_operational_status.py Changes applied to all: - Replaced chr(10) with '\n' for clarity - Replaced emoji markers with text: ❌ -> [FAIL], ✅ -> [PASS] This prevents encoding issues in CI logs and improves code readability. * fix(tests): extract f-string join to variable for Python 3.11 compatibility Python 3.11 does not allow backslash escapes inside f-string expressions. The previous commit replaced chr(10) with '\n' but left it inside the f-string expression, causing a SyntaxError. This fix extracts the join operation to a separate variable before using it in the f-string: failures_text = '\n'.join(failures) f"{failures_text}\n\n" * fix(tests): add proxy bypass for localhost in integration tests Add session-scoped autouse fixture to ensure 127.0.0.1 is included in the no_proxy/NO_PROXY environment variables. This prevents the mock API server requests from being routed through corporate proxies, which was causing 504 Gateway Timeout errors in environments with proxy configured. The fixture preserves original proxy settings and restores them after the test session completes. * refactor(tests): split robot_pabot tests into focused modules Split the 493-line test_integration_robot_pabot.py into 4 focused test modules following Single Responsibility Principle: - test_cli_basic.py: Basic CLI execution tests (6 tests) - test_cli_rendering.py: Template rendering tests (8 tests) - test_cli_ordering.py: Robot test ordering tests (4 tests) - test_cli_extra_args.py: Extra arguments tests (5 tests) Improvements: - Added Google-style docstrings to all 23 test functions - Renamed generic test names to descriptive names - Added assertion messages to all assert statements - Extracted magic number to named constant (ROBOT_ARGUMENT_ERROR_EXIT_CODE) * add management_ip_variable to test fixture data --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: danischm <danischm@cisco.com> Co-authored-by: Oliver Boehmer <oli@spine.de> Co-authored-by: Oliver Boehmer <oboehmer@cisco.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Justyna Chowaniec <79261946+juchowan@users.noreply.github.com>
oboehmer
added a commit
that referenced
this pull request
Mar 19, 2026
* Bump pluggy from 1.5.0 to 1.6.0 (#365) Bumps [pluggy](https://github.com/pytest-dev/pluggy) from 1.5.0 to 1.6.0. - [Changelog](https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pluggy@1.5.0...1.6.0) --- updated-dependencies: - dependency-name: pluggy dependency-version: 1.6.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump click from 8.2.0 to 8.2.1 (#368) Bumps [click](https://github.com/pallets/click) from 8.2.0 to 8.2.1. - [Release notes](https://github.com/pallets/click/releases) - [Changelog](https://github.com/pallets/click/blob/main/CHANGES.rst) - [Commits](pallets/click@8.2.0...8.2.1) --- updated-dependencies: - dependency-name: click dependency-version: 8.2.1 dependency-type: indirect update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump typer from 0.15.3 to 0.16.0 (#371) Bumps [typer](https://github.com/fastapi/typer) from 0.15.3 to 0.16.0. - [Release notes](https://github.com/fastapi/typer/releases) - [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md) - [Commits](fastapi/typer@0.15.3...0.16.0) --- updated-dependencies: - dependency-name: typer dependency-version: 0.16.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump requests from 2.32.3 to 2.32.4 (#372) Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.3...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump urllib3 from 2.4.0 to 2.5.0 (#374) Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.4.0...2.5.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.5.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump rpds-py from 0.24.0 to 0.26.0 (#375) Bumps [rpds-py](https://github.com/crate-py/rpds) from 0.24.0 to 0.26.0. - [Release notes](https://github.com/crate-py/rpds/releases) - [Commits](crate-py/rpds@v0.24.0...v0.26.0) --- updated-dependencies: - dependency-name: rpds-py dependency-version: 0.26.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cryptography from 44.0.3 to 45.0.5 (#376) Bumps [cryptography](https://github.com/pyca/cryptography) from 44.0.3 to 45.0.5. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@44.0.3...45.0.5) --- updated-dependencies: - dependency-name: cryptography dependency-version: 45.0.5 dependency-type: indirect update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Migrate to uv * Refactor error handling * Various linter updates * Dependency updates * Update readme * Update dependencies and pyproject * Fix dependency * Fix robot dependency * Update versions * Bump astral-sh/setup-uv from 6 to 7 (#388) Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 6 to 7. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@v6...v7) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * License header updates * Update changelog * Update dependabot config for uv * Bump nac-yaml version * 1.1.0 version bump * Bump actions/upload-artifact from 4 to 5 (#391) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 5. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4...v5) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix lint workflow for fork prs * Expose pabot's process option to nac-test cli (#393) * Increase robot loglevel to debug when verbosity=DEBUG is used (#394) * Add robotframework-jmespath dependency (#395) * Update README to include jmespath and add test for 3rd party libs (#396) * Add support for test case parallelization for capable suites (#390) * Switch from pabot.main to pabot.main_program and handle exit codes (#400) Change nac-test to use pabot's main_program() instead of main() to avoid sys.exit() calls and properly handle exit codes. This allows nac-test to control the exit behavior and return appropriate exit codes to the caller. Key changes: - Update run_pabot() to call pabot.pabot.main_program() instead of main() - Change run_pabot() return type from None to int - Return exit code from pabot execution - Update CLI main() to capture and use exit code - Remove try/except wrapper in CLI main() - let exceptions propagate - Disable typer pretty exceptions for cleaner error output Benefits: - Proper exit code handling (test failures, errors, etc.) - No unexpected sys.exit() calls - Cleaner error messages without verbose typer tracebacks - Better integration with CI/CD systems 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * Add support for passing Robot Framework options to nac-test (#399) * Add support for passing Robot Framework options to nac-test Users can now pass additional Robot Framework options directly to nac-test, which are validated and appended to the pabot invocation. This enables advanced use cases like custom variables, listeners, and logging configuration without modifying nac-test's core options. Key changes: - Enable extra args in CLI via typer context settings - Add parse_and_validate_extra_args() to validate Robot Framework options - Reject pabot-specific options (--processes, --testlevelsplit, etc.) - Reject datasources/test files in extra arguments - Return exit code 252 for invalid extra arguments - Separate pabot_args and robot_args for proper argument ordering - Add comprehensive unit tests (14 tests covering all scenarios) - Add type hints and fix all mypy errors - Update CLI help text to document extra args feature Implementation details: - Uses pabot's parse_args() to validate options against Robot Framework - Filters out datasources and pabot-specific options - Appends validated args to robot_args before pabot invocation - Proper error handling with logging and exit codes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Remove pabot.main_program changes from robot-args branch This commit removes the pabot-specific changes (main_program, exit codes, exception handling) that belong in the separate pabot-call branch. The robot-args branch now focuses solely on the extra arguments feature. Changes: - Revert run_pabot() return type from int to None - Use pabot.pabot.main() instead of main_program() - Remove exit code handling (no return 252) - Keep try/except in CLI main() for proper error handling - Update tests to use pabot.pabot.main mock - Update tests to expect exceptions instead of error codes The extra args validation and appending logic remains unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Revert "Remove pabot.main_program changes from robot-args branch" This reverts commit df21b98. --------- Co-authored-by: Claude <noreply@anthropic.com> * Add support for chunking templates (#398) * Update changelog and readms * Update lock file * Expand test coverage for chunk split test (#401) * Fix typo in readme link * Remove library version numbers from readme * 1.2.0 version bump * Support rendering on windows (#404) * Bump actions/checkout from 5 to 6 (#402) Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update changelog * 1.2.1 version bump * Bump actions/upload-artifact from 5 to 6 (#418) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v5...v6) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: resolve all ruff linting errors and warnings Fix code quality issues identified by ruff to ensure compliance with project style guidelines and best practices. Changes: - Fix syntax error in pabot.py (space in != operator) - Add missing filecmp import in test_integration.py - Add proper exception chaining (from e) for B904 violations - Rename unused loop variables to prefix with underscore (B007) - Remove trailing whitespace from blank lines (W293) - Modernize Union type annotations to use | operator (UP007) - Remove unused Union import after migration to | syntax - Exclude test fixture directories from ruff checks in pyproject.toml All ruff checks now pass successfully. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix pabot module import in tests/unit/test_pabot_args.py * no longer test for RESTinstance robot lib * add fixture to set bogus ACI_xxx environment for nac-test to pass rendering tests * add/fix --processes feature * fix/add ordering_file logic to robot orchestrator * fix/add passing extra cli args to robot orchestrator * add integration test for extra args * update README.md * fix ruff config changes broken during merge conflict resolution * fix: resolve all mypy type checking errors This commit addresses 11 mypy type errors across 4 files: - connection_utils.py: Add type annotations for dict parameters and update function signature to accept None values that are validated - subprocess_runner.py: Remove unreachable code branch in output processing that was flagged due to redundant None check - connection_broker.py: Add type annotations for testbed attribute, AsyncIterator return type, and assertion for loader result - robot_writer.py: Remove duplicate method definition and add explicit type annotation to prevent incorrect type narrowing All files now pass mypy strict type checking without errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: restore custom_data handling in render_template for chunked templates The refactoring that moved data conversion to initialization (for performance) inadvertently broke the chunked template functionality by not using the custom_data parameter when provided. The render_template method now correctly checks if custom_data is provided (used for chunked templates) and converts it on-demand, otherwise uses the pre-converted self.template_data. This fixes the test_nac_test_list_chunked test which was failing after merging main into the feature branch. Fixes functionality added in commits 18da546 and ced5983. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * ruff formatting fixes * fix: add type stub dependencies to pre-commit mypy hook The pre-commit mypy hook runs in an isolated environment and needs type stubs explicitly specified via additional_dependencies. Without these, mypy complains about missing type information for yaml, aiofiles, and markdown modules. Added: - types-PyYAML>=6.0.12.20250822 - types-aiofiles>=24.1.0.20250822 - types-markdown>=3.8.0.20250809 This aligns the pre-commit environment with the project dependencies and allows mypy to properly type-check imports from these libraries. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: add type annotations to unit test functions Add proper type annotations to all test functions in tests/unit to resolve mypy strict type checking errors. Changes include: - Add -> None return type annotations to all test methods - Add type annotations for mock parameters (mock_logger, mock_connection_class, etc.) using typing.Any - Add explicit type annotation for connections list variable - Add type: ignore comment for intentional type mismatch in test This ensures tests pass mypy strict type checking while maintaining test clarity and avoiding over-specification of mock types. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller tests from shell environment variables Add autouse fixture to clear all controller-related environment variables (ACI_*, SDWAN_*, CC_*, MERAKI_*, FMC_*, ISE_*) before each test in TestCombinedOrchestratorController. This ensures tests run in a clean environment and results are not influenced by the caller's shell environment, preventing false failures when developer has controller credentials set in their shell. The fixture uses pytest's monkeypatch to safely remove environment variables only for the duration of each test. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: migrate integration test fixture to use monkeypatch Replace manual environment variable setup/teardown with pytest's monkeypatch fixture for safer and more robust test isolation. Benefits of monkeypatch approach: - Automatic cleanup guaranteed even if test fails - Safe deletion with raising=False (no KeyError if var missing) - Preserves and restores original environment variable values - Less boilerplate code (no yield/try-finally needed) - Thread-safe state management handled by pytest The fixture now uses monkeypatch.setenv() which automatically saves the original value (if any) and restores it after the test, ensuring the caller's shell environment is never permanently modified. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller detection integration tests from shell environment Add autouse fixture to clear all controller-related environment variables before each test in TestControllerDetectionIntegration. This prevents test failures when developers have controller credentials set in their shell environment. Without this fix, tests would fail with "Multiple controller credentials detected" errors when shell variables conflicted with test-specific environment setup. The fixture uses pytest's monkeypatch.delenv() with raising=False for safe deletion that won't fail if a variable doesn't exist. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * add license headers * add missing test fixture template for extra args test * security: replace hardcoded /tmp with tempfile.gettempdir() in ConnectionBroker Fix Bandit B108 vulnerability by using tempfile.gettempdir() instead of hardcoded /tmp path for output_dir default value. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: add usedforsecurity=False to MD5 hash in AuthCache Fix Bandit B324 vulnerability by explicitly marking MD5 usage as not for security purposes. The hash is only used to generate cache filenames from URLs, not for cryptographic security. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp references in emergency dump Fix Bandit B108 vulnerabilities by: - Using tempfile.gettempdir() instead of hardcoded /tmp path - Dynamically checking if file is in temp directory instead of string matching "/tmp/" - Updated docstrings and log messages to reflect system temp dir usage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in AUTH_CACHE_DIR constant Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for AUTH_CACHE_DIR. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in batching reporter overflow dir Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for default overflow directory. This ensures cross-platform compatibility while still allowing override via NAC_TEST_OVERFLOW_DIR environment variable. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: enable Jinja2 autoescape to prevent XSS vulnerabilities Fix Bandit B701 vulnerability by enabling autoescape for HTML, XML, and .j2 files using select_autoescape(). This prevents potential XSS attacks when rendering user-controlled data in HTML reports. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * gitignore bandit-security-report.json created during pipeline run * Update readme cli help output * rename robot/pabot integration test file * Add netconf related packages and update dependencies * 1.2.2 version bump * refactor: centralize DEBUG_MODE constant for progressive disclosure Consolidates the NAC_TEST_DEBUG environment variable check into a single constant in core/constants.py, eliminating repeated inline checks. Changes: - Add DEBUG_MODE constant to nac_test/core/constants.py - Update cli/main.py to use DEBUG_MODE for pretty_exceptions_enable - Update combined_orchestrator.py to use DEBUG_MODE for exception handling - Update batching_reporter.py to use DEBUG_MODE (2 locations) This enables progressive disclosure: customers get clean error output by default, while developers can set NAC_TEST_DEBUG=true for full exception context and verbose tracebacks when debugging. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: danischm <danischm@cisco.com> Co-authored-by: Oliver Boehmer <oli@spine.de> Co-authored-by: Oliver Boehmer <oboehmer@cisco.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Justyna Chowaniec <79261946+juchowan@users.noreply.github.com>
oboehmer
added a commit
that referenced
this pull request
Mar 19, 2026
* Bump pluggy from 1.5.0 to 1.6.0 (#365) Bumps [pluggy](https://github.com/pytest-dev/pluggy) from 1.5.0 to 1.6.0. - [Changelog](https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pluggy@1.5.0...1.6.0) --- updated-dependencies: - dependency-name: pluggy dependency-version: 1.6.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump click from 8.2.0 to 8.2.1 (#368) Bumps [click](https://github.com/pallets/click) from 8.2.0 to 8.2.1. - [Release notes](https://github.com/pallets/click/releases) - [Changelog](https://github.com/pallets/click/blob/main/CHANGES.rst) - [Commits](pallets/click@8.2.0...8.2.1) --- updated-dependencies: - dependency-name: click dependency-version: 8.2.1 dependency-type: indirect update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump typer from 0.15.3 to 0.16.0 (#371) Bumps [typer](https://github.com/fastapi/typer) from 0.15.3 to 0.16.0. - [Release notes](https://github.com/fastapi/typer/releases) - [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md) - [Commits](fastapi/typer@0.15.3...0.16.0) --- updated-dependencies: - dependency-name: typer dependency-version: 0.16.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump requests from 2.32.3 to 2.32.4 (#372) Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.3...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump urllib3 from 2.4.0 to 2.5.0 (#374) Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.4.0...2.5.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.5.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump rpds-py from 0.24.0 to 0.26.0 (#375) Bumps [rpds-py](https://github.com/crate-py/rpds) from 0.24.0 to 0.26.0. - [Release notes](https://github.com/crate-py/rpds/releases) - [Commits](crate-py/rpds@v0.24.0...v0.26.0) --- updated-dependencies: - dependency-name: rpds-py dependency-version: 0.26.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cryptography from 44.0.3 to 45.0.5 (#376) Bumps [cryptography](https://github.com/pyca/cryptography) from 44.0.3 to 45.0.5. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@44.0.3...45.0.5) --- updated-dependencies: - dependency-name: cryptography dependency-version: 45.0.5 dependency-type: indirect update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Migrate to uv * Refactor error handling * Various linter updates * Dependency updates * Update readme * Update dependencies and pyproject * Fix dependency * Fix robot dependency * Update versions * Bump astral-sh/setup-uv from 6 to 7 (#388) Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 6 to 7. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@v6...v7) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * License header updates * Update changelog * Update dependabot config for uv * Bump nac-yaml version * 1.1.0 version bump * Bump actions/upload-artifact from 4 to 5 (#391) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 5. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4...v5) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix lint workflow for fork prs * Expose pabot's process option to nac-test cli (#393) * Increase robot loglevel to debug when verbosity=DEBUG is used (#394) * Add robotframework-jmespath dependency (#395) * Update README to include jmespath and add test for 3rd party libs (#396) * Add support for test case parallelization for capable suites (#390) * Switch from pabot.main to pabot.main_program and handle exit codes (#400) Change nac-test to use pabot's main_program() instead of main() to avoid sys.exit() calls and properly handle exit codes. This allows nac-test to control the exit behavior and return appropriate exit codes to the caller. Key changes: - Update run_pabot() to call pabot.pabot.main_program() instead of main() - Change run_pabot() return type from None to int - Return exit code from pabot execution - Update CLI main() to capture and use exit code - Remove try/except wrapper in CLI main() - let exceptions propagate - Disable typer pretty exceptions for cleaner error output Benefits: - Proper exit code handling (test failures, errors, etc.) - No unexpected sys.exit() calls - Cleaner error messages without verbose typer tracebacks - Better integration with CI/CD systems 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> * Add support for passing Robot Framework options to nac-test (#399) * Add support for passing Robot Framework options to nac-test Users can now pass additional Robot Framework options directly to nac-test, which are validated and appended to the pabot invocation. This enables advanced use cases like custom variables, listeners, and logging configuration without modifying nac-test's core options. Key changes: - Enable extra args in CLI via typer context settings - Add parse_and_validate_extra_args() to validate Robot Framework options - Reject pabot-specific options (--processes, --testlevelsplit, etc.) - Reject datasources/test files in extra arguments - Return exit code 252 for invalid extra arguments - Separate pabot_args and robot_args for proper argument ordering - Add comprehensive unit tests (14 tests covering all scenarios) - Add type hints and fix all mypy errors - Update CLI help text to document extra args feature Implementation details: - Uses pabot's parse_args() to validate options against Robot Framework - Filters out datasources and pabot-specific options - Appends validated args to robot_args before pabot invocation - Proper error handling with logging and exit codes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Remove pabot.main_program changes from robot-args branch This commit removes the pabot-specific changes (main_program, exit codes, exception handling) that belong in the separate pabot-call branch. The robot-args branch now focuses solely on the extra arguments feature. Changes: - Revert run_pabot() return type from int to None - Use pabot.pabot.main() instead of main_program() - Remove exit code handling (no return 252) - Keep try/except in CLI main() for proper error handling - Update tests to use pabot.pabot.main mock - Update tests to expect exceptions instead of error codes The extra args validation and appending logic remains unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Revert "Remove pabot.main_program changes from robot-args branch" This reverts commit df21b98. --------- Co-authored-by: Claude <noreply@anthropic.com> * Add support for chunking templates (#398) * Update changelog and readms * Update lock file * Expand test coverage for chunk split test (#401) * Fix typo in readme link * Remove library version numbers from readme * 1.2.0 version bump * feat(path_setup): auto-discover pyats_common for test imports Add find_pyats_common_parent() to search test directory tree for pyats_common subdirectory and add its parent to PYTHONPATH. This enables test files to use simple imports like: from pyats_common.apic_base_test import APICTestBase 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * add verbose option to pyats when running at DEBUG level, log pyats output as well when in DEBUG * current state, not yet working * add simple test for pipeline - WIP * add integration tests and support for mock devices * use easypy.run to invoke script * do not assume a default CONTROLLER_TYPE * remove assumption for CONTROLLER_TYPE being ACI in base_test too * adjust test * add some debug log statements * add warning about potential issues with python3.11 on macOS * connect using log_stdout=False * fix simple test, and ruff formatting * simple test changes, WIP * Support rendering on windows (#404) * Bump actions/checkout from 5 to 6 (#402) Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update changelog * 1.2.1 version bump * update with mock flask server - WIP * remove debug logs * add mock-server to perform API testing in integration tests * add integration tests for quicksilver-generated templates * add type hints * Bump actions/upload-artifact from 5 to 6 (#418) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v5...v6) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add sdwan mock api results * adjust mock-server file, create mock-server test * fix: resolve all ruff linting errors and warnings Fix code quality issues identified by ruff to ensure compliance with project style guidelines and best practices. Changes: - Fix syntax error in pabot.py (space in != operator) - Add missing filecmp import in test_integration.py - Add proper exception chaining (from e) for B904 violations - Rename unused loop variables to prefix with underscore (B007) - Remove trailing whitespace from blank lines (W293) - Modernize Union type annotations to use | operator (UP007) - Remove unused Union import after migration to | syntax - Exclude test fixture directories from ruff checks in pyproject.toml All ruff checks now pass successfully. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix pabot module import in tests/unit/test_pabot_args.py * no longer test for RESTinstance robot lib * add fixture to set bogus ACI_xxx environment for nac-test to pass rendering tests * add/fix --processes feature * fix/add ordering_file logic to robot orchestrator * fix/add passing extra cli args to robot orchestrator * add integration test for extra args * update README.md * fix ruff config changes broken during merge conflict resolution * fix: resolve all mypy type checking errors This commit addresses 11 mypy type errors across 4 files: - connection_utils.py: Add type annotations for dict parameters and update function signature to accept None values that are validated - subprocess_runner.py: Remove unreachable code branch in output processing that was flagged due to redundant None check - connection_broker.py: Add type annotations for testbed attribute, AsyncIterator return type, and assertion for loader result - robot_writer.py: Remove duplicate method definition and add explicit type annotation to prevent incorrect type narrowing All files now pass mypy strict type checking without errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: restore custom_data handling in render_template for chunked templates The refactoring that moved data conversion to initialization (for performance) inadvertently broke the chunked template functionality by not using the custom_data parameter when provided. The render_template method now correctly checks if custom_data is provided (used for chunked templates) and converts it on-demand, otherwise uses the pre-converted self.template_data. This fixes the test_nac_test_list_chunked test which was failing after merging main into the feature branch. Fixes functionality added in commits 18da546 and ced5983. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * ruff formatting fixes * fix: add type stub dependencies to pre-commit mypy hook The pre-commit mypy hook runs in an isolated environment and needs type stubs explicitly specified via additional_dependencies. Without these, mypy complains about missing type information for yaml, aiofiles, and markdown modules. Added: - types-PyYAML>=6.0.12.20250822 - types-aiofiles>=24.1.0.20250822 - types-markdown>=3.8.0.20250809 This aligns the pre-commit environment with the project dependencies and allows mypy to properly type-check imports from these libraries. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: add type annotations to unit test functions Add proper type annotations to all test functions in tests/unit to resolve mypy strict type checking errors. Changes include: - Add -> None return type annotations to all test methods - Add type annotations for mock parameters (mock_logger, mock_connection_class, etc.) using typing.Any - Add explicit type annotation for connections list variable - Add type: ignore comment for intentional type mismatch in test This ensures tests pass mypy strict type checking while maintaining test clarity and avoiding over-specification of mock types. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller tests from shell environment variables Add autouse fixture to clear all controller-related environment variables (ACI_*, SDWAN_*, CC_*, MERAKI_*, FMC_*, ISE_*) before each test in TestCombinedOrchestratorController. This ensures tests run in a clean environment and results are not influenced by the caller's shell environment, preventing false failures when developer has controller credentials set in their shell. The fixture uses pytest's monkeypatch to safely remove environment variables only for the duration of each test. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: migrate integration test fixture to use monkeypatch Replace manual environment variable setup/teardown with pytest's monkeypatch fixture for safer and more robust test isolation. Benefits of monkeypatch approach: - Automatic cleanup guaranteed even if test fails - Safe deletion with raising=False (no KeyError if var missing) - Preserves and restores original environment variable values - Less boilerplate code (no yield/try-finally needed) - Thread-safe state management handled by pytest The fixture now uses monkeypatch.setenv() which automatically saves the original value (if any) and restores it after the test, ensuring the caller's shell environment is never permanently modified. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: isolate controller detection integration tests from shell environment Add autouse fixture to clear all controller-related environment variables before each test in TestControllerDetectionIntegration. This prevents test failures when developers have controller credentials set in their shell environment. Without this fix, tests would fail with "Multiple controller credentials detected" errors when shell variables conflicted with test-specific environment setup. The fixture uses pytest's monkeypatch.delenv() with raising=False for safe deletion that won't fail if a variable doesn't exist. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * add license headers * add missing test fixture template for extra args test * security: replace hardcoded /tmp with tempfile.gettempdir() in ConnectionBroker Fix Bandit B108 vulnerability by using tempfile.gettempdir() instead of hardcoded /tmp path for output_dir default value. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: add usedforsecurity=False to MD5 hash in AuthCache Fix Bandit B324 vulnerability by explicitly marking MD5 usage as not for security purposes. The hash is only used to generate cache filenames from URLs, not for cryptographic security. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp references in emergency dump Fix Bandit B108 vulnerabilities by: - Using tempfile.gettempdir() instead of hardcoded /tmp path - Dynamically checking if file is in temp directory instead of string matching "/tmp/" - Updated docstrings and log messages to reflect system temp dir usage Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in AUTH_CACHE_DIR constant Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for AUTH_CACHE_DIR. This ensures cross-platform compatibility and respects system-specific temporary directory configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: replace hardcoded /tmp in batching reporter overflow dir Fix Bandit B108 vulnerability by using os.path.join with tempfile.gettempdir() instead of hardcoded /tmp path for default overflow directory. This ensures cross-platform compatibility while still allowing override via NAC_TEST_OVERFLOW_DIR environment variable. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * security: enable Jinja2 autoescape to prevent XSS vulnerabilities Fix Bandit B701 vulnerability by enabling autoescape for HTML, XML, and .j2 files using select_autoescape(). This prevents potential XSS attacks when rendering user-controlled data in HTML reports. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * gitignore bandit-security-report.json created during pipeline run * Update readme cli help output * rename robot/pabot integration test file * Add netconf related packages and update dependencies * 1.2.2 version bump * adjust and fix aci integration tests * remove environment vars after testing * fix license check * remove unfinished tests * use tmpdir * add debugging for CI troubleshooting * attempt to fix event loop issue in CI * add more debug code to troubleshoot CI failure * add missing nac-test-pyats-common to CI pipeline * sum up passed/failed for check (in case we have multiple results.json files) * re-organize test case fixture directories use a common base dir, and then add specific archs as subdirectories to avoid directory bloat in fixture directory * Fix Python 3.12 asyncio compatibility with get_or_create_event_loop Python 3.12 changed asyncio.get_event_loop() behavior to raise RuntimeError when called from the main thread without an active event loop, breaking async operations that run synchronous PyATS/Unicon calls via loop.run_in_executor(). Previous behavior (Python 3.10-3.11): - asyncio.get_event_loop() automatically created a new event loop if none existed New behavior (Python 3.12+): - asyncio.get_event_loop() raises RuntimeError if no event loop exists - Must explicitly call asyncio.new_event_loop() and set_event_loop() Solution: Created get_or_create_event_loop() helper that handles both behaviors: - Tries to get existing event loop - If closed or RuntimeError, creates and sets a new loop - Works across Python 3.10-3.12+ Replaced 9 instances of asyncio.get_event_loop() across 4 modules: - nac_test/pyats_core/broker/connection_broker.py (3 calls) - nac_test/pyats_core/common/ssh_base_test.py (3 calls) - nac_test/pyats_core/ssh/connection_manager.py (2 calls) - nac_test/pyats_core/reporting/multi_archive_generator.py (1 call) These calls are used to run blocking Unicon operations (connect, disconnect, execute) in thread pools via run_in_executor() to prevent blocking the async event loop. * add sdwan integration tests (both api and d2d) * add license header * test: remove test_mock_server.py - tests mock infrastructure, not business logic These tests only verify that the MockAPIServer returns what it was configured to return: - test_mock_api_server_endpoint_matching: configures endpoints, asserts they return configured data - test_nac_test_with_mock_api_complex_urls: same pattern with complex URLs - test_nac_test_with_mock_api_dynamic: adds endpoint with response X, asserts response is X This violates the principle: "What MUST NEVER be Tested: whether mocks return what you told them to return." The mock server is infrastructure to enable testing real business logic, not something that needs its own test coverage. * test: remove test_controller_detection_consistency - tests Python determinism This test calls detect_controller_type() three times with identical environment and asserts all results are equal. This tests whether Python functions return consistent results when called with the same inputs - which is testing Python's deterministic execution, not business logic. A pure function will always return the same result for the same inputs. Testing this adds no value unless there's caching or stateful behavior that could cause inconsistency (there isn't). Violates: "What MUST NEVER be Tested: Whether Python's standard library works" * test: remove mock assertion from test_end_to_end_controller_detection Removed the pattern that mocks EnvironmentValidator and asserts it was called with the correct argument: with patch("...EnvironmentValidator") as mock_validator: orchestrator.validate_environment() mock_validator.validate_controller_env.assert_called_once_with("SDWAN") This tests whether validate_environment() calls EnvironmentValidator with the right argument - essentially testing "whether one-line wrapper functions call the functions they wrap." The real test is already done: assert orchestrator.controller_type == "SDWAN" which validates the business logic (controller was correctly detected). Violates: "What MUST NEVER be Tested: whether one-line wrapper functions call the functions they wrap" * fix(tests): remove conflicting copyright header and fix resource leak mock_unicon.py changes: - Removed Cisco proprietary copyright header (lines 5-7), keeping only MPL-2.0 - Fixed resource leak: file opened without context manager now uses 'with' statement Before: states = yaml.safe_load(open(os.path.join(mock_data_dir, file))) or [] After: with open(os.path.join(mock_data_dir, file)) as f: states = yaml.safe_load(f) or [] * fix(tests): implement proper server startup and shutdown for MockAPIServer mock_server.py changes: - Replaced time.sleep(0.5) race condition with proper readiness polling - Added _wait_for_server_ready() that polls until server responds or timeout - Implemented proper stop() method using werkzeug.serving.make_server - Added reset_endpoints() method to clear dynamic endpoints while preserving YAML-loaded baseline (enables test isolation with session-scoped fixture) - Added SERVER_STARTUP_TIMEOUT_SECONDS and SERVER_POLL_INTERVAL_SECONDS constants This fixes flaky test failures caused by server not being ready, and prevents port conflicts from improper shutdown. * fix(tests): add endpoint reset and isolated fixture for test isolation conftest.py changes: - Updated mock_api_server fixture to call reset_endpoints() after yield - Added mock_api_server_isolated fixture (function-scoped) for tests that add dynamic endpoints, preventing cross-test pollution The session-scoped mock_api_server now resets to baseline state between tests, while mock_api_server_isolated provides per-test isolation. * fix(tests): use absolute path and fix PEP8 naming in test_pyats_standard test_pyats_standard.py changes: - Fixed hardcoded relative path that breaks when run from different directories Now uses Path(__file__).parent to construct absolute path to mock_unicon.py - Renamed classes tc_one -> TcOne, tc_two -> TcTwo (PEP8 CamelCase) - Fixed decorator spacing: '@ aetest.test' -> '@aetest.test' * fix(tests): replace debug prints with logging and use monkeypatch test_integration_pyats.py changes: - Replaced all print() debug statements with logger.debug() calls - Moved 'import json' from inside function to module level - Refactored environment variable handling to use monkeypatch.setenv() instead of manual os.environ manipulation with try/finally - Removed commented-out debug code for static output directories This improves test maintainability and ensures proper cleanup via pytest's monkeypatch fixture. * fix(tests): modernize type hints and use monkeypatch in robot_pabot tests test_integration_robot_pabot.py changes: - Changed all tmpdir: str parameters to tmp_path: Path (modern pytest fixture) - Refactored environment variable handling to use monkeypatch.setenv() instead of manual os.environ with try/finally cleanup - Updated fixture references from tmpdir to tmp_path throughout - Used Path methods (.exists(), .touch()) instead of os.path equivalents This improves type safety and ensures proper test isolation via pytest's automatic cleanup of monkeypatched environment variables. * fix(tests): replace obfuscated code and emojis in PyATS quicksilver tests templates_pyats_qs test changes (3 files): - verify_sdwanmanager_all_sd_wan_edge_configurations_are_in_sync.py - verify_iosxe_all_sd_wan_control_connections_are_up.py - verify_aci_apic_appliance_operational_status.py Changes applied to all: - Replaced chr(10) with '\n' for clarity - Replaced emoji markers with text: ❌ -> [FAIL], ✅ -> [PASS] This prevents encoding issues in CI logs and improves code readability. * fix(tests): extract f-string join to variable for Python 3.11 compatibility Python 3.11 does not allow backslash escapes inside f-string expressions. The previous commit replaced chr(10) with '\n' but left it inside the f-string expression, causing a SyntaxError. This fix extracts the join operation to a separate variable before using it in the f-string: failures_text = '\n'.join(failures) f"{failures_text}\n\n" * fix(tests): add proxy bypass for localhost in integration tests Add session-scoped autouse fixture to ensure 127.0.0.1 is included in the no_proxy/NO_PROXY environment variables. This prevents the mock API server requests from being routed through corporate proxies, which was causing 504 Gateway Timeout errors in environments with proxy configured. The fixture preserves original proxy settings and restores them after the test session completes. * refactor(tests): split robot_pabot tests into focused modules Split the 493-line test_integration_robot_pabot.py into 4 focused test modules following Single Responsibility Principle: - test_cli_basic.py: Basic CLI execution tests (6 tests) - test_cli_rendering.py: Template rendering tests (8 tests) - test_cli_ordering.py: Robot test ordering tests (4 tests) - test_cli_extra_args.py: Extra arguments tests (5 tests) Improvements: - Added Google-style docstrings to all 23 test functions - Renamed generic test names to descriptive names - Added assertion messages to all assert statements - Extracted magic number to named constant (ROBOT_ARGUMENT_ERROR_EXIT_CODE) * add management_ip_variable to test fixture data --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: danischm <danischm@cisco.com> Co-authored-by: Oliver Boehmer <oli@spine.de> Co-authored-by: Oliver Boehmer <oboehmer@cisco.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Justyna Chowaniec <79261946+juchowan@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
had to add
setuptools<81asRESTinstancelib failed to load in python 3.12