prevent interference by other pytest plugins

[alvin-r]

User description

Two ways:

1. Prevent discovery of tests with benchmark fixtures during test discovery.

• This is necessary for any plugin that uses fixtures. If you disable it, the fixtures will throw an error.

2. Add a blocklist of plugins

• when running test, simply use -p no: {pluginname} to prevent the plugin from being used.

---

PR Type

• Enhancement
• Tests

---

Description

• Added hook to skip tests with benchmark fixtures.

• Injected plugin blocklist flags in test runner commands.

• Created unit test to verify benchmark test exclusion.

---

Changes walkthrough 📝

	Relevant files
Enhancement	pytest_new_process_discovery.py Add hook to skip benchmark tests                                                  codeflash/discovery/pytest_new_process_discovery.py Introduced pytest_collection_modifyitems hook. Applies skip marker for benchmark tests. Adjusted pytest.main command parameters. +6/-1      test_runner.py Integrate plugin blocklist into test runner                            codeflash/verification/test_runner.py Defined constants for plugin blocklists. Added blocklist args to test subprocess commands. Improved test execution filtering. +9/-3
Tests	test_unit_test_discovery.py Add unit test for benchmark exclusion                                        tests/test_unit_test_discovery.py Added new test for benchmark test discovery. Confirms benchmark tests are skipped. Validates correct test function discovery. +42/-0

---

Need help?

Type /help how to ... in the comments thread for any questions about PR-Agent usage.

Check out the documentation for more information.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Fixture Filtering Verify that the new marker addition in the pytest_collection_modifyitems hook correctly identifies all benchmark fixture cases. Consider edge cases such as partial matches or case sensitivity. def pytest_collection_modifyitems(config, items): skip_benchmark = pytest.mark.skip(reason="Skipping benchmark tests") for item in items: if "benchmark" in item.fixturenames: item.add_marker(skip_benchmark) Plugin Blocklist Integration Review the concatenation of blocklist arguments with other pytest command-line arguments to ensure that disabling of plugins works reliably and does not conflict with existing options. blocklist_args = [f"-p no:{plugin}" for plugin in BEHAVIORAL_BLOCKLISTED_PLUGINS if plugin != "cov"] results = execute_test_subprocess( coverage_cmd + common_pytest_args + blocklist_args + result_args + test_files, cwd=cwd, env=pytest_test_env, timeout=600 ) logger.debug( f"Result return code: {results.returncode}, " f"{'Result stderr:' + str(results.stderr) if results.stderr else ''}" ) else: blocklist_args = [f"-p no:{plugin}" for plugin in BEHAVIORAL_BLOCKLISTED_PLUGINS] results = execute_test_subprocess( pytest_cmd_list + common_pytest_args + blocklist_args + result_args + test_files, cwd=cwd,

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Fix method signature Add the missing self parameter to the hook method so it adheres to the expected instance method signature. codeflash/discovery/pytest_new_process_discovery.py [19-23] -def pytest_collection_modifyitems(config, items): +def pytest_collection_modifyitems(self, config, items): skip_benchmark = pytest.mark.skip(reason="Skipping benchmark tests") for item in items: if "benchmark" in item.fixturenames: item.add_marker(skip_benchmark) Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies that the hook method should include the 'self' parameter to be consistent with its class-based implementation, addressing a likely runtime issue.	High