Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should run full test if there is API, SDK or CLI version bump #329

Open
jiasli opened this issue Mar 29, 2022 · 1 comment
Open

Should run full test if there is API, SDK or CLI version bump #329

jiasli opened this issue Mar 29, 2022 · 1 comment

Comments

@jiasli
Copy link
Member

jiasli commented Mar 29, 2022

Currently, for SDK or API version bump, only incremental tests are run. This frequently leads to dev branch Batched CI failure because the new API or SDK conflicts with other modules, mainly due to API version mismatch, such as Azure/azure-cli#21681 (comment):

https://dev.azure.com/azure-sdk/public/_build/results?buildId=1466412&view=logs&j=ad51cf76-b294-5bde-777e-be77cc5f3050&t=0e02ab23-4098-514d-f7b9-ab422251bbb2

2022-03-29T02:16:29.3378803Z         except CannotOverwriteExistingCassetteException as ex:
2022-03-29T02:16:29.3379238Z >           raise AssertionError(ex)
2022-03-29T02:16:29.3380350Z E           AssertionError: Can't overwrite existing cassette ('/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/monitor/tests/latest/recordings/test_metric_alert_for_sql_database_scope.yaml') in your current record mode ('once').
2022-03-29T02:16:29.3382083Z E           No match for the request (<Request (PUT) https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli_test_metric_alert_v1_2000001/providers/Microsoft.Sql/servers/clitestservermatricalertA000002/databases/cliautomationdb01?api-version=2021-05-01-preview>) was found.
2022-03-29T02:16:29.3383114Z E           Found 1 similar requests with 1 different matcher(s) :
2022-03-29T02:16:29.3383519Z E           
2022-03-29T02:16:29.3384713Z E           1 - (<Request (PUT) https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli_test_metric_alert_v1_2000001/providers/Microsoft.Sql/servers/clitestservermatricalertA000002/databases/cliautomationdb01?api-version=2021-02-01-preview>).
2022-03-29T02:16:29.3385863Z E           Matchers succeeded : ['method', 'scheme', 'host', 'port', 'path']
2022-03-29T02:16:29.3386340Z E           Matchers failed :
2022-03-29T02:16:29.3386940Z E           _custom_request_query_matcher - assertion failure :
2022-03-29T02:16:29.3387367Z E           None
2022-03-29T02:16:29.3387562Z 
2022-03-29T02:16:29.3388186Z /opt/az/lib/python3.8/site-packages/azure/cli/testsdk/base.py:299: AssertionError
2022-03-29T02:16:29.3389118Z ----------------------------- Captured stdout call -----------------------------
2022-03-29T02:16:29.3389583Z None
2022-03-29T02:16:29.3390202Z ------------ generated xml file: /azure_cli_test_result/monitor.xml ------------
2022-03-29T02:16:29.3390794Z =========================== short test summary info ============================
2022-03-29T02:16:29.3391462Z FAILED tests/latest/test_monitor_metric_alert_scenarios.py::MonitorTests::test_metric_alert_for_sql_database_scope

The solution is to run full test if contents in these files are changed:

  • src/azure-cli-core/azure/cli/core/profiles/_shared.py
  • src/azure-cli/requirements.py3.Darwin.txt
  • src/azure-cli/requirements.py3.Linux.txt
  • src/azure-cli/requirements.py3.windows.txt
@jiasli
Copy link
Member Author

jiasli commented Apr 1, 2022

CLI version bump also deserves a full test, otherwise deprecation-related linter check will fail: Azure/azure-cli#21892

@jiasli jiasli changed the title Should run full test if there is API or SDK version bump Should run full test if there is API, SDK or CLI version bump Apr 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant