Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AzureML tests: Durations, disable warnings, logs less verbose #1864

Merged
merged 26 commits into from
Feb 17, 2023

Conversation

miguelgfierro
Copy link
Collaborator

@miguelgfierro miguelgfierro commented Nov 29, 2022

Description

Add parameters to pytest: durations and disable warnings

Related Issues

Fix #1857

References

Checklist:

  • I have followed the contribution guidelines and code style for this project.
  • I have added tests covering my contributions.
  • I have updated the documentation accordingly.
  • This PR is being made to staging branch and not to main branch.

@miguelgfierro
Copy link
Collaborator Author

miguelgfierro commented Nov 29, 2022

Error:

Run python tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py --clustername cpu-cluster --subid *** --reponame "recommenders" --branch refs/pull/1864/merge --rg recommenders_project_resources --wsname azureml-test-workspace --expname unit_tests_group_cpu_001 --testlogs "test_logs.log" --testkind unit --conda_pkg_python "python=3.7" --testgroup group_cpu_001 --pytestargs "--durations 0 --disable-warnings" || exit -1
'enabled' is deprecated. Please use the azureml.core.runconfig.DockerConfiguration object with the 'use_docker' param instead.
Traceback (most recent call last):
  File "tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py", line 496, in <module>
    run = submit_experiment_to_azureml(
  File "tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py", line 279, in submit_experiment_to_azureml
    run = experiment.submit(script_run_config)
  File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/azureml/core/experiment.py", line 238, in submit
    run = submit_func(config, self.workspace, self.name, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/azureml/core/script_run_config.py", line 63, in submit
    run = _commands.start_run(project, run_config,
  File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/azureml/_execution/_commands.py", line 120, in start_run
    return _start_internal(project_object, run_config_object, prepare_check=check,
  File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/azureml/_execution/_commands.py", line 429, in _start_internal
    _raise_request_error(response, "starting run")
  File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/azureml/_execution/_commands.py", line 570, in _raise_request_error
    raise ExperimentExecutionException(response_message)
azureml.exceptions._azureml_exception.ExperimentExecutionException: ExperimentExecutionException:
	Message: ***
    "error_details": ***
        "componentName": "execution",
        "correlation": ***
            "operation": "bbf2bc1f2857aa1eed9cb0763cae70c1",
            "request": "60d4b5b6588c36c0"
        ***,
        "environment": "eastus",
        "error": ***
            "code": "UserError",
            "message": "Error when parsing request; unable to deserialize request body"
        ***,
        "location": "eastus",
        "time": "2022-11-29T11:00:27.1901787+00:00"
    ***,
    "status_code": 400,
    "url": "https://eastus.experiments.azureml.net/execution/v1.0/subscriptions/***/resourceGroups/recommenders_project_resources/providers/Microsoft.MachineLearningServices/workspaces/azureml-test-workspace/experiments/unit_tests_group_cpu_001/snapshotrun?runId=unit_tests_group_cpu_001_1669719624_ba26ad72"
***
	InnerException None
	ErrorResponse 
***
    "error": ***
        "message": "***\n    \"error_details\": ***\n        \"componentName\": \"execution\",\n        \"correlation\": ***\n            \"operation\": \"bbf2bc1f2857aa1eed9cb0763cae70c1\",\n            \"request\": \"60d4b5b6588c36c0\"\n        ***,\n        \"environment\": \"eastus\",\n        \"error\": ***\n            \"code\": \"UserError\",\n            \"message\": \"Error when parsing request; unable to deserialize request body\"\n        ***,\n        \"location\": \"eastus\",\n        \"time\": \"2022-11-29T11:00:27.1901787+00:00\"\n    ***,\n    \"status_code\": 400,\n    \"url\": 

fixed adding the argument as a string:

    parser.add_argument(
        "--pytestargs",
        action="store",
        default="",
        help="Additional arguments to pass to pytest as a string. Multiple arguments should be separated by space.",
    )

@miguelgfierro
Copy link
Collaborator Author

miguelgfierro commented Nov 30, 2022

Error with pytest_exit_code = pytest.main(args=[test_group, args.pytestargs]) in the file run_groupwise_pytest.py.

INFO:submit_groupwise_azureml_pytest.py:Tests to be executed
INFO:submit_groupwise_azureml_pytest.py:['tests/unit/recommenders/datasets/test_movielens.py::test_load_pandas_df_mock_100__with_default_param__succeed', 'tests/unit/recommenders/datasets/test_dataset.py::test_maybe_download_wrong_bytes', 'tests/unit/recommenders/datasets/test_movielens.py::test_mock_movielens_schema__has_default_col_names', 'tests/unit/recommenders/datasets/test_movielens.py::test_load_pandas_df_mock_100__with_custom_param__succeed', 'tests/unit/recommenders/datasets/test_dataset.py::test_maybe_download_retry', 'tests/unit/recommenders/datasets/test_movielens.py::test_mock_movielens_schema__get_df__return_success', 'tests/unit/recommenders/utils/test_timer.py::test_timer', 'tests/unit/recommenders/tuning/test_ncf_utils.py::test_compute_test_results__return_success', 'tests/unit/recommenders/datasets/test_movielens.py::test_mock_movielens_schema__get_df_remove_default_col__return_success', 'tests/unit/recommenders/models/test_geoimc.py::test_imcproblem', 'tests/unit/recommenders/datasets/test_wikidata.py::test_find_wikidata_id', 'tests/unit/recommenders/models/test_sar_singlenode.py::test_sar_item_similarity', 'tests/unit/recommenders/models/test_tfidf_utils.py::test_tokenize_text', 'tests/unit/recommenders/models/test_tfidf_utils.py::test_get_tokens', 'tests/unit/recommenders/models/test_cornac_utils.py::test_recommend_k_items', 'tests/unit/recommenders/evaluation/test_python_evaluation_time_performance.py']
Traceback (most recent call last):
  File "tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py", line 496, in <module>
    run = submit_experiment_to_azureml(
  File "tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py", line 281, in submit_experiment_to_azureml
    run.wait_for_completion(show_output=True, wait_post_processing=True)
  File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/azureml/core/run.py", line 843, in wait_for_completion
    self._stream_run_output(
  File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/azureml/core/run.py", line 1096, in _stream_run_output
    raise ActivityFailedException(error_details=json.dumps(error, indent=4))
azureml.exceptions._azureml_exception.ActivityFailedException: ActivityFailedException:
	Message: Activity Failed:
***
    "error": ***
        "code": "UserError",
        "message": "***\"NonCompliant\":\"Process '/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/bin/python' exited with code 1 and error message 'Execution failed. Process exited with status code 1. Error:   File \\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/helpconfig.py\\\", line 103, in pytest_cmdline_parse\\n    config: Config = outcome.get_result()\\n  File \\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/pluggy/_result.py\\\", line 60, in get_result\\n    raise ex[1].with_traceback(ex[2])\\n  File \\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/pluggy/_callers.py\\\", line 39, in _multicall\\n    res = hook_impl.function(*args)\\n  File \\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py\\\", line 1058, in pytest_cmdline_parse\\n    self.parse(args)\\n  File \\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py\\\", line 1346, in parse\\n    self._preparse(args, addopts=addopts)\\n  File \\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py\\\", line 1214, in _preparse\\n    self._initini(args)\\n  File \\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py\\\", line 1130, in _initini\\n    ns, unknown_args = self._parser.parse_known_and_unknown_args(\\n  File \\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/argparsing.py\\\", line 172, in parse_known_and_unknown_args\\n    strargs = [os.fspath(x) for x in args]\\n  File \\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/argparsing.py\\\", line 172, in <listcomp>\\n    strargs = [os.fspath(x) for x in args]\\nTypeError: expected str, bytes or os.PathLike object, not list\\n\\n'. Please check the log file 'user_logs/std_log.txt' for more details.\"***\n***\n  \"code\": \"ExecutionFailed\",\n  \"target\": \"\",\n  \"category\": \"UserError\",\n  \"error_details\": [\n    ***\n      \"key\": \"exit_codes\",\n      \"value\": \"1\"\n    ***\n  ]\n***",
        "messageParameters": ***,
        "details": []
    ***,
    "time": "0001-01-01T00:00:00.000Z"
***
	InnerException None
	ErrorResponse 
***
    "error": ***
        "message": "Activity Failed:\n***\n    \"error\": ***\n        \"code\": \"UserError\",\n        \"message\": \"***\\\"NonCompliant\\\":\\\"Process '/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/bin/python' exited with code 1 and error message 'Execution failed. Process exited with status code 1. Error:   File \\\\\\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/helpconfig.py\\\\\\\", line 103, in pytest_cmdline_parse\\\\n    config: Config = outcome.get_result()\\\\n  File \\\\\\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/pluggy/_result.py\\\\\\\", line 60, in get_result\\\\n    raise ex[1].with_traceback(ex[2])\\\\n  File \\\\\\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/pluggy/_callers.py\\\\\\\", line 39, in _multicall\\\\n    res = hook_impl.function(*args)\\\\n  File \\\\\\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py\\\\\\\", line 1058, in pytest_cmdline_parse\\\\n    self.parse(args)\\\\n  File \\\\\\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py\\\\\\\", line 1346, in parse\\\\n    self._preparse(args, addopts=addopts)\\\\n  File \\\\\\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py\\\\\\\", line 1214, in _preparse\\\\n    self._initini(args)\\\\n  File \\\\\\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py\\\\\\\", line 1130, in _initini\\\\n    ns, unknown_args = self._parser.parse_known_and_unknown_args(\\\\n  File \\\\\\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/argparsing.py\\\\\\\", line 172, in parse_known_and_unknown_args\\\\n    strargs = [os.fspath(x) for x in args]\\\\n  File \\\\\\\"/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/argparsing.py\\\\\\\", line 172, in <listcomp>\\\\n    strargs = [os.fspath(x) for x in args]\\\\nTypeError: expected str, bytes or os.PathLike object, not list\\\\n\\\\n'. Please check the log file 'user_logs/std_log.txt' for more details.\\\"***\\n***\\n  \\\"code\\\": \\\"ExecutionFailed\\\",\\n  \\\"target\\\": \\\"\\\",\\n  \\\"category\\\": \\\"UserError\\\",\\n  \\\"error_details\\\": [\\n    ***\\n      \\\"key\\\": \\\"exit_codes\\\",\\n      \\\"value\\\": \\\"1\\\"\\n    ***\\n  ]\\n***\",\n        \"messageParameters\": ***,\n        \"details\": []\n    ***,\n    \"time\": \"0001-01-01T00:00:00.000Z\"\n***"
    ***
***
INFO:azureml._restclient.clientbase:Created a worker pool for first use
INFO:submit_groupwise_azureml_pytest.py:Python version 
INFO:submit_groupwise_azureml_pytest.py:3.9.12 (main, Jun  1 2022, 11:38:51) 
[GCC 7.5.0]
INFO:submit_groupwise_azureml_pytest.py:Executing tests now...
Cleaning up all outstanding Run operations, waiting 300.0 seconds
1 items cleaning up...
Cleanup took 0.07751345634460449 seconds
Traceback (most recent call last):
  File "tests/ci/azureml_tests/run_groupwise_pytest.py", line 64, in <module>
    pytest_exit_code = pytest.main(args=[test_group, args.pytestargs])
  File "/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py", line 148, in main
    config = _prepareconfig(args, plugins)
  File "/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py", line [329](https://github.com/microsoft/recommenders/actions/runs/3582687553/jobs/6027215045#step:3:337), in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/helpconfig.py", line 103, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1058, in pytest_cmdline_parse
    self.parse(args)
  File "/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1[346](https://github.com/microsoft/recommenders/actions/runs/3582687553/jobs/6027215045#step:3:354), in parse
    self._preparse(args, addopts=addopts)
  File "/azureml-envs/azureml_ee6cda[386](https://github.com/microsoft/recommenders/actions/runs/3582687553/jobs/6027215045#step:3:394)de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1214, in _preparse
    self._initini(args)
  File "/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1130, in _initini
    ns, unknown_args = self._parser.parse_known_and_unknown_args(
  File "/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/argparsing.py", line 172, in parse_known_and_unknown_args
    strargs = [os.fspath(x) for x in args]
  File "/azureml-envs/azureml_ee6cda386de85b635b43b98d5d27dc4b/lib/python3.9/site-packages/_pytest/config/argparsing.py", line 172, in <listcomp>
    strargs = [os.fspath(x) for x in args]
TypeError: expected str, bytes or os.PathLike object, not list

Tried [DIDN'T WORK]:

    if " " in args.pytestargs:
        pytest_string = test_group + args.pytestargs.split(" ")
    else:
        pytest_string = test_group + [args.pytestargs]
    print(pytest_string)
    pytest_exit_code = pytest.main(pytest_string)
  • It works with python playground.py -a "--disable-warnings --durations 0"
  • It works with python playground.py -a "--durations 0"
  • I get an error when I do python playground.py -a "--disable-warnings": error: argument --pytestargs/-a: expected one argument
  • It works with python playground.py -a "--durations 0 --disable-warnings"

Tried:

    # Flag to indicate whether to turn off the warnings
    parser.add_argument(
        "--disable-warnings",
        action="store_true",
        help="Turn off warnings",
    )
    parser.add_argument(
        "--durations",
        action="store",
        default=0,
        help="Add the duration of the test to the output",
    )

however, I still see the warning: https://github.com/microsoft/recommenders/actions/runs/4064811280/jobs/6998690821

Also, right now the tests are broken due to #1876

action.yml executes python tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py --clustername ${{inputs.CPU_CLUSTER_NAME}} --subid ${{inputs.AZUREML_TEST_SUBID}} --reponame "recommenders" --branch ${{ github.ref }} --rg ${{inputs.RG}} --wsname ${{inputs.WS}} --expname ${{inputs.EXP_NAME}}_${{inputs.TEST_GROUP}} --testlogs ${{inputs.TEST_LOGS_PATH}} --testkind ${{inputs.TEST_KIND}} --conda_pkg_python ${{inputs.PYTHON_VERSION}} --testgroup ${{inputs.TEST_GROUP}} --durations 0 --disable-warnings

In submit_groupwise_azureml_pytest.py it is called submit_experiment_to_azureml

FIXED in a9baaa8

@miguelgfierro
Copy link
Collaborator Author

image

@miguelgfierro
Copy link
Collaborator Author

miguelgfierro commented Feb 1, 2023

Error in

          python tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py --clustername ${{inputs.CPU_CLUSTER_NAME}}
          --subid ${{inputs.AZUREML_TEST_SUBID}} --reponame "recommenders" --branch ${{ steps.extract_branch.outputs.branch }}
          --rg ${{inputs.RG}} --wsname ${{inputs.WS}} --expname ${{inputs.EXP_NAME}}_${{inputs.TEST_GROUP}}
          --testlogs ${{inputs.TEST_LOGS_PATH}} --add_spark_dependencies --testkind ${{inputs.TEST_KIND}}
          --conda_pkg_python ${{inputs.PYTHON_VERSION}} --testgroup ${{inputs.TEST_GROUP}}
          --pytestargs "--durations 0 --disable-warnings"

I get:

Run python tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py --clustername cpu-cluster --subid *** --reponame "recommenders" --branch  --rg recommenders_project_resources --wsname azureml-test-workspace --expname unit_tests_group_notebooks_spark_001 --testlogs "test_logs.log" --add_spark_dependencies --testkind unit --conda_pkg_python "python=3.7" --testgroup group_notebooks_spark_001 --pytestargs "--durations 0 --disable-warnings"
usage: submit_groupwise_azureml_pytest.py [-h] [--test TEST]
                                          [--maxnodes MAXNODES]
                                          [--testgroup TESTGROUP] [--rg RG]
                                          [--wsname WSNAME]
                                          [--clustername CLUSTERNAME]
                                          [--vmsize VMSIZE]
                                          [--dockerproc DOCKERPROC]
                                          [--subid SUBID] [--expname EXPNAME]
                                          [--location LOCATION]
                                          [--reponame REPONAME]
                                          [--branch BRANCH] [--pr PR]
                                          [--add_gpu_dependencies]
                                          [--add_spark_dependencies]
                                          [--testlogs TESTLOGS]
                                          [--conda_pkg_cudatoolkit CONDA_PKG_CUDATOOLKIT]
                                          [--conda_pkg_cudnn CONDA_PKG_CUDNN]
                                          [--conda_pkg_jdk CONDA_PKG_JDK]
                                          [--conda_pkg_python CONDA_PKG_PYTHON]
                                          [--testkind TESTKIND]
                                          [--pytestargs PYTESTARGS]
submit_groupwise_azureml_pytest.py: error: argument --branch: expected one argument
Error: Process completed with exit code 2.

The issue came because I modified the deprecated code:

    - name: Extract branch name
      shell: bash
      run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF#refs/heads/})"
      id: extract_branch

this saves the current branch to a variable, that can be later obtained via steps.extract_branch.outputs.branch. In this case the output of the branch is --branch refs/pull/1864/merge.

Tried:

@miguelgfierro miguelgfierro changed the title AzureML tests: Durations, disable warnings and exit -1 AzureML tests: Durations, disable warnings Feb 10, 2023
@miguelgfierro miguelgfierro marked this pull request as ready for review February 10, 2023 06:30
@miguelgfierro miguelgfierro changed the title AzureML tests: Durations, disable warnings AzureML tests: Durations, disable warnings, logs less verbose Feb 10, 2023
@miguelgfierro
Copy link
Collaborator Author

@simonzhaoms @pradnyeshjoshi this PR is ready, it adds durations, disable warnings and the output is less verbose. However, there is an error with AzureML #1876.

We can either put this into staging, and then fix #1876 or we can wait to fix #1876, update this branch and then merge it.

What do you think?

@miguelgfierro
Copy link
Collaborator Author

After merging #1879, I updated this branch

@miguelgfierro miguelgfierro merged commit ddfe110 into staging Feb 17, 2023
@miguelgfierro miguelgfierro deleted the miguel/review_tests branch February 17, 2023 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants