Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

单元测试缺陷: 不稳定的用例(test_parse_parameters_testcase) #1151

Open
zhengtong0898 opened this issue Jan 15, 2022 · 0 comments
Open

单元测试缺陷: 不稳定的用例(test_parse_parameters_testcase) #1151

zhengtong0898 opened this issue Jan 15, 2022 · 0 comments
Assignees
@debugtalk
Labels
pending pending for confirmation

Comments

@zhengtong0898
Copy link

zhengtong0898 commented Jan 15, 2022 •
edited

问题描述

不稳定的单元测试用例: test_parse_parameters_testcase
运行10次, 有接近50%的失败率.

版本信息

请提供如下版本信息：

  • 操作系统类型: macos
  • Python 版本 3.8.2
  • HttpRunner 版本 3.6.1
  • 设备 ID: 119394988060799

运行命令 && 堆栈信息

请提供在命令行中运行测试时所在的目录和命令，以及报错时的详细堆栈内容。

$ pwd
/Users/xxxxxxxx/PycharmProjects/httprunner-316/

$ python3.8 main.py 
['main.py', 'run', '--showlocals', 'tests', '--verbose']
2022-01-15 21:01:32.531 | INFO     | httprunner.make:__make:517 - make path: /Users/xxxxxxxx/PycharmProjects/httprunner-316/tests
2022-01-15 21:01:32.535 | WARNING  | httprunner.make:__make:539 - Invalid test file: /Users/xxxxxxxx/PycharmProjects/httprunner-316/tests/data/a-b.c/中文case.yml
reason: test content not in dict format.
2022-01-15 21:01:32.539 | INFO     | httprunner.compat:ensure_testcase_v3:225 - ensure compatibility with testcase format v2
2022-01-15 21:01:32.543 | INFO     | httprunner.make:make_testcase:354 - start to make testcase: /Users/xxxxxxxx/PycharmProjects/httprunner-316/tests/data/a-b.c/1.yml
2022-01-15 21:01:32.544 | INFO     | httprunner.make:make_testcase:447 - generated testcase: /Users/xxxxxxxx/PycharmProjects/httprunner-316/tests/data/a_b_c/T1_test.py
2022-01-15 21:01:32.547 | INFO     | httprunner.compat:ensure_testcase_v3:225 - ensure compatibility with testcase format v2
2022-01-15 21:01:32.548 | INFO     | httprunner.make:make_testcase:354 - start to make testcase: /Users/xxxxxxxx/PycharmProjects/httprunner-316/tests/data/a-b.c/2 3.yml
2022-01-15 21:01:32.552 | INFO     | httprunner.compat:ensure_testcase_v3:225 - ensure compatibility with testcase format v2
2022-01-15 21:01:32.552 | INFO     | httprunner.make:make_testcase:354 - start to make testcase: /Users/xxxxxxxx/PycharmProjects/httprunner-316/tests/data/a-b.c/1.yml
2022-01-15 21:01:32.552 | INFO     | httprunner.make:make_testcase:447 - generated testcase: /Users/xxxxxxxx/PycharmProjects/httprunner-316/tests/data/a_b_c/T2_3_test.py
2022-01-15 21:01:32.552 | INFO     | httprunner.make:format_pytest_with_black:170 - format pytest cases with black ...
reformatted /Users/xxxxxxxx/PycharmProjects/httprunner-316/tests/data/a_b_c/T1_test.py
reformatted /Users/xxxxxxxx/PycharmProjects/httprunner-316/tests/data/a_b_c/T2_3_test.py
All done! ✨ 🍰 ✨
2 files reformatted.
2022-01-15 21:01:33.083 | INFO     | httprunner.cli:main_run:56 - start to run tests with pytest. HttpRunner version: 3.1.6
=============================================================== test session starts ================================================================
platform darwin -- Python 3.8.12, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 -- /Users/xxxxxxxx/Library/Caches/pypoetry/virtualenvs/httprunner-01q-NmAB-py3.8/bin/python3.8
cachedir: .pytest_cache
metadata: {'Python': '3.8.12', 'Platform': 'macOS-10.15.5-x86_64-i386-64bit', 'Packages': {'pytest': '5.4.3', 'py': '1.11.0', 'pluggy': '0.13.1'}, 'Plugins': {'metadata': '1.9.0', 'html': '2.1.1', 'anyio': '3.3.4'}, 'JAVA_HOME': '/Library/Java/JavaVirtualMachines/jdk1.8.0_291.jdk/Contents/Home'}
rootdir: /Users/xxxxxxxx/PycharmProjects/httprunner-316
plugins: metadata-1.9.0, html-2.1.1, anyio-3.3.4
collected 92 items                                                                                                                                 

tests/data/a_b_c/T1_test.py::TestCaseT1::test_start PASSED                                          [  1%]
tests/parser_test.py::TestParserBasic::test_extract_functions PASSED                                [  2%]
tests/parser_test.py::TestParserBasic::test_extract_variables PASSED                                [  3%]
tests/parser_test.py::TestParserBasic::test_parse_data_func_abnormal PASSED                         [  4%]
tests/parser_test.py::TestParserBasic::test_parse_data_func_var_duplicate PASSED                    [  5%]
tests/parser_test.py::TestParserBasic::test_parse_data_multiple_identical_variables PASSED          [  6%]
tests/parser_test.py::TestParserBasic::test_parse_data_request PASSED                               [  7%]
tests/parser_test.py::TestParserBasic::test_parse_data_string_with_functions PASSED                 [  8%]
tests/parser_test.py::TestParserBasic::test_parse_data_string_with_variables PASSED                 [  9%]
tests/parser_test.py::TestParserBasic::test_parse_data_testcase PASSED                              [ 10%]
tests/parser_test.py::TestParserBasic::test_parse_function_params PASSED                            [ 11%]
tests/parser_test.py::TestParserBasic::test_parse_parameters_testcase FAILED                        [ 13%]
tests/parser_test.py::TestParserBasic::test_parse_string_value PASSED                               [ 14%]
tests/parser_test.py::TestParserBasic::test_parse_variables_mapping PASSED                          [ 15%]
tests/parser_test.py::TestParserBasic::test_parse_variables_mapping_exception PASSED                [ 16%]
tests/parser_test.py::TestParserBasic::test_regex_findall_variables PASSED                          [ 17%]
tests/runner_test.py::TestHttpRunner::test_run_testcase_by_path_ref_testcase PASSED                 [ 18%]
tests/runner_test.py::TestHttpRunner::test_run_testcase_by_path_request_only PASSED                 [ 19%]
tests/runner_test.py::TestHttpRunner::test_run_testcase_with_abnormal_path PASSED                   [ 20%]
tests/ext/har2case/har_utils_test.py::TestHar2CaseUtils::test_convert_list_to_dict PASSED           [ 21%]
tests/ext/har2case/har_utils_test.py::TestHar2CaseUtils::test_convert_x_www_form_urlencoded_to_dict PASSED                                   [ 22%]
tests/ext/har2case/har_utils_test.py::TestHar2CaseUtils::test_load_har_log_empty_error PASSED       [ 23%]
tests/ext/har2case/har_utils_test.py::TestHar2CaseUtils::test_load_har_log_entries PASSED           [ 25%]
tests/ext/har2case/har_utils_test.py::TestHar2CaseUtils::test_load_har_log_key_error PASSED         [ 26%]
tests/make_test.py::TestMake::test_convert_testcase_path PASSED                                     [ 27%]
tests/make_test.py::TestMake::test_ensure_file_path_valid PASSED                                    [ 28%]
tests/make_test.py::TestMake::test_make_config_chain_style PASSED                                   [ 29%]
tests/make_test.py::TestMake::test_make_requests_with_json_chain_style PASSED                       [ 30%]
tests/make_test.py::TestMake::test_make_testcase PASSED                                             [ 31%]
tests/make_test.py::TestMake::test_make_testcase_folder PASSED                                      [ 32%]
tests/make_test.py::TestMake::test_make_testcase_with_ref PASSED                                    [ 33%]
tests/make_test.py::TestMake::test_make_teststep_chain_style PASSED                                 [ 34%]
tests/make_test.py::TestMake::test_make_testsuite PASSED                                            [ 35%]
tests/app/debug_test.py::TestDebug::test_debug_single_testcase PASSED                               [ 38%]
tests/compat_test.py::TestCompat::test_convert_extractors PASSED                                    [ 39%]
tests/compat_test.py::TestCompat::test_convert_jmespath PASSED                                      [ 40%]
tests/compat_test.py::TestCompat::test_convert_validators PASSED                                    [ 41%]
tests/compat_test.py::TestCompat::test_convert_variables PASSED                                     [ 42%]
tests/compat_test.py::TestCompat::test_ensure_cli_args PASSED                                       [ 43%]
tests/compat_test.py::TestCompat::test_ensure_file_path PASSED                                      [ 44%]
tests/compat_test.py::TestCompat::test_ensure_testcase_v3 PASSED                                    [ 45%]
tests/compat_test.py::TestCompat::test_ensure_testcase_v3_api PASSED                                [ 46%]
tests/loader_test.py::TestLoader::test_load_csv_file_multiple_parameters PASSED                     [ 47%]
tests/loader_test.py::TestLoader::test_load_csv_file_one_parameter PASSED                           [ 48%]
tests/loader_test.py::TestLoader::test_load_custom_dot_env_file PASSED                              [ 50%]
tests/loader_test.py::TestLoader::test_load_env_path_not_exist PASSED                               [ 51%]
tests/loader_test.py::TestLoader::test_load_folder_files PASSED                                     [ 52%]
tests/loader_test.py::TestLoader::test_load_json_file_file_format_error PASSED                      [ 53%]
tests/loader_test.py::TestLoader::test_load_testcase_file PASSED                                    [ 54%]
tests/loader_test.py::TestLoader::test_load_testcases_bad_filepath PASSED                           [ 55%]
tests/loader_test.py::TestLoader::test_locate_file PASSED                                           [ 56%]
tests/cli_test.py::TestCli::test_debug_pytest PASSED                                                [ 57%]
tests/cli_test.py::TestCli::test_show_help PASSED                                                   [ 58%]
tests/cli_test.py::TestCli::test_show_version PASSED                                                [ 59%]
tests/ext/har2case/core_test.py::TestHar::test_convert_list_to_dict PASSED                          [ 60%]
tests/ext/har2case/core_test.py::TestHar::test_convert_x_www_form_urlencoded_to_dict PASSED         [ 61%]
tests/ext/har2case/core_test.py::TestHar::test_exclude PASSED                                       [ 63%]
tests/ext/har2case/core_test.py::TestHar::test_exclude_multiple PASSED                              [ 64%]
tests/ext/har2case/core_test.py::TestHar::test_filter PASSED                                        [ 65%]
tests/ext/har2case/core_test.py::TestHar::test_gen_testcase_json PASSED                             [ 66%]
tests/ext/har2case/core_test.py::TestHar::test_gen_testcase_yaml PASSED                             [ 67%]
tests/ext/har2case/core_test.py::TestHar::test_load_har_log_empty_error PASSED                      [ 68%]
tests/ext/har2case/core_test.py::TestHar::test_load_har_log_entries PASSED                          [ 69%]
tests/ext/har2case/core_test.py::TestHar::test_load_har_log_key_error PASSED                        [ 70%]
tests/ext/har2case/core_test.py::TestHar::test_make_request_data_json PASSED                        [ 71%]
tests/ext/har2case/core_test.py::TestHar::test_make_request_data_params PASSED                      [ 72%]
tests/ext/har2case/core_test.py::TestHar::test_make_request_data_text_empty PASSED                  [ 73%]
tests/ext/har2case/core_test.py::TestHar::test_make_testcase PASSED                                 [ 75%]
tests/ext/har2case/core_test.py::TestHar::test_make_validate PASSED                                 [ 76%]
tests/ext/har2case/core_test.py::TestHar::test_prepare_teststep PASSED                              [ 77%]
tests/ext/har2case/core_test.py::TestHar::test_prepare_teststeps PASSED                             [ 78%]
tests/ext/har2case/core_test.py::TestHar2CaseUtils::test_convert_list_to_dict PASSED                [ 79%]
tests/ext/har2case/core_test.py::TestHar2CaseUtils::test_convert_x_www_form_urlencoded_to_dict PASSED                                        [ 80%]
tests/ext/har2case/core_test.py::TestHar2CaseUtils::test_load_har_log_empty_error PASSED            [ 81%]
tests/ext/har2case/core_test.py::TestHar2CaseUtils::test_load_har_log_entries PASSED                [ 82%]
tests/ext/har2case/core_test.py::TestHar2CaseUtils::test_load_har_log_key_error PASSED              [ 83%]
tests/response_test.py::TestResponse::test_extract PASSED                                           [ 84%]
tests/response_test.py::TestResponse::test_validate PASSED                                          [ 85%]
tests/response_test.py::TestResponse::test_validate_functions PASSED                                [ 86%]
tests/response_test.py::TestResponse::test_validate_variables PASSED                                [ 88%]
tests/utils_test.py::TestUtils::test_cartesian_product_empty PASSED                                 [ 89%]
tests/utils_test.py::TestUtils::test_cartesian_product_multiple PASSED                              [ 90%]
tests/utils_test.py::TestUtils::test_cartesian_product_one PASSED                                   [ 91%]
tests/utils_test.py::TestUtils::test_lower_dict_keys PASSED                                         [ 92%]
tests/utils_test.py::TestUtils::test_override_config_variables PASSED                               [ 93%]
tests/utils_test.py::TestUtils::test_print_info PASSED                                              [ 94%]
tests/utils_test.py::TestUtils::test_safe_dump_json PASSED                                          [ 95%]
tests/utils_test.py::TestUtils::test_set_os_environ PASSED                                          [ 96%]
tests/utils_test.py::TestUtils::test_sort_dict_by_custom_order PASSED                               [ 97%]
tests/utils_test.py::TestUtils::test_validators PASSED                                              [ 98%]
tests/data/a_b_c/T2_3_test.py::TestCaseT23::test_start PASSED                                       [100%]

===================================================================== FAILURES =====================================================================
__________________________________________________ TestParserBasic.test_parse_parameters_testcase __________________________________________________
tests/parser_test.py:475: in test_parse_parameters_testcase
    parsed_params = parser.parse_parameters(parameters)
        parameters = {'sum': '${calculate_two_nums(1, 2)}', 'user_agent': ['iOS/10.1', 'iOS/10.2'], 'username-password': '${parameterize(request_methods/account.csv)}'}
        self       = <tests.parser_test.TestParserBasic testMethod=test_parse_parameters_testcase>
httprunner/parser.py:519: in parse_parameters
    parsed_parameter_content: List = parse_data(
        functions_mapping = {'get_httprunner_version': <function get_httprunner_version at 0x7faa920b7700>, 'get_variables': <function get_variables at 0x7faa91e6b1f0>, 'sum_two': <function sum_two at 0x7faa91e6b310>}
        parameter_content = '${parameterize(request_methods/account.csv)}'
        parameter_content_dict = {'user_agent': 'iOS/10.2'}
        parameter_content_list = [{'user_agent': 'iOS/10.1'}, {'user_agent': 'iOS/10.2'}]
        parameter_item = ['iOS/10.2']
        parameter_name = 'username-password'
        parameter_name_list = ['username', 'password']
        parameters = {'sum': '${calculate_two_nums(1, 2)}', 'user_agent': ['iOS/10.1', 'iOS/10.2'], 'username-password': '${parameterize(request_methods/account.csv)}'}
        parsed_parameters_list = [[{'user_agent': 'iOS/10.1'}, {'user_agent': 'iOS/10.2'}]]
        project_meta = ProjectMeta(debugtalk_py='', debugtalk_path='/Users/xxxxxxxx/PycharmProjects/httprunner-316/tests/data/debugtalk.py'...ction get_variables at 0x7faa91e6b1f0>}, env={}, RootDir='/Users/xxxxxxxx/PycharmProjects/httprunner-316/tests/data')
httprunner/parser.py:405: in parse_data
    return parse_string(raw_data, variables_mapping, functions_mapping)
        functions_mapping = {'get_httprunner_version': <function get_httprunner_version at 0x7faa920b7700>, 'get_variables': <function get_variables at 0x7faa91e6b1f0>, 'sum_two': <function sum_two at 0x7faa91e6b310>}
        raw_data   = '${parameterize(request_methods/account.csv)}'
        variables_mapping = {}
httprunner/parser.py:340: in parse_string
    func_eval_value = func(*parsed_args, **parsed_kwargs)
        args       = ['request_methods/account.csv']
        dollar_match = None
        func       = <function load_csv_file at 0x7faa90e84d30>
        func_match = <re.Match object; span=(0, 44), match='${parameterize(request_methods/account.csv)}'>
        func_name  = 'parameterize'
        func_params_str = 'request_methods/account.csv'
        function_meta = {'args': ['request_methods/account.csv'], 'kwargs': {}}
        functions_mapping = {'get_httprunner_version': <function get_httprunner_version at 0x7faa920b7700>, 'get_variables': <function get_variables at 0x7faa91e6b1f0>, 'sum_two': <function sum_two at 0x7faa91e6b310>}
        kwargs     = {}
        match_start_position = 0
        parsed_args = ['request_methods/account.csv']
        parsed_kwargs = {}
        parsed_string = ''
        raw_string = '${parameterize(request_methods/account.csv)}'
        variables_mapping = {}
httprunner/loader.py:182: in load_csv_file
    raise exceptions.CSVNotFound(csv_file)
E   httprunner.exceptions.CSVNotFound: /Users/xxxxxxxx/PycharmProjects/httprunner-316/tests/data/request_methods/account.csv
        csv_file   = '/Users/xxxxxxxx/PycharmProjects/httprunner-316/tests/data/request_methods/account.csv'
--------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------
2022-01-15 21:01:37.331 | ERROR    | httprunner.parser:parse_string:342 - call function error:
func_name: parameterize
args: ['request_methods/account.csv']
kwargs: {}
CSVNotFound: /Users/xxxxxxxx/PycharmProjects/httprunner-316/tests/data/request_methods/account.csv
@zhengtong0898 zhengtong0898 added the pending pending for confirmation label Jan 15, 2022
zhengtong0898 added a commit to zhengtong0898/httprunner that referenced this issue Jan 15, 2022
@zhengtong0898
fix: httprunner#1151 unit test unstable
27081c3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@debugtalk debugtalk

Labels
pending pending for confirmation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@debugtalk @zhengtong0898