Add suite outputs as annotations #392

EnricoMi · 2023-01-03T11:56:40Z

Adds suite stdout and strerr outputs as annotations. This is disabled by default.

Fixes #374.

python/publish/junit.py

github-actions · 2023-01-03T12:02:08Z

Test Results (setup-python)

      69 files ±    0       69 suites ±0 43m 29s ⏱️ + 14m 13s
    394 tests +  13     394 ✔️ +  13     0 💤 ±0 0 ❌ ±0
27 186 runs +897 26 646 ✔️ +897 540 💤 ±0 0 ❌ ±0

Results for commit aa81b1e. ± Comparison against base commit 2937053.

This pull request removes 1 and adds 14 tests. Note that renamed tests count towards both.

test_publisher.TestPublisher ‑ test_publish_data

test_action_script.Test ‑ test_get_settings_json_suite_details
test_action_script.Test ‑ test_get_settings_json_test_case_results
test_action_script.Test ‑ test_get_settings_report_suite_logs
test_action_script.Test ‑ test_main
test_action_script.Test ‑ test_parse_files_with_suite_details
test_junit.Test ‑ test_get_settings_json_suite_details
test_junit.Test ‑ test_get_settings_json_test_case_results
test_junit.Test ‑ test_get_settings_report_suite_logs
test_junit.Test ‑ test_main
test_junit.Test ‑ test_parse_files_with_suite_details
…

♻️ This comment has been updated with latest results.

github-actions · 2023-01-03T12:02:27Z

Test Results (Test File)

97 tests ±0 80 ✔️ ±0 3m 25s ⏱️ ±0s
  1 suites ±0 17 💤 ±0
  1 files ±0   0 ❌ ±0

Results for commit aa81b1e. ± Comparison against base commit 2937053.

♻️ This comment has been updated with latest results.

github-actions · 2023-01-03T12:02:30Z

Test Results (Dockerfile)

      69 files ±    0       69 suites ±0 43m 29s ⏱️ + 14m 13s
    394 tests +  13     394 ✔️ +  13     0 💤 ±0 0 ❌ ±0
27 186 runs +897 26 646 ✔️ +897 540 💤 ±0 0 ❌ ±0

Results for commit aa81b1e. ± Comparison against base commit 2937053.

This pull request removes 1 and adds 14 tests. Note that renamed tests count towards both.

test_publisher.TestPublisher ‑ test_publish_data

test_action_script.Test ‑ test_get_settings_json_suite_details
test_action_script.Test ‑ test_get_settings_json_test_case_results
test_action_script.Test ‑ test_get_settings_report_suite_logs
test_action_script.Test ‑ test_main
test_action_script.Test ‑ test_parse_files_with_suite_details
test_junit.Test ‑ test_get_settings_json_suite_details
test_junit.Test ‑ test_get_settings_json_test_case_results
test_junit.Test ‑ test_get_settings_report_suite_logs
test_junit.Test ‑ test_main
test_junit.Test ‑ test_parse_files_with_suite_details
…

♻️ This comment has been updated with latest results.

github-actions · 2023-01-03T12:02:57Z

Test Results (Test Files)

    70 files +1       6 errors 361 suites +1 1h 6m 7s ⏱️ ±0s
1 786 tests +1 1 499 ✔️ ±0   71 💤 ±0 208 ❌ ±0 8 🔥 +1
2 041 runs +1 1 700 ✔️ ±0 106 💤 ±0 226 ❌ ±0 9 🔥 +1

For more details on these parsing errors, failures and errors, see this check.

Results for commit aa81b1e. ± Comparison against base commit 2937053.

♻️ This comment has been updated with latest results.

github-actions · 2023-01-03T12:02:59Z

Test Results (Docker Image)

      69 files ±    0       69 suites ±0 43m 29s ⏱️ + 14m 13s
    394 tests +  13     394 ✔️ +  13     0 💤 ±0 0 ❌ ±0
27 186 runs +897 26 646 ✔️ +897 540 💤 ±0 0 ❌ ±0

Results for commit aa81b1e. ± Comparison against base commit 2937053.

This pull request removes 1 and adds 14 tests. Note that renamed tests count towards both.

test_publisher.TestPublisher ‑ test_publish_data

test_action_script.Test ‑ test_get_settings_json_suite_details
test_action_script.Test ‑ test_get_settings_json_test_case_results
test_action_script.Test ‑ test_get_settings_report_suite_logs
test_action_script.Test ‑ test_main
test_action_script.Test ‑ test_parse_files_with_suite_details
test_junit.Test ‑ test_get_settings_json_suite_details
test_junit.Test ‑ test_get_settings_json_test_case_results
test_junit.Test ‑ test_get_settings_report_suite_logs
test_junit.Test ‑ test_main
test_junit.Test ‑ test_parse_files_with_suite_details
…

♻️ This comment has been updated with latest results.

github-actions · 2023-01-03T12:05:23Z

Test Results (macOS python 3.8)

      69 files ±    0       69 suites ±0 43m 29s ⏱️ + 14m 13s
    394 tests +  13     394 ✔️ +  13     0 💤 ±0 0 ❌ ±0
27 186 runs +897 26 646 ✔️ +897 540 💤 ±0 0 ❌ ±0

Results for commit aa81b1e. ± Comparison against base commit 2937053.

This pull request removes 1 and adds 14 tests. Note that renamed tests count towards both.

test_publisher.TestPublisher ‑ test_publish_data

test_action_script.Test ‑ test_get_settings_json_suite_details
test_action_script.Test ‑ test_get_settings_json_test_case_results
test_action_script.Test ‑ test_get_settings_report_suite_logs
test_action_script.Test ‑ test_main
test_action_script.Test ‑ test_parse_files_with_suite_details
test_junit.Test ‑ test_get_settings_json_suite_details
test_junit.Test ‑ test_get_settings_json_test_case_results
test_junit.Test ‑ test_get_settings_report_suite_logs
test_junit.Test ‑ test_main
test_junit.Test ‑ test_parse_files_with_suite_details
…

♻️ This comment has been updated with latest results.

github-actions · 2023-01-03T12:05:30Z

Test Results (macOS 12 python installed)

      69 files ±    0       69 suites ±0 43m 29s ⏱️ + 14m 13s
    394 tests +  13     394 ✔️ +  13     0 💤 ±0 0 ❌ ±0
27 186 runs +897 26 646 ✔️ +897 540 💤 ±0 0 ❌ ±0

Results for commit aa81b1e. ± Comparison against base commit 2937053.

This pull request removes 1 and adds 14 tests. Note that renamed tests count towards both.

test_publisher.TestPublisher ‑ test_publish_data

test_action_script.Test ‑ test_get_settings_json_suite_details
test_action_script.Test ‑ test_get_settings_json_test_case_results
test_action_script.Test ‑ test_get_settings_report_suite_logs
test_action_script.Test ‑ test_main
test_action_script.Test ‑ test_parse_files_with_suite_details
test_junit.Test ‑ test_get_settings_json_suite_details
test_junit.Test ‑ test_get_settings_json_test_case_results
test_junit.Test ‑ test_get_settings_report_suite_logs
test_junit.Test ‑ test_main
test_junit.Test ‑ test_parse_files_with_suite_details
…

♻️ This comment has been updated with latest results.

github-actions · 2023-01-03T12:05:37Z

Test Results (macOS python installed)

      69 files ±    0       69 suites ±0 43m 29s ⏱️ + 14m 13s
    394 tests +  13     394 ✔️ +  13     0 💤 ±0 0 ❌ ±0
27 186 runs +897 26 646 ✔️ +897 540 💤 ±0 0 ❌ ±0

Results for commit aa81b1e. ± Comparison against base commit 2937053.

This pull request removes 1 and adds 14 tests. Note that renamed tests count towards both.

test_publisher.TestPublisher ‑ test_publish_data

test_action_script.Test ‑ test_get_settings_json_suite_details
test_action_script.Test ‑ test_get_settings_json_test_case_results
test_action_script.Test ‑ test_get_settings_report_suite_logs
test_action_script.Test ‑ test_main
test_action_script.Test ‑ test_parse_files_with_suite_details
test_junit.Test ‑ test_get_settings_json_suite_details
test_junit.Test ‑ test_get_settings_json_test_case_results
test_junit.Test ‑ test_get_settings_report_suite_logs
test_junit.Test ‑ test_main
test_junit.Test ‑ test_parse_files_with_suite_details
…

♻️ This comment has been updated with latest results.

github-actions · 2023-01-03T12:06:00Z

Test Results (Linux python 3.8)

      69 files ±    0       69 suites ±0 43m 29s ⏱️ + 14m 13s
    394 tests +  13     394 ✔️ +  13     0 💤 ±0 0 ❌ ±0
27 186 runs +897 26 646 ✔️ +897 540 💤 ±0 0 ❌ ±0

Results for commit aa81b1e. ± Comparison against base commit 2937053.

This pull request removes 1 and adds 14 tests. Note that renamed tests count towards both.

test_publisher.TestPublisher ‑ test_publish_data

test_action_script.Test ‑ test_get_settings_json_suite_details
test_action_script.Test ‑ test_get_settings_json_test_case_results
test_action_script.Test ‑ test_get_settings_report_suite_logs
test_action_script.Test ‑ test_main
test_action_script.Test ‑ test_parse_files_with_suite_details
test_junit.Test ‑ test_get_settings_json_suite_details
test_junit.Test ‑ test_get_settings_json_test_case_results
test_junit.Test ‑ test_get_settings_report_suite_logs
test_junit.Test ‑ test_main
test_junit.Test ‑ test_parse_files_with_suite_details
…

♻️ This comment has been updated with latest results.

github-actions · 2023-01-03T12:06:13Z

Test Results (Linux 22.04 python installed)

      69 files ±    0       69 suites ±0 43m 29s ⏱️ + 14m 13s
    394 tests +  13     394 ✔️ +  13     0 💤 ±0 0 ❌ ±0
27 186 runs +897 26 646 ✔️ +897 540 💤 ±0 0 ❌ ±0

Results for commit aa81b1e. ± Comparison against base commit 2937053.

This pull request removes 1 and adds 14 tests. Note that renamed tests count towards both.

test_publisher.TestPublisher ‑ test_publish_data

test_action_script.Test ‑ test_get_settings_json_suite_details
test_action_script.Test ‑ test_get_settings_json_test_case_results
test_action_script.Test ‑ test_get_settings_report_suite_logs
test_action_script.Test ‑ test_main
test_action_script.Test ‑ test_parse_files_with_suite_details
test_junit.Test ‑ test_get_settings_json_suite_details
test_junit.Test ‑ test_get_settings_json_test_case_results
test_junit.Test ‑ test_get_settings_report_suite_logs
test_junit.Test ‑ test_main
test_junit.Test ‑ test_parse_files_with_suite_details
…

♻️ This comment has been updated with latest results.

github-actions · 2023-01-03T12:06:18Z

Test Results (Linux python installed)

      69 files ±    0       69 suites ±0 43m 29s ⏱️ + 14m 13s
    394 tests +  13     394 ✔️ +  13     0 💤 ±0 0 ❌ ±0
27 186 runs +897 26 646 ✔️ +897 540 💤 ±0 0 ❌ ±0

Results for commit aa81b1e. ± Comparison against base commit 2937053.

This pull request removes 1 and adds 14 tests. Note that renamed tests count towards both.

test_publisher.TestPublisher ‑ test_publish_data

test_action_script.Test ‑ test_get_settings_json_suite_details
test_action_script.Test ‑ test_get_settings_json_test_case_results
test_action_script.Test ‑ test_get_settings_report_suite_logs
test_action_script.Test ‑ test_main
test_action_script.Test ‑ test_parse_files_with_suite_details
test_junit.Test ‑ test_get_settings_json_suite_details
test_junit.Test ‑ test_get_settings_json_test_case_results
test_junit.Test ‑ test_get_settings_report_suite_logs
test_junit.Test ‑ test_main
test_junit.Test ‑ test_parse_files_with_suite_details
…

♻️ This comment has been updated with latest results.

github-actions · 2023-01-03T12:07:53Z

Test Results (Windows python installed)

      69 files ±    0       69 suites ±0 43m 29s ⏱️ + 14m 13s
    394 tests +  13     394 ✔️ +  13     0 💤 ±0 0 ❌ ±0
27 186 runs +897 26 646 ✔️ +897 540 💤 ±0 0 ❌ ±0

Results for commit aa81b1e. ± Comparison against base commit 2937053.

This pull request removes 1 and adds 14 tests. Note that renamed tests count towards both.

test_publisher.TestPublisher ‑ test_publish_data

test_action_script.Test ‑ test_get_settings_json_suite_details
test_action_script.Test ‑ test_get_settings_json_test_case_results
test_action_script.Test ‑ test_get_settings_report_suite_logs
test_action_script.Test ‑ test_main
test_action_script.Test ‑ test_parse_files_with_suite_details
test_junit.Test ‑ test_get_settings_json_suite_details
test_junit.Test ‑ test_get_settings_json_test_case_results
test_junit.Test ‑ test_get_settings_report_suite_logs
test_junit.Test ‑ test_main
test_junit.Test ‑ test_parse_files_with_suite_details
…

♻️ This comment has been updated with latest results.

github-actions · 2023-01-03T12:08:46Z

Test Results (reference)

      69 files ±    0       69 suites ±0 43m 29s ⏱️ + 14m 13s
    394 tests +  13     394 ✔️ +  13     0 💤 ±0 0 ❌ ±0
27 186 runs +897 26 646 ✔️ +897 540 💤 ±0 0 ❌ ±0

Results for commit aa81b1e. ± Comparison against base commit 2937053.

This pull request removes 1 and adds 14 tests. Note that renamed tests count towards both.

test_publisher.TestPublisher ‑ test_publish_data

test_action_script.Test ‑ test_get_settings_json_suite_details
test_action_script.Test ‑ test_get_settings_json_test_case_results
test_action_script.Test ‑ test_get_settings_report_suite_logs
test_action_script.Test ‑ test_main
test_action_script.Test ‑ test_parse_files_with_suite_details
test_junit.Test ‑ test_get_settings_json_suite_details
test_junit.Test ‑ test_get_settings_json_test_case_results
test_junit.Test ‑ test_get_settings_report_suite_logs
test_junit.Test ‑ test_main
test_junit.Test ‑ test_parse_files_with_suite_details
…

♻️ This comment has been updated with latest results.

EnricoMi · 2023-01-06T18:53:37Z

Works nicely:

sfc-gh-zpeng

Looks fantastic! Thank you!

sonatype-lift bot reviewed Jan 3, 2023

View reviewed changes

python/publish/junit.py Show resolved Hide resolved

EnricoMi force-pushed the branch-add-suite-outputs branch 2 times, most recently from 00b31c3 to ac08d26 Compare January 3, 2023 13:00

EnricoMi added 8 commits January 4, 2023 11:56

Add suite stdout and stderr as annotations

218dd12

Include suite details in expected result files

80b85b5

Add missing bazel test files

88149b9

Add suite output annotations to CI

7da6cf5

Fix variable retyping

1e523f6

Allow for stdout and stderr annotations individually

2acffa7

Test get_suite_annotations, drop whitespace-only logs

9b3ed82

Adjust expectations for faulty macOX-Python versions

75ca81b

EnricoMi force-pushed the branch-add-suite-outputs branch from a481105 to 75ca81b Compare January 4, 2023 11:01

EnricoMi added 5 commits January 4, 2023 16:47

Rename junit bazel test files

1013689

Remove suite details from reduced stats dict, not stats dict

4c987ce

Log settings

227c690

Fully test main method

509140d

Configure suite details contained in json file

997c96d

Adjust expectation in test_main for broken macOS-Python version

9ae41f3

EnricoMi force-pushed the branch-add-suite-outputs branch 2 times, most recently from ff3f0e0 to 7d56994 Compare January 4, 2023 16:39

Use absolute file patterns, incl. trx, fix assertions in test_main

eeb3b3e

EnricoMi force-pushed the branch-add-suite-outputs branch from 7d56994 to eeb3b3e Compare January 4, 2023 16:45

EnricoMi marked this pull request as ready for review January 6, 2023 18:34

EnricoMi added 2 commits January 6, 2023 19:36

Move suite logs annotations into their own config var

b60734e

Generate test annotations with suite logs

aa81b1e

EnricoMi force-pushed the branch-add-suite-outputs branch from c43f9c5 to aa81b1e Compare January 6, 2023 18:37

EnricoMi mentioned this pull request Jan 6, 2023

Also include test suite stdout/stderr in annotation details #374

Closed

sfc-gh-zpeng approved these changes Jan 6, 2023

View reviewed changes

EnricoMi merged commit e3fdce4 into master Jan 7, 2023

EnricoMi deleted the branch-add-suite-outputs branch January 7, 2023 16:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add suite outputs as annotations #392

Add suite outputs as annotations #392

EnricoMi commented Jan 3, 2023

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

EnricoMi commented Jan 6, 2023

sfc-gh-zpeng left a comment

Add suite outputs as annotations #392

Add suite outputs as annotations #392

Conversation

EnricoMi commented Jan 3, 2023

github-actions bot commented Jan 3, 2023 • edited

Test Results (setup-python)

github-actions bot commented Jan 3, 2023 • edited

Test Results (Test File)

github-actions bot commented Jan 3, 2023 • edited

Test Results (Dockerfile)

github-actions bot commented Jan 3, 2023 • edited

Test Results (Test Files)

github-actions bot commented Jan 3, 2023 • edited

Test Results (Docker Image)

github-actions bot commented Jan 3, 2023 • edited

Test Results (macOS python 3.8)

github-actions bot commented Jan 3, 2023 • edited

Test Results (macOS 12 python installed)

github-actions bot commented Jan 3, 2023 • edited

Test Results (macOS python installed)

github-actions bot commented Jan 3, 2023 • edited

Test Results (Linux python 3.8)

github-actions bot commented Jan 3, 2023 • edited

Test Results (Linux 22.04 python installed)

github-actions bot commented Jan 3, 2023 • edited

Test Results (Linux python installed)

github-actions bot commented Jan 3, 2023 • edited

Test Results (Windows python installed)

github-actions bot commented Jan 3, 2023 • edited

Test Results (reference)

EnricoMi commented Jan 6, 2023

sfc-gh-zpeng left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited

github-actions bot commented Jan 3, 2023 •

edited