Reporting to ga4 #4040

DenysGonchar · 2023-06-17T22:50:35Z

This PR introduces the following changes:

reporting of the failed to GA4:
- GH Actions reporting
- CircleCI reporting
- see test cases counters.
- some minor improvements were required to ct_markdown_errors_hook to cover failures in init_per_testcase/end_per_testcases.
improved a bit ct_groups_summary_hook:
- covered situation when repeat_until_all_ok flag is missing (on the last attempt to repeat the test)
- it's still not enough to cover correctly all the edge cases of eventually passing tests, this hook should be completely replaced/reworked in the upcoming PR.
introduced test hook and suit to investigate corner cases (these 2 to be removed in the upcoming PR, once ct_groups_summary_hook is replaced).
minor clean-up of the existing hooks:
- remote logging is moved into ct_mongoose_log_hook. Previously, there were 2 redundant implementations in ct_mongoose_hook and ct_tty_hook hooks.
- ct_mim_config_hook is removed completely, mim_data_dir config is now set by ct_mongoose_hook.
- dummy ct hooks callbacks are removed from all hooks.
- halting logic is moved from ct_tty_hook:on_tc_skip/3 to ct_groups_summary_hook:on_tc_skip/3. I didn't test that code, but ct_tty_hook is definitely an improper place for it.
some minor fixes test execution scripts.

UPDATE:

reporting to GA4 is done for master branch only.

chrzaszcz

Please make the tests pass. Could you hide the stale reports with failed tests as well? It's a lot to scroll down. Thanks!

chrzaszcz

Looks good in general, I added some comments.

big_tests/failure_repoting_testing.spec

big_tests/run_common_test.erl

big_tests/src/ct_groups_summary_hook.erl

tools/summarise-ct-results

tools/test.sh

big_tests/src/ct_groups_summary_hook.erl

DenysGonchar · 2023-07-11T08:38:25Z

oh my god, so many typos in this PR.

for testing use the following commands: . tools/test-runner-complete.sh test-runner.sh --skip-preset --skip-build-mim --skip-small-tests --skip-start-nodes --skip-stop-nodes --db --skip-cover --skip-preset --dev-nodes --spec failure_reporting_testing.spec

ct_mongoose_log_hook already does it, and does it in a better way.

it's quite an overhead to have an extra hook just to set the mim_data_dir parameter, so the hook is merged with ct_mongoose_hook

…ry_hook:on_tc_skip/3

testing package building for ubuntu_xenial as it's not tested by CircleCI

chrzaszcz

It looks good. I only suggest changing the target to a feature branch and following with a fix for these errors that appear multiple times in the comments to avoid affecting other PR's:

codecov · 2023-07-12T09:24:49Z

Codecov Report

Patch coverage has no change and project coverage change: +0.04 🎉

Comparison is base (563f592) 83.82% compared to head (15d30b5) 83.87%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4040      +/-   ##
==========================================
+ Coverage   83.82%   83.87%   +0.04%     
==========================================
  Files         526      526              
  Lines       33133    33133              
==========================================
+ Hits        27773    27789      +16     
+ Misses       5360     5344      -16

see 15 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

mongoose-im · 2023-07-12T09:32:20Z

small_tests_24 / small_tests / 15d30b5
Reports root / small

elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / 15d30b5
Reports root/ big
OK: 369 / Failed: 0 / User-skipped: 38 / Auto-skipped: 0

small_tests_25_arm64 / small_tests / 15d30b5
Reports root / small

small_tests_25 / small_tests / 15d30b5
Reports root / small

ldap_mnesia_24 / ldap_mnesia / 15d30b5
Reports root/ big
OK: 2250 / Failed: 0 / User-skipped: 826 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 15d30b5
Reports root/ big
OK: 4212 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

mam_SUITE:rdbms_async_cache_muc_all:muc_configurable_archiveid:muc_only_stanzaid

{failed,
  {mam_SUITE,end_per_testcase,
    {'EXIT',
      {{room_archive_size,0,[{times,200,1}],ok},
       [{mongoose_helper,do_wait_until,2,
          [{file,
             "/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
           {line,357}]},
        {mam_helper,wait_for_room_archive_size,3,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,789}]},
        {mam_helper,clean_room_archive,1,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,733}]},
        {mam_helper,destroy_room,1,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,726}]},
        {mam_SUITE,end_per_testcase,2,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_SUITE.erl"},
           {line,965}]},
        {test_server,do_end_per_testcase,4,
          [{file,"test_server.erl"},{line,1627}]},
        {test_server,run_test_case_eval1,6,
          [{file,"test_server.erl"},{line,1335}]},
        {test_server,run_test_case_eval,9,
          [{file,"test_server.erl"},{line,1224}]}]}}}}

Report log

ldap_mnesia_25 / ldap_mnesia / 15d30b5
Reports root/ big
OK: 2249 / Failed: 1 / User-skipped: 826 / Auto-skipped: 0

mod_global_distrib_SUITE:hosts_refresher:test_host_refreshing

{error,
  {{trees_for_connections_present,true,[{times,50,false}],ok},
   [{mongoose_helper,do_wait_until,2,
      [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
       {line,357}]},
    {mod_global_distrib_SUITE,test_host_refreshing,1,
      [{file,
         "/home/circleci/project/big_tests/tests/mod_global_distrib_SUITE.erl"},
       {line,384}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1291}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1223}]}]}}

Report log

dynamic_domains_mysql_redis_25 / mysql_redis / 15d30b5
Reports root/ big
OK: 4186 / Failed: 0 / User-skipped: 108 / Auto-skipped: 0

pgsql_mnesia_24 / pgsql_mnesia / 15d30b5
Reports root/ big
OK: 4595 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / 15d30b5
Reports root/ big
OK: 4209 / Failed: 0 / User-skipped: 85 / Auto-skipped: 0

internal_mnesia_25 / internal_mnesia / 15d30b5
Reports root/ big
OK: 2396 / Failed: 0 / User-skipped: 680 / Auto-skipped: 0

mysql_redis_25 / mysql_redis / 15d30b5
Reports root/ big
OK: 4581 / Failed: 0 / User-skipped: 103 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / 15d30b5
Reports root/ big
OK: 4595 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / 15d30b5
Reports root/ big
OK: 4586 / Failed: 1 / User-skipped: 92 / Auto-skipped: 5

bosh_SUITE:essential_https:accept_higher_hold_value

{error,
  {{assertEqual,
     [{module,bosh_SUITE},
      {line,260},
      {expression,"get_bosh_sessions ( )"},
      {expected,[]},
      {value,
        [{bosh_session,<<"227142cad1242272923402949f73dca646676058">>,
           <9322.10168.0>}]}]},
   [{bosh_SUITE,accept_higher_hold_value,1,
      [{file,"/home/circleci/project/big_tests/tests/bosh_SUITE.erl"},
       {line,260}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1291}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1223}]}]}}

Report log

ldap_mnesia_25 / ldap_mnesia / 15d30b5
Reports root/ big
OK: 2250 / Failed: 0 / User-skipped: 826 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / 15d30b5
Reports root/ big
OK: 4592 / Failed: 0 / User-skipped: 92 / Auto-skipped: 0

DenysGonchar · 2023-07-12T10:49:17Z

It looks good. I only suggest changing the target to a feature branch and following with a fix for these errors that appear multiple times in the comments to avoid affecting other PR's:

I have fixed metrics_roster_SUITE:end_per_testcase/2 and it won't spam other PRs, but I would rather want to merge the changes into the master, so we can finally start collecting TC failure statistics for GH Actions.

chrzaszcz

Thanks for fixing it!

DenysGonchar force-pushed the reporting-to-ga4 branch 2 times, most recently from 95aed9f to cc6768c Compare June 17, 2023 23:28

DenysGonchar force-pushed the reporting-to-ga4 branch from cc6768c to 8a234f2 Compare June 26, 2023 22:35

DenysGonchar force-pushed the reporting-to-ga4 branch 4 times, most recently from c832c43 to 791c3d5 Compare July 7, 2023 13:20

DenysGonchar marked this pull request as ready for review July 7, 2023 13:41

DenysGonchar requested a review from chrzaszcz July 7, 2023 14:52

chrzaszcz reviewed Jul 10, 2023

View reviewed changes

esl deleted a comment from mongoose-im Jul 10, 2023

esl deleted a comment from codecov bot Jul 10, 2023

esl deleted a comment from mongoose-im Jul 10, 2023

esl deleted a comment from codecov bot Jul 10, 2023

DenysGonchar force-pushed the reporting-to-ga4 branch from 27d9a2e to 2332d47 Compare July 10, 2023 20:38

esl deleted a comment from codecov bot Jul 10, 2023

esl deleted a comment from mongoose-im Jul 10, 2023

chrzaszcz reviewed Jul 11, 2023

View reviewed changes

DenysGonchar added 9 commits July 11, 2023 12:10

improving ct_groups_summary_hook

b6c35a4

improving truncated_counter_file reporting at ct_markdown_errors_hook

6c0f28f

removing redundant logging from ct_mongoose_hook and ct_tty_hook.

4b31075

ct_mongoose_log_hook already does it, and does it in a better way.

removing ct_mim_config_hook

61140cd

it's quite an overhead to have an extra hook just to set the mim_data_dir parameter, so the hook is merged with ct_mongoose_hook

removing dummy optional CT hook callbacks from big_tests hooks

30b5798

minor improvements to ct_tty_hook

d6b3420

moving halting logic from ct_tty_hook:on_tc_skip/3 to ct_groups_summa…

adec35a

…ry_hook:on_tc_skip/3

adding optional CT suite/group/testcase logging at MIM node

1547306

updating pkg GH Actions job

5bbe8dd

testing package building for ubuntu_xenial as it's not tested by CircleCI

DenysGonchar force-pushed the reporting-to-ga4 branch 3 times, most recently from f460f45 to d38b2fc Compare July 11, 2023 12:26

report failed test cases to GA4 only for master branch

ea1c44d

DenysGonchar force-pushed the reporting-to-ga4 branch from d38b2fc to ea1c44d Compare July 11, 2023 13:43

esl deleted a comment from mongoose-im Jul 11, 2023

esl deleted a comment from codecov bot Jul 11, 2023

esl deleted a comment from mongoose-im Jul 11, 2023

esl deleted a comment from codecov bot Jul 11, 2023

esl deleted a comment from mongoose-im Jul 11, 2023

chrzaszcz reviewed Jul 12, 2023

View reviewed changes

fixing failures in metrics_roster_SUITE:end_per_testcase/2

15d30b5

esl deleted a comment from mongoose-im Jul 12, 2023

chrzaszcz approved these changes Jul 12, 2023

View reviewed changes

chrzaszcz merged commit d87eb4c into master Jul 12, 2023
22 of 24 checks passed

chrzaszcz deleted the reporting-to-ga4 branch July 12, 2023 10:57

chrzaszcz added this to the 6.2.0 milestone Dec 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reporting to ga4 #4040

Reporting to ga4 #4040

DenysGonchar commented Jun 17, 2023 •

edited

chrzaszcz left a comment

chrzaszcz left a comment

DenysGonchar commented Jul 11, 2023

chrzaszcz left a comment

codecov bot commented Jul 12, 2023 •

edited

mongoose-im commented Jul 12, 2023 •

edited

DenysGonchar commented Jul 12, 2023

chrzaszcz left a comment

Reporting to ga4 #4040

Reporting to ga4 #4040

Conversation

DenysGonchar commented Jun 17, 2023 • edited

chrzaszcz left a comment

Choose a reason for hiding this comment

chrzaszcz left a comment

Choose a reason for hiding this comment

DenysGonchar commented Jul 11, 2023

chrzaszcz left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 12, 2023 • edited

Codecov Report

mongoose-im commented Jul 12, 2023 • edited

DenysGonchar commented Jul 12, 2023

chrzaszcz left a comment

Choose a reason for hiding this comment

DenysGonchar commented Jun 17, 2023 •

edited

codecov bot commented Jul 12, 2023 •

edited

mongoose-im commented Jul 12, 2023 •

edited