Comment failures to a separate issue #3361

arcusfelis · 2021-10-25T11:14:57Z

This PR addresses "getting some statistical info"

Proposed changes include:

A separate issue to gather a list of failed cases + their URLs

Issue with the comments #3360

codecov · 2021-10-25T11:28:32Z

Codecov Report

Merging #3361 (5980790) into master (e09d453) will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3361      +/-   ##
==========================================
+ Coverage   80.66%   80.67%   +0.01%     
==========================================
  Files         397      397              
  Lines       32441    32441              
==========================================
+ Hits        26168    26172       +4     
+ Misses       6273     6269       -4

Impacted Files	Coverage Δ
src/ejabberd.erl	`45.00% <0.00%> (-10.00%)`	⬇️
src/elasticsearch/mongoose_elasticsearch.erl	`76.92% <0.00%> (-7.70%)`	⬇️
src/event_pusher/mod_event_pusher_sns.erl	`84.21% <0.00%> (-5.27%)`	⬇️
src/domain/mongoose_domain_loader.erl	`77.35% <0.00%> (-1.89%)`	⬇️
src/mam/mod_mam_elasticsearch_arch.erl	`85.08% <0.00%> (-1.76%)`	⬇️
src/inbox/mod_inbox_rdbms.erl	`92.50% <0.00%> (-0.63%)`	⬇️
src/pubsub/mod_pubsub_db_rdbms.erl	`95.09% <0.00%> (-0.26%)`	⬇️
src/mod_roster.erl	`78.85% <0.00%> (-0.24%)`	⬇️
src/mod_muc_log.erl	`77.88% <0.00%> (ø)`
src/mod_muc_room.erl	`77.20% <0.00%> (+0.05%)`	⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e09d453...5980790. Read the comment docs.

mongoose-im · 2021-10-25T11:28:40Z

small_tests_24 / small_tests / 5980790
Reports root / small

internal_mnesia_24 / internal_mnesia / 5980790
Reports root/ big
OK: 1589 / Failed: 0 / User-skipped: 297 / Auto-skipped: 0

small_tests_23 / small_tests / 5980790
Reports root / small

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 5980790
Reports root/ big
OK: 2702 / Failed: 0 / User-skipped: 184 / Auto-skipped: 0

ldap_mnesia_24 / ldap_mnesia / 5980790
Reports root/ big
OK: 1486 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0

dynamic_domains_mysql_redis_24 / mysql_redis / 5980790
Reports root/ big
OK: 2685 / Failed: 0 / User-skipped: 201 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_23 / pgsql_mnesia / 5980790
Reports root/ big
OK: 2702 / Failed: 0 / User-skipped: 184 / Auto-skipped: 0

ldap_mnesia_23 / ldap_mnesia / 5980790
Reports root/ big
OK: 1486 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_24 / odbc_mssql_mnesia / 5980790
Reports root/ big
OK: 2702 / Failed: 0 / User-skipped: 184 / Auto-skipped: 0

elasticsearch_and_cassandra_24 / elasticsearch_and_cassandra_mnesia / 5980790
Reports root/ big
OK: 1862 / Failed: 0 / User-skipped: 323 / Auto-skipped: 0

pgsql_mnesia_24 / pgsql_mnesia / 5980790
Reports root/ big
OK: 3071 / Failed: 0 / User-skipped: 211 / Auto-skipped: 0

mysql_redis_24 / mysql_redis / 5980790
Reports root/ big
OK: 3054 / Failed: 0 / User-skipped: 228 / Auto-skipped: 0

mssql_mnesia_24 / odbc_mssql_mnesia / 5980790
Reports root/ big
OK: 3070 / Failed: 1 / User-skipped: 211 / Auto-skipped: 0

inbox_SUITE:one_to_one:other_resources_do_not_interfere

{error,
  {test_case_failed,
    #{check_params => #{},
      error => {badmatch,<<"Hello">>},
      expected_items =>
        [#{content => <<"How are you">>,
         from =>
           <<"kate_other_resources_do_not_interfere_34.620002@localhost/res1">>,
         to =>
           <<"mike_other_resources_do_not_interfere_34.620002@localhost/res1">>,
         unread => 0,verify => #Fun<inbox_SUITE.118.122357169>}],
      inbox_items =>
        [<<"<message from='kate_other_resources_do_not_interfere_34.620002@localhost' to='kate_other_resources_do_not_interfere_34.620002@localhost/res1' id='bc12a1a6-67e7-43d3-aad0-6556805c2b7c'><result xmlns='erlang-solutions.com:xmpp:inbox:0' unread='0' queryid='09fc9f160255d91102939c873b7d29d6'><forwarded xmlns='urn:xmpp:forward:0'><delay xmlns='urn:xmpp:delay' stamp='2021-10-25T11:33:55.369358Z'/><message from='kate_other_resources_do_not_interfere_34.620002@localhost/res1' xml:lang='en' to='mike_other_resources_do_not_interfere_34.620002@localhost/res1' type='chat' id='bbd16ec411ec856f32f9651d658fff41'><body>Hello</body></message></forwarded><archive>false</archive><mute>0</mute></result></message>">>],
      query_params => #{},reason => inbox_mismatch,
      stacktrace =>
        [{inbox_helper,process_inbox_message,4,
           [{file,
            "/home/circleci/app/big_tests/tests/inbox_helper.erl"},
          {line,199}]},
         {lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
         {inbox_helper,check_inbox,4,
          ...

Report log

pgsql_mnesia_23 / pgsql_mnesia / 5980790
Reports root/ big
OK: 3071 / Failed: 0 / User-skipped: 211 / Auto-skipped: 0

riak_mnesia_24 / riak_mnesia / 5980790
Reports root/ big
OK: 1718 / Failed: 2 / User-skipped: 326 / Auto-skipped: 0

mod_event_pusher_rabbit_SUITE:group_chat_message_publish:group_chat_message_received_event_properly_formatted

{error,
  {{assertMatch,
     [{module,mod_event_pusher_rabbit_SUITE},
      {line,435},
      {expression,
        "get_decoded_message_from_rabbit ( AliceGroupChatMsgRecvRK )"},
      {pattern,
        "# { << \"from_user_id\" >> := BobRoomJID , << \"to_user_id\" >> := AliceFullJID , << \"message\" >> := Message }"},
      {value,
        #{<<"from_user_id">> =>
          <<"muc_publish@muc.localhost/bOb_unnamed_20.271388">>,
        <<"message">> => <<"Hi there!">>,
        <<"to_user_id">> =>
          <<"alice_unnamed_20.676332@localhost/res1">>}}]},
   [{mod_event_pusher_rabbit_SUITE,
      '-group_chat_message_received_event_properly_formatted/1-fun-1-',3,
      [{file,
         "/home/circleci/app/big_tests/tests/mod_event_pusher_rabbit_SUITE.erl"},
       {line,435}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1292}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1224}]}]}}

Report log

mod_ping_SUITE:server_ping:server_ping_pong

{error,{{badmatch,[{[<<"localhost">>,mod_ping,ping_response],
          {expected_diff,5},
          {before_story,0},
          {after_story,4}}]},
    [{escalus_mongooseim,post_story_check_metrics,1,
               [{file,"/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_mongooseim.erl"},
                {line,74}]},
     {escalus_mongooseim,maybe_check_metrics_post_story,1,
               [{file,"/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_mongooseim.erl"},
                {line,51}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,75}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1292}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1224}]}]}}

Report log

NelsonVides · 2021-10-25T11:56:08Z

More than one test failed but only the first preset wrote a message on the issue, can you fix that? :)

arcusfelis · 2021-10-25T12:12:25Z

these two in riak_mnesia_24 has eventually passed:

    2 tests eventually passed
    1718 tests passed
    2 tests failed
    324 tests skipped by user
    0 tests skipped automatically

They are not reported, because the build is green. We could collect that info though

NelsonVides · 2021-10-25T12:13:26Z

these two in riak_mnesia_24 has eventually passed:
    2 tests eventually passed
    1718 tests passed
    2 tests failed
    324 tests skipped by user
    0 tests skipped automatically
They are not reported, because the build is green. We could collect that info though

Ahh, all right all right

NelsonVides

Ok, let's do this for now

Comment failures to a separate issue

5980790

mongoose-im mentioned this pull request Oct 25, 2021

CI build failures summary #3360

Closed

NelsonVides approved these changes Oct 25, 2021

View reviewed changes

NelsonVides merged commit 5f82ff6 into master Oct 25, 2021

NelsonVides deleted the mu-summary-commenter branch October 25, 2021 12:19

Premwoik modified the milestone: 5.1.0 May 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comment failures to a separate issue #3361

Comment failures to a separate issue #3361

arcusfelis commented Oct 25, 2021 •

edited

codecov bot commented Oct 25, 2021 •

edited

mongoose-im commented Oct 25, 2021 •

edited

NelsonVides commented Oct 25, 2021

arcusfelis commented Oct 25, 2021

NelsonVides commented Oct 25, 2021

NelsonVides left a comment

Comment failures to a separate issue #3361

Comment failures to a separate issue #3361

Conversation

arcusfelis commented Oct 25, 2021 • edited

codecov bot commented Oct 25, 2021 • edited

Codecov Report

mongoose-im commented Oct 25, 2021 • edited

NelsonVides commented Oct 25, 2021

arcusfelis commented Oct 25, 2021

NelsonVides commented Oct 25, 2021

NelsonVides left a comment

Choose a reason for hiding this comment

arcusfelis commented Oct 25, 2021 •

edited

codecov bot commented Oct 25, 2021 •

edited

mongoose-im commented Oct 25, 2021 •

edited