Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simple MAM queries with complete result flag #3734

Merged
merged 5 commits into from
Aug 17, 2022

Conversation

arcusfelis
Copy link
Contributor

This PR addresses "complete flag is not always set".

Proposed changes include:

  • New shared function mod_mam_utils:lookup/3
  • There are two strategies to determine complete flag:
    • is_complete_result_page_using_offset
    • set_complete_result_page_using_extra_message for simple queries.
  • set_complete_result_page_using_extra_message asks for an extra message on a page from a DB. After that it hides the message from the user, but sets complete flag properly

@mongoose-im

This comment was marked as outdated.

@codecov
Copy link

codecov bot commented Aug 17, 2022

Codecov Report

Merging #3734 (f6ae3c3) into master (1b9802b) will increase coverage by 0.04%.
The diff coverage is 95.83%.

@@            Coverage Diff             @@
##           master    #3734      +/-   ##
==========================================
+ Coverage   82.28%   82.32%   +0.04%     
==========================================
  Files         526      526              
  Lines       33844    33862      +18     
==========================================
+ Hits        27847    27876      +29     
+ Misses       5997     5986      -11     
Impacted Files Coverage Δ
src/mam/mod_mam.erl 97.40% <ø> (ø)
src/mam/mod_mam_muc_cassandra_arch.erl 84.79% <ø> (+0.92%) ⬆️
src/mam/mod_mam_utils.erl 89.40% <95.45%> (+0.37%) ⬆️
src/mam/mod_mam_muc.erl 75.23% <100.00%> (-0.12%) ⬇️
src/mam/mod_mam_pm.erl 89.51% <100.00%> (-0.05%) ⬇️
src/mam/mod_mam_muc_rdbms_arch_async.erl 82.85% <0.00%> (-2.86%) ⬇️
src/pubsub/mod_pubsub_db_mnesia.erl 92.40% <0.00%> (-0.43%) ⬇️
src/mod_muc_log.erl 62.82% <0.00%> (ø)
src/pubsub/node_pep.erl 79.62% <0.00%> (ø)
src/inbox/mod_inbox_rdbms_async.erl 72.05% <0.00%> (ø)
... and 9 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@mongoose-im
Copy link
Collaborator

mongoose-im commented Aug 17, 2022

small_tests_24 / small_tests / f6ae3c3
Reports root / small


small_tests_25 / small_tests / f6ae3c3
Reports root / small


dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / f6ae3c3
Reports root/ big
OK: 3501 / Failed: 0 / User-skipped: 88 / Auto-skipped: 0


ldap_mnesia_24 / ldap_mnesia / f6ae3c3
Reports root/ big
OK: 1897 / Failed: 0 / User-skipped: 520 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / f6ae3c3
Reports root/ big
OK: 3501 / Failed: 0 / User-skipped: 88 / Auto-skipped: 0


ldap_mnesia_25 / ldap_mnesia / f6ae3c3
Reports root/ big
OK: 1897 / Failed: 0 / User-skipped: 520 / Auto-skipped: 0


dynamic_domains_mysql_redis_25 / mysql_redis / f6ae3c3
Reports root/ big
OK: 3484 / Failed: 0 / User-skipped: 105 / Auto-skipped: 0


internal_mnesia_25 / internal_mnesia / f6ae3c3
Reports root/ big
OK: 2018 / Failed: 0 / User-skipped: 399 / Auto-skipped: 0


dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / f6ae3c3
Reports root/ big
OK: 3501 / Failed: 0 / User-skipped: 88 / Auto-skipped: 0


pgsql_mnesia_24 / pgsql_mnesia / f6ae3c3
Reports root/ big
OK: 3875 / Failed: 0 / User-skipped: 97 / Auto-skipped: 0


elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / f6ae3c3
Reports root/ big
OK: 2358 / Failed: 0 / User-skipped: 394 / Auto-skipped: 0


pgsql_mnesia_25 / pgsql_mnesia / f6ae3c3
Reports root/ big
OK: 3875 / Failed: 0 / User-skipped: 97 / Auto-skipped: 0


mysql_redis_25 / mysql_redis / f6ae3c3
Reports root/ big
OK: 3869 / Failed: 1 / User-skipped: 102 / Auto-skipped: 0

muc_SUITE:hibernation:hibernated_room_can_be_queried_for_archive
{error,{{assertion_failed,assert,is_groupchat_message,
              [<<"Restorable message">>],
              undefined,"undefined"},
    [{escalus_new_assert,assert_true,2,
               [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
                {line,84}]},
     {muc_SUITE,wait_for_mam_result,3,
          [{file,"/home/circleci/project/big_tests/tests/muc_SUITE.erl"},
           {line,4394}]},
     {muc_SUITE,'-hibernated_room_can_be_queried_for_archive/1-fun-0-',3,
          [{file,"/home/circleci/project/big_tests/tests/muc_SUITE.erl"},
           {line,4130}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {muc_SUITE,hibernated_room_can_be_queried_for_archive,1,
          [{file,"/home/circleci/project/big_tests/tests/muc_SUITE.erl"},
           {line,4126}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log


mssql_mnesia_25 / odbc_mssql_mnesia / f6ae3c3
Reports root/ big
OK: 3875 / Failed: 0 / User-skipped: 97 / Auto-skipped: 0


riak_mnesia_24 / riak_mnesia / f6ae3c3
Reports root/ big
OK: 2215 / Failed: 1 / User-skipped: 387 / Auto-skipped: 0

pubsub_SUITE:tree+basic:subscribe_options_deliver_option_test
{error,
  {{badmatch,
     [{xmlel,<<"message">>,
        [{<<"from">>,<<"pubsub.localhost">>},
         {<<"to">>,
        <<"bob_subscribe_options_deliver_option_test_1531@localhost/res1">>},
         {<<"type">>,<<"headline">>}],
        [{xmlel,<<"event">>,
           [{<<"xmlns">>,
           <<"http://jabber.org/protocol/pubsub#event">>}],
           [{xmlel,<<"items">>,
            [{<<"node">>,<<"princely_musings_9JahZBKl9mA=">>}],
            [{xmlel,<<"item">>,
               [{<<"id">>,<<"item1">>}],
               [{xmlel,<<"entry">>,
                  [{<<"xmlns">>,
                  <<"http://www.w3.org/2005/Atom">>}],
                  []}]}]}]},
         {xmlel,<<"headers">>,
           [{<<"xmlns">>,<<"http://jabber.org/protocol/shim">>}],
           []}]}]},
   [{pubsub_SUITE,'-subscribe_options_deliver_option_test/1-fun-0-',3,
      [{file,"/home/circleci/project/big_tests/tests/pubsub_SUITE.erl"},
       {line,479}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1292}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1224}]}]}}

Report log


mysql_redis_25 / mysql_redis / f6ae3c3
Reports root/ big
OK: 3870 / Failed: 0 / User-skipped: 102 / Auto-skipped: 0

@NelsonVides NelsonVides self-requested a review August 17, 2022 15:30
Copy link
Collaborator

@NelsonVides NelsonVides left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to merge, but just some small comments.

end.

remove_extra_message(Params, Messages) ->
case maps:get(ordering_direction, Params, forward) of
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be pattern-matched on the function clause, it'll be even faster (see maps performance guide if curious).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, in case key exists - yea

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And in the case it does not, a second catch-all clause just throws the default.

Other
end.

patch_fun_to_make_result_as_map(F) ->
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh god, this is patching extreme dynamic typing 😱 anyway, for future major refactorings.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I knew you will love it!
Yea, lookup_result_map should replace the tuple. Bad news is that it is like 50 places we use the tuple :D

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I imagined 😱

@@ -1155,19 +1156,75 @@ is_policy_violation(TotalCount, Offset, MaxResultLimit, LimitPassed) ->
LookupResult :: mod_mam:lookup_result(),
R :: {ok, mod_mam:lookup_result()} | {error, item_not_found}.
check_for_item_not_found(#rsm_in{direction = before, id = ID},
PageSize, {TotalCount, Offset, MessageRows}) ->
_PageSize, {TotalCount, Offset, MessageRows}) ->
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this parameter is not needed it can be removed, maybe?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

of course xd
Remained for compat/lazy.
Will get refactored with the tuple. And we probably wanna pass Params, instead of just RSM.

@NelsonVides NelsonVides merged commit e65aace into master Aug 17, 2022
@NelsonVides NelsonVides deleted the mu-mam-simple-with-complete-result-flag branch August 17, 2022 19:43
@chrzaszcz chrzaszcz added this to the 6.0.0 milestone Dec 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants