Dynamic lookup of GDPR-related modules for retrieval #2297

fenek · 2019-05-08T11:05:28Z

This PR replaces hardcoded GDPR modules list with a dynamic lookup. It also changes the retrieval implementations to query all module backends.

Implementation
Tests?

aleklisi

PR is slightly in Erlang hacker style, but I don't see better way to handle the issue.

aleklisi · 2019-05-08T11:15:34Z

src/mongoose_lib.erl

-    bin_to_int(T, (X*10)+(H-$0));
-bin_to_int(Bin, X) ->
-    {X, Bin}.
+%% WARNING! For simplicity, this function searches only MongooseIM code dir


Would it work for MIM installed from binaries?

I'm 99.9% certain it will because we're asking for a path for mongooseim lib and beam files are always in ebin subdir.

aleklisi · 2019-05-08T11:55:52Z

src/mongoose_lib.erl

+-spec find_behaviour_implementations(Behaviour :: module()) -> [module()].
+find_behaviour_implementations(Behaviour) ->
+    {ok, EbinFiles} = file:list_dir(code:lib_dir(mongooseim, ebin)),
+    Mods = [ list_to_existing_atom(filename:rootname(File))


This line may return badarg if there does not already exist an atom whose text representation is String. See: http://erlang.org/doc/man/erlang.html#list_to_existing_atom-1
find_behaviour_implementations/1 is used in mod_vcard:get_personal_data/2 without try catch an than there is top try catch in src/admin_extra/service_admin_extra_gdpr.erl but this will not work for following case:
Let's assume:

module mod_m with backends mod_m_a, mod_m_b, mod_m_c

mod_m_a is loaded but without data

mod_m_b is not loaded and there is no data in this backend

mod_m_c is loaded and there is some data in this backend
Than:
Generating list Mods will crush on mod_m_b and will exit with reason badarg so no further code will be executed and the data from backend provided by mod_m_c will never be retrieved.
Correct me if I am wrong.

Another case would be to start MIM and BEAM with some module and after saving some data in the backend for this module, stop BEAM and when restarting it do it without the module data is in, so there the module is not is not in atoms list but there is data in the backend.

I am pretty sure there are some other cases that will need attention, but I cannot think of any more right now.

This is correct, I tried to be overcautious.

aleklisi · 2019-05-08T12:03:07Z

src/mongoose_lib.erl

+             || File <- EbinFiles, filename:extension(File) == ".beam" ],
+    lists:filter(fun(M) ->
+                         try lists:keyfind([Behaviour], 2, M:module_info(attributes)) of
+                             {Key, _} when Key == behaviour; Key == behavior -> true;


{behaviour, _} -> true; {behavior, _} -> true;

??

I wanted to save 1 line :D

The code is cool, mine is no better, I just showed different version for you to pick which one you like more ;)

codecov · 2019-05-08T12:47:54Z

Codecov Report

Merging #2297 into gdpr-retrieve-clean will decrease coverage by 0.05%.
The diff coverage is 87.5%.

@@                  Coverage Diff                   @@
##           gdpr-retrieve-clean   #2297      +/-   ##
======================================================
- Coverage                78.95%   78.9%   -0.06%     
======================================================
  Files                      334     334              
  Lines                    29038   28918     -120     
======================================================
- Hits                     22928   22817     -111     
+ Misses                    6110    6101       -9

Impacted Files	Coverage Δ
src/mod_vcard.erl	`78.26% <100%> (+0.19%)`	⬆️
src/admin_extra/service_admin_extra_gdpr.erl	`93.22% <100%> (ø)`	⬆️
src/mongoose_lib.erl	`78.57% <77.77%> (+10.38%)`	⬆️
src/pubsub/mod_pubsub.erl	`71.95% <87.5%> (+0.14%)`	⬆️
...bal_distrib/mod_global_distrib_hosts_refresher.erl	`82.22% <0%> (-4.45%)`	⬇️
src/mam/mod_mam_rdbms_prefs.erl	`92.52% <0%> (-3.74%)`	⬇️
src/mod_bosh.erl	`92.85% <0%> (-2.15%)`	⬇️
src/rdbms/mongoose_rdbms.erl	`69.38% <0%> (-2.05%)`	⬇️
src/ejabberd_admin.erl	`58.76% <0%> (-1.99%)`	⬇️
src/mod_muc_room.erl	`76.62% <0%> (-1%)`	⬇️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd91e0a...c28612d. Read the comment docs.

mongoose-im · 2019-05-09T08:59:55Z

6444.1 / Erlang 19.3 / small_tests / d55510e
Reports root / small

6444.6 / Erlang 19.3 / elasticsearch_and_cassandra_mnesia / d55510e
Reports root/ big
OK: 469 / Failed: 0 / User-skipped: 8 / Auto-skipped: 0

6444.2 / Erlang 19.3 / internal_mnesia / d55510e
Reports root/ big
OK: 1221 / Failed: 1 / User-skipped: 66 / Auto-skipped: 0

pubsub_SUITE:tree+node_config:disable_notifications_test

{error,{{badmatch,false},
    [{pubsub_tools,check_response,2,
             [{file,"pubsub_tools.erl"},{line,468}]},
     {pubsub_tools,receive_response,3,
             [{file,"pubsub_tools.erl"},{line,458}]},
     {pubsub_tools,receive_and_check_response,4,
             [{file,"pubsub_tools.erl"},{line,448}]},
     {escalus_story,story,4,
            [{file,"/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1529}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1045}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,977}]}]}}

Report log

6444.3 / Erlang 19.3 / mysql_redis / d55510e
Reports root/ big
OK: 3085 / Failed: 0 / User-skipped: 232 / Auto-skipped: 0

6444.4 / Erlang 19.3 / odbc_mssql_mnesia / d55510e
Reports root/ big
OK: 3087 / Failed: 0 / User-skipped: 230 / Auto-skipped: 0

6444.5 / Erlang 19.3 / ldap_mnesia / d55510e
Reports root/ big
OK: 1199 / Failed: 1 / User-skipped: 102 / Auto-skipped: 0

sm_SUITE:parallel:subscription_requests_are_buffered_properly

{error,{{badmatch,false},
    [{escalus_session,stream_management,2,
              [{file,"/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_session.erl"},
               {line,227}]},
     {escalus_connection,connection_step,2,
               [{file,"/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_connection.erl"},
                {line,134}]},
     {lists,foldl,3,[{file,"lists.erl"},{line,1263}]},
     {escalus_connection,start,2,
               [{file,"/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_connection.erl"},
                {line,118}]},
     {sm_SUITE,'-subscription_requests_are_buffered_properly/1-fun-3-',6,
           [{file,"sm_SUITE.erl"},{line,848}]},
     {escalus_story,story,4,
            [{file,"/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1529}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1045}]}]}}

Report log

6444.8 / Erlang 20.0 / pgsql_mnesia / d55510e
Reports root/ big / small
OK: 3119 / Failed: 0 / User-skipped: 198 / Auto-skipped: 0

6444.9 / Erlang 21.0 / riak_mnesia / d55510e
Reports root/ big / small
OK: 1449 / Failed: 0 / User-skipped: 63 / Auto-skipped: 0

erszcz · 2019-05-09T14:25:58Z

src/mod_vcard.erl

+                                            [{Jid, SerializedRecords}];
+                                        _ -> []
+                                    catch
+                                        _:_ ->


Silent catch-all error - is there really a reason not to log it? This will be a nightmare to debug.

I thought about the same thing after merging and decided to fix it after all other PRs are merged.

erszcz · 2019-05-09T15:00:13Z

src/pubsub/mod_pubsub.erl

+                                Result when is_list(Result) -> Result;
+                                _ -> []
+                            catch
+                                _:_ -> []


FYI, a similar silent catch-all

* Find GDPR modules dynamically * Find module backends for GDPR retrieval dynamically * Fix behaviours' impls lookup * Add test case for GDPR retrieval with disabled modules * Fix GDPR suite

fenek added WIP 🚧 waiting-for-ci GDPR labels May 8, 2019

This comment has been minimized.

Sign in to view

aleklisi reviewed May 8, 2019

View reviewed changes

This comment has been minimized.

Sign in to view

fenek added 4 commits May 8, 2019 17:19

Find GDPR modules dynamically

cdc6925

Find module backends for GDPR retrieval dynamically

5967b34

Fix behaviours' impls lookup

3db2b47

Add test case for GDPR retrieval with disabled modules

2f3ccee

fenek force-pushed the gdpr-retrieve-dynamic branch from c0083fb to 2f3ccee Compare May 8, 2019 15:19

This comment has been minimized.

Sign in to view

Fix GDPR suite

c28612d

fenek added waiting-for-review and removed WIP 🚧 waiting-for-ci labels May 9, 2019

fenek merged commit de52634 into gdpr-retrieve-clean May 9, 2019

fenek deleted the gdpr-retrieve-dynamic branch May 9, 2019 11:31

erszcz reviewed May 9, 2019

View reviewed changes

src/pubsub/mod_pubsub.erl

Result when is_list(Result) -> Result;

_ -> []

catch

_:_ -> []

Copy link

Member

erszcz May 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, a similar silent catch-all

fenek added this to the MongooseIM 3.3.0++ milestone Jun 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic lookup of GDPR-related modules for retrieval #2297

Dynamic lookup of GDPR-related modules for retrieval #2297

fenek commented May 8, 2019 •

edited

This comment has been minimized.

aleklisi left a comment

aleklisi May 8, 2019

fenek May 8, 2019

aleklisi May 8, 2019

aleklisi May 8, 2019

aleklisi May 8, 2019

fenek May 8, 2019

aleklisi May 8, 2019

fenek May 8, 2019

aleklisi May 8, 2019

codecov bot commented May 8, 2019 •

edited

This comment has been minimized.

This comment has been minimized.

mongoose-im commented May 9, 2019 •

edited

erszcz May 9, 2019 •

edited

fenek May 9, 2019

erszcz May 9, 2019

Dynamic lookup of GDPR-related modules for retrieval #2297

Dynamic lookup of GDPR-related modules for retrieval #2297

Conversation

fenek commented May 8, 2019 • edited

This comment has been minimized.

aleklisi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 8, 2019 • edited

Codecov Report

This comment has been minimized.

This comment has been minimized.

mongoose-im commented May 9, 2019 • edited

erszcz May 9, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fenek commented May 8, 2019 •

edited

codecov bot commented May 8, 2019 •

edited

mongoose-im commented May 9, 2019 •

edited

erszcz May 9, 2019 •

edited