Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(mnesia_acl): introduce optimized schema and migration process #5885

Merged
merged 4 commits into from Oct 27, 2021

Conversation

savonarola
Copy link
Contributor

@savonarola savonarola commented Oct 7, 2021

This is an optimization to fix #5506

@savonarola savonarola marked this pull request as draft October 7, 2021 17:19
apps/emqx_auth_mnesia/src/emqx_acl_mnesia_cli.erl Outdated Show resolved Hide resolved
apps/emqx_auth_mnesia/src/emqx_acl_mnesia_cli.erl Outdated Show resolved Hide resolved
apps/emqx_auth_mnesia/src/emqx_acl_mnesia_cli.erl Outdated Show resolved Hide resolved
apps/emqx_auth_mnesia/src/emqx_acl_mnesia_cli.erl Outdated Show resolved Hide resolved
apps/emqx_auth_mnesia/src/emqx_acl_mnesia_cli.erl Outdated Show resolved Hide resolved
apps/emqx_auth_mnesia/src/emqx_acl_mnesia_migrator.erl Outdated Show resolved Hide resolved
apps/emqx_auth_mnesia/src/emqx_acl_mnesia_migrator.erl Outdated Show resolved Hide resolved
@savonarola savonarola requested a review from a team October 7, 2021 17:38
@qzhuyan
Copy link
Contributor

qzhuyan commented Oct 8, 2021

maybe it is safer not to remove the old table and just leave it or?

@qzhuyan
Copy link
Contributor

qzhuyan commented Oct 8, 2021

In general, there is one assumption that all nodes in the cluster should participate in the migration but I wonder if users is ok with that since normally they try a new version on one of the nodes then decide if they should go for it or fallback, especially when they want to try the fix in this PR on the new node only.

@qzhuyan
Copy link
Contributor

qzhuyan commented Oct 8, 2021

In general, there is one assumption that all nodes in the cluster should participate in the migration but I wonder if users is ok with that since normally they try a new version on one of the nodes then decide if they should go for it or fallback, especially when they want to try the fix in this PR on the new node only.

I am just putting the limitation here and I am not asking for a change.

@zmstone
Copy link
Member

zmstone commented Oct 8, 2021

@k32 do you remember where is the code for data export & import ?
since we do not plan to export any data from 4.3, so it's fine not to support it.
but for 4.2 data import, we'll need to be compatible.

@k32
Copy link
Contributor

k32 commented Oct 8, 2021

Code for the data import/export is located in this module: apps/emqx_management/src/emqx_mgmt_data_backup.erl

@savonarola savonarola force-pushed the fix-acl-schema branch 3 times, most recently from a9a7368 to e1c09bc Compare October 13, 2021 10:57
@savonarola savonarola force-pushed the fix-acl-schema branch 2 times, most recently from 8f7d1c2 to 2609814 Compare October 15, 2021 18:06
.ci/acl_migration_test/build.sh Outdated Show resolved Hide resolved
apps/emqx_auth_mnesia/src/emqx_auth_mnesia_api.erl Outdated Show resolved Hide resolved
apps/emqx_auth_mnesia/src/emqx_auth_mnesia_api.erl Outdated Show resolved Hide resolved
apps/emqx_auth_mnesia/test/emqx_acl_mnesia_SUITE.erl Outdated Show resolved Hide resolved
apps/emqx_auth_mnesia/src/emqx_acl_mnesia_migrator.erl Outdated Show resolved Hide resolved
@savonarola savonarola marked this pull request as ready for review October 18, 2021 16:32
@zmstone
Copy link
Member

zmstone commented Oct 18, 2021

Great job!

Copy link
Contributor

@qzhuyan qzhuyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@qzhuyan
Copy link
Contributor

qzhuyan commented Oct 25, 2021

I am not familiar with QLC, someone should review that part.

Comment on lines 224 to 228
login_match_spec_new(LoginSpec) ->
[{{?ACL_TABLE2, LoginSpec, '_'}, [], ['$_']}].

login_match_spec_old(LoginSpec) ->
[{{?ACL_TABLE, {LoginSpec, '_'}, '_', '_', '_'}, [], ['$_']}].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: would it make sense to use a pattern that does not depend on field order or changes?

Like:

login_match_spec_new(LoginSpec) ->
    [{#?ACL_TABLE2{_ = '_', who = LoginSpec}, [], ['$_']}].

login_match_spec_old(LoginSpec) ->
    [{#?ACL_TABLE{_ = '_', filter = {LoginSpec, '_'}}, [], ['$_']}].

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now dialyzer hates it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One trick to use record definition for ets match, and still have sane types is the following:

-record(foo, {a, b}). %% Record definition doesn't specify field types

-type foo() :: #foo{a :: foo(), b :: bar()}. %% Create a typedef that specifies field types and use it in the specs instead of #foo{} type

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But then we will loose useful warnings too 🤔
I mean, when we put #foo{b = NotABar} unintentionally.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't know that was bad for dialyzer, sorry. 🙊

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using record is better.

we can suppress dialyzer warnings. e.g. -dialyzer({nowarn_function,foo/3})

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems that we can't just disable warnings for single funs, because dialyzer looses their typings and fails in other places, like https://github.com/emqx/emqx/runs/4013130567?check_suite_focus=true#step:6:87

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed a variant with ets:fun2ms that also allows to construct matchspecs without concerning about unspecified fields.
Additionally, the external API of emqx_acl_mnesia_db module now does not receive match_head parts.

@savonarola savonarola force-pushed the fix-acl-schema branch 4 times, most recently from d6dae1f to 0860f90 Compare October 26, 2021 19:33
@savonarola savonarola merged commit 49c7eae into emqx:main-v4.3 Oct 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants