feat: implement configuration and user data export/import #10676

SergeTupchiy · 2023-05-11T17:40:43Z

Fixes EMQX-9203

Summary

`🤖 Generated by Copilot at f1dce5c`

This pull request adds data backup features for various EMQ X modules and applications, allowing users to export and import their configurations, authentication sources, authorization rules, users, api keys, banned clients, listeners, bridges, and auto subscribe topics as JSON files. It also updates some application versions, comment styles, and formatting functions. It mainly uses base64 encoding and decoding, JSON encoding and decoding, and file reading and writing functions to handle the data backup logic. The main files involved are emqx_authn.erl, emqx_authz.erl, emqx_bridge.erl, emqx_dashboard_admin.erl, emqx_exhook_mgr.erl, emqx_gateway_conf.erl, emqx_listeners.erl, emqx_mgmt_api_api_keys.erl, emqx_mgmt_auth.erl, emqx_mgmt_cli.erl, emqx_mgmt_data_backup.erl, and emqx_tls_lib.erl.

PR Checklist

Please convert it to a draft if any of the following conditions are not met. Reviewers may skip over until all the items are checked:

Added tests for the changes
Changed lines covered in coverage report
Change log has been added to changes/{ce,ee}/(feat|perf|fix)-<PR-id>.en.md files
For internal contributor: there is a jira ticket to track this change
If there should be document changes, a PR to emqx-docs.git is sent, or a jira ticket is created to follow up
Schema changes are backward compatible

Checklist for CI (.github/workflows) changes

If changed package build workflow, pass this action (manual trigger)
Change log has been added to changes/ dir for user-facing artifacts update

coveralls · 2023-05-11T18:48:37Z

Pull Request Test Coverage Report for Build 5214393224

383 of 456 (83.99%) changed or added relevant lines in 24 files are covered.
53 unchanged lines in 16 files lost coverage.
Overall coverage increased (+0.2%) to 81.921%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
apps/emqx_authz/src/emqx_authz.erl	18	19	94.74%
apps/emqx_auto_subscribe/src/emqx_auto_subscribe.erl	9	10	90.0%
apps/emqx_exhook/src/emqx_exhook_mgr.erl	8	9	88.89%
apps/emqx_gateway/src/emqx_gateway_conf.erl	7	8	87.5%
apps/emqx_utils/src/emqx_utils.erl	8	9	88.89%
apps/emqx_authn/src/emqx_authn.erl	9	11	81.82%
apps/emqx_management/src/emqx_mgmt_listeners_conf.erl	22	24	91.67%
apps/emqx_bridge/src/emqx_bridge.erl	27	31	87.1%
apps/emqx_psk/src/emqx_psk.erl	2	8	25.0%
lib-ee/emqx_ee_schema_registry/src/emqx_ee_schema_registry.erl	15	21	71.43%

Files with Coverage Reduction	New Missed Lines	%
apps/emqx_bridge_kafka/src/emqx_bridge_kafka_impl_consumer.erl	1	81.13%
apps/emqx_gateway_mqttsn/src/emqx_mqttsn_frame.erl	1	63.48%
apps/emqx/src/emqx_connection.erl	1	83.24%
apps/emqx/src/emqx_limiter/src/emqx_limiter_schema.erl	1	81.48%
apps/emqx/src/emqx_limiter/src/emqx_limiter_server.erl	1	90.07%
apps/emqx/src/emqx_quic_data_stream.erl	1	75.21%
apps/emqx/src/emqx_session.erl	1	87.25%
apps/emqx/src/emqx_channel.erl	2	87.71%
apps/emqx/src/emqx_logger_textfmt.erl	2	56.82%
apps/emqx/src/emqx_trace/emqx_trace.erl	2	88.18%

Totals
Change from base Build 5211972019:	0.2%
Covered Lines:	29620
Relevant Lines:	36157

💛 - Coveralls

zhongwencool · 2023-05-24T02:58:56Z

apps/emqx_authn/src/emqx_authn.erl

+    AuthnList = maps:get(<<"authentication">>, RawConf, []),
+    lists:foreach(
+        fun(Authn) ->
+            case update_conf(update_req(Authn)) of
+                %% Assume not_found error
+                {error, _} ->
+                    {ok, _} = update_conf(create_req(Authn));
+                {ok, _} ->
+                    ok
+            end
+        end,
+        AuthnList
+    ).


we need support delete action, for deleting authn not exist in rawConf.

Note: it has been agreed to merge imported data with the existing one (without deleting) in the first iteration to keep it consistent with EMQX4.4 data import behaviour (https://emqx.atlassian.net/browse/EMQX-9203?focusedCommentId=28667).
Replacing (restore backup) may be implemented later, as it was discussed on the demo meeting.

SergeTupchiy · 2023-06-01T07:37:35Z

apps/emqx_management/src/emqx_mgmt_data_backup.erl

+                true ->
+                    %% 4.x backup files are anyway not compatible and will be treated as invalid,
+                    %% before this step.
+                    maybe_print(


This can be done differently.
My point was that: it's hard to ensure forward compatibility, but at the same time, disallowing newer version imports may be too strict.
Alternatively, we can add some info about forward-compatibility to META.hocon, e.g.

min_major_version=5 min_minor_version=1

And when we import the backup, we can compare it with EMQX version and disallow import if it's not supported.

Group review feedback: disallow 'downgrade' import (if backup is newer than EMQX - don't allow import).

Agreed to disallow importing data if backup version is newer than EMQX version.

SergeTupchiy · 2023-06-01T07:40:17Z

apps/emqx_management/src/emqx_mgmt_data_backup.erl

+    RawConf1 = emqx_authz:maybe_write_acl_file(RawConf),
+    emqx_hocon:check(
+        emqx_conf:schema_module(),
+        maps:merge(emqx:get_raw_config([]), RawConf1),


The idea is to validate the whole config before trying to import its individual root keys.
I use maps:merge/2 just to put existing values for required keys like node (no need for deep merging).
I wonder if there are better alternatives.. 🤔

Discussed and agreed that it must be fine.

apps/emqx_management/src/emqx_mgmt_data_backup.erl

savonarola · 2023-06-01T09:10:29Z

apps/emqx_management/src/emqx_mgmt_data_backup.erl

+prepare_new_backup(Opts) ->
+    {{Y, M, D}, {H, MM, S}} = _DateTime = local_datetime(),
+    BackupBaseName = str(
+        io_lib:format("emqx-export-~p-~p-~p-~p-~p-~p", [Y, M, D, H, MM, S])


Nit: Maybe smth like

io_lib:format( "~0p.~2..0b.~2..0b.~2..0b.~2..0b.~2..0b.~3..0b", [Y, M, D, HH, MM, SS, Ts rem 1000] )

as in recent Stone's commit to make e.g. names more sorting-friendly?

Thanks, great idea!
But I think it's better to keep dashes as separators for date/time, e,g:

io_lib:format("emqx-export-~0p-~2..0b-~2..0b-~2..0b-~2..0b-~2..0b.~3..0b", [Y, M, D, HH, MM, SS, Ts rem 1000]) %% emqx-export-2023-06-01-13-06-49.135.tar.gz.

apps/emqx_management/src/emqx_mgmt_data_backup.erl

apps/emqx_management/src/emqx_mgmt_listeners_conf.erl

savonarola · 2023-06-01T10:33:33Z

apps/emqx_authn/src/emqx_authn.erl

+            end
+        end,
+        {[], #{}},
+        AuthnList


EMQX schema also supports individual authenticator config as "authentication" value, so it may appear in the raw config:

emqx/apps/emqx/src/emqx_schema.erl

Line 2872 in b5f24c4

converter => fun ensure_array/2,

Do we handle this?

Added a helper to handle it, thanks!

savonarola · 2023-06-01T11:06:05Z

apps/emqx_authn/src/emqx_authn.erl

+-define(IMPORT_OPTS, #{rawconf_with_defaults => true, override_to => cluster}).
+
+import_data(RawConf) ->
+    AuthnList = maps:get(<<"authentication">>, RawConf, []),


Suggested change

AuthnList = maps:get(<<"authentication">>, RawConf, []),

AuthnList = maps:get(?CONF_NS_BINARY, RawConf, []),

Thanks, changed all the occurrences in authn and authz

apps/emqx/src/bhvrs/emqx_config_backup.erl

apps/emqx_management/src/emqx_mgmt_data_backup.erl

…r its lifetime

apps/emqx_authn/src/emqx_authn.erl

apps/emqx_authz/src/emqx_authz.erl

apps/emqx/src/emqx_listeners.erl

zmstone · 2023-06-09T11:06:35Z

changes/ce/feat-10676.en.md

@@ -0,0 +1,4 @@
+Implement configuration and user data import/export CLI.
+
+The tool allows to export configuration and built-in database data from a running


nit:

Suggested change

The tool allows to export configuration and built-in database data from a running

The `emqx ctl export` and `emqx ctl import` commands allow to export configuration and built-in database data from a running

fixed, thanks

Closes: EMQX-9203

yanzhiemq · 2023-06-13T06:02:24Z

Enhancements

Added CLI commands emqx ctl export and emqx ctl import for importing/exporting configuration and user data. This allows exporting configurations and built-in database data from a running EMQX cluster and importing them into the same or another running EMQX cluster.

yanzhiemq · 2023-06-13T06:02:26Z

增强

新增了用于导入/导出配置和用户数据的命令 emqx ctl export 和 emqx ctl import，允许从正在运行的 EMQX 集群中导出配置和内置数据库数据，然后将其导入到相同或另一个正在运行的 EMQX 集群中。

SergeTupchiy force-pushed the EMQX-9203-config-backup branch from f1dce5c to 4a63a5a Compare May 11, 2023 17:47

SergeTupchiy force-pushed the EMQX-9203-config-backup branch 6 times, most recently from bb72066 to af5231f Compare May 15, 2023 10:51

SergeTupchiy force-pushed the EMQX-9203-config-backup branch 4 times, most recently from 0241e41 to 4ae42de Compare May 23, 2023 19:05

zhongwencool reviewed May 24, 2023

View reviewed changes

SergeTupchiy force-pushed the EMQX-9203-config-backup branch 4 times, most recently from 85039da to a796f46 Compare May 29, 2023 18:38

SergeTupchiy force-pushed the EMQX-9203-config-backup branch 3 times, most recently from ef5367c to 28cab51 Compare May 31, 2023 18:20

SergeTupchiy changed the title ~~feat: implement configuration and user data backup/restore~~ feat: implement configuration and user data export/import May 31, 2023

SergeTupchiy marked this pull request as ready for review May 31, 2023 20:12

SergeTupchiy requested review from a team, kjellwinblad, lafirest, JimMoen, sstrigler and savonarola as code owners May 31, 2023 20:12

SergeTupchiy force-pushed the EMQX-9203-config-backup branch from 28cab51 to a55a2ec Compare May 31, 2023 21:04

SergeTupchiy commented Jun 1, 2023

View reviewed changes

apps/emqx_management/src/emqx_mgmt_data_backup.erl Show resolved Hide resolved

savonarola reviewed Jun 1, 2023

View reviewed changes

apps/emqx_management/src/emqx_mgmt_data_backup.erl Outdated Show resolved Hide resolved

savonarola reviewed Jun 1, 2023

View reviewed changes

apps/emqx_management/src/emqx_mgmt_listeners_conf.erl Outdated Show resolved Hide resolved

savonarola reviewed Jun 1, 2023

View reviewed changes

SergeTupchiy commented Jun 1, 2023

View reviewed changes

apps/emqx/src/bhvrs/emqx_config_backup.erl Outdated Show resolved Hide resolved

SergeTupchiy commented Jun 1, 2023

View reviewed changes

apps/emqx_management/src/emqx_mgmt_data_backup.erl Outdated Show resolved Hide resolved

SergeTupchiy force-pushed the EMQX-9203-config-backup branch 2 times, most recently from b272c1a to 8aeebc4 Compare June 1, 2023 17:20

SergeTupchiy added 3 commits June 8, 2023 20:42

style: use double percent (%%) comments

5f526d5

test(emqx_common_test_helpers): keep EMQX ENV vars on a child node fo…

33fa053

…r its lifetime

fix(emqx_listeners): convert authentication certs

53ec832

SergeTupchiy force-pushed the EMQX-9203-config-backup branch from 8aeebc4 to ba64428 Compare June 8, 2023 18:10

zmstone reviewed Jun 9, 2023

View reviewed changes

apps/emqx_authn/src/emqx_authn.erl Show resolved Hide resolved

zhongwencool reviewed Jun 9, 2023

View reviewed changes

apps/emqx_authz/src/emqx_authz.erl Outdated Show resolved Hide resolved

apps/emqx/src/emqx_listeners.erl Show resolved Hide resolved

zhongwencool previously approved these changes Jun 9, 2023

View reviewed changes

zmstone reviewed Jun 9, 2023

View reviewed changes

zmstone previously approved these changes Jun 9, 2023

View reviewed changes

feat: implement configuration and user data export/import CLI

e4d09d4

Closes: EMQX-9203

SergeTupchiy dismissed stale reviews from zmstone and zhongwencool via e4d09d4 June 9, 2023 11:12

SergeTupchiy force-pushed the EMQX-9203-config-backup branch from ba64428 to e4d09d4 Compare June 9, 2023 11:12

zmstone approved these changes Jun 9, 2023

View reviewed changes

SergeTupchiy merged commit e61b210 into emqx:master Jun 9, 2023
41 checks passed

SergeTupchiy mentioned this pull request Jun 9, 2023

fix(emqx_mgmt_data_backup): avoid writing temporary Mnesia backup file #10993

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement configuration and user data export/import #10676

feat: implement configuration and user data export/import #10676

SergeTupchiy commented May 11, 2023 •

edited

coveralls commented May 11, 2023 •

edited

zhongwencool May 24, 2023

SergeTupchiy May 26, 2023

SergeTupchiy Jun 1, 2023

SergeTupchiy Jun 1, 2023

SergeTupchiy Jun 1, 2023

SergeTupchiy Jun 1, 2023

SergeTupchiy Jun 1, 2023

savonarola Jun 1, 2023

SergeTupchiy Jun 1, 2023

SergeTupchiy Jun 1, 2023

savonarola Jun 1, 2023

SergeTupchiy Jun 1, 2023

savonarola Jun 1, 2023

SergeTupchiy Jun 1, 2023

zmstone Jun 9, 2023

SergeTupchiy Jun 9, 2023

yanzhiemq commented Jun 13, 2023 •

edited by Meggielqk

yanzhiemq commented Jun 13, 2023 •

edited by Meggielqk

	AuthnList = maps:get(<<"authentication">>, RawConf, []),
	AuthnList = maps:get(?CONF_NS_BINARY, RawConf, []),

		@@ -0,0 +1,4 @@
		Implement configuration and user data import/export CLI.

		The tool allows to export configuration and built-in database data from a running

	The tool allows to export configuration and built-in database data from a running
	The `emqx ctl export` and `emqx ctl import` commands allow to export configuration and built-in database data from a running

feat: implement configuration and user data export/import #10676

feat: implement configuration and user data export/import #10676

Conversation

SergeTupchiy commented May 11, 2023 • edited

Summary

🤖 Generated by Copilot at f1dce5c

PR Checklist

Checklist for CI (.github/workflows) changes

coveralls commented May 11, 2023 • edited

Pull Request Test Coverage Report for Build 5214393224

💛 - Coveralls

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yanzhiemq commented Jun 13, 2023 • edited by Meggielqk

Enhancements

yanzhiemq commented Jun 13, 2023 • edited by Meggielqk

增强

SergeTupchiy commented May 11, 2023 •

edited

`🤖 Generated by Copilot at f1dce5c`

coveralls commented May 11, 2023 •

edited

yanzhiemq commented Jun 13, 2023 •

edited by Meggielqk

yanzhiemq commented Jun 13, 2023 •

edited by Meggielqk