feat: 配置文件导入导出功能 (#524) #858

alexwanglei · 2022-12-09T09:33:53Z

fix issue (【配置中心】配置组或者配置文件的导入导出功能 #524)

polaris-test-bot · 2022-12-09T09:34:12Z

2022-12-09 17:34:11: =========TEST START==========
2022-12-09 17:34:50: Pre-Merge check success! Now we will perform a full automated test which is expected to take 2 hours, thank you for your patience.
2022-12-09 17:42:46: =========TEST START==========
2022-12-09 17:43:11: Pre-Merge check success! Now we will perform a full automated test which is expected to take 2 hours, thank you for your patience.
2022-12-09 21:59:31: =========TEST START==========
2022-12-09 21:59:58: Pre-Merge check success! Now we will perform a full automated test which is expected to take 2 hours, thank you for your patience.
2023-01-03 17:22:22: =========TEST START==========
2023-01-03 17:22:52: Pre-Merge check success! Now we will perform a full automated test which is expected to take 2 hours, thank you for your patience.
2023-01-03 17:43:56: =========TEST START==========
2023-01-03 17:44:26: Pre-Merge check success! Now we will perform a full automated test which is expected to take 2 hours, thank you for your patience.
2023-02-02 15:57:37: =========TEST START==========
2023-02-02 15:58:05: Pre-Merge check success! Now we will perform a full automated test which is expected to take 2 hours, thank you for your patience.
2023-02-02 16:11:58: [STANDALONE VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/sta/qta-report.html
2023-02-02 16:12:02: [CLUSTER VERSION] PR autotest FAILED! We executed 20 test cases and 19 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/clu/qta-report.html
2023-02-02 16:12:06: [KUBENETES VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/k8s/qta-report.html
2023-02-02 16:20:43: =========TEST START==========
2023-02-02 16:21:12: Pre-Merge check success! Now we will perform a full automated test which is expected to take 2 hours, thank you for your patience.
2023-02-02 16:35:57: [STANDALONE VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/sta/qta-report.html
2023-02-02 16:36:01: [CLUSTER VERSION] PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/clu/qta-report.html
2023-02-02 16:36:05: [KUBENETES VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/k8s/qta-report.html
2023-02-03 17:01:22: =========TEST START==========
2023-02-03 17:01:52: Pre-Merge check success! Now we will perform a full automated test which is expected to take 2 hours, thank you for your patience.
2023-02-03 17:15:26: [STANDALONE VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/sta/qta-report.html
2023-02-03 17:15:30: [CLUSTER VERSION] PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/clu/qta-report.html
2023-02-03 17:15:34: [KUBENETES VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/k8s/qta-report.html
2023-02-03 19:41:30: =========TEST START==========
2023-02-03 19:41:59: Pre-Merge check success! Now we will perform a full automated test which is expected to take 2 hours, thank you for your patience.
2023-02-03 19:55:36: [STANDALONE VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/sta/qta-report.html
2023-02-03 19:55:40: [CLUSTER VERSION] PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/clu/qta-report.html
2023-02-03 19:55:44: [KUBENETES VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/k8s/qta-report.html
2023-02-07 10:29:46: =========TEST START==========
2023-02-07 10:30:15: Pre-Merge check success! Now we will perform a full automated test which is expected to take 2 hours, thank you for your patience.
2023-02-07 10:45:47: [STANDALONE VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/sta/qta-report.html
2023-02-07 10:45:51: [CLUSTER VERSION] PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/clu/qta-report.html
2023-02-07 10:45:54: [KUBENETES VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/k8s/qta-report.html
2023-02-08 11:09:13: =========TEST START==========
2023-02-08 11:09:41: Pre-Merge check success! Now we will perform a full automated test which is expected to take 2 hours, thank you for your patience.
2023-02-08 11:24:36: [STANDALONE VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/sta/qta-report.html
2023-02-08 11:24:39: [CLUSTER VERSION] PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/clu/qta-report.html
2023-02-08 11:24:43: [KUBENETES VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/k8s/qta-report.html
2023-02-08 14:10:38: =========TEST START==========
2023-02-08 14:11:08: Pre-Merge check success! Now we will perform a full automated test which is expected to take 2 hours, thank you for your patience.
2023-02-08 14:24:29: [STANDALONE VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/sta/qta-report.html
2023-02-08 14:24:33: [CLUSTER VERSION] PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/clu/qta-report.html
2023-02-08 14:24:37: [KUBENETES VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/k8s/qta-report.html
2023-02-08 16:09:28: =========TEST START==========
2023-02-08 16:09:58: Pre-Merge check success! Now we will perform a full automated test which is expected to take 2 hours, thank you for your patience.
2023-02-08 16:24:09: [STANDALONE VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/sta/qta-report.html
2023-02-08 16:24:13: [CLUSTER VERSION] PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/clu/qta-report.html
2023-02-08 16:24:17: [KUBENETES VERSION]PR autotest PASSED! We executed 20 test cases and 20 were passed. You can see test report in https://polaris-opentest-1306863770.cos.ap-guangzhou.myqcloud.com/pr1155744609/k8s/qta-report.html

config/common.go

lepdou · 2022-12-09T10:03:02Z

导出的zip包格式，现在是只能用导出来的包吗？

是不是可以支持，用户随意的把本地磁盘的一个目录压缩之后，上传到一个 file group 下吗？这种场景下，tag那些都可以不要。然后把目录转化成 / 的文件名？

比如用户有一个这样的目录:
config
-- a.txt
--- | common
-- b.yml

直接就可以创建出 config/a.txt 和 config/common/b.yml 文件。

支持这种场景，是不是应用更广泛一点？比如用户直接把项目里的resource 目录下整个传上去，初始化迁移就很方便。

lepdou · 2022-12-09T10:03:58Z

另外，导入的时候，文件冲突是怎么解决的？直接覆盖还是？可能要考虑一下

conf/polaris-server.yaml

chuntaojun · 2022-12-09T10:16:52Z

导出的zip包格式，现在是只能用导出来的包吗？

是不是可以支持，用户随意的把本地磁盘的一个目录压缩之后，上传到一个 file group 下吗？这种场景下，tag那些都可以不要。然后把目录转化成 / 的文件名？

比如用户有一个这样的目录: config -- a.txt --- | common -- b.yml

直接就可以创建出 config/a.txt 和 config/common/b.yml 文件。

支持这种场景，是不是应用更广泛一点？比如用户直接把项目里的resource 目录下整个传上去，初始化迁移就很方便。

感觉有很多种？

直接导入，比如期望同时导入到多个 ns 当中去
期望导入的时候，可以根据一些方式直接导入到我期望的多个group
单个 group 下的导入导出

chuntaojun · 2022-12-09T10:19:42Z

为了避免贡献者太多的代码改的，可以先在相关 issue 再次讨论下，确定下最终导入导出的方案

chuntaojun · 2022-12-09T10:20:55Z

另外，导入的时候，文件冲突是怎么解决的？直接覆盖还是？可能要考虑一下

这个确实需要考虑，是不是走页面提示比较好？或者导入的时候用户可以选择导入规则

冲突不覆盖
冲突覆盖

无论选择哪一种，都需要给出冲突的列表以及冲突的最终处理结果？

codecov · 2022-12-09T10:45:59Z

Codecov Report

Merging #858 (086cbe3) into main (8054cc6) will increase coverage by 0.07%.
The diff coverage is 56.17%.

@@            Coverage Diff             @@
##             main     #858      +/-   ##
==========================================
+ Coverage   54.34%   54.42%   +0.07%     
==========================================
  Files         182      182              
  Lines       33281    33497     +216     
==========================================
+ Hits        18086    18230     +144     
- Misses      12753    12818      +65     
- Partials     2442     2449       +7

Impacted Files	Coverage Δ
config/config_file_group.go	`38.25% <0.00%> (-0.32%)`	⬇️
store/mysql/config_file.go	`57.14% <ø> (+4.57%)`	⬆️
config/config_file.go	`51.20% <56.30%> (+2.98%)`	⬆️
config/config_file_authibility.go	`65.57% <70.00%> (+0.86%)`	⬆️
cache/config_file.go	`71.19% <0.00%> (-1.64%)`	⬇️
cache/routing_config.go	`63.49% <0.00%> (-1.20%)`	⬇️
store/boltdb/instance.go	`62.90% <0.00%> (+0.54%)`	⬆️
store/mysql/routing_config_v2.go	`51.40% <0.00%> (+1.40%)`	⬆️
store/boltdb/config_file.go	`65.93% <0.00%> (+2.62%)`	⬆️
... and 2 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

alexwanglei · 2022-12-09T13:26:47Z

另外，导入的时候，文件冲突是怎么解决的？直接覆盖还是？可能要考虑一下

这个确实需要考虑，是不是走页面提示比较好？或者导入的时候用户可以选择导入规则

冲突不覆盖

冲突覆盖

无论选择哪一种，都需要给出冲突的列表以及冲突的最终处理结果？

在同命名空间的同组下导入同名的配置文件，在创建配置文件时会报资源已存在的错误。这块时复用的创建配置文件的逻辑。

alexwanglei · 2022-12-09T13:32:42Z

导出的zip包格式，现在是只能用导出来的包吗？

是不是可以支持，用户随意的把本地磁盘的一个目录压缩之后，上传到一个 file group 下吗？这种场景下，tag那些都可以不要。然后把目录转化成 / 的文件名？

比如用户有一个这样的目录: config -- a.txt --- | common -- b.yml

直接就可以创建出 config/a.txt 和 config/common/b.yml 文件。

支持这种场景，是不是应用更广泛一点？比如用户直接把项目里的resource 目录下整个传上去，初始化迁移就很方便。

现在就是按照zip包中的目录结构导入的，namespace/group/file，META文件主要为了记录配置文件的Tags信息，没有META文件也可以导入。

alexwanglei · 2022-12-09T13:39:02Z

导出的zip包格式，现在是只能用导出来的包吗？
是不是可以支持，用户随意的把本地磁盘的一个目录压缩之后，上传到一个 file group 下吗？这种场景下，tag那些都可以不要。然后把目录转化成 / 的文件名？
比如用户有一个这样的目录: config -- a.txt --- | common -- b.yml
直接就可以创建出 config/a.txt 和 config/common/b.yml 文件。
支持这种场景，是不是应用更广泛一点？比如用户直接把项目里的resource 目录下整个传上去，初始化迁移就很方便。

感觉有很多种？

直接导入，比如期望同时导入到多个 ns 当中去

期望导入的时候，可以根据一些方式直接导入到我期望的多个group

单个 group 下的导入导出

若要支持各种灵活的导入场景，导入接口可能需要增加比较复杂的参数。目前只有一个zip文件参数，等于约定了按照zip文件目录结构去映射命名空间-组-配置文件的方式去导入。

lepdou · 2022-12-12T02:16:24Z

要不先搞简单一点，先支持跨namespace的导出导入

chuntaojun · 2022-12-12T02:25:41Z

要不先搞简单一点，先支持跨namespace的导出导入

导入的格式呢？是这种让用户按照规则组织一下，还是怎么处理？

lepdou · 2022-12-15T11:13:50Z

要不先搞简单一点，先支持跨namespace的导出导入

导入的格式呢？是这种让用户按照规则组织一下，还是怎么处理？

现在应该只能信先导出，再导入。用户不需要自己处理规则把？

andrewshan · 2023-01-30T08:07:31Z

hi,alex. We have move all the proto into https://github.com/polarismesh/specification, pls make your change into this repo

apiserver/httpserver/config_console_access.go

common/utils/config_file.go

config/config_file.go

* fix issue (polarismesh#524)

config/config_file.go

chuntaojun · 2023-02-08T09:58:43Z

等 1.14.0 分支打出来之后，在进行合并改 PR

alexwanglei force-pushed the feat/config_file_export_import branch from b032c79 to f229bf3 Compare December 9, 2022 09:42

lepdou reviewed Dec 9, 2022

View reviewed changes

config/common.go Outdated Show resolved Hide resolved

chuntaojun reviewed Dec 9, 2022

View reviewed changes

conf/polaris-server.yaml Outdated Show resolved Hide resolved

alexwanglei force-pushed the feat/config_file_export_import branch from f229bf3 to 19bdc9c Compare December 9, 2022 13:59

chuntaojun added this to the v1.15.0 milestone Dec 28, 2022

alexwanglei force-pushed the feat/config_file_export_import branch 3 times, most recently from 018e1bb to bc9c481 Compare January 3, 2023 09:51

alexwanglei force-pushed the feat/config_file_export_import branch 4 times, most recently from 076f9c6 to 90d2c3e Compare February 7, 2023 02:29

alexwanglei marked this pull request as ready for review February 7, 2023 03:40

chuntaojun reviewed Feb 7, 2023

View reviewed changes

alexwanglei force-pushed the feat/config_file_export_import branch 2 times, most recently from 1321d47 to 934b785 Compare February 8, 2023 06:10

feat: 配置文件导入导出功能 (polarismesh#524)

086cbe3

* fix issue (polarismesh#524)

alexwanglei force-pushed the feat/config_file_export_import branch from 934b785 to 086cbe3 Compare February 8, 2023 08:09

chuntaojun requested changes Feb 8, 2023

View reviewed changes

config/config_file.go Show resolved Hide resolved

chuntaojun added enhancement New feature or request config Configuration Center labels Feb 8, 2023

chuntaojun approved these changes Feb 8, 2023

View reviewed changes

chuntaojun added the in processed processing label Feb 8, 2023

chuntaojun merged commit 733a19a into polarismesh:main Feb 15, 2023

chuntaojun pushed a commit to chuntaojun/polaris that referenced this pull request Feb 24, 2023

feat: 配置文件导入导出功能 (polarismesh#524) (polarismesh#858)

5b63b8b

chuntaojun pushed a commit to chuntaojun/polaris that referenced this pull request Feb 28, 2023

feat: 配置文件导入导出功能 (polarismesh#524) (polarismesh#858)

65b39d5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: 配置文件导入导出功能 (#524) #858

feat: 配置文件导入导出功能 (#524) #858

alexwanglei commented Dec 9, 2022

polaris-test-bot commented Dec 9, 2022 •

edited

lepdou commented Dec 9, 2022

lepdou commented Dec 9, 2022

chuntaojun commented Dec 9, 2022

chuntaojun commented Dec 9, 2022

chuntaojun commented Dec 9, 2022 •

edited

codecov bot commented Dec 9, 2022 •

edited

alexwanglei commented Dec 9, 2022

alexwanglei commented Dec 9, 2022

alexwanglei commented Dec 9, 2022

lepdou commented Dec 12, 2022

chuntaojun commented Dec 12, 2022

lepdou commented Dec 15, 2022

andrewshan commented Jan 30, 2023

chuntaojun commented Feb 8, 2023

feat: 配置文件导入导出功能 (#524) #858

feat: 配置文件导入导出功能 (#524) #858

Conversation

alexwanglei commented Dec 9, 2022

polaris-test-bot commented Dec 9, 2022 • edited

lepdou commented Dec 9, 2022

lepdou commented Dec 9, 2022

chuntaojun commented Dec 9, 2022

chuntaojun commented Dec 9, 2022

chuntaojun commented Dec 9, 2022 • edited

codecov bot commented Dec 9, 2022 • edited

Codecov Report

alexwanglei commented Dec 9, 2022

alexwanglei commented Dec 9, 2022

alexwanglei commented Dec 9, 2022

lepdou commented Dec 12, 2022

chuntaojun commented Dec 12, 2022

lepdou commented Dec 15, 2022

andrewshan commented Jan 30, 2023

chuntaojun commented Feb 8, 2023

polaris-test-bot commented Dec 9, 2022 •

edited

chuntaojun commented Dec 9, 2022 •

edited

codecov bot commented Dec 9, 2022 •

edited