Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ISSUE #2986] Support for multiple ACL files in a fixed directory #3761

Merged
merged 13 commits into from
Feb 19, 2022
Merged

[ISSUE #2986] Support for multiple ACL files in a fixed directory #3761

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
b8640b9
acl temp
sunxi92 Jun 23, 2021
645d35a
acl
sunxi92 Jun 26, 2021
b56b237
fix test case
sunxi92 Dec 17, 2021
2e848e9
fix code style issues
sunxi92 Jan 14, 2022
30f7363
add considerations on compatibility to the original one ACL config fi…
sunxi92 Jan 23, 2022
e6e3548
fix test case testWatch
sunxi92 Jan 24, 2022
6248b47
1.fix some issues
sunxi92 Jan 28, 2022
d4629ac
Add warn log when the accesskey is repeated in multiple ACL files.
sunxi92 Feb 12, 2022
2d877e6
1.Change the folder of acl configuration to conf/acl
sunxi92 Feb 15, 2022
b323e8e
Add a parameter in AclFileWatchService constructor.
sunxi92 Feb 15, 2022
a5691ab
Add logic to determine if path exists in the getAllAclFiles(String pa…
sunxi92 Feb 15, 2022
05d1207
Fix the serialization problem of allAclFileVersion field in clusterAc…
sunxi92 Feb 17, 2022
4dae014
1.Fix the serialization problem of allAclFileVersion field in cluster…
sunxi92 Feb 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
13 changes: 13 additions & 0 deletions acl/src/main/java/org/apache/rocketmq/acl/AccessValidator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,10 @@
package org.apache.rocketmq.acl;

import java.util.List;
import java.util.Map;

import org.apache.rocketmq.common.AclConfig;
import org.apache.rocketmq.common.DataVersion;
import org.apache.rocketmq.common.PlainAccessConfig;
import org.apache.rocketmq.remoting.protocol.RemotingCommand;

Expand Down Expand Up @@ -60,17 +63,27 @@ public interface AccessValidator {
*
* @return
*/
@Deprecated
String getAclConfigVersion();
Comment on lines +66 to 67
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pls add more description about:

  • the reason to deprecate String getAclConfigVersion() and replace it with Map<String, DataVersion> getAllAclConfigVersion()
  • What is the object or scope that version describe? In previous implementation, version is used to mark the whole config. In this multi-filed feature, each config file has its own version. And if ACL config are stored in DB, each record may have its own version. We should define version precisely.
  • In Map<String, DataVersion> getAllAclConfigVersion(), what is the key of Map? Users may curious of it. If this is the full path of config files, it brings implementation to interface.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.


/**
* Update globalWhiteRemoteAddresses in acl yaml config file
*
* @return
*/
boolean updateGlobalWhiteAddrsConfig(List<String> globalWhiteAddrsList);

/**
* get broker cluster acl config information
*
* @return
*/
AclConfig getAllAclConfig();

/**
* get all access resource config version information
*
* @return
*/
Map<String, DataVersion> getAllAclConfigVersion();
}
19 changes: 17 additions & 2 deletions acl/src/main/java/org/apache/rocketmq/acl/plain/PlainAccessValidator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import org.apache.rocketmq.acl.common.Permission;
import org.apache.rocketmq.acl.common.SessionCredentials;
import org.apache.rocketmq.common.AclConfig;
import org.apache.rocketmq.common.DataVersion;
import org.apache.rocketmq.common.MixAll;
import org.apache.rocketmq.common.PlainAccessConfig;
import org.apache.rocketmq.common.protocol.RequestCode;
Expand Down Expand Up @@ -127,7 +128,7 @@ public AccessResource parse(RemotingCommand request, String remoteAddr) {
SortedMap<String, String> map = new TreeMap<String, String>();
for (Map.Entry<String, String> entry : request.getExtFields().entrySet()) {
if (!SessionCredentials.SIGNATURE.equals(entry.getKey())
&& !MixAll.UNIQUE_MSG_QUERY_FLAG.equals(entry.getKey())) {
&& !MixAll.UNIQUE_MSG_QUERY_FLAG.equals(entry.getKey())) {
map.put(entry.getKey(), entry.getValue());
}
}
Expand All @@ -150,7 +151,7 @@ public boolean deleteAccessConfig(String accesskey) {
return aclPlugEngine.deleteAccessConfig(accesskey);
}

@Override public String getAclConfigVersion() {
@Override public String getAclConfigVersion() {
return aclPlugEngine.getAclConfigDataVersion();
}

Expand All @@ -161,4 +162,18 @@ public boolean deleteAccessConfig(String accesskey) {
@Override public AclConfig getAllAclConfig() {
return aclPlugEngine.getAllAclConfig();
}

public Map<String, Object> createAclAccessConfigMap(Map<String, Object> existedAccountMap,
PlainAccessConfig plainAccessConfig) {
return aclPlugEngine.createAclAccessConfigMap(existedAccountMap, plainAccessConfig);
}

public Map<String, Object> updateAclConfigFileVersion(Map<String, Object> updateAclConfigMap) {
return aclPlugEngine.updateAclConfigFileVersion(updateAclConfigMap);
}

@Override
public Map<String, DataVersion> getAllAclConfigVersion() {
return aclPlugEngine.getDataVersionMap();
}
}
427 changes: 330 additions & 97 deletions acl/src/main/java/org/apache/rocketmq/acl/plain/PlainPermissionManager.java
View file
Open in desktop

Large diffs are not rendered by default.

13 changes: 4 additions & 9 deletions acl/src/test/java/org/apache/rocketmq/acl/common/AclUtilsTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,23 +294,18 @@ public void getYamlDataIgnoreFileNotFoundExceptionTest() {
Assert.assertTrue(yamlDataObject == null);
}

@Test(expected = Exception.class)
public void getYamlDataExceptionTest() {

AclUtils.getYamlDataObject("src/test/resources/conf/plain_acl_format_error.yml", Map.class);
}

@Test
public void getAclRPCHookTest() {

RPCHook errorContRPCHook = AclUtils.getAclRPCHook("src/test/resources/conf/plain_acl_format_error.yml");
Assert.assertNull(errorContRPCHook);
//RPCHook errorContRPCHook = AclUtils.getAclRPCHook("src/test/resources/conf/plain_acl_format_error.yml");
//Assert.assertNull(errorContRPCHook);

RPCHook noFileRPCHook = AclUtils.getAclRPCHook("src/test/resources/plain_acl_format_error1.yml");
Assert.assertNull(noFileRPCHook);

RPCHook emptyContRPCHook = AclUtils.getAclRPCHook("src/test/resources/conf/plain_acl_null.yml");
Assert.assertNull(emptyContRPCHook);
//RPCHook emptyContRPCHook = AclUtils.getAclRPCHook("src/test/resources/conf/plain_acl_null.yml");
//Assert.assertNull(emptyContRPCHook);

RPCHook incompleteContRPCHook = AclUtils.getAclRPCHook("src/test/resources/conf/plain_acl_incomplete.yml");
Assert.assertNull(incompleteContRPCHook);
Expand Down
494 changes: 323 additions & 171 deletions acl/src/test/java/org/apache/rocketmq/acl/plain/PlainAccessValidatorTest.java
View file
Open in desktop

Large diffs are not rendered by default.

59 changes: 25 additions & 34 deletions acl/src/test/java/org/apache/rocketmq/acl/plain/PlainPermissionManagerTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,9 @@ public void init() throws NoSuchFieldException, SecurityException, IOException {
ANYPlainAccessResource = clonePlainAccessResource(Permission.ANY);
DENYPlainAccessResource = clonePlainAccessResource(Permission.DENY);

System.setProperty("rocketmq.home.dir", "src/test/resources");
System.setProperty("rocketmq.acl.plain.file", "/conf/plain_acl.yml");
File file = new File("src/test/resources");
System.setProperty("rocketmq.home.dir", file.getAbsolutePath());

plainPermissionManager = new PlainPermissionManager();

}
Expand Down Expand Up @@ -117,7 +117,7 @@ public void buildPlainAccessResourceTest() {
Assert.assertEquals(resourcePermMap.size(), 3);

Assert.assertEquals(resourcePermMap.get(PlainAccessResource.getRetryTopic("groupA")).byteValue(), Permission.DENY);
Assert.assertEquals(resourcePermMap.get(PlainAccessResource.getRetryTopic("groupB")).byteValue(), Permission.PUB|Permission.SUB);
Assert.assertEquals(resourcePermMap.get(PlainAccessResource.getRetryTopic("groupB")).byteValue(), Permission.PUB | Permission.SUB);
Assert.assertEquals(resourcePermMap.get(PlainAccessResource.getRetryTopic("groupC")).byteValue(), Permission.PUB);

List<String> topics = new ArrayList<String>();
Expand All @@ -130,7 +130,7 @@ public void buildPlainAccessResourceTest() {
Assert.assertEquals(resourcePermMap.size(), 6);

Assert.assertEquals(resourcePermMap.get("topicA").byteValue(), Permission.DENY);
Assert.assertEquals(resourcePermMap.get("topicB").byteValue(), Permission.PUB|Permission.SUB);
Assert.assertEquals(resourcePermMap.get("topicB").byteValue(), Permission.PUB | Permission.SUB);
Assert.assertEquals(resourcePermMap.get("topicC").byteValue(), Permission.PUB);
}

Expand All @@ -157,13 +157,15 @@ public void checkPerm() {
plainPermissionManager.checkPerm(plainAccessResource, ANYPlainAccessResource);

}

@Test(expected = AclException.class)
public void checkErrorPermDefaultValueNotMatch() {

plainAccessResource = new PlainAccessResource();
plainAccessResource.addResourceAndPerm("topicF", Permission.PUB);
plainPermissionManager.checkPerm(plainAccessResource, SUBPlainAccessResource);
}

@Test(expected = AclException.class)
public void accountNullTest() {
plainAccessConfig.setAccessKey(null);
Expand All @@ -184,25 +186,20 @@ public void passWordtNullTest() {

@Test(expected = AclException.class)
public void passWordThanTest() {
plainAccessConfig.setAccessKey("123");
plainAccessConfig.setSecretKey("123");
plainPermissionManager.buildPlainAccessResource(plainAccessConfig);
}

@Test(expected = AclException.class)
public void testPlainAclPlugEngineInit() {
System.setProperty("rocketmq.home.dir", "");
new PlainPermissionManager().load();
}

@SuppressWarnings("unchecked")
@Test
public void cleanAuthenticationInfoTest() throws IllegalAccessException {
// PlainPermissionManager.addPlainAccessResource(plainAccessResource);
Map<String, List<PlainAccessResource>> plainAccessResourceMap = (Map<String, List<PlainAccessResource>>) FieldUtils.readDeclaredField(plainPermissionManager, "plainAccessResourceMap", true);
Map<String, Map<String, PlainAccessResource>> plainAccessResourceMap = (Map<String, Map<String, PlainAccessResource>>) FieldUtils.readDeclaredField(plainPermissionManager, "aclPlainAccessResourceMap", true);
Assert.assertFalse(plainAccessResourceMap.isEmpty());

plainPermissionManager.clearPermissionInfo();
plainAccessResourceMap = (Map<String, List<PlainAccessResource>>) FieldUtils.readDeclaredField(plainPermissionManager, "plainAccessResourceMap", true);
plainAccessResourceMap = (Map<String, Map<String, PlainAccessResource>>) FieldUtils.readDeclaredField(plainPermissionManager, "aclPlainAccessResourceMap", true);
Assert.assertTrue(plainAccessResourceMap.isEmpty());
// RemoveDataVersionFromYamlFile("src/test/resources/conf/plain_acl.yml");
}
Expand All @@ -213,34 +210,36 @@ public void isWatchStartTest() {
PlainPermissionManager plainPermissionManager = new PlainPermissionManager();
Assert.assertTrue(plainPermissionManager.isWatchStart());
// RemoveDataVersionFromYamlFile("src/test/resources/conf/plain_acl.yml");

}


@Test
public void testWatch() throws IOException, IllegalAccessException ,InterruptedException{
System.setProperty("rocketmq.home.dir", "src/test/resources");
System.setProperty("rocketmq.acl.plain.file", "/conf/plain_acl-test.yml");
String fileName =System.getProperty("rocketmq.home.dir", "src/test/resources")+System.getProperty("rocketmq.acl.plain.file", "/conf/plain_acl.yml");
public void testWatch() throws IOException, IllegalAccessException, InterruptedException {
File file = new File("src/test/resources");
System.setProperty("rocketmq.home.dir", file.getAbsolutePath());

String fileName = System.getProperty("rocketmq.home.dir") + File.separator + "/conf/acl/plain_acl_test.yml";
File transport = new File(fileName);
transport.delete();
transport.createNewFile();
FileWriter writer = new FileWriter(transport);
writer.write("accounts:\r\n");
writer.write("- accessKey: watchrocketmq\r\n");
writer.write("- accessKey: watchrocketmqx\r\n");
writer.write(" secretKey: 12345678\r\n");
writer.write(" whiteRemoteAddress: 127.0.0.1\r\n");
writer.write(" admin: true\r\n");
writer.flush();
writer.close();

Thread.sleep(1000);

PlainPermissionManager plainPermissionManager = new PlainPermissionManager();
Assert.assertTrue(plainPermissionManager.isWatchStart());

Map<String, String> accessKeyTable = (Map<String, String>) FieldUtils.readDeclaredField(plainPermissionManager, "accessKeyTable", true);
String aclFileName = accessKeyTable.get("watchrocketmqx");
{
Map<String, PlainAccessResource> plainAccessResourceMap = (Map<String, PlainAccessResource>) FieldUtils.readDeclaredField(plainPermissionManager, "plainAccessResourceMap", true);
PlainAccessResource accessResource = plainAccessResourceMap.get("watchrocketmq");
Map<String, Map<String, PlainAccessResource>> plainAccessResourceMap = (Map<String, Map<String, PlainAccessResource>>) FieldUtils.readDeclaredField(plainPermissionManager, "aclPlainAccessResourceMap", true);
PlainAccessResource accessResource = plainAccessResourceMap.get(aclFileName).get("watchrocketmqx");
Assert.assertNotNull(accessResource);
Assert.assertEquals(accessResource.getSecretKey(), "12345678");
Assert.assertTrue(accessResource.isAdmin());
Expand All @@ -251,30 +250,22 @@ public void testWatch() throws IOException, IllegalAccessException ,InterruptedE
List<Map<String, Object>> accounts = (List<Map<String, Object>>) updatedMap.get("accounts");
accounts.get(0).remove("accessKey");
accounts.get(0).remove("secretKey");
accounts.get(0).put("accessKey", "watchrocketmq1");
accounts.get(0).put("accessKey", "watchrocketmq1y");
accounts.get(0).put("secretKey", "88888888");
accounts.get(0).put("admin", "false");
// Update file and flush to yaml file
AclUtils.writeDataObject(fileName, updatedMap);

Thread.sleep(1000);
Thread.sleep(10000);
{
Map<String, PlainAccessResource> plainAccessResourceMap = (Map<String, PlainAccessResource>) FieldUtils.readDeclaredField(plainPermissionManager, "plainAccessResourceMap", true);
PlainAccessResource accessResource = plainAccessResourceMap.get("watchrocketmq1");
Map<String, Map<String, PlainAccessResource>> plainAccessResourceMap = (Map<String, Map<String, PlainAccessResource>>) FieldUtils.readDeclaredField(plainPermissionManager, "aclPlainAccessResourceMap", true);
PlainAccessResource accessResource = plainAccessResourceMap.get(aclFileName).get("watchrocketmq1y");
Assert.assertNotNull(accessResource);
Assert.assertEquals(accessResource.getSecretKey(), "88888888");
Assert.assertFalse(accessResource.isAdmin());

}
transport.delete();
System.setProperty("rocketmq.home.dir", "src/test/resources");
System.setProperty("rocketmq.acl.plain.file", "/conf/plain_acl.yml");
}

@Test(expected = AclException.class)
public void initializeTest() {
System.setProperty("rocketmq.acl.plain.file", "/conf/plain_acl_null.yml");
new PlainPermissionManager();

}
}
32 changes: 25 additions & 7 deletions ...resources/conf/plain_acl_format_error.yml → ...src/test/resources/conf/acl/plain_acl.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,30 @@

## suggested format

date 2015-02-01
accounts:
- name: Jai
globalWhiteRemoteAddresses:
- 10.10.103.*
- 192.168.0.*

accounts:
- accessKey: RocketMQ
secretKey: 12345678
whiteRemoteAddress: 192.168.0.*
admin: false
- accessKey: RocketMQ
secretKey: 12345678
whiteRemoteAddress: 192.168.0.*
admin: false
defaultTopicPerm: DENY
defaultGroupPerm: SUB
topicPerms:
- topicA=DENY
- topicB=PUB|SUB
- topicC=SUB
groupPerms:
# the group should convert to retry topic
- groupA=DENY
- groupB=SUB
- groupC=SUB

- accessKey: rocketmq2
secretKey: 12345678
whiteRemoteAddress: 192.168.1.*
# if it is admin, it could access all resources
admin: true

18 changes: 0 additions & 18 deletions acl/src/test/resources/conf/plain_acl_null.yml
View file
Open in desktop

This file was deleted.

1 change: 1 addition & 0 deletions broker/src/main/java/org/apache/rocketmq/broker/processor/AdminBrokerProcessor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -429,6 +429,7 @@ private RemotingCommand getBrokerAclConfigVersion(ChannelHandlerContext ctx, Rem
try {
AccessValidator accessValidator = this.brokerController.getAccessValidatorMap().get(PlainAccessValidator.class);

responseHeader.setAllAclFileVersion(JSON.toJSONString(accessValidator.getAllAclConfigVersion()));
responseHeader.setVersion(accessValidator.getAclConfigVersion());
responseHeader.setBrokerAddr(this.brokerController.getBrokerAddr());
responseHeader.setBrokerName(this.brokerController.getBrokerConfig().getBrokerName());
Expand Down