External config override for rest-api looks discarding optional second level of control for endpoint groups

[ghost]

• Members have below config initially

<rest-api enabled="true">
            <endpoint-group name="CLUSTER_READ" enabled="false"/>
            <endpoint-group name="CLUSTER_WRITE" enabled="false"/>
            <endpoint-group name="HEALTH_CHECK" enabled="false"/>
            <endpoint-group name="HOT_RESTART" enabled="false"/>
            <endpoint-group name="WAN" enabled="false"/>
            <endpoint-group name="DATA" enabled="false"/>
        </rest-api>

• When we start with above we got below logs from members when checking hazelcast/health

2020-10-04T07:16:55.864856501Z java.lang.IllegalStateException: REST API is not enabled.
2020-10-04T07:16:55.864859526Z 	at com.hazelcast.internal.server.tcp.UnifiedProtocolDecoder.onRead(UnifiedProtocolDecoder.java:106)
2020-10-04T07:16:55.864862391Z 	at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:137)
2020-10-04T07:16:55.864865236Z 	at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383)
2020-10-04T07:16:55.864868003Z 	at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368)
2020-10-04T07:16:55.864870752Z 	at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294)
2020-10-04T07:16:55.864873460Z 	at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249)
2020-10-04T07:16:55.864876270Z 	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
2020-10-04T07:16:55.864878960Z 

• When config overridden for below with -Dhz.network.rest-api.enabled=true

<rest-api enabled="false">
            <endpoint-group name="CLUSTER_READ" enabled="false"/>
            <endpoint-group name="CLUSTER_WRITE" enabled="false"/>
            <endpoint-group name="HEALTH_CHECK" enabled="false"/>
            <endpoint-group name="HOT_RESTART" enabled="false"/>
            <endpoint-group name="WAN" enabled="false"/>
            <endpoint-group name="DATA" enabled="false"/>
        </rest-api>

2020-10-04T07:24:03.903891161Z Oct 04, 2020 7:24:03 AM com.hazelcast.internal.config.override.ExternalConfigurationOverride
2020-10-04T07:24:03.903917626Z INFO: Detected external configuration overrides in system properties: [hazelcast.network.rest-api.enabled=true]

• And members reports hazelcast/health, yet we have second level disabled as endpoint-group name="HEALTH_CHECK" enabled="false"

• kindly check or guide.

[pivovarit]

I can confirm that I managed to reproduce the issue with the below setup:

<?xml version="1.0" encoding="UTF-8"?>
<hazelcast xmlns="http://www.hazelcast.com/schema/config"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://www.hazelcast.com/schema/config
           http://www.hazelcast.com/schema/config/hazelcast-config-4.0.xsd">
    <network>
        <rest-api enabled="false">
            <endpoint-group name="CLUSTER_READ" enabled="false"/>
            <endpoint-group name="CLUSTER_WRITE" enabled="false"/>
            <endpoint-group name="HEALTH_CHECK" enabled="false"/>
            <endpoint-group name="HOT_RESTART" enabled="false"/>
            <endpoint-group name="WAN" enabled="false"/>
            <endpoint-group name="DATA" enabled="false"/>
        </rest-api>
    </network>
</hazelcast>

// System.setProperty("hz.network.rest-api.enabled", "true");
Config load = Config.load();
load.getNetworkConfig().getRestApiConfig().setEnabled(true);
System.out.println(load.getNetworkConfig().getRestApiConfig().getEnabledGroups());

When the first line is commented out, the response is [], and when it's uncommented, the response it [CLUSTER_READ, HEALTH_CHECK]. It looks like it's reverting to default settings (?)

However, I have no clue what might be causing it. I will let you know as soon as I discover something.

[pivovarit]

Ok, I think I got it. The issue is with our configuration parser that throws away the existing config and always replaces it with a new one. On it.