Update sumer-ospp 10376, to keep path with the newest commits in deve…

…lop branch (#10942) * [ISSUE #10734] Implement http request param check filter and http param extractors (#10758) * For #10734,Implement grpc server interceptor and grpc param extractors * For #10734,add unit test for grpc server interceptor and grpc param extractors * For #10734,alter the test case * For #10734,delete the ConnectionSetupRequestParamExtractor * For #10734,add the naming http request param check filter and implement the naming http request param extractors * For #10734,add unit test for naming http request param extractors * For #10734,Implement grpc server interceptor and grpc param extractors * For #10734,add unit test for grpc server interceptor and grpc param extractors * For #10734,delete the ConnectionSetupRequestParamExtractor * For #10734,add the naming http request param check filter and implement the naming http request param extractors * For #10734,add unit test for naming http request param extractors * For #10734,add the config http request param check filter and implement the config http request param extractors and unit test * For #10734,add the console http request param check filter and implement the console http request param extractors and unit test * For #10734,fix code style * For #10734,alter the logic of exception handle in filter * For #10734,fix code style * dump change check task submit (#10755) * dump change check task submit * delete config nid convert error fix * fix test case * checkstyle * Refactor grpc tls (#10759) * Move Tls negotiator to GrpcSdkServer. * use protocol negotiator builder replace directly create. * use SPI load negotiator and set tls as default negotiator. * Remove tlsconfig in BaseRpcServer. * Add some ut. * For checkstyle. * fix word spelling in `AuthenticationManagerDelegator` (#10777) * fix(#10427): When the execution of handleServerRequest() encounters an exception, record the log and throw an exception, then quickly response to the server errResponse (#10770) * fix(#10585): selectInstances and selectOneHealthyInstance methods, if the parameter subscribe is true. Subscription is required when clientProxy.isSubscribe() is false. (#10805) * disable check port input when use registered port (#10799) * Add the handle to overload connection (#10783) * add the handle to overload connection * fast return * [ISSUE #10662]Prometheus-sd add namespace and service api (#10663) * add apis * add tests * fix checkstyle * param namespace to namespaceId * namespace to namespaceId * fix test case * fix test case * Service instance should display related color when healthy or unhealthy (#10811) * fix a couple of invaild props (#10810) * feat(#10539): When the operation configuration fails, log. (#10804) * [ISSUE #10744]feat:Add HealthControllerV2 and HealthControllerV2Test (#10786) * [ISSUE #10744]feat:Add HealthControllerV2 and HealthControllerV2Test * fix:V2 api return Result * [ISSUE #10734] Refactor the ParamChecker and ParamExtractor (#10775) * For #10734,refactor the paramextractor and ParamChecker * For #10734,alter the rules of ParamCheck * For #10734,alter the rules of ParamCheck * For #10734,fix bug * For #10734,fix bug and alter the ParamCheckRules.java * For #10734,fix code style * For #10734,fix the param check rules * For #10734,implement the server param check config * For #10734,optimize the logic * For #10734,optimize the logic * For #10734,optimize the logic * Refactor Prometheus Module (#10827) * Refactor Prometheus Module * Complete Test Case * format * format * For #10734,fix the param check rule (#10826) * [ISSUE #10792]When nacos client use endpoint, the registration center should support configuring context-path and cluster-name like the configuration center (#10793) * Reactor code in datasource-plugin (#10791) * Reactor code in datasource-plugin * Fix Abstract Mapper Test Case * Add Empty Check * Fix Checkstyle * fix checkstyle * fix check style * fix check style * Fix CheckStyle * Fix SQL Blank * bugfix for PersistentClientOperationServiceImpl log (#10825) * For #10734,fix the param check rule (#10858) * fix(#10831): When using the deregisterInstance method to remove one of multiple instances registered by batchRegisterInstance, all instances registered by batchRegisterInstance will be removed. (#10836) * UnsupportedFeatureError (#10860) * fix(distro): fix issue#10880. (#10881) * feat(#5608 && #10223): When the custom instance id is empty, the id will be automatically generated. (#10812) * fix: test-code branch (#10904) * add nacos ci * delete client version of nacos ci * fix: test-code branch * console-ui 新增 toml 语言支持，修复之前 properties 主题未生效的问题 (#10896) * console-ui 新增 toml 语言支持，修复之前 properties 主题未生效的问题 * console-ui 新增 toml 语言支持，修复之前 properties 主题未生效的问题 * 老规矩，要编译一波 * feat(10891): Provide a configuration item for the maximum number of push retries, instead of directly hardcoding it to 50 times in the code. (#10895) * [ISSUE #10824] Remove udp port param for v1-client (#10914) * Remove UDP Param * Fix gRPC client * fix test case * fix test case * fix test case * fix test case * Fix login failed when close auth. --------- Co-authored-by: Sunrisea <49605583+Sunrisea@users.noreply.github.com> Co-authored-by: nov.lzf <liuzunfei@gmail.com> Co-authored-by: 杨翊 SionYang <xiweng.yy@alibaba-inc.com> Co-authored-by: ZhangShenao <15201440436@163.com> Co-authored-by: blake.qiu <46370663+Bo-Qiu@users.noreply.github.com> Co-authored-by: Joey777210 <53630996+Joey777210@users.noreply.github.com> Co-authored-by: chenyiqin <83362909+Daydreamer-ia@users.noreply.github.com> Co-authored-by: maoling <11016631+maoling@users.noreply.github.com> Co-authored-by: DiligenceLai <96920907+DiligenceLai@users.noreply.github.com> Co-authored-by: huhongjie2014 <139250842+huhongjie2014@users.noreply.github.com> Co-authored-by: lu-xiaoshuang <121755080+lu-xiaoshuang@users.noreply.github.com> Co-authored-by: zt9788 <zt9788@users.noreply.github.com> Co-authored-by: 阿魁 <670569467@qq.com> Co-authored-by: wuyfee <30968107+Wuyunfan-BUPT@users.noreply.github.com> Co-authored-by: Darren Luo <jn296@qq.com> Co-authored-by: xxc <xchaos8@126.com>
alibaba · Aug 10, 2023 · 5714bed · 5714bed
1 parent f62cf16
commit 5714bed
Show file tree

Hide file tree

Showing 173 changed files with 3,817 additions and 1,309 deletions.
diff --git a/.github/workflows/pr-e2e-test.yml b/.github/workflows/pr-e2e-test.yml
@@ -171,7 +171,7 @@ jobs:
           ask-config: "${{ secrets.ASK_CONFIG_VIRGINA }}"
           test-version: "${{ matrix.version }}"
           test-code-git: "https://github.com/nacos-group/nacos-e2e.git"
-          test-code-branch: "master"
+          test-code-branch: "main"
           test-code-path: "java/nacos-2X"
           test-cmd: 'mvn clean test -B'
           job-id: ${{ strategy.job-index }}

diff --git a/api/src/main/java/com/alibaba/nacos/api/config/ConfigType.java b/api/src/main/java/com/alibaba/nacos/api/config/ConfigType.java
@@ -58,6 +58,11 @@ public enum ConfigType {
      */
     YAML("yaml"),
 
+    /**
+     * config type is "toml".
+     */
+    TOML("toml"),
+
     /**
      * not a real type.
      */

diff --git a/client/src/main/java/com/alibaba/nacos/client/naming/NacosNamingService.java b/client/src/main/java/com/alibaba/nacos/client/naming/NacosNamingService.java
@@ -54,7 +54,7 @@
 @SuppressWarnings("PMD.ServiceOrDaoClassShouldEndWithImplRule")
 public class NacosNamingService implements NamingService {
 
-    private static final String DEFAULT_NAMING_LOG_FILE_PATH =  "naming.log";
+    private static final String DEFAULT_NAMING_LOG_FILE_PATH = "naming.log";
 
     private static final String UP = "UP";
 
@@ -93,13 +93,14 @@ private void init(Properties properties) throws NacosException {
         InitUtils.initSerialization();
         InitUtils.initWebRootContext(nacosClientProperties);
         initLogName(nacosClientProperties);
-
+        
         this.notifierEventScope = UUID.randomUUID().toString();
         this.changeNotifier = new InstancesChangeNotifier(this.notifierEventScope);
         NotifyCenter.registerToPublisher(InstancesChangeEvent.class, 16384);
         NotifyCenter.registerSubscriber(changeNotifier);
         this.serviceInfoHolder = new ServiceInfoHolder(namespace, this.notifierEventScope, nacosClientProperties);
-        this.clientProxy = new NamingClientProxyDelegate(this.namespace, serviceInfoHolder, nacosClientProperties, changeNotifier);
+        this.clientProxy = new NamingClientProxyDelegate(this.namespace, serviceInfoHolder, nacosClientProperties,
+                changeNotifier);
     }
 
     private void initLogName(NacosClientProperties properties) {
@@ -241,7 +242,7 @@ public List<Instance> getAllInstances(String serviceName, String groupName, List
                 serviceInfo = clientProxy.subscribe(serviceName, groupName, clusterString);
             }
         } else {
-            serviceInfo = clientProxy.queryInstancesOfService(serviceName, groupName, clusterString, 0, false);
+            serviceInfo = clientProxy.queryInstancesOfService(serviceName, groupName, clusterString, false);
         }
         List<Instance> list;
         if (serviceInfo == null || CollectionUtils.isEmpty(list = serviceInfo.getHosts())) {
@@ -298,11 +299,11 @@ public List<Instance> selectInstances(String serviceName, String groupName, List
         String clusterString = StringUtils.join(clusters, ",");
         if (subscribe) {
             serviceInfo = serviceInfoHolder.getServiceInfo(serviceName, groupName, clusterString);
-            if (null == serviceInfo) {
+            if (null == serviceInfo || !clientProxy.isSubscribed(serviceName, groupName, clusterString)) {
                 serviceInfo = clientProxy.subscribe(serviceName, groupName, clusterString);
             }
         } else {
-            serviceInfo = clientProxy.queryInstancesOfService(serviceName, groupName, clusterString, 0, false);
+            serviceInfo = clientProxy.queryInstancesOfService(serviceName, groupName, clusterString, false);
         }
         return selectInstances(serviceInfo, healthy);
     }
@@ -368,13 +369,12 @@ public Instance selectOneHealthyInstance(String serviceName, String groupName, L
         String clusterString = StringUtils.join(clusters, ",");
         if (subscribe) {
             ServiceInfo serviceInfo = serviceInfoHolder.getServiceInfo(serviceName, groupName, clusterString);
-            if (null == serviceInfo) {
+            if (null == serviceInfo || !clientProxy.isSubscribed(serviceName, groupName, clusterString)) {
                 serviceInfo = clientProxy.subscribe(serviceName, groupName, clusterString);
             }
             return Balancer.RandomByWeight.selectHost(serviceInfo);
         } else {
-            ServiceInfo serviceInfo = clientProxy
-                    .queryInstancesOfService(serviceName, groupName, clusterString, 0, false);
+            ServiceInfo serviceInfo = clientProxy.queryInstancesOfService(serviceName, groupName, clusterString, false);
             return Balancer.RandomByWeight.selectHost(serviceInfo);
         }
     }
@@ -467,6 +467,6 @@ public void shutDown() throws NacosException {
         serviceInfoHolder.shutdown();
         clientProxy.shutdown();
         NotifyCenter.deregisterSubscriber(changeNotifier);
-
+        
     }
 }
diff --git a/client/src/main/java/com/alibaba/nacos/client/naming/core/ServerListManager.java b/client/src/main/java/com/alibaba/nacos/client/naming/core/ServerListManager.java
@@ -25,6 +25,8 @@
 import com.alibaba.nacos.client.naming.utils.CollectionUtils;
 import com.alibaba.nacos.client.naming.utils.InitUtils;
 import com.alibaba.nacos.client.naming.utils.NamingHttpUtil;
+import com.alibaba.nacos.client.utils.ContextPathUtil;
+import com.alibaba.nacos.client.utils.ParamUtil;
 import com.alibaba.nacos.common.executor.NameThreadFactory;
 import com.alibaba.nacos.common.http.HttpRestResult;
 import com.alibaba.nacos.common.http.client.NacosRestTemplate;
@@ -50,7 +52,6 @@
 import java.util.concurrent.atomic.AtomicInteger;
 
 import static com.alibaba.nacos.client.utils.LogUtils.NAMING_LOGGER;
-import static com.alibaba.nacos.common.constant.RequestUrlConstants.HTTP_PREFIX;
 
 /**
  * Server list manager.
@@ -74,6 +75,10 @@ public class ServerListManager implements ServerListFactory, Closeable {
     private ScheduledExecutorService refreshServerListExecutor;
 
     private String endpoint;
+
+    private String contentPath = ParamUtil.getDefaultContextPath();
+
+    private String serverListName = ParamUtil.getDefaultNodesPath();
 
     private String nacosDomain;
 
@@ -96,6 +101,16 @@ public ServerListManager(NacosClientProperties properties, String namespace) {
     private void initServerAddr(NacosClientProperties properties) {
         this.endpoint = InitUtils.initEndpoint(properties);
         if (StringUtils.isNotEmpty(endpoint)) {
+
+            String contentPathTmp = properties.getProperty(PropertyKeyConst.CONTEXT_PATH);
+            if (!StringUtils.isBlank(contentPathTmp)) {
+                this.contentPath = contentPathTmp;
+            }
+            String serverListNameTmp = properties.getProperty(PropertyKeyConst.CLUSTER_NAME);
+            if (!StringUtils.isBlank(serverListNameTmp)) {
+                this.serverListName = serverListNameTmp;
+            }
+
             this.serversFromEndpoint = getServerListFromEndpoint();
             refreshServerListExecutor = new ScheduledThreadPoolExecutor(1,
                     new NameThreadFactory("com.alibaba.nacos.client.naming.server.list.refresher"));
@@ -115,7 +130,10 @@ private void initServerAddr(NacosClientProperties properties) {
 
     private List<String> getServerListFromEndpoint() {
         try {
-            String urlString = HTTP_PREFIX + endpoint + "/nacos/serverlist";
+            StringBuilder addressServerUrlTem = new StringBuilder(
+                    String.format("http://%s%s/%s", this.endpoint,
+                            ContextPathUtil.normalizeContextPath(this.contentPath), this.serverListName));
+            String urlString = addressServerUrlTem.toString();
             Header header = NamingHttpUtil.builderHeader();
             Query query = StringUtils.isNotBlank(namespace)
                     ? Query.newInstance().addParam("namespace", namespace)

diff --git a/client/src/main/java/com/alibaba/nacos/client/naming/core/ServiceInfoUpdateService.java b/client/src/main/java/com/alibaba/nacos/client/naming/core/ServiceInfoUpdateService.java
@@ -77,7 +77,8 @@ private boolean isAsyncQueryForSubscribeService(NacosClientProperties properties
         if (properties == null || !properties.containsKey(PropertyKeyConst.NAMING_ASYNC_QUERY_SUBSCRIBE_SERVICE)) {
             return false;
         }
-        return ConvertUtils.toBoolean(properties.getProperty(PropertyKeyConst.NAMING_ASYNC_QUERY_SUBSCRIBE_SERVICE), false);
+        return ConvertUtils.toBoolean(properties.getProperty(PropertyKeyConst.NAMING_ASYNC_QUERY_SUBSCRIBE_SERVICE),
+                false);
     }
 
     private int initPollingThreadCount(NacosClientProperties properties) {
@@ -188,14 +189,14 @@ public void run() {
 
                 ServiceInfo serviceObj = serviceInfoHolder.getServiceInfoMap().get(serviceKey);
                 if (serviceObj == null) {
-                    serviceObj = namingClientProxy.queryInstancesOfService(serviceName, groupName, clusters, 0, false);
+                    serviceObj = namingClientProxy.queryInstancesOfService(serviceName, groupName, clusters, false);
                     serviceInfoHolder.processServiceInfo(serviceObj);
                     lastRefTime = serviceObj.getLastRefTime();
                     return;
                 }
 
                 if (serviceObj.getLastRefTime() <= lastRefTime) {
-                    serviceObj = namingClientProxy.queryInstancesOfService(serviceName, groupName, clusters, 0, false);
+                    serviceObj = namingClientProxy.queryInstancesOfService(serviceName, groupName, clusters, false);
                     serviceInfoHolder.processServiceInfo(serviceObj);
                 }
                 lastRefTime = serviceObj.getLastRefTime();

diff --git a/client/src/main/java/com/alibaba/nacos/client/naming/remote/NamingClientProxy.java b/client/src/main/java/com/alibaba/nacos/client/naming/remote/NamingClientProxy.java
@@ -91,13 +91,12 @@ public interface NamingClientProxy extends Closeable {
      * @param serviceName service name
      * @param groupName   group name
      * @param clusters    clusters
-     * @param udpPort     udp port
      * @param healthyOnly healthy only
      * @return service info
      * @throws NacosException nacos exception
      */
-    ServiceInfo queryInstancesOfService(String serviceName, String groupName, String clusters, int udpPort,
-            boolean healthyOnly) throws NacosException;
+    ServiceInfo queryInstancesOfService(String serviceName, String groupName, String clusters, boolean healthyOnly)
+            throws NacosException;
 
     /**
      * Query Service.

diff --git a/client/src/main/java/com/alibaba/nacos/client/naming/remote/NamingClientProxyDelegate.java b/client/src/main/java/com/alibaba/nacos/client/naming/remote/NamingClientProxyDelegate.java
@@ -65,8 +65,8 @@ public class NamingClientProxyDelegate implements NamingClientProxy {
 
     private ScheduledExecutorService executorService;
 
-    public NamingClientProxyDelegate(String namespace, ServiceInfoHolder serviceInfoHolder, NacosClientProperties properties,
-            InstancesChangeNotifier changeNotifier) throws NacosException {
+    public NamingClientProxyDelegate(String namespace, ServiceInfoHolder serviceInfoHolder,
+            NacosClientProperties properties, InstancesChangeNotifier changeNotifier) throws NacosException {
         this.serviceInfoUpdateService = new ServiceInfoUpdateService(properties, serviceInfoHolder, this,
                 changeNotifier);
         this.serverListManager = new ServerListManager(properties, namespace);
@@ -88,9 +88,8 @@ private void initSecurityProxy(NacosClientProperties properties) {
         });
         final Properties nacosClientPropertiesView = properties.asProperties();
         this.securityProxy.login(nacosClientPropertiesView);
-        this.executorService
-                .scheduleWithFixedDelay(() -> securityProxy.login(nacosClientPropertiesView), 0, SECURITY_INFO_REFRESH_INTERVAL_MILLS,
-                        TimeUnit.MILLISECONDS);
+        this.executorService.scheduleWithFixedDelay(() -> securityProxy.login(nacosClientPropertiesView), 0,
+                SECURITY_INFO_REFRESH_INTERVAL_MILLS, TimeUnit.MILLISECONDS);
     }
 
     @Override
@@ -131,9 +130,9 @@ public void updateInstance(String serviceName, String groupName, Instance instan
     }
 
     @Override
-    public ServiceInfo queryInstancesOfService(String serviceName, String groupName, String clusters, int udpPort,
+    public ServiceInfo queryInstancesOfService(String serviceName, String groupName, String clusters,
             boolean healthyOnly) throws NacosException {
-        return grpcClientProxy.queryInstancesOfService(serviceName, groupName, clusters, udpPort, healthyOnly);
+        return grpcClientProxy.queryInstancesOfService(serviceName, groupName, clusters, healthyOnly);
     }
 
     @Override
@@ -178,8 +177,8 @@ public ServiceInfo subscribe(String serviceName, String groupName, String cluste
 
     @Override
     public void unsubscribe(String serviceName, String groupName, String clusters) throws NacosException {
-        NAMING_LOGGER
-                .debug("[UNSUBSCRIBE-SERVICE] service:{}, group:{}, cluster:{} ", serviceName, groupName, clusters);
+        NAMING_LOGGER.debug("[UNSUBSCRIBE-SERVICE] service:{}, group:{}, cluster:{} ", serviceName, groupName,
+                clusters);
         serviceInfoUpdateService.stopUpdateIfContain(serviceName, groupName, clusters);
         grpcClientProxy.unsubscribe(serviceName, groupName, clusters);
     }

diff --git a/client/src/main/java/com/alibaba/nacos/client/naming/remote/gprc/NamingGrpcClientProxy.java b/client/src/main/java/com/alibaba/nacos/client/naming/remote/gprc/NamingGrpcClientProxy.java
@@ -54,8 +54,8 @@
 import com.alibaba.nacos.common.remote.ConnectionType;
 import com.alibaba.nacos.common.remote.client.RpcClient;
 import com.alibaba.nacos.common.remote.client.RpcClientFactory;
-import com.alibaba.nacos.common.remote.client.ServerListFactory;
 import com.alibaba.nacos.common.remote.client.RpcClientTlsConfig;
+import com.alibaba.nacos.common.remote.client.ServerListFactory;
 import com.alibaba.nacos.common.utils.CollectionUtils;
 import com.alibaba.nacos.common.utils.JacksonUtils;
 
@@ -139,8 +139,10 @@ public void batchRegisterService(String serviceName, String groupName, List<Inst
     @Override
     public void batchDeregisterService(String serviceName, String groupName, List<Instance> instances)
             throws NacosException {
-        List<Instance> retainInstance = getRetainInstance(serviceName, groupName, instances);
-        batchRegisterService(serviceName, groupName, retainInstance);
+        synchronized (redoService.getRegisteredInstances()) {
+            List<Instance> retainInstance = getRetainInstance(serviceName, groupName, instances);
+            batchRegisterService(serviceName, groupName, retainInstance);
+        }
     }
 
     /**
@@ -218,11 +220,20 @@ public void doRegisterService(String serviceName, String groupName, Instance ins
 
     @Override
     public void deregisterService(String serviceName, String groupName, Instance instance) throws NacosException {
-        NAMING_LOGGER
-                .info("[DEREGISTER-SERVICE] {} deregistering service {} with instance: {}", namespaceId, serviceName,
-                        instance);
-        redoService.instanceDeregister(serviceName, groupName);
-        doDeregisterService(serviceName, groupName, instance);
+        NAMING_LOGGER.info("[DEREGISTER-SERVICE] {} deregistering service {} with instance: {}", namespaceId,
+                serviceName, instance);
+        String key = NamingUtils.getGroupedName(serviceName, groupName);
+        InstanceRedoData instanceRedoData = redoService.getRegisteredInstancesByKey(key);
+        if (instanceRedoData instanceof BatchInstanceRedoData) {
+            List<Instance> instances = new ArrayList<>();
+            if (null != instance) {
+                instances.add(instance);
+            }
+            batchDeregisterService(serviceName, groupName, instances);
+        } else {
+            redoService.instanceDeregister(serviceName, groupName);
+            doDeregisterService(serviceName, groupName, instance);
+        }
     }
 
     /**
@@ -246,12 +257,11 @@ public void updateInstance(String serviceName, String groupName, Instance instan
     }
 
     @Override
-    public ServiceInfo queryInstancesOfService(String serviceName, String groupName, String clusters, int udpPort,
+    public ServiceInfo queryInstancesOfService(String serviceName, String groupName, String clusters,
             boolean healthyOnly) throws NacosException {
         ServiceQueryRequest request = new ServiceQueryRequest(namespaceId, serviceName, groupName);
         request.setCluster(clusters);
         request.setHealthyOnly(healthyOnly);
-        request.setUdpPort(udpPort);
         QueryServiceResponse response = requestToServer(request, QueryServiceResponse.class);
         return response.getServiceInfo();
     }
@@ -321,8 +331,8 @@ public ServiceInfo doSubscribe(String serviceName, String groupName, String clus
     @Override
     public void unsubscribe(String serviceName, String groupName, String clusters) throws NacosException {
         if (NAMING_LOGGER.isDebugEnabled()) {
-            NAMING_LOGGER
-                    .debug("[GRPC-UNSUBSCRIBE] service:{}, group:{}, cluster:{} ", serviceName, groupName, clusters);
+            NAMING_LOGGER.debug("[GRPC-UNSUBSCRIBE] service:{}, group:{}, cluster:{} ", serviceName, groupName,
+                    clusters);
         }
         redoService.subscriberDeregister(serviceName, groupName, clusters);
         doUnsubscribe(serviceName, groupName, clusters);

diff --git a/...src/main/java/com/alibaba/nacos/client/naming/remote/gprc/redo/NamingGrpcRedoService.java b/...src/main/java/com/alibaba/nacos/client/naming/remote/gprc/redo/NamingGrpcRedoService.java
@@ -68,6 +68,10 @@ public NamingGrpcRedoService(NamingGrpcClientProxy clientProxy) {
                 DEFAULT_REDO_DELAY, TimeUnit.MILLISECONDS);
     }
 
+    public ConcurrentMap<String, InstanceRedoData> getRegisteredInstances() {
+        return registeredInstances;
+    }
+
     public boolean isConnected() {
         return connected;
     }