Skip to content

Commit

Permalink
support Nacos cluster (#2640)
Browse files Browse the repository at this point in the history
* Fix #2546

* Change the stackdepth type

* Fix bug

* Fix CI failed

* fix #2599

* fix ci

* add license

* add ActiveTag error methods

* fix

* fix

* nacos

* nacos

* nacos

* nacos

* fix ci

* support nacos cluster

* fix yml

* fix license

* fix ci

* remove optional

* update license

* update license and doc

* fix
  • Loading branch information
IanCao authored and peng-yongsheng committed May 14, 2019
1 parent b4ee1f1 commit 91ffae2
Show file tree
Hide file tree
Showing 17 changed files with 636 additions and 2 deletions.
2 changes: 2 additions & 0 deletions apm-dist/release-docs/LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,7 @@ The text of each license is also included at licenses/LICENSE-[project].txt.

Apache: groovy 2.4.5-indy: https://github.com/apache/groovy, Apache 2.0
Apache: httpcomponents 4.x.x: http://hc.apache.org/index.html, Apache 2.0
Apache: fastjson 1.2.47: https://github.com/alibaba/fastjson, Apache 2.0

========================================================================
Apache 2.0 licenses
Expand Down Expand Up @@ -307,6 +308,7 @@ The text of each license is the standard Apache 2.0 license.
consul-client 1.2.6: https://github.com/rickfast/consul-client, Apache 2.0
okhttp 3.9.0: https://github.com/square/okhttp, Apache 2.0
prometheus client_java 0.6.0: https://github.com/prometheus/client_java, Apache 2.0
nacos 1.0.0: https://github.com/alibaba/nacos, Apache 2.0
proto files from istio/istio: https://github.com/istio/istio Apache 2.0
proto files from istio/api: https://github.com/istio/api Apache 2.0
proto files from envoyproxy/data-plane-api: https://github.com/envoyproxy/data-plane-api Apache 2.0
Expand Down
38 changes: 38 additions & 0 deletions apm-dist/release-docs/NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -831,3 +831,41 @@ from and not be held liable to the user for any such damages as noted
above as far as the program is concerned.

------

===========================================================================
nacos-1.0.0 Notice
===========================================================================
Nacos
Copyright 2018-2019 The Apache Software Foundation

This product includes software developed at
The Alibaba MiddleWare Group.

------
This product has a bundle Spring Boot:
The Spring Boot Project
=================

Please visit the Spring Boot web site for more information:

* https://spring.io/projects/spring-boot

Copyright 2014 The Spring Boot Project

The Spring Boot Project licenses this file to you under the Apache License,
version 2.0 (the "License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at:

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.

Also, please refer to each LICENSE.<component>.txt file, which is located in
the 'license' directory of the distribution file, for the license terms of the
components that this product depends on.

------
13 changes: 13 additions & 0 deletions apm-dist/release-docs/licenses/LICENSE-fastjson.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
Copyright 1999-2019 Alibaba Group Holding Ltd.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
3 changes: 3 additions & 0 deletions docker/config/application.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ cluster:
# serviceName: ${SW_SERVICE_NAME:"SkyWalking_OAP_Cluster"}
# Consul cluster nodes, example: 10.0.0.1:8500,10.0.0.2:8500,10.0.0.3:8500
# hostPort: ${SW_CLUSTER_CONSUL_HOST_PORT:localhost:8500}
# nacos:
# serviceName: ${SW_SERVICE_NAME:"SkyWalking_OAP_Cluster"}
# hostPort: ${SW_CLUSTER_NACOS_HOST_PORT:localhost:8848}
core:
default:
# Mixed: Receive agent data, Level 1 aggregate, Level 2 aggregate
Expand Down
17 changes: 15 additions & 2 deletions docs/en/setup/backend/backend-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ with each other.
- [Kubernetes](#kubernetes). When backend cluster are deployed inside kubernetes, you could choose this
by using k8s native APIs to manage cluster.
- [Consul](#consul). Use Consul as backend cluster management implementor, to coordinate backend instances.

- [Nacos](#nacos). Use Nacos to coordinate backend instances.

## Zookeeper coordinator
Zookeeper is a very common and wide used cluster coordinator. Set the **cluster** module's implementor
Expand Down Expand Up @@ -77,4 +77,17 @@ Same as Zookeeper coordinator,
in some cases, oap default gRPC host and port in core are not suitable for internal communication among the oap nodes.
The following setting are provided to set the hot and port manually, based on your own LAN env.
- internalComHost, the host registered and other oap node use this to communicate with current node.
- internalComPort, the port registered and other oap node use this to communicate with current node.
- internalComPort, the port registered and other oap node use this to communicate with current node.


## Nacos
Set the **cluster** module's implementor to **nacos** in
the yml to active.

```yaml
cluster:
nacos:
serviceName: ${SW_SERVICE_NAME:"SkyWalking_OAP_Cluster"}
# Nacos cluster nodes, example: 10.0.0.1:8848,10.0.0.2:8848,10.0.0.3:8848
hostPort: ${SW_CLUSTER_NACOS_HOST_PORT:localhost:8848}
```
71 changes: 71 additions & 0 deletions oap-server/server-cluster-plugin/cluster-nacos-plugin/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
~
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>server-cluster-plugin</artifactId>
<groupId>org.apache.skywalking</groupId>
<version>6.2.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>cluster-nacos-plugin</artifactId>
<properties>
<nacos.version>1.0.0</nacos.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>server-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.alibaba.nacos</groupId>
<artifactId>nacos-client</artifactId>
<version>${nacos.version}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
</exclusion>
<exclusion>
<groupId>net.jcip</groupId>
<artifactId>jcip-annotations</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/

package org.apache.skywalking.oap.server.cluster.plugin.nacos;

import lombok.Getter;
import lombok.Setter;
import org.apache.skywalking.oap.server.library.module.ModuleConfig;

/**
* @author caoyixiong
*/
public class ClusterModuleNacosConfig extends ModuleConfig {
@Setter @Getter private String serviceName;
@Setter @Getter private String hostPort;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/

package org.apache.skywalking.oap.server.cluster.plugin.nacos;

import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.naming.NamingFactory;
import com.alibaba.nacos.api.naming.NamingService;
import org.apache.skywalking.oap.server.core.CoreModule;
import org.apache.skywalking.oap.server.core.cluster.ClusterModule;
import org.apache.skywalking.oap.server.core.cluster.ClusterNodesQuery;
import org.apache.skywalking.oap.server.core.cluster.ClusterRegister;
import org.apache.skywalking.oap.server.library.module.*;

/**
* @author caoyixiong
*/
public class ClusterModuleNacosProvider extends ModuleProvider {

private final ClusterModuleNacosConfig config;
private NamingService namingService;

public ClusterModuleNacosProvider() {
super();
this.config = new ClusterModuleNacosConfig();
}

@Override
public String name() {
return "nacos";
}

@Override
public Class<? extends ModuleDefine> module() {
return ClusterModule.class;
}

@Override
public ModuleConfig createConfigBeanIfAbsent() {
return config;
}

@Override
public void prepare() throws ServiceNotProvidedException, ModuleStartException {
try {
namingService = NamingFactory.createNamingService(config.getHostPort());
} catch (NacosException e) {
throw new ModuleStartException(e.getMessage(), e);
}
NacosCoordinator coordinator = new NacosCoordinator(namingService, config);
this.registerServiceImplementation(ClusterRegister.class, coordinator);
this.registerServiceImplementation(ClusterNodesQuery.class, coordinator);
}

@Override
public void start() throws ServiceNotProvidedException {

}

@Override
public void notifyAfterCompleted() throws ServiceNotProvidedException {

}

@Override
public String[] requiredModules() {
return new String[]{CoreModule.NAME};
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/

package org.apache.skywalking.oap.server.cluster.plugin.nacos;

import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.pojo.Instance;
import org.apache.skywalking.oap.server.core.cluster.*;
import org.apache.skywalking.oap.server.core.remote.client.Address;
import org.apache.skywalking.oap.server.library.util.CollectionUtils;

import java.util.ArrayList;
import java.util.List;
import java.util.Objects;

/**
* @author caoyixiong
*/
public class NacosCoordinator implements ClusterRegister, ClusterNodesQuery {

private final NamingService namingService;
private final ClusterModuleNacosConfig config;
private volatile Address selfAddress;

public NacosCoordinator(NamingService namingService, ClusterModuleNacosConfig config) {
this.namingService = namingService;
this.config = config;
}

@Override
public List<RemoteInstance> queryRemoteNodes() {
List<RemoteInstance> result = new ArrayList<>();
try {
List<Instance> instances = namingService.selectInstances(config.getServiceName(), true);
if (CollectionUtils.isNotEmpty(instances)) {
instances.forEach(instance -> {
if (Objects.nonNull(selfAddress)) {
if (selfAddress.getHost().equals(instance.getIp()) && selfAddress.getPort() == instance.getPort()) {
result.add(new RemoteInstance(new Address(instance.getIp(), instance.getPort(), true)));
} else {
result.add(new RemoteInstance(new Address(instance.getIp(), instance.getPort(), false)));
}
}
});
}
} catch (NacosException e) {
throw new ServiceQueryException(e.getErrMsg());
}
return result;
}

@Override
public void registerRemote(RemoteInstance remoteInstance) throws ServiceRegisterException {
String host = remoteInstance.getAddress().getHost();
int port = remoteInstance.getAddress().getPort();
try {
namingService.registerInstance(config.getServiceName(), host, port);
} catch (Exception e) {
throw new ServiceRegisterException(e.getMessage());
}
this.selfAddress = remoteInstance.getAddress();
}
}
Loading

0 comments on commit 91ffae2

Please sign in to comment.