Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Introduce testcontainer based cluster integration tests (#2055)
* Introduce testcontainer cluster tests *Motivation* Introduce testcontainer based cluster tests for replacing existing arquilian tests. testcontainer is more flexible and more nature to how we write unit tests. arquilian is configuration based and hard to change cluster at runtime. it is possible to add bookies/brokers/proxies at testing runtime using testcontainers. current arquilian based integration tests can only run in linux environment (which is using host network). it is impossible to run arquillian tests in mac environment, where host network is meaningless. changing to testcontainers is allowing running integration tests on mac to speed up development. *Changes* This mirrors the exact same setup as existing arquillian tests - 1 zk container, 1 cs container, 3 bk containers, 2 broker containers and 1 proxy container. add a semantic integration test module to using the new testcontainer test base `PulsarClusterTestBase`. *Result* - both arquillian and testcontainers exist. - new integration tests are encouraged to be written using `PulsarClusterTestBase`. It is just as easy as how you write normal unit tests. - existing arquillian tests will be migrated in future PRs. * all modules under `tests/integration` should only be run after `-DintegrationTests` is specified
- Loading branch information
Showing
17 changed files
with
948 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
30 changes: 30 additions & 0 deletions
30
...ration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/BKContainer.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| /** | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
| package org.apache.pulsar.tests.containers; | ||
|
|
||
| /** | ||
| * A pulsar container that runs bookkeeper. | ||
| */ | ||
| public class BKContainer extends PulsarContainer<BKContainer> { | ||
|
|
||
| public BKContainer(String clusterName, String hostName) { | ||
| super( | ||
| clusterName, hostName, hostName, "bin/run-bookie.sh", BOOKIE_PORT, INVALID_PORT); | ||
| } | ||
| } |
30 changes: 30 additions & 0 deletions
30
...on-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/BrokerContainer.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| /** | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
| package org.apache.pulsar.tests.containers; | ||
|
|
||
| /** | ||
| * A pulsar container that runs bookkeeper. | ||
| */ | ||
| public class BrokerContainer extends PulsarContainer<BrokerContainer> { | ||
|
|
||
| public BrokerContainer(String clusterName, String hostName) { | ||
| super( | ||
| clusterName, hostName, hostName, "bin/run-broker.sh", BROKER_PORT, INVALID_PORT); | ||
| } | ||
| } |
37 changes: 37 additions & 0 deletions
37
...ration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/CSContainer.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| /** | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
| package org.apache.pulsar.tests.containers; | ||
|
|
||
| /** | ||
| * A pulsar container that runs configuration store. | ||
| */ | ||
| public class CSContainer extends PulsarContainer<CSContainer> { | ||
|
|
||
| public static final String NAME = "configuration-store"; | ||
|
|
||
| public CSContainer(String clusterName) { | ||
| super( | ||
| clusterName, | ||
| NAME, | ||
| NAME, | ||
| "bin/run-global-zk.sh", | ||
| CS_PORT, | ||
| INVALID_PORT); | ||
| } | ||
| } |
116 changes: 116 additions & 0 deletions
116
...ion-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/ChaosContainer.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| /** | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
| package org.apache.pulsar.tests.containers; | ||
|
|
||
| import static java.nio.charset.StandardCharsets.UTF_8; | ||
|
|
||
| import com.github.dockerjava.api.command.LogContainerCmd; | ||
| import com.github.dockerjava.api.model.Frame; | ||
| import com.github.dockerjava.core.command.LogContainerResultCallback; | ||
| import java.util.Objects; | ||
| import java.util.concurrent.CompletableFuture; | ||
| import java.util.concurrent.TimeUnit; | ||
| import lombok.extern.slf4j.Slf4j; | ||
| import org.apache.commons.lang.StringUtils; | ||
| import org.testcontainers.containers.GenericContainer; | ||
|
|
||
| /** | ||
| * A base container provides chaos capability. | ||
| */ | ||
| @Slf4j | ||
| public class ChaosContainer<SelfT extends ChaosContainer<SelfT>> extends GenericContainer<SelfT> { | ||
|
|
||
| protected final String clusterName; | ||
|
|
||
| protected ChaosContainer(String clusterName, String image) { | ||
| super(image); | ||
| this.clusterName = clusterName; | ||
| } | ||
|
|
||
| public void tailContainerLog() { | ||
| CompletableFuture.runAsync(() -> { | ||
| while (null == containerId) { | ||
| try { | ||
| TimeUnit.MILLISECONDS.sleep(100); | ||
| } catch (InterruptedException e) { | ||
| return; | ||
| } | ||
| } | ||
|
|
||
| LogContainerCmd logContainerCmd = this.dockerClient.logContainerCmd(containerId); | ||
| logContainerCmd.withStdOut(true).withStdErr(true).withFollowStream(true); | ||
| logContainerCmd.exec(new LogContainerResultCallback() { | ||
| @Override | ||
| public void onNext(Frame item) { | ||
| log.info(new String(item.getPayload(), UTF_8)); | ||
| } | ||
| }); | ||
| }); | ||
| } | ||
|
|
||
| public String getContainerLog() { | ||
| StringBuilder sb = new StringBuilder(); | ||
|
|
||
| LogContainerCmd logContainerCmd = this.dockerClient.logContainerCmd(containerId); | ||
| logContainerCmd.withStdOut(true).withStdErr(true); | ||
| try { | ||
| logContainerCmd.exec(new LogContainerResultCallback() { | ||
| @Override | ||
| public void onNext(Frame item) { | ||
| sb.append(new String(item.getPayload(), UTF_8)); | ||
| } | ||
| }).awaitCompletion(); | ||
| } catch (InterruptedException e) { | ||
|
|
||
| } | ||
| return sb.toString(); | ||
| } | ||
|
|
||
| public ExecResult execCmd(String... cmd) throws Exception { | ||
| String cmdString = StringUtils.join(cmd, " "); | ||
|
|
||
| log.info("DOCKER.exec({}:{}): Executing ...", containerId, cmdString); | ||
|
|
||
| ExecResult result = execInContainer(cmd); | ||
|
|
||
| log.info("Docker.exec({}:{}): Done", containerId, cmdString); | ||
| log.info("Docker.exec({}:{}): Stdout -\n{}", containerId, cmdString, result.getStdout()); | ||
| log.info("Docker.exec({}:{}): Stderr -\n{}", containerId, cmdString, result.getStderr()); | ||
|
|
||
| return result; | ||
| } | ||
|
|
||
| @Override | ||
| public boolean equals(Object o) { | ||
| if (!(o instanceof ChaosContainer)) { | ||
| return false; | ||
| } | ||
|
|
||
| ChaosContainer another = (ChaosContainer) o; | ||
| return clusterName.equals(another.clusterName) | ||
| && super.equals(another); | ||
| } | ||
|
|
||
| @Override | ||
| public int hashCode() { | ||
| return 31 * super.hashCode() + Objects.hash( | ||
| clusterName); | ||
| } | ||
|
|
||
| } |
38 changes: 38 additions & 0 deletions
38
...ion-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/ProxyContainer.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| /** | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
| package org.apache.pulsar.tests.containers; | ||
|
|
||
| /** | ||
| * A pulsar container that runs bookkeeper. | ||
| */ | ||
| public class ProxyContainer extends PulsarContainer<ProxyContainer> { | ||
|
|
||
| public ProxyContainer(String clusterName, String hostName) { | ||
| super( | ||
| clusterName, hostName, hostName, "bin/run-proxy.sh", BROKER_PORT, BROKER_HTTP_PORT); | ||
| } | ||
|
|
||
| public String getPlainTextServiceUrl() { | ||
| return "pulsar://" + getContainerIpAddress() + ":" + getMappedPort(BROKER_PORT); | ||
| } | ||
|
|
||
| public String getHttpServiceUrl() { | ||
| return "http://" + getContainerIpAddress() + ":" + getMappedPort(BROKER_HTTP_PORT); | ||
| } | ||
| } |
Oops, something went wrong.