[IOTDB-1559] Refactor the IT framework (#4371)

.github/workflows/cluster.yml

@@ -0,0 +1,50 @@
+name: Cluster Test
+
+on:
+  push:
+    branches:
+      - master
+      - 'rel/*'
+    paths-ignore:
+      - 'docs/**'
+  pull_request:
+    branches:
+      - master
+      - 'rel/*'
+    paths-ignore:
+      - 'docs/**'
+  # allow manually run the action:
+  workflow_dispatch:
+
+env:
+  MAVEN_OPTS: -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3
+
+jobs:
+  unix:
+    strategy:
+      fail-fast: false
+      max-parallel: 20
+      matrix:
+        java: [ 8 ]
+        os: [ ubuntu-latest ]
+    runs-on: ${{ matrix.os}}
+
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up JDK ${{ matrix.java }}
+        uses: actions/setup-java@v1
+        with:
+          java-version: ${{ matrix.java }}
+      - name: Cache Maven packages
+        uses: actions/cache@v2
+        with:
+          path: ~/.m2
+          key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
+          restore-keys: ${{ runner.os }}-m2-
+      - name: Check Apache Rat
+        run: mvn -B apache-rat:check -P site -P code-coverage
+      - name: IT/UT Test
+        shell: bash
+        # we do not compile client-cpp for saving time, it is tested in client.yml
+        # we can skip influxdb-protocol because it has been tested separately in influxdb-protocol.yml
+        run: mvn -B clean verify -Dsession.test.skip=true -Diotdb.test.skip=true -Dcluster.test.skip=true -Dtsfile.test.skip=true -pl integration -am -PCluster

.github/workflows/main-unix.yml

@@ -53,4 +53,4 @@ jobs:
         shell: bash
         # we do not compile client-cpp for saving time, it is tested in client.yml
         # we can skip influxdb-protocol because it has been tested separately in influxdb-protocol.yml
-        run: mvn -B clean post-integration-test -Dtest.port.closed=true -P '!testcontainer,!influxdb-protocol'
+        run: mvn -B clean verify -Dtest.port.closed=true -P '!testcontainer,!influxdb-protocol'

.github/workflows/main-win.yml

@@ -109,5 +109,5 @@ jobs:
         shell: bash
         if: ${{ matrix.it_task == 'others'}}
         # we can skip influxdb-protocol because it has been tested separately in influxdb-protocol.yml
-        run: source ~/.bash_profile && mvn -B clean integration-test -Dtest.port.closed=true -Diotdb.test.skip=true -Dcluster.test.skip=true -P '!influxdb-protocol'
+        run: source ~/.bash_profile && mvn -B clean verify -Dtest.port.closed=true -Diotdb.test.skip=true -Dcluster.test.skip=true -P '!influxdb-protocol'
 

docs/UserGuide/Integration-Test/Integration-Test-refactoring-tutorial.md

@@ -0,0 +1,161 @@
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+    
+        http://www.apache.org/licenses/LICENSE-2.0
+    
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+-->
+
+# Integration Test refactoring tutorial
+
+- Step 0. Prerequisites
+
+  - **IT Location has been moved**;  all Integration Tests have been moved to the integration module.
+  - **Test case MUST label**; the test classification label is the junit's `category`, which determines the test environment or process in which the test case is tested.
+  - **Code related to the test environment MAY need to be refactored**; this determines whether the current test environment can correctly test the test case. The corresponding statement needs to be refactored.
+
+
+
+- Step 1. Labeling test cases
+
+  - Add the appropriate `Category` before the test case class or test case method, which can collect any desired test category labels.
+
+  - The `Category` of the following three test categories are all real and effective,
+
+    ```java
+    @Category({LocalStandaloneTest.class, ClusterTest.class, RemoteTest.class})
+    public class IoTDBAliasIT {
+      ......
+    }
+
+
+    @Category({LocalStandaloneTest.class, ClusterTest.class})
+    public class IoTDBAlignByDeviceIT {
+      ......
+    }
+
+
+    @Category({LocalStandaloneTest.class})
+    public class IoTDBArithmeticIT {
+      ......
+    }
+    ```
+
+  - You can also add `Category` at the test method level.
+
+    ```java
+    @Category({LocalStandaloneTest.class})
+    public class IoTDBExampleIT {
+
+     // This case can ONLY test in environment of local.
+     @Test
+     public void theStandaloneCase() {
+       ......
+     }
+
+     // This case can test in environment of local, cluster and remote.
+     @Test
+     @Category({ClusterTest.class, RemoteTest.class})
+     public void theAllEnvCase() {
+       ......
+     }
+    }
+    ```
+
+  - At present, all test cases must at least add the `Category` of the stand-alone test, namely `LocalStandaloneTest.class`.
+
+
+
+- Step 2. Environmental code refactoring
+
+  - If the test case needs to be tested in the Cluster or Remote environment, the environment-related code MUST be refactored accordingly. If it is only tested in the LocalStandalone environment, modifications are only recommended. (Not all test cases can be tested in the Cluster or Remote environment because statements that are limited by some functions, such as local file operations, cannot be refactored.)
+
+    |                            | LocalStandalone |   Cluster   |   Remote    |
+    | :------------------------- | :-------------: | :---------: | :---------: |
+    | setUp and tearDown         |    Recommend    |    Must     |    Must     |
+    | getConnection              |    Recommend    |    Must     |    Must     |
+    | change config              |    Recommend    |    Must     | Not support |
+    | Local file operation       |  Won't change   | Not support | Not support |
+    | Local descriptor operation |  Won't change   | Not support | Not support |
+    | restart operation          |  Won't change   | Not support | Not support |
+
+    
+
+  - The `setUp` and `tearDown` methods must be refactored in the Cluster and Remote environment
+
+    ```java
+    @Category({LocalStandaloneTest.class, ClusterTest.class, RemoteTest.class})
+    public class IoTDBAliasIT {
+    
+      @BeforeClass
+      public static void setUp() throws Exception {
+        // EnvironmentUtils.closeStatMonitor(); // orginal setup code
+        // EnvironmentUtils.envSetUp(); // orginal setup code
+        EnvFactory.getEnv().initBeforeClass(); // new initBeforeClass code
+    
+        insertData();
+      }
+    
+      @AfterClass
+      public static void tearDown() throws Exception {
+        // EnvironmentUtils.cleanEnv(); // orginal code
+        EnvFactory.getEnv().cleanAfterClass(); // new cleanAfterClass code
+      }
+    }
+    ```
+
+    
+
+  - The `getConnection` must be refactored in Cluster and Remote environments
+
+    ```java
+    private static void insertData() throws ClassNotFoundException {
+        // Class.forName(Config.JDBC_DRIVER_NAME); // orginal connection code
+        // try (Connection connection =  
+        //         DriverManager.getConnection( 
+        //             Config.IOTDB_URL_PREFIX + "127.0.0.1:6667/", "root", "root");
+        try (Connection connection = EnvFactory.getEnv().getConnection(); // new code
+            Statement statement = connection.createStatement()) {
+    
+          for (String sql : sqls) {
+            statement.execute(sql);
+          }
+        } catch (Exception e) {
+          e.printStackTrace();
+        }
+      }
+    ```
+  
+  - The method of changing the IoTDB configuration must be refactored in the Cluster environment. (As the remote environment cannot change the configuration remotely at present, the test cases that change the configuration will not support testing in the remote environment)
+    - In the Cluster environment, as the configuration cannot be changed dynamically, only the configuration changes before the environment init are effective.
+    - The refactoring has included most of the configuration changes, which can be changed through the method of `ConfigFactory.getConfig()`.
+
+    ```java
+      @Category({LocalStandaloneTest.class, ClusterTest.class})
+      public class IoTDBCompleteIT {
+        private int prevVirtualStorageGroupNum;
+    
+        @Before
+        public void setUp() {
+          prevVirtualStorageGroupNum =
+              IoTDBDescriptor.getInstance().getConfig().getVirtualStorageGroupNum();
+          // IoTDBDescriptor.getInstance().getConfig().setVirtualStorageGroupNum(16); // orginal code
+          ConfigFactory.getConfig().setVirtualStorageGroupNum(16); // new code
+          EnvFactory.getEnv().initBeforeClass();
+        }
+    ```
+  
+    - If the configuration item has not been included in the method of `ConfigFactory.getConfig()`, it needs to be defined in the BaseConfig.java interface file and implemented in StandaloneEnvConfig.java and ClusterEnvConfig.java, respectively. This part is not very common. For specific, please refer to the realized part.

docs/zh/UserGuide/Integration-Test/Integration-Test-refactoring-tutorial.md

@@ -0,0 +1,163 @@
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+    
+        http://www.apache.org/licenses/LICENSE-2.0
+    
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+-->
+
+# IoTDB社区Integration Test改造说明
+
+- 步骤0. 前提须知
+  - **位置已移动**；所有的Integration Test已被移动至单独的integration模块。
+  - **测试用例必须打分类标签**； `Category` 即测试分类标签，决定了该测试用例在哪套测试环境或流程中被测试。
+  - **涉及测试环境的代码可能要重构**；决定了该测试用例是否能被当前测试环境正确测试，需要根据相应的环境重构相应的代码。
+
+
+
+- 步骤1. 测试用例打标签
+
+  - 在测试用例类或者测试用例方法前加上合适的`Category`，可以是任意期望的测试分类标签的集合。
+
+  - 真实样例，下面三个测试类的`Category`都是真实有效的，
+
+    ```java
+    @Category({LocalStandaloneTest.class, ClusterTest.class, RemoteTest.class})
+    public class IoTDBAliasIT {
+      ......
+    }
+
+
+    @Category({LocalStandaloneTest.class, ClusterTest.class})
+    public class IoTDBAlignByDeviceIT {
+      ......
+    }
+
+
+    @Category({LocalStandaloneTest.class})
+    public class IoTDBArithmeticIT {
+      ......
+    }
+    ```
+
+  - 甚至，你还可以在测试方法级别加`Category`。
+
+    ```java
+    @Category({LocalStandaloneTest.class})
+    public class IoTDBExampleIT {
+
+     // This case can ONLY test in environment of local.
+     @Test
+     public void theStandaloneCase() {
+       ......
+     }
+
+     // This case can test in environment of local, cluster and remote.
+     @Test
+     @Category({ClusterTest.class, RemoteTest.class})
+     public void theAllEnvCase() {
+       ......
+     }
+    }
+    ```
+
+  - 目前，所有测试用例至少要加上单机测试的测试分类，即`LocalStandaloneTest.class`。
+
+
+
+- 步骤2. 环境代码重构
+
+  - 如果测试用例需要在Cluster或者Remote环境下被测试，那么必须对环境相关的代码作相应重构，如果是仅在LocalStandalone环境下测试，则只推荐修改。（不是所有的测试用例可以在Cluster或者Remote环境下被测试，因为受限于部分功能的语句比如本地文件操作，这些代码不能被重构。）
+
+    |                            | LocalStandalone |   Cluster   |   Remote    |
+    | :------------------------- | :-------------: | :---------: | :---------: |
+    | setUp and tearDown         |    Recommend    |    Must     |    Must     |
+    | getConnection              |    Recommend    |    Must     |    Must     |
+    | change config              |    Recommend    |    Must     | Not support |
+    | Local file operation       |  Won't change   | Not support | Not support |
+    | Local descriptor operation |  Won't change   | Not support | Not support |
+    | restart operation          |  Won't change   | Not support | Not support |
+  
+    
+  
+  - `setUp` 和`tearDown` 方法内的重构，在Cluster和Remote环境下是必须更改的
+  
+    ```java
+    @Category({LocalStandaloneTest.class, ClusterTest.class, RemoteTest.class})
+    public class IoTDBAliasIT {
+    
+      @BeforeClass
+      public static void setUp() throws Exception {
+        // EnvironmentUtils.closeStatMonitor(); // orginal setup code
+        // EnvironmentUtils.envSetUp(); // orginal setup code
+        EnvFactory.getEnv().initBeforeClass(); // new initBeforeClass code
+    
+        insertData();
+      }
+    
+      @AfterClass
+      public static void tearDown() throws Exception {
+        // EnvironmentUtils.cleanEnv(); // orginal code
+        EnvFactory.getEnv().cleanAfterClass(); // new cleanAfterClass code
+      }
+    }
+    ```
+  
+    
+
+  - `getConnection` 的重构，在Cluster和Remote环境下是必须更改
+  
+    ```java
+      private static void insertData() throws ClassNotFoundException {
+        // Class.forName(Config.JDBC_DRIVER_NAME); // orginal connection code
+        // try (Connection connection =  
+        //         DriverManager.getConnection( 
+        //             Config.IOTDB_URL_PREFIX + "127.0.0.1:6667/", "root", "root");
+        try (Connection connection = EnvFactory.getEnv().getConnection(); // new code
+            Statement statement = connection.createStatement()) {
+    
+          for (String sql : sqls) {
+            statement.execute(sql);
+          }
+        } catch (Exception e) {
+          e.printStackTrace();
+        }
+      }
+    ```
+  
+  
+  
+  - 更改配置的方法，在Cluster环境下是必须重构的。（由于目前Remote环境无法远程更改配置，更改配置的测试用例将不支持Remote环境下测试）
+  
+    - 在Cluster环境下，由于无法动态更改配置，只有环境init前的配置更改才有效。
+    - 重构已包含了大部分的配置更改，通过`ConfigFactory.getConfig()` 的方法可以进行链式更改。
+  
+    ```java
+    @Category({LocalStandaloneTest.class, ClusterTest.class})
+    public class IoTDBCompleteIT {
+      private int prevVirtualStorageGroupNum;
+    
+      @Before
+      public void setUp() {
+        prevVirtualStorageGroupNum =
+            IoTDBDescriptor.getInstance().getConfig().getVirtualStorageGroupNum();
+        // IoTDBDescriptor.getInstance().getConfig().setVirtualStorageGroupNum(16); // orginal code
+        ConfigFactory.getConfig().setVirtualStorageGroupNum(16); // new code
+        EnvFactory.getEnv().initBeforeClass();
+      }
+    ```
+    
+    - 若配置项尚未在`ConfigFactory.getConfig()` 的方法中包含，需要在BaseConfig.java接口文件中定义，在StandaloneEnvConfig.java和ClusterEnvConfig.java中分别实现，这部分不是很常用，具体方法可以参考已实现的部分，目前暂不列出。