Skip to content

Commit

Permalink
📑 Update source database performance test docs (#8980)
Browse files Browse the repository at this point in the history
* Add missing pr number

* Update source database performance test docs

* Update doc

* Update script paths
  • Loading branch information
tuliren committed Dec 21, 2021
1 parent a6c2d05 commit 9b75a76
Show file tree
Hide file tree
Showing 10 changed files with 62 additions and 49 deletions.
30 changes: 18 additions & 12 deletions airbyte-integrations/connectors/source-mssql/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,30 @@

## Performance Test

### Running performance tests with CPU and Memory limits for the container
To run performance tests in commandline:
```shell
./gradlew :airbyte-integrations:connectors:source-mssql:performanceTest [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
```

In order to run performance tests with CPU and Memory limits, you need to run the performance test start command with
additional parameters **cpulimit=cpulimit/YOU_CPU_LIMIT** and **memorylimit=memorylimit/YOU_MEMORY_LIMIT**.
**YOU_MEMORY_LIMIT** - RAM limit. Be sure to indicate the limit in MB or GB at the end. Minimum size - 6MB.
**YOU_CPU_LIMIT** - CPU limit. Minimum size - 2.
These parameters are optional and can be used separately from each other.
For example, if you use only **memorylimit=memorylimit/2GB**, only the memory limit for the container will be set, the CPU will not be limited.
Also, if you do not use both of these parameters, then performance tests will run without memory and CPU limitations.
In pull request:
```shell
/test-performance connector=connectors/source-mssql [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
```

- `cpulimit`: Limit the number of CPUs. The minimum is `2`. E.g. `--cpulimit=cpulimit/2`.
- `memorylimit`: Limit the size of the memory. Must include the unit at the end (e.g. `MB`, `GB`). The minimum size is `6MB`. E.g. `--memorylimit=memorylimit/4GB`.
- When none of the CPU or memory limit is provided, the performance tests will run without memory or CPU limitations. The available resource will be bound that those specified in `ResourceRequirements.java`.

### Use MsSQL script to populate the benchmark database

In order to create a database with a certain number of tables, and a certain number of records in each of them,
you need to follow a few simple steps.

1. Create a new database.
2. Follow the TODOs in **mssql-script.sql** to change the number of tables, and the number of records of different sizes.
3. Execute the script with your changes for the new database.
You can run the script use the MySQL command line client: - **mysql -h hostname -u user database < path/to/script/mssql-script.sql**
After the script finishes its work, you will receive the number of tables specified in the script, with names starting with **test_0** and ending with **test_(the number of tables minus 1)**.
2. Follow the TODOs in [create_mssql_benchmarks.sql](src/test-performance/sql/create_mssql_benchmarks.sql) to change the number of tables, and the number of records of different sizes.
3. Execute the script with your changes for the new database. You can run the script with the MySQL client:
```bash
cd airbyte-integrations/connectors/source-mssql
sqlcmd -S Serverinstance -E -i src/test-performance/sql/create_mssql_benchmarks.sql
```
4. After the script finishes its work, you will receive the number of tables specified in the script, with names starting with **test_0** and ending with **test_(the number of tables minus 1)**.
41 changes: 21 additions & 20 deletions airbyte-integrations/connectors/source-mysql/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,37 +25,38 @@ the Dockerfile.
## Testing
We use `JUnit` for Java tests.

### Test Configuration
#### Acceptance Tests
### Acceptance Tests
To run acceptance and custom integration tests:
```
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest
```

#### Performance Tests
To run performance tests:
### Performance Tests

To run performance tests in commandline:
```shell
./gradlew :airbyte-integrations:connectors:source-mysql:performanceTest [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
```
./gradlew :airbyte-integrations:connectors:source-mysql:performanceTest

In pull request:
```shell
/test-performance connector=connectors/source-mysql [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
```
### Running performance tests with CPU and Memory limits for the container

In order to run performance tests with CPU and Memory limits, you need to run the performance test start command with
additional parameters **cpulimit=cpulimit/YOU_CPU_LIMIT** and **memorylimit=memorylimit/YOU_MEMORY_LIMIT**.
**YOU_MEMORY_LIMIT** - RAM limit. Be sure to indicate the limit in MB or GB at the end. Minimum size - 6MB.
**YOU_CPU_LIMIT** - CPU limit. Minimum size - 2.
These parameters are optional and can be used separately from each other.
For example, if you use only **memorylimit=memorylimit/2GB**, only the memory limit for the container will be set, the CPU will not be limited.
Also, if you do not use both of these parameters, then performance tests will run without memory and CPU limitations.
- `cpulimit`: Limit the number of CPUs. The minimum is `2`. E.g. `--cpulimit=cpulimit/2`.
- `memorylimit`: Limit the size of the memory. Must include the unit at the end (e.g. `MB`, `GB`). The minimum size is `6MB`. E.g. `--memorylimit=memorylimit/4GB`.
- When none of the CPU or memory limit is provided, the performance tests will run without memory or CPU limitations. The available resource will be bound that those specified in `ResourceRequirements.java`.

### Use MySQL script to populate the benchmark database
#### Use MySQL script to populate the benchmark database

In order to create a database with a certain number of tables, and a certain number of records in each of them,
you need to follow a few simple steps.

1. Create a new database.
2. Follow the TODOs in **mssql-script.sql** to change the number of tables, and the number of records of different sizes.
3. Execute the script with your changes for the new database.
You can run the script use the MsSQL command line client: - **sqlcmd -S Serverinstance -E -i path/to/script/mssql-script.sql**
After the script finishes its work, you will receive the number of tables specified in the script, with names starting with **test_0** and ending with **test_(the number of tables minus 1)**.


2. Follow the TODOs in [create_mysql_benchmarks.sql](src/test-performance/sql/create_mysql_benchmarks.sql) to change the number of tables, and the number of records of different sizes.
3. Execute the script with your changes for the new database. You can run the script using the MySQL command line client:
```shell
cd airbyte-integrations/connectors/source-mysql
mysql -h hostname -u user database < src/test-performance/sql/create_mysql_benchmarks.sql
```
4. After the script finishes its work, you will receive the number of tables specified in the script, with names starting with **test_0** and ending with **test_(the number of tables minus 1)**.
36 changes: 21 additions & 15 deletions airbyte-integrations/connectors/source-postgres/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,32 @@

## Performance Test

### Running performance tests with CPU and Memory limits for the container
To run performance tests in commandline:
```shell
./gradlew :airbyte-integrations:connectors:source-postgres:performanceTest [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
```

In order to run performance tests with CPU and Memory limits, you need to run the performance test start command with
additional parameters **cpulimit=cpulimit/YOU_CPU_LIMIT** and **memorylimit=memorylimit/YOU_MEMORY_LIMIT**.
**YOU_MEMORY_LIMIT** - RAM limit. Be sure to indicate the limit in MB or GB at the end. Minimum size - 6MB.
**YOU_CPU_LIMIT** - CPU limit. Minimum size - 2.
These parameters are optional and can be used separately from each other.
For example, if you use only **memorylimit=memorylimit/2GB**, only the memory limit for the container will be set, the CPU will not be limited.
Also, if you do not use both of these parameters, then performance tests will run without memory and CPU limitations.
In pull request:
```shell
/test-performance connector=connectors/source-postgres [--cpulimit=cpulimit/<limit>] [--memorylimit=memorylimit/<limit>]
```

- `cpulimit`: Limit the number of CPUs. The minimum is `2`. E.g. `--cpulimit=cpulimit/2`.
- `memorylimit`: Limit the size of the memory. Must include the unit at the end (e.g. `MB`, `GB`). The minimum size is `6MB`. E.g. `--memorylimit=memorylimit/4GB`.
- When none of the CPU or memory limit is provided, the performance tests will run without memory or CPU limitations. The available resource will be bound that those specified in `ResourceRequirements.java`.

### Use Postgres script to populate the benchmark database

In order to create a database with a certain number of tables, and a certain number of records in each of them,
you need to follow a few simple steps.

1. Create a new database.
2. On the new database, run script **1-create-copy-tables-procedure.sql** to create the table copy procedure.
You can run the script use the Postgres command line client: - **psql -h host -d userstoreis -U admin -p port -a -q -f /path/to/script/1-create-copy-tables-procedure.sql**
3. Run script **2-create-insert-rows-to-table-procedure.sql** to create a procedure for creating a table with the specified number of records.
4. Follow the TODOs in **3-run-script.sql** to change the number of tables, and the number of records of different sizes.
5. Execute the **3-run-script.sql** script with your changes for the new database. After the script finishes its work, you will receive the number of tables specified in the script, with names starting with **test_0** and ending with **test_(the number of tables minus 1)**.


2. Follow the TODOs in [3-run-script.sql](src/test-performance/sql/3-run-script.sql) to change the number of tables, and the number of records of different sizes.
3. On the new database, run the following script:
```shell
cd airbyte-integrations/connectors/source-postgres
psql -h <host> -d <db-name> -U <username> -p <port> -a -q -f src/test-performance/sql/1-create-copy-tables-procedure.sql
psql -h <host> -d <db-name> -U <username> -p <port> -a -q -f src/test-performance/sql/2-create-insert-rows-to-table-procedure.sql
psql -h <host> -d <db-name> -U <username> -p <port> -a -q -f src/test-performance/sql/3-run-script.sql
```
4. After the script finishes, you will receive the number of tables specified in the script, with names starting with **test_0** and ending with **test_(the number of tables minus 1)**.
2 changes: 1 addition & 1 deletion docs/integrations/destinations/gcs.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ Under the hood, an Airbyte data stream in Json schema is first converted to an A

| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.1.16 | 2021-12-20 | [\#](https://github.com/airbytehq/airbyte/pull/) | Release a new version to ensure there is no excessive logging. |
| 0.1.16 | 2021-12-20 | [\#8974](https://github.com/airbytehq/airbyte/pull/8974) | Release a new version to ensure there is no excessive logging. |
| 0.1.15 | 2021-12-03 | [\#8386](https://github.com/airbytehq/airbyte/pull/8386) | Add new GCP regions |
| 0.1.14 | 2021-12-01 | [\#7732](https://github.com/airbytehq/airbyte/pull/7732) | Support timestamp in Avro and Parquet |
| 0.1.13 | 2021-11-03 | [\#7288](https://github.com/airbytehq/airbyte/issues/7288) | Support Json `additionalProperties`. |
Expand Down
2 changes: 1 addition & 1 deletion docs/integrations/destinations/s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ Under the hood, an Airbyte data stream in Json schema is first converted to an A

| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.2.1 | 2021-12-20 | [\#](https://github.com/airbytehq/airbyte/pull/) | Release a new version to ensure there is no excessive logging. |
| 0.2.1 | 2021-12-20 | [\#8974](https://github.com/airbytehq/airbyte/pull/8974) | Release a new version to ensure there is no excessive logging. |
| 0.2.0 | 2021-12-15 | [\#8607](https://github.com/airbytehq/airbyte/pull/8607) | Change the output filename for CSV files - it's now `bucketPath/namespace/streamName/timestamp_epochMillis_randomUuid.csv` |
| 0.1.16 | 2021-12-10 | [\#8562](https://github.com/airbytehq/airbyte/pull/8562) | Swap dependencies with destination-jdbc. |
| 0.1.15 | 2021-12-03 | [\#8501](https://github.com/airbytehq/airbyte/pull/8501) | Remove excessive logging for Avro and Parquet invalid date strings. |
Expand Down

0 comments on commit 9b75a76

Please sign in to comment.