Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added benchmarks scripts with small instruction #8529

Merged
merged 24 commits into from
Dec 12, 2021
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f77238f
updated source-mongodb-v2 performance
andriikorotkov Nov 15, 2021
6ac8c93
updated code style
andriikorotkov Nov 15, 2021
6e3f202
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Nov 15, 2021
9ff2bfd
fixed remarks
andriikorotkov Nov 15, 2021
4eb9d09
fixed remarks
andriikorotkov Nov 15, 2021
451fe71
fixed remarks
andriikorotkov Nov 15, 2021
ae3c06e
updated strict encrypt source mongodb version
andriikorotkov Nov 18, 2021
5f19252
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Nov 18, 2021
a1d23ff
updated source mongodb work with empty collections
andriikorotkov Nov 18, 2021
799c877
updated source mongodb timestamp cursor
andriikorotkov Nov 21, 2021
a5d444c
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Nov 21, 2021
e83f8d3
updated mongodb source perfomance
andriikorotkov Nov 21, 2021
9a8dce4
fix code style
andriikorotkov Nov 22, 2021
18fd7b7
fix code style
andriikorotkov Nov 22, 2021
70ccff2
updated tests and documentation
andriikorotkov Nov 22, 2021
868d7e9
updated tests and documentation
andriikorotkov Nov 22, 2021
a3c6284
updated tests and documentation
andriikorotkov Nov 22, 2021
d6399b3
added vudangngoc changes
andriikorotkov Nov 23, 2021
d427834
updated code style
andriikorotkov Nov 23, 2021
7a8c313
updated code style
andriikorotkov Nov 23, 2021
01c6b4d
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Dec 6, 2021
8a4a964
added benchmarks scripts with small instruction
andriikorotkov Dec 6, 2021
d34d65c
fixed remarks
andriikorotkov Dec 9, 2021
9f64c10
updated ci performance test sh script
andriikorotkov Dec 11, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
17 changes: 17 additions & 0 deletions docs/benchmarks-scripts/mssql/mssql-script.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# How to use MsSQL Script?

In order to create a database with a certain number of tables, and a certain number of records in each of them,
you need to follow a few simple steps.

1. Create a new database.
2. In the **mssql-script.sql** file, you need to change a few points to create the required number of tables and records in them.
1. Change the value for the number of tables.

![](images/table_count_mssql.png)
2. You can add records of different sizes to the table - 50 bytes, 500 bytes, 10 kilobytes, and 100 kilobytes. To do this,
you need to specify the number of records for each size.

![](images/rows_count_mssql.png)
andriikorotkov marked this conversation as resolved.
Show resolved Hide resolved
3. Execute the script with your changes for the new database. After the script finishes its work, you will receive the number of tables specified in the script, with names starting with **test_0** and ending with **test_(the number of tables minus 1)**.
andriikorotkov marked this conversation as resolved.
Show resolved Hide resolved


250 changes: 250 additions & 0 deletions docs/benchmarks-scripts/mssql/mssql-script.sql

Large diffs are not rendered by default.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
17 changes: 17 additions & 0 deletions docs/benchmarks-scripts/mysql/mysql-script.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# How to use MySQL Script?

In order to create a database with a certain number of tables, and a certain number of records in each of them,
you need to follow a few simple steps.

1. Create a new database.
2. In the **mysql-script.sql** file, you need to change a few points to create the required number of tables and records in them.
1. Change the value for the number of tables.

![](images/table_count_mysql.png)
2. You can add records of different sizes to the table - 50 bytes, 500 bytes, 10 kilobytes, and 100 kilobytes. To do this,
you need to specify the number of records for each size.

![](images/rows_count_mysql.png)
3. Execute the script with your changes for the new database. After the script finishes its work, you will receive the number of tables specified in the script, with names starting with **test_0** and ending with **test_(the number of tables minus 1)**.


245 changes: 245 additions & 0 deletions docs/benchmarks-scripts/mysql/mysql-script.sql

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
create or replace procedure copy_table(tablecount int)
language plpgsql
as $$
declare v_max_table int; v_counter_table int; v_tnamee VARCHAR(255);
begin
v_max_table := tablecount;
v_counter_table := 1;
while v_counter_table < v_max_table loop
EXECUTE format('create table test_%s as (select * from test t)', v_counter_table);
v_counter_table := v_counter_table + 1;
end loop;
commit;
end;$$

Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
create or replace procedure insert_rows(allrows int, insertcount int, value text)
language plpgsql
as $$
declare dummyIpsum varchar(255); fieldText text; vmax int; vmaxx int; vmaxoneinsert int; counter int;
declare lastinsertcounter int; lastinsert int; fullloop int; fullloopcounter int; insertTable text; insertTableLasted text;

begin
fieldText := value;
dummyIpsum = '''dummy_ipsum''';
vmax = allrows;
vmaxx = allrows;
vmaxoneinsert = insertcount;
counter = 0;
lastinsertcounter = 0;
lastinsert = 0;
fullloop = 0;
fullloopcounter = 0;

while vmaxx < vmaxoneinsert loop
vmaxoneinsert := vmaxx;
fullloop := fullloop + 1;
end loop;
commit;

while vmax > vmaxoneinsert loop
fullloop := fullloop + 1;
vmax := vmax - vmaxoneinsert;
lastinsert := vmax;
end loop;
commit;

insertTable := 'insert into test (varchar1, varchar2, varchar3, varchar4, varchar5, longblobfield, timestampfield) values (';
while counter < vmaxoneinsert loop
insertTable := concat(insertTable, dummyIpsum, ', ', dummyIpsum, ', ', dummyIpsum, ', ', dummyIpsum, ', ', dummyIpsum, ', ', fieldText, ', CURRENT_TIMESTAMP), (');
counter := counter + 1;
end loop;
commit;
insertTable := concat(insertTable, dummyIpsum, ', ', dummyIpsum, ', ', dummyIpsum, ', ', dummyIpsum, ', ', dummyIpsum, ', ', fieldText, ', CURRENT_TIMESTAMP);');

while vmax < 1 loop
fullloop := 0;
vmax := 1;
end loop;
commit;

while fullloopcounter < fullloop loop
EXECUTE insertTable;
fullloopcounter := fullloopcounter + 1;
end loop;
commit;

insertTableLasted := 'insert into test (varchar1, varchar2, varchar3, varchar4, varchar5, longblobfield, timestampfield) values (';
while lastinsertcounter < lastinsert loop
insertTableLasted := concat(insertTableLasted, dummyIpsum, ', ', dummyIpsum, ', ', dummyIpsum, ', ', dummyIpsum, ', ', dummyIpsum, ', ', fieldText, ', CURRENT_TIMESTAMP), (');
lastinsertcounter := lastinsertcounter + 1;
end loop;
commit;
insertTableLasted := concat(insertTableLasted, dummyIpsum, ', ', dummyIpsum, ', ', dummyIpsum, ', ', dummyIpsum, ', ', dummyIpsum, ', ', fieldText, ', CURRENT_TIMESTAMP);');

while lastinsert > 0 loop
EXECUTE insertTableLasted;
lastinsert := 0;
end loop;
commit;
end;$$


136 changes: 136 additions & 0 deletions docs/benchmarks-scripts/postgres/3-run-script.sql

Large diffs are not rendered by default.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
19 changes: 19 additions & 0 deletions docs/benchmarks-scripts/postgres/postgres-script.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# How to use Postgres Script?

In order to create a database with a certain number of tables, and a certain number of records in each of them,
you need to follow a few simple steps.

1. Create a new database.
2. On the new database, run script **1-create-copy-tables-procedure.sql** to create the table copy procedure.
3. Run script **2-create-insert-rows-to-table-procedure.sql** to create a procedure for creating a table with the specified number of records.
4. In the **3-run-script.sql** file, you need to change a few points to create the required number of tables and records in them.
1. Change the value for the number of tables.

![](images/table_count_postgres.png)
2. You can add records of different sizes to the table - 50 bytes, 500 bytes, 10 kilobytes, and 100 kilobytes. To do this,
you need to specify the number of records for each size.

![](images/rows_count_postgres.png)
5. Execute the **3-run-script.sql** script with your changes for the new database. After the script finishes its work, you will receive the number of tables specified in the script, with names starting with **test_0** and ending with **test_(the number of tables minus 1)**.