Creating big query tables for perf tests - 3 #1450

Tulsishah · 2023-10-18T05:58:35Z

Description

Changing python script, to upload data in big query tables(Added functions).
Added licence files.

Link to the issue in case of a bug fix.

NA

Testing details

Manual - Tested build script manually, Working fine uploading data on gsheet.
Unit tests - NA
Integration tests - NA

1. It chunkifies DiffFiles utility function. This rationalizes the memory requirement of file-op of this utility. This utility earlier used to the read the complete files in memory, but now only a chunk is read and compared at a time. 2. Rename DiffFiles to AreFilesIdentical Renames file-op utility DiffFiles to AreFilesIdentical and changes it to return bool instead of int.

…1380) * adding print log file statement * failure test * testing error * undo testing changes * formating * fix comment * adding test error * small fix * small fix * testing * adding action artifacts * small fix * small fix * removing error statement * testing * testing * testing * undo changes * undo testing changes * fixing comments

* Upgrading go version to 1.21.1 * Temp changes to run perf/integration test * Revert "Temp changes to run perf/integration test" This reverts commit 18dd533.

For now, it just prints stack-trace whenever anyone calls logger.Fatalf(...). This needs to be enhanced further to include more scenarios (i.e. more sources of crashes).

Also, improved the doc for config-file.

…#1413)

Upgrade from golang 1.21.1 to 1.21.2

* update go yaml package version from v2 to v3 in integration tests * Empty-Commit

#1430) * Passing gcsfuse flags from build.sh to mount the test bucket for performing list benchmarking * removing __pycache * small fix * small fix * unmount after fio tests * small fix * empty commit * testing kokoro perf tests * undo testing changes * adding upload flag * small fix * small fix * upload to upload_gs * unnecessary change

perfmetrics/scripts/continuous_test/gcp_ubuntu/build.sh

perfmetrics/scripts/fio/fio_metrics.py

perfmetrics/scripts/fetch_metrics.py

raj-prince

Few minor comments.

Tulsishah · 2023-10-30T04:27:42Z

Few minor comments.

Fixed and tested again.

gargnitingoogle · 2023-11-01T10:15:29Z

@mustvicky@gmail.com test

gargnitingoogle · 2023-11-01T10:24:01Z

@mustvicky@gmail.com this is changes that we talked about offline. This is a change being merged into the master branch just 3 days before release (coming Monday). This has no impact on a user, and is purely in our regression jobs. So, this should be okay to go ahead, and not be blocked from merge into master. Needed your confirmation.

cc @Tulsishah

mustvicky · 2023-11-01T10:27:27Z

Confirmed to let the commit go to Master. My understanding is that if there is a problem this causes in regression test, we should rollback this chaneg to not release a possibly incorrect software, even if the software is not in the main logic.

Also, in general, we should tag master before runnign the longhaul tests release the build from that tag. I understand we are not there yet.

* Fix DiffFiles utility function 1. It chunkifies DiffFiles utility function. This rationalizes the memory requirement of file-op of this utility. This utility earlier used to the read the complete files in memory, but now only a chunk is read and compared at a time. 2. Rename DiffFiles to AreFilesIdentical Renames file-op utility DiffFiles to AreFilesIdentical and changes it to return bool instead of int. * Copying logs on kokoro artifacts incase of integration test failure. (#1380) * adding print log file statement * failure test * testing error * undo testing changes * formating * fix comment * adding test error * small fix * small fix * testing * adding action artifacts * small fix * small fix * removing error statement * testing * testing * testing * undo changes * undo testing changes * fixing comments * Upgrading go version from 1.21.0 to 1.21.1 (#1401) * Upgrading go version to 1.21.1 * Temp changes to run perf/integration test * Revert "Temp changes to run perf/integration test" This reverts commit 18dd533. * Updating semantics docs (#1407) * Print stack-trace on crash For now, it just prints stack-trace whenever anyone calls logger.Fatalf(...). This needs to be enhanced further to include more scenarios (i.e. more sources of crashes). * Throw error in case of unexpected fields (#1416) Also, improved the doc for config-file. * Upgraded the fuse library (#1419) * add "create-empty-file: true" tests to run_tests_mounted_directory.sh (#1413) * Upgrade golang from 1.21.1 to 1.21.2 (#1431) Upgrade from golang 1.21.1 to 1.21.2 * Update yaml package version from v2 to v3 in integration tests (#1434) * update go yaml package version from v2 to v3 in integration tests * Empty-Commit * Passing gcsfuse flags from build.sh to mount the test bucket for perf… (#1430) * Passing gcsfuse flags from build.sh to mount the test bucket for performing list benchmarking * removing __pycache * small fix * small fix * unmount after fio tests * small fix * empty commit * testing kokoro perf tests * undo testing changes * adding upload flag * small fix * small fix * upload to upload_gs * unnecessary change * adding big query table setup * changes for small pr * adding requirements and setup scripts * indentation fix * removing unnecessary changes * adding start build time * adding requirements and setup scripts * python file changes * small fix * testing * testing gsheet upload * perfmetrics/scripts/ls_metrics/listing_benchmark.py * testing * removing unnecessary changes * Creating big query tables for perf tests - 1 (#1444) * adding big query table setup * changes for small pr * indentation fix * removing unnecessary changes * adding start build time * changes for small pr * indentation fix * removing unnecessary changes * adding requirements and setup scripts * merge with parent * testing * testing gsheet upload * merge with parent * removing unnecessary changes * fixing requirements.in * Creating big query tables for perf tests - 2 (#1445) * adding big query table setup * changes for small pr * adding requirements and setup scripts * fixing requirements.in * merge * merge * small fixes * small fixes * small fix * undo testing changes * formating * testing changes * undo testing changes * remove unnecessary functions * fixing indentation * small fix * adding testing changes * small fix * undo testing changes * formating * fixing unit tests * formatting * fixing commets and adding test changes * undo testing changes --------- Co-authored-by: Nitin Garg <gargnitin@google.com> Co-authored-by: Prince Kumar <princer@google.com> Co-authored-by: Ayush Sethi <ayushsethi@google.com> Co-authored-by: Ashmeen Kaur <57195160+ashmeenkaur@users.noreply.github.com> Co-authored-by: Nitin Garg <113666283+gargnitingoogle@users.noreply.github.com>

gargnitingoogle and others added 14 commits October 10, 2023 05:24

Upgrading go version from 1.21.0 to 1.21.1 (#1401)

d38a7f1

* Upgrading go version to 1.21.1 * Temp changes to run perf/integration test * Revert "Temp changes to run perf/integration test" This reverts commit 18dd533.

Updating semantics docs (#1407)

bf09c7c

Print stack-trace on crash

241e4c1

For now, it just prints stack-trace whenever anyone calls logger.Fatalf(...). This needs to be enhanced further to include more scenarios (i.e. more sources of crashes).

Throw error in case of unexpected fields (#1416)

14736a2

Also, improved the doc for config-file.

Upgraded the fuse library (#1419)

f11347a

add "create-empty-file: true" tests to run_tests_mounted_directory.sh (…

5fba84e

…#1413)

Upgrade golang from 1.21.1 to 1.21.2 (#1431)

7448f21

Upgrade from golang 1.21.1 to 1.21.2

Update yaml package version from v2 to v3 in integration tests (#1434)

0ae1b1f

* update go yaml package version from v2 to v3 in integration tests * Empty-Commit

adding big query table setup

8ac2103

changes for small pr

7a7afbd

adding requirements and setup scripts

ebb0d61

Tulsishah requested review from sethiay, vadlakondaswetha and raj-prince as code owners October 18, 2023 05:58

Tulsishah changed the base branch from master to nov_2023_release October 18, 2023 05:58

Tulsishah force-pushed the create_big_query_tables_3 branch from 8b0a261 to 411dbc4 Compare October 18, 2023 06:25

Tulsishah added 2 commits October 18, 2023 06:49

indentation fix

cd50167

removing unnecessary changes

5e988ea

Tulsishah changed the title ~~Creating big query tables for perf tests - 3~~ [Don't Review] Creating big query tables for perf tests - 3 Oct 18, 2023

Tulsishah force-pushed the create_big_query_tables_3 branch 2 times, most recently from b1ad848 to 3467c94 Compare October 18, 2023 09:29

Tulsishah added 5 commits October 18, 2023 09:51

adding start build time

62d439b

adding requirements and setup scripts

caab8cb

python file changes

2953cb7

small fix

1f9d7d4

testing

59d321d

Tulsishah force-pushed the create_big_query_tables_3 branch from 3467c94 to a3da04b Compare October 18, 2023 09:57

Tulsishah added 4 commits October 26, 2023 04:39

small fix

93166fa

undo testing changes

3497883

formating

fb9f165

fixing unit tests

794118b

Tulsishah force-pushed the create_big_query_tables_3 branch 3 times, most recently from 3ffaebc to 1be2b96 Compare October 26, 2023 09:56

formatting

cd03358

Tulsishah force-pushed the create_big_query_tables_3 branch from 1be2b96 to cd03358 Compare October 26, 2023 10:17

Tulsishah added 2 commits October 27, 2023 02:43

merge with master

fbdad0d

merge with master

c26b6cc

raj-prince reviewed Oct 30, 2023

View reviewed changes

perfmetrics/scripts/continuous_test/gcp_ubuntu/build.sh Outdated Show resolved Hide resolved

raj-prince reviewed Oct 30, 2023

View reviewed changes

perfmetrics/scripts/fio/fio_metrics.py Outdated Show resolved Hide resolved

raj-prince reviewed Oct 30, 2023

View reviewed changes

perfmetrics/scripts/fetch_metrics.py Outdated Show resolved Hide resolved

raj-prince requested changes Oct 30, 2023

View reviewed changes

Tulsishah added 2 commits October 30, 2023 03:38

fixing commets and adding test changes

6e4a05e

undo testing changes

48367ad

Tulsishah requested a review from raj-prince October 30, 2023 04:27

raj-prince approved these changes Oct 30, 2023

View reviewed changes

gargnitingoogle mentioned this pull request Nov 1, 2023

Add exec-mode to script files #1486

Merged

Tulsishah merged commit 7fed6ea into master Nov 1, 2023
4 checks passed

p-linnane mentioned this pull request Dec 18, 2023

gcsfuse 1.3.0 Homebrew/homebrew-core#157693

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Creating big query tables for perf tests - 3 #1450

Creating big query tables for perf tests - 3 #1450

Tulsishah commented Oct 18, 2023 •

edited

raj-prince left a comment

Tulsishah commented Oct 30, 2023

gargnitingoogle commented Nov 1, 2023

gargnitingoogle commented Nov 1, 2023 •

edited

mustvicky commented Nov 1, 2023

Creating big query tables for perf tests - 3 #1450

Creating big query tables for perf tests - 3 #1450

Conversation

Tulsishah commented Oct 18, 2023 • edited

Description

Link to the issue in case of a bug fix.

Testing details

raj-prince left a comment

Choose a reason for hiding this comment

Tulsishah commented Oct 30, 2023

gargnitingoogle commented Nov 1, 2023

gargnitingoogle commented Nov 1, 2023 • edited

mustvicky commented Nov 1, 2023

Tulsishah commented Oct 18, 2023 •

edited

gargnitingoogle commented Nov 1, 2023 •

edited