Create scripts to launch benchmark testing to Google cloud #496
Conversation
dimberman
force-pushed
the
dimberman-issue-432
branch
from
0e17c9a
to
8d877aa
Compare
|
Closing-reopening to see if it triggeres CI |
Codecov Report
@@ Coverage Diff @@
## main #496 +/- ##
=======================================
Coverage 92.02% 92.02%
=======================================
Files 40 40
Lines 1342 1342
Branches 182 182
=======================================
Hits 1235 1235
Misses 85 85
Partials 22 22 Continue to review full report at Codecov.
|
|
@kaxil I can confirm that this runs smoothly on GKE. I was able to do the entire flow from beginning to end. Glad to demo tomorrow if you like. |
dimberman
force-pushed
the
dimberman-issue-432
branch
from
a6cbdfc
to
e17bb34
Compare
|
@kaxil I'm gonna run this script one more time and merge if everything works. |
tests/benchmark/config.json
Outdated
| "conn_id": "bigquery", | ||
| "file_type": "ndjson", | ||
| "name": "one_gb", | ||
| "path": "gs://astro-sdk/benchmark/trimmed/stackoverflow/stackoverflow_posts_1g.ndjson", | ||
| "rows": 385817, | ||
| "size": "1G" |
There was a problem hiding this comment.
@kaxil I'm not sure hwo that happened. I added it back in.
Description
What is the current behavior?
Currently users are running benchmark tests on their local machines. This is a) inconsistent as much of the time might be tied to network time and b) bad as it discourages consistently running benchmarks (E.g. as a daily job or as a part of CI).
closes: #432
this PR is following up on the PR: #471
What is the new behavior?
Now users will have the ability to launch a GKE cluster, push an image for the astro SDK, and launch the job to run on GKE
Does this introduce a breaking change?
No
Checklist