Refactor/pkg/improve job performance #2061

vankichi · 2023-06-14T02:55:11Z

Description:

I have refactored the followings:

Add "original" enum for dataset
Refactor HDF5 package to improve performance
Refactor benchmark jobs

Related Issue:

Versions:

Go Version: 1.20.3
Docker Version: 20.10.8
Kubernetes Version: 1.22.0
NGT Version: 2.0.11

Checklist:

I have read the CONTRIBUTING document and completed our CLA agreement.
I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer:

Signed-off-by: vankichi <kyukawa315@gmail.com>

vdaas-ci · 2023-06-14T02:56:37Z

[CHATOPS:HELP] ChatOps commands.

🙆‍♀️ /approve - approve
💌 /changelog - replace the PR body by changelog details
🍱 /format - format codes and add licenses
✅ /gen-test - generate test codes
🏷️ /label - add labels
⏪ /rebase - rebase main
🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

codecov · 2023-06-14T03:35:09Z

Codecov Report

Patch coverage: 4.76% and project coverage change: +0.11 🎉

Comparison is base (fd47989) 31.55% compared to head (9182107) 31.66%.

❗ Current head 9182107 differs from pull request most recent head 82623b1. Consider uploading reports for the commit 82623b1 to get more accurate results

Additional details and impacted files

@@                                     Coverage Diff                                      @@
##           refactor/internal-charts-pkg/refactor-benchmark-operator    #2061      +/-   ##
============================================================================================
+ Coverage                                                     31.55%   31.66%   +0.11%     
============================================================================================
  Files                                                           354      354              
  Lines                                                         33253    33253              
============================================================================================
+ Hits                                                          10494    10531      +37     
+ Misses                                                        22273    22240      -33     
+ Partials                                                        486      482       -4

Impacted Files	Coverage Δ
pkg/tools/benchmark/job/config/config.go	`8.62% <0.00%> (-0.16%)`	⬇️
internal/test/data/hdf5/hdf5.go	`9.03% <5.00%> (+3.42%)`	⬆️

... and 7 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

Signed-off-by: vankichi <kyukawa315@gmail.com>

cloudflare-pages · 2023-06-14T05:18:57Z

Deploying with Cloudflare Pages

Latest commit:	`82623b1`
Status:	✅ Deploy successful!
Preview URL:	https://f8822cc4.vald.pages.dev
Branch Preview URL:	https://refactor-pkg-improve-job-per.vald.pages.dev

View logs

* Implement base of continuous benchmark tool (#1776) * Create Continuous Bench Search Job tool (#1733) * ✨ create bench job search tools Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add load hdf5 functions Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix docker and use hdf5 data Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix proto Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 add benchmark job image build ci Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 invest Signed-off-by: vankichi <kyukawa315@gmail.com> * Revert "💚 invest" This reverts commit f0f585c. * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply code review Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * ✨ apply from feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * Update internal/config/benchmark.go Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ change directory path Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * Add crds for continuous benchmark tools (#1789) * ✨ add crds for continuous benchmark operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add benchmark operator/job scheme Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ rename package names and add doc.go Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create runtime object Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> * Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825) * ✨ ♻️ add Job reconciler & use scenario instead of operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format & rename file Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Add benchmark operator framework (#1916) * ✨ impl benchmark reconciler Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create benchmark operator framework Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ remove unness changes Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * impl reconcile logic for create benchmark job (#1923) * ✨ impl reconcile logic for create benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark's crds Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark job logic Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ update charts Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ rafactor con bench config and bug fix reconcile logic Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Bugfix: fix typo and recall function logic Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor pkg benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ✨ impl status handle of continuous benchmark crds (#1955) Signed-off-by: vankichi <kyukawa315@gmail.com> * Impl benchmark jobs (#1977) * Implement base of continuous benchmark tool (#1776) * Create Continuous Bench Search Job tool (#1733) * ✨ create bench job search tools Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add load hdf5 functions Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix docker and use hdf5 data Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix proto Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 add benchmark job image build ci Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 invest Signed-off-by: vankichi <kyukawa315@gmail.com> * Revert "💚 invest" This reverts commit f0f585c. * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply code review Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * ✨ apply from feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * Update internal/config/benchmark.go Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ change directory path Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * Add crds for continuous benchmark tools (#1789) * ✨ add crds for continuous benchmark operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add benchmark operator/job scheme Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ rename package names and add doc.go Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create runtime object Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> * Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825) * ✨ ♻️ add Job reconciler & use scenario instead of operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format & rename file Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Add benchmark operator framework (#1916) * ✨ impl benchmark reconciler Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create benchmark operator framework Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ remove unness changes Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * impl reconcile logic for create benchmark job (#1923) * ✨ impl reconcile logic for create benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark's crds Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark job logic Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ update charts Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ rafactor con bench config and bug fix reconcile logic Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Bugfix: fix typo and recall function logic Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor pkg benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ✨ impl benchmark jobs Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ♻️ Refactor helm template and operator logic Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Add download original dataset URL option Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Set docker image location at the benchmark operator configmap and use it when information is set Signed-off-by: vankichi <kyukawa315@gmail.com> * add search algorithm benchmark and update search aggregation algo Signed-off-by: kpango <kpango@vdaas.org> * ✨ Add search result aggregation option Signed-off-by: vankichi <kyukawa315@gmail.com> * style: Format code with prettier and gofumpt * Improve job performance (#2061) * 🐛 Fix job function to apply rate limiter * ♻️ Add pyroscope setting Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Fix build error Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Fix docker file and add concurrencyLimit for job goroutine Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Fix job_template.go by feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Fix job logic by feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Fix Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: kpango <kpango@vdaas.org> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* Implement base of continuous benchmark tool (#1776) * Create Continuous Bench Search Job tool (#1733) * ✨ create bench job search tools Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add load hdf5 functions Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix docker and use hdf5 data Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix proto Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 add benchmark job image build ci Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 invest Signed-off-by: vankichi <kyukawa315@gmail.com> * Revert "💚 invest" This reverts commit f0f585c. * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply code review Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * ✨ apply from feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * Update internal/config/benchmark.go Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ change directory path Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * Add crds for continuous benchmark tools (#1789) * ✨ add crds for continuous benchmark operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add benchmark operator/job scheme Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ rename package names and add doc.go Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create runtime object Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> * Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825) * ✨ ♻️ add Job reconciler & use scenario instead of operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format & rename file Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Add benchmark operator framework (#1916) * ✨ impl benchmark reconciler Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create benchmark operator framework Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ remove unness changes Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * impl reconcile logic for create benchmark job (#1923) * ✨ impl reconcile logic for create benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark's crds Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark job logic Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ update charts Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ rafactor con bench config and bug fix reconcile logic Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Bugfix: fix typo and recall function logic Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor pkg benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ✨ impl status handle of continuous benchmark crds (#1955) Signed-off-by: vankichi <kyukawa315@gmail.com> * Impl benchmark jobs (#1977) * Implement base of continuous benchmark tool (#1776) * Create Continuous Bench Search Job tool (#1733) * ✨ create bench job search tools Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add load hdf5 functions Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix docker and use hdf5 data Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix proto Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 add benchmark job image build ci Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 invest Signed-off-by: vankichi <kyukawa315@gmail.com> * Revert "💚 invest" This reverts commit f0f585c. * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply code review Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * ✨ apply from feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * Update internal/config/benchmark.go Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ change directory path Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * Add crds for continuous benchmark tools (#1789) * ✨ add crds for continuous benchmark operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add benchmark operator/job scheme Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ rename package names and add doc.go Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create runtime object Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> * Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825) * ✨ ♻️ add Job reconciler & use scenario instead of operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format & rename file Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Add benchmark operator framework (#1916) * ✨ impl benchmark reconciler Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create benchmark operator framework Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ remove unness changes Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * impl reconcile logic for create benchmark job (#1923) * ✨ impl reconcile logic for create benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark's crds Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark job logic Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ update charts Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ rafactor con bench config and bug fix reconcile logic Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Bugfix: fix typo and recall function logic Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor pkg benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ✨ impl benchmark jobs Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * create helm template for benchmark operator (#2027) * ✨ create helm template for benchmark operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ refactor helm template Signed-off-by: vankichi <kyukawa315@gmail.com> * style: Format code with gofumpt and prettier --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * Refactor helm template and operator logic (#2043) * Implement base of continuous benchmark tool (#1776) * Create Continuous Bench Search Job tool (#1733) * ✨ create bench job search tools Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add load hdf5 functions Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix docker and use hdf5 data Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix proto Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 add benchmark job image build ci Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 invest Signed-off-by: vankichi <kyukawa315@gmail.com> * Revert "💚 invest" This reverts commit f0f585c. * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply code review Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * ✨ apply from feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * Update internal/config/benchmark.go Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ change directory path Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * Add crds for continuous benchmark tools (#1789) * ✨ add crds for continuous benchmark operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add benchmark operator/job scheme Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ rename package names and add doc.go Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create runtime object Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> * Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825) * ✨ ♻️ add Job reconciler & use scenario instead of operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format & rename file Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Add benchmark operator framework (#1916) * ✨ impl benchmark reconciler Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create benchmark operator framework Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ remove unness changes Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * impl reconcile logic for create benchmark job (#1923) * ✨ impl reconcile logic for create benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark's crds Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark job logic Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ update charts Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ rafactor con bench config and bug fix reconcile logic Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Bugfix: fix typo and recall function logic Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor pkg benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ✨ impl status handle of continuous benchmark crds (#1955) Signed-off-by: vankichi <kyukawa315@gmail.com> * Impl benchmark jobs (#1977) * Implement base of continuous benchmark tool (#1776) * Create Continuous Bench Search Job tool (#1733) * ✨ create bench job search tools Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add load hdf5 functions Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix docker and use hdf5 data Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix proto Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 add benchmark job image build ci Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 invest Signed-off-by: vankichi <kyukawa315@gmail.com> * Revert "💚 invest" This reverts commit f0f585c. * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply code review Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * ✨ apply from feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * Update internal/config/benchmark.go Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ change directory path Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * Add crds for continuous benchmark tools (#1789) * ✨ add crds for continuous benchmark operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add benchmark operator/job scheme Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ rename package names and add doc.go Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create runtime object Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> * Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825) * ✨ ♻️ add Job reconciler & use scenario instead of operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format & rename file Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Add benchmark operator framework (#1916) * ✨ impl benchmark reconciler Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create benchmark operator framework Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ remove unness changes Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * impl reconcile logic for create benchmark job (#1923) * ✨ impl reconcile logic for create benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark's crds Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark job logic Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ update charts Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ rafactor con bench config and bug fix reconcile logic Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Bugfix: fix typo and recall function logic Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor pkg benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ✨ impl benchmark jobs Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ♻️ Refactor helm template and operator logic Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Add download original dataset URL option Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Set docker image location at the benchmark operator configmap and use it when information is set Signed-off-by: vankichi <kyukawa315@gmail.com> * add search algorithm benchmark and update search aggregation algo Signed-off-by: kpango <kpango@vdaas.org> * ✨ Add search result aggregation option Signed-off-by: vankichi <kyukawa315@gmail.com> * style: Format code with prettier and gofumpt * Improve job performance (#2061) * 🐛 Fix job function to apply rate limiter * ♻️ Add pyroscope setting Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Fix build error Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Fix docker file and add concurrencyLimit for job goroutine Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Fix job_template.go by feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Fix job logic by feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Fix Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: kpango <kpango@vdaas.org> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * Impl benchmark jobs (#1977) * Implement base of continuous benchmark tool (#1776) * Create Continuous Bench Search Job tool (#1733) * ✨ create bench job search tools Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add load hdf5 functions Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix docker and use hdf5 data Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix proto Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 add benchmark job image build ci Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 invest Signed-off-by: vankichi <kyukawa315@gmail.com> * Revert "💚 invest" This reverts commit f0f585c. * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply code review Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * ✨ apply from feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * Update internal/config/benchmark.go Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ change directory path Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * Add crds for continuous benchmark tools (#1789) * ✨ add crds for continuous benchmark operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add benchmark operator/job scheme Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ rename package names and add doc.go Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create runtime object Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> * Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825) * ✨ ♻️ add Job reconciler & use scenario instead of operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format & rename file Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Add benchmark operator framework (#1916) * ✨ impl benchmark reconciler Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create benchmark operator framework Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ remove unness changes Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * impl reconcile logic for create benchmark job (#1923) * ✨ impl reconcile logic for create benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark's crds Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark job logic Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ update charts Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ rafactor con bench config and bug fix reconcile logic Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Bugfix: fix typo and recall function logic Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor pkg benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ✨ impl benchmark jobs Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * Refactor helm template and operator logic (#2043) * Implement base of continuous benchmark tool (#1776) * Create Continuous Bench Search Job tool (#1733) * ✨ create bench job search tools Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add load hdf5 functions Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix docker and use hdf5 data Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix proto Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 add benchmark job image build ci Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 invest Signed-off-by: vankichi <kyukawa315@gmail.com> * Revert "💚 invest" This reverts commit f0f585c. * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply code review Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * ✨ apply from feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * Update internal/config/benchmark.go Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ change directory path Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * Add crds for continuous benchmark tools (#1789) * ✨ add crds for continuous benchmark operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add benchmark operator/job scheme Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ rename package names and add doc.go Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create runtime object Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> * Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825) * ✨ ♻️ add Job reconciler & use scenario instead of operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format & rename file Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Add benchmark operator framework (#1916) * ✨ impl benchmark reconciler Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create benchmark operator framework Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ remove unness changes Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * impl reconcile logic for create benchmark job (#1923) * ✨ impl reconcile logic for create benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark's crds Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark job logic Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ update charts Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ rafactor con bench config and bug fix reconcile logic Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Bugfix: fix typo and recall function logic Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor pkg benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ✨ impl status handle of continuous benchmark crds (#1955) Signed-off-by: vankichi <kyukawa315@gmail.com> * Impl benchmark jobs (#1977) * Implement base of continuous benchmark tool (#1776) * Create Continuous Bench Search Job tool (#1733) * ✨ create bench job search tools Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add load hdf5 functions Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix docker and use hdf5 data Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix proto Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 add benchmark job image build ci Signed-off-by: vankichi <kyukawa315@gmail.com> * 💚 invest Signed-off-by: vankichi <kyukawa315@gmail.com> * Revert "💚 invest" This reverts commit f0f585c. * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply code review Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * ✨ apply from feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * Update internal/config/benchmark.go Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ change directory path Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> * Add crds for continuous benchmark tools (#1789) * ✨ add crds for continuous benchmark operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ add benchmark operator/job scheme Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ rename package names and add doc.go Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create runtime object Signed-off-by: vankichi <kyukawa315@gmail.com> * Apply suggestions from code review Co-authored-by: Yusuke Kato <kpango@vdaas.org> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> * Add Job reconciler & Change directory constitution of internal/k8s for benchmark (#1825) * ✨ ♻️ add Job reconciler & use scenario instead of operator Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix format & rename file Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Add benchmark operator framework (#1916) * ✨ impl benchmark reconciler Signed-off-by: vankichi <kyukawa315@gmail.com> * ✨ create benchmark operator framework Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ remove unness changes Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * impl reconcile logic for create benchmark job (#1923) * ✨ impl reconcile logic for create benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark's crds Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ resolve error due to update conn bench crds for pkg/tools/benchmark/job Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor continuous benchmark job logic Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ update charts Signed-off-by: vankichi <kyukawa315@gmail.com> * Format code with prettier and gofumpt * ♻️ rafactor con bench config and bug fix reconcile logic Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Bugfix: fix typo and recall function logic Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor pkg benchmark job Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ✨ impl benchmark jobs Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ apply feedback Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ♻️ Refactor helm template and operator logic Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Add download original dataset URL option Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Set docker image location at the benchmark operator configmap and use it when information is set Signed-off-by: vankichi <kyukawa315@gmail.com> * add search algorithm benchmark and update search aggregation algo Signed-off-by: kpango <kpango@vdaas.org> * ✨ Add search result aggregation option Signed-off-by: vankichi <kyukawa315@gmail.com> * style: Format code with prettier and gofumpt * Improve job performance (#2061) * 🐛 Fix job function to apply rate limiter * ♻️ Add pyroscope setting Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Fix build error Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> * 🐛 Fix docker file and add concurrencyLimit for job goroutine Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Fix job_template.go by feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Fix job logic by feedback Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ Fix Signed-off-by: vankichi <kyukawa315@gmail.com> --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: kpango <kpango@vdaas.org> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * ♻️ Add noise to vector when update/upsert and update crd Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ add deepmerge func for override default config by user-defined config Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ refactor deepmerge Signed-off-by: vankichi <kyukawa315@gmail.com> * style: Format code with prettier and gofumpt * style: Format code with prettier and gofumpt * ♻️ change add noise func Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ use golang/sync/error for prevent OOM KILL Signed-off-by: vankichi <kyukawa315@gmail.com> * ♻️ fix config adn add test Signed-off-by: vankichi <kyukawa315@gmail.com> * style: Format code with prettier and gofumpt --------- Signed-off-by: vankichi <kyukawa315@gmail.com> Signed-off-by: kpango <kpango@vdaas.org> Co-authored-by: Yusuke Kato <kpango@vdaas.org> Co-authored-by: Hiroto Funakoshi <hiroto.funakoshi.hiroto@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>