Design of Continuous Benchmark

[kmrmt]

Background

• refer notion

Functional requirements

must

• Execute the whole method in Vald (Read: Search, GetObject, Exists and Write: Insert, Update, Remove, Upsert)
  • Multi and Stream are the same as above
• plugable request pattern
  • load continuously
  • emulate user access pattern
    • specify RPC of each operation
      • e.g. peak time, running some periodic batches, etc.
    • a condition of starting operation
      • elapsed time
      • number of the vectors in Vald
    • error ratio
• calculate measurable metrics in the benchmark tool
  • Latency, Throughput(whole methods)
  • Recall (Search)
• use request data easily (1)
  • random values (uniform, gaussian)
  • ann-benchmarks
    • use the same vector with a different ID many times

nice to have

• Kubernetes Job
• use request data easily (2)
  • big-ann-benchmarks
    • same as ann-benchmarks
  • real dataset
    • s3, hbase, MySQL, etc.
• plugable request pattern with a configuration file
• result visualization
• create test data
  • create using LinearSearch API
• support Ingress/Egress Filter
  • request image or text directly
• log
  • execution log
• notification
  • e.g. slack

[kmrmt]

discussion

• repository

  • a cluster operator can use easily

• focus on benchmark

  • provisioning is other task

• tools

  • benchmarker
    • operator (not k8s)
    • job
  • metrics collector
  • visualizer
  • notificator
  • etc.

[kmrmt]

benchmark tool design

package structure

job

job package has the benchmarks of each vald methods

• job/read
  • Search
  • Exists
  • GetObject
• job/write
  • Insert
  • Update
  • Remove
  • Upsert

senario

senario package represents the benchmark request pattern.

• operator
  • constant
  • once
  • repeat
  • random
  • probability

metrics

metrics package has the ANN's benchmark metrics and their collector.

• collector
• result
• metrics/calculate
  • Latency
  • Throughput
  • Recall
• metrics/grpc
  • intercepter

dataset

dataset package has the request dataset generators and loaders

• dataset/random
  • uniform
  • gaussian
• dataset/pretrained
  • fashion-mnist
  • glove
  • mnist
  • sift
  • deep1B
  • etc.
• dataset/external
  • s3
  • hbase
  • MySQL
  • etc.

visualizer

visualizer package has benchmark result visualize methods.

• visualize

notify

notify package has the benchmark progress notifiers for the users.

• slack
• email
• etc.

dependencies

graph TD
    subgraph benchmarker
        benchmark --build--> senario/operator
        senario/operator -- output--> result
        senario/operator --create--> job1
        senario/operator --create--> job2
        senario/operator --create--> job3
        benchmark --load--> dataset
        benchmark --initialize--> metrics/grpc/interceptor
        metrics/grpc/interceptor --send--> metrics/collector
        senario/operator --> notify
    end
    subgraph vald-cluster
        job1 --request--> vald-gateway
        job2 --request--> vald-gateway
        job3 --request--> vald-gateway
        vald-gateway --> vald-agent
    end
    subgraph visualizer
        visualize --load--> result
    end
        notify --send--> slack
        notify --send--> email

relationships

erDiagram
    benchmark ||--|{ senario-operator : build
    benchmark ||--|| dataset: load
    senario-operator ||--|{ job : create
    senario-operator ||--|| notify : call
    senario-operator ||--|| result : output
    job ||--|| dataset : read
    benchmark ||--|| grpc-interceptor : init
    grpc-interceptor ||--|{ metrics-collector : send
    grpc-interceptor ||--|| job : use

priority

1. operator, job, dataset(random)
2. metrics/grpc/interceptor, metrics/collector
3. visualize
4. notifier

[kmrmt]

@vankichi metrics/grpc/interceptor measures latency and throughput.
And I try to describe the relationship.

@kevindiu There is no reason, so it is good for me to implement flat package.
If there is no problem about the package architecture, we will prototype the interfaces.

[kmrmt]

add relationships

[kevindiu]

@kmrmt sorry but can you let me know why we split job/read and job/write?

[hlts2]

@kmrmt Thank you for your design.
I have one question. How many components are there? 🤔
benchmark-operator, benchmark-job ??

[vankichi]

@kevindiu
IMO, it is just package design, not component layer design.

@hlts2
I suppose it is one component benchmark.
The benchmark component creates the job Pod for each job.
My rough image is below.

+-----------+    +-------------+      +--------------+
| benchmark | -> | k8s Job Pod |  <-> | Vald cluster |
+-----------+    +-------------+      +--------------+
                        |
                 +-----------+
                 | Prometeus |
                 +-----------+

[kmrmt]

benchmark tool design

package structure

job

job package has the benchmarks of each vald methods.

job can execute benchmark standalone.

And job output logs for calculate metrics.

• job/read
  • Search
  • Exists
  • GetObject
• job/write
  • Insert
  • Update
  • Remove
  • Upsert
• configuration

senario

senario package represents the benchmark request pattern.
senario-operator reconciles the jobs.

• operator
  • constant
  • once
  • repeat
  • random
  • probability
  • configuration

dataset

dataset package has the request dataset generators and loaders.

• dataset/random
  • uniform
  • gaussian
• dataset/pretrained
  • fashion-mnist
  • glove
  • mnist
  • sift
  • deep1B
  • etc.
• dataset/external
  • s3
  • hbase
  • MySQL
  • etc.

notify

notify package has the benchmark progress notifiers for the users.

• slack
• email
• etc.

grafaba-dashboard

This is not Go program.

dependencies

graph TD
    subgraph benchmark
        senario-operator --reconcile--> job
        notify-operator --watch--> job
        job --read--> dataset
    end
    job -- output--> log
    prometheus --read and calculate--> log
    subgraph visualize
        grafana --read--> prometheus
    end
    subgraph vald-cluster
        job --request--> vald-gateway
        vald-gateway --> vald-agent
    end
    notify-operator --send--> slack
    notify-operator --send--> email

relationships

erDiagram
    senario-operator ||--|{ job : ""
    job ||--|| dataset : ""
    job ||--|{ log : ""
    notify-operator ||--|{ job : ""
    prometheus ||--|{ log : ""
    grafana ||--|{ prometheus : ""

priority

1. job, dataset(random)
2. senario-operator
3. grafana-dashboard
4. notify-operator