Skip to content
負荷試験
Scala Shell HCL Other
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci
boot/src/main
domain/src/main/scala/com/github/BambooTuna/LoadTest/domain
e2e-test/src/it/scala
infrastructure/staging/gcp
interface/src/main/scala/com/github/BambooTuna/LoadTest/adaptor
project
tools
useCase/src/main/scala/com/github/BambooTuna/LoadTest
.gitignore
.scalafmt.conf
README.md
build.sbt
create_html_from_logs.sh
deploy.sh
docker-compose.yml
sample_user.csv
setup-local-k8s.sh

README.md

LoadTest

高負荷に耐えるためにはどうしたらいいかを調べていく アドテクコンペでDSPサーバーを作ったので、それで負荷試験を行う

Test

Local setup

  1. Build Use sbt-native-packager to build docker image.
$ sbt docker:publishLocal

//$ sbt docker:stage
  1. Run
// Use k8s
$ sh setup-local-k8s.sh

// Use docker-compose
$ docker-compose up -d --build
...

E2E Test

$ sbt it:test

Curl test

$ curl -X POST -H "Content-Type: application/json" -d '{"data":[{"device_id":"1","advertiser_id":1,"game_install_count":1},{"device_id":"2","advertiser_id":1,"game_install_count":1}]}' http://localhost:8080/user/add
$ curl -X POST -H "Content-Type: application/json" -d '{"device_id":"1"}' http://localhost:8080/user/get
$ curl -X POST -H "Content-Type: application/json" -d '{"id":"1","timestamp":1234567890,"device_id":"1","banner_size":1,"media_id":1,"os_type":1,"banner_position":1,"floor_price":1.0}' http://localhost:8080/bid_request


$ curl -X POST -H "Content-Type: application/json" -d '{"id":"","price":1.21,"is_click":0}' http://localhost:8080/win

  • SetBudget
curl -X POST -H "Content-Type: application/json" -d '{"advertiser_id":1,"event_type":0,"price":10000}' http://localhost:8080/budget/set
curl -X POST -H "Content-Type: application/json" -d '{"advertiser_id":1,"event_type":1,"price":0}' http://localhost:8080/budget/set
  • Mysql
$ curl localhost:8080/ping
{"message":"pong","error_messages":[]}

$ curl -X POST -H "Content-Type: application/json" -d '{"name":"bambootuna","age":20}' localhost:8080/user/add
{"id":{"user_id":-4934742503425198802},"error_messages":[]}

$ curl -X GET -H "Content-Type: application/json" -d '{"user_id":1}' localhost:8080/user/get
{"user":{"user_id":6795494702222648856,"name":"bambootuna","age":20},"error_messages":[]}
  • Redis
$ curl -X POST -H "Content-Type: application/json" -d '{"name":"bambootuna","age":20}' localhost:8080/redis/user/add
{"id":{"user_id":-4934742503425198802},"error_messages":[]}

$ curl -X GET -H "Content-Type: application/json" -d '{"user_id":1}' localhost:8080/redis/user/get
{"user":{"user_id":6795494702222648856,"name":"bambootuna","age":20},"error_messages":[]}
  • Aerospike
$ curl -X POST -H "Content-Type: application/json" -d '{"name":"bambootuna","age":20}' localhost:8080/aerospike/user/add
{"id":{"user_id":-4934742503425198802},"error_messages":[]}

$ curl -X GET -H "Content-Type: application/json" -d '{"user_id":1}' localhost:8080/aerospike/user/get
{"user":{"user_id":6795494702222648856,"name":"bambootuna","age":20},"error_messages":[]}

コンテナに入ってデータの確認をする SELECT 1でデータベースを指定している所に注意!

$ docker exec -it [Container name] sh
$ redis-cli
127.0.0.1:6379> SELECT 1
127.0.0.1:6379[1]> SET [Key name] [Value]
127.0.0.1:6379[1]> GET [Key name]

Gatling Test

  • From local
$ sbt gatling-runner/docker:publishLocal
...

$ docker run \
-e GATLING_BASE_URL=http://35.200.14.159:8080 \
-e GATLING_USERS=1000 \
--rm bambootuna/gatling-runner
...
$ docker cp [Conteiner ID]:/opt/docker/tools/gatling-runner/target/gatling ./tools/gatling-runner/target/

//See docker status(cpu : memory)
$ docker stats [Conteiner ID]
  • from server
$ gcloud container clusters get-credentials gatling-runner-cluster --zone asia-northeast1-a --project [Your project id]
$ cd infrastructure/staging/gcp/k8s
$ kubectl apply -f ./gatling-job.yml
$ kubectl delete job gatling-runner
  • Create html report from simulation.log tools/gatling-runner/target/gatling/pingsimulation-oooooooooooooooにログファイルを入れる
    名前が被らないように適当にインデックスをつけて以下を実行するとHTML形式ファイルが生成される
$ sbt gatling-runner/gatling:generateReport

$ sh create_html_from_logs.sh

Locust Test

$ cd tools/locust
$ LOCUST_FILE_PATH="main.py" LOCUST_HOST=http://localhost:8080 sh setup.sh

Access localhost:8089 and run the test.

Other

  • See JMX Metrics
$ jconsole service:jmx:rmi:///jndi/rmi://localhost:8999/jmxrmi
  • Too many connections error(mysql)
    • Check max connections show variables like "%max_connections%";

    • set max connections set global max_connections = 1000;

    • Check connection process show processlist;

    • Change my.cnf and apply service mysqld restart or systemctl restart mysqld

Staging

CommonSetting

  1. enable Compute Engine API
    Go to API & Service and enable Compute Engine API.

  2. Adding credentials to a project Using API => Computer Engine API Are you planning to use this API on App Engine or Compute Engine? => Yes, using one or both.

  3. Create Service Account
    Go to credentials and create service account.
    Service Account => Compute Engin default service account
    Key Type => Json
    Rename credentials json file to account.json.
    Move account.json to infrastructure/staging/gcp/terraform/account.json

  4. Install and setting Google-Cloud-SDK

$ brew cask install google-cloud-sdk
$ gcloud config set project [your project id]
$ gcloud config set compute/region asia-northeast1
$ gcloud config set compute/zone asia-northeast1-a
$ gcloud auth activate-service-account --key-file=infrastructure/staging/gcp/terraform/account.json
  1. Add role IAM Compute Engine default service account <- Kubernetes Engine 管理者

CircleCI->GCR->GKE

CircleCI
|
|-Use terraform to create GKE[k8s-cluster]
|
|-Push docker-image to GCR
|
|-Use helm to deploy datadog-agent to GKE
|
|-Use deployment.yml to deploy db, app to GKE
|
Done

  1. Enable google services
// GKE
$ gcloud services enable container.googleapis.com
// GCR
$ gcloud services enable containerregistry.googleapis.com
// GRM
$ gcloud services enable cloudresourcemanager.googleapis.com
  1. Encode account json to string
$ base64 -i infrastructure/staging/gcp/terraform/account.json
1234567890asdfghjkl
  1. Set encoded text to CircleCI env.
KeyName -> GCLOUD_SERVICE_KEY
Value -> 1234567890asdfghjkl

// [Datadog](https://app.datadoghq.com/account/settings#api) -> API Keys -> Create API Keys
KeyName -> DD_API_KEY
Value -> 1234567890asdfghjkl
  1. Init terraform from local If you have not created [GKE GRM] yet, do next command.
//Install terraform.
$ brew install terraform
...

//Create your environment useing terraform.
$ sh infrastructure/staging/gcp/terraform/init.sh
...
It will take some time.
...

GoogleCloudBuilder->GKE

  1. Enable some services
// GCB
$ gcloud services enable cloudbuild.googleapis.com
// GKE
$ gcloud services enable container.googleapis.com
// GCR
$ gcloud services enable containerregistry.googleapis.com
// GSR
$ gcloud services enable sourcerepo.googleapis.com
  1. Register your Github repository
    GSR
    The registered repository name would be used in [6].
    ex. https://source.cloud.google.com/[your project id]/[repository name]

  2. Install terraform

$ brew install terraform
$ cd infrastructure/staging/gcp/terraform
  1. Edit main.tf
    The repo_name should be replaced to [repository name].
variable "repo_name" {
    default = "[repository name]"
}
  1. Run terraform
$ terraform init
...
$ terraform plan
...
$ terraform apply
...
  1. Let's push something to your Github repository! After pushed, see next page.
    GCB
    Cloud builder should e moving.

References

You can’t perform that action at this time.