-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'master' into pre-agg-status
- Loading branch information
Showing
113 changed files
with
1,498 additions
and
398 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| .cubestore | ||
| .postgres |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| # Cube Store benchmark template | ||
|
|
||
| ## Setting up | ||
|
|
||
| * Update `cubejs-cubestore/docker-compose.yml` and `cubejs-postgres/docker-compose.yml` with data source credentials (see `TODO` comments) | ||
| * Update `cubejs-cubestore/schema` and `cubejs-postgres/schema` with relevant data schema that matches your data source | ||
| * Update `loadtest/queries.js` with relevant queries | ||
|
|
||
| ## Running | ||
|
|
||
| * Go to `cubejs-cubestore` and run `docker-compose -p cubejs-cubestore up` | ||
| * Go to `cubejs-postgres` and run `docker-compose -p cubejs-postgres up` | ||
| * Go to `loadtest` and run `npm install` | ||
| * Then, start the relay server using `npm start` | ||
| * Then, run the load test using `RPS=<requests per second> DURATION=<duration, seconds>s npm test` (e.g., `RPS=10 DURATION=10s npm test`) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| module.exports = { | ||
| orchestratorOptions: { | ||
| rollupOnlyMode: !process.env.CUBEJS_SCHEDULED_REFRESH_TIMER, | ||
| queryCacheOptions: { | ||
| externalQueueOptions: { | ||
| concurrency: 20 | ||
| } | ||
| }, | ||
| preAggregationsOptions: { | ||
| externalRefresh: process.env.CUBEJS_SCHEDULED_REFRESH_TIMER !== 'true' | ||
| } | ||
| } | ||
| } |
103 changes: 103 additions & 0 deletions
103
examples/cubestore-benchmark/cubejs-cubestore/docker-compose.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| version: '2.2' | ||
|
|
||
| services: | ||
| cube: | ||
| image: cubejs/cube:latest | ||
| ports: | ||
| - 4001:4000 | ||
| - 3001:3000 | ||
| environment: | ||
| - CUBEJS_DB_TYPE=bigquery | ||
| # TODO: Fill in the values below | ||
| - CUBEJS_DB_BQ_PROJECT_ID= | ||
| - CUBEJS_DB_BQ_CREDENTIALS= | ||
| - CUBEJS_DB_BQ_EXPORT_BUCKET= | ||
|
|
||
| - CUBEJS_EXT_DB_TYPE=cubestore | ||
| - CUBEJS_EXT_DB_HOST=cubestore | ||
| - CUBEJS_EXT_DB_PORT=3030 | ||
|
|
||
| - CUBEJS_REDIS_URL=redis://redis:6379 | ||
|
|
||
| - CUBEJS_API_SECRET=SECRET | ||
| volumes: | ||
| - .:/cube/conf | ||
| depends_on: | ||
| - refresh_worker | ||
| - cubestore | ||
| - redis | ||
|
|
||
| refresh_worker: | ||
| image: cubejs/cube:latest | ||
| environment: | ||
| - CUBEJS_SCHEDULED_REFRESH_TIMER=true | ||
| - CUBEJS_DB_TYPE=bigquery | ||
| # TODO: Fill in the values below | ||
| - CUBEJS_DB_BQ_PROJECT_ID= | ||
| - CUBEJS_DB_BQ_CREDENTIALS= | ||
| - CUBEJS_DB_BQ_EXPORT_BUCKET= | ||
|
|
||
| - CUBEJS_EXT_DB_TYPE=cubestore | ||
| - CUBEJS_EXT_DB_HOST=cubestore | ||
| - CUBEJS_EXT_DB_PORT=3030 | ||
|
|
||
| - CUBEJS_REDIS_URL=redis://redis:6379 | ||
|
|
||
| - CUBEJS_API_SECRET=SECRET | ||
| volumes: | ||
| - .:/cube/conf | ||
| depends_on: | ||
| - cubestore | ||
| - redis | ||
|
|
||
| cubestore: | ||
| image: cubejs/cubestore:latest | ||
| environment: | ||
| - CUBESTORE_WORKERS=cubestore_worker_1:10001,cubestore_worker_2:10002,cubestore_worker_3:10003,cubestore_worker_4:10004 | ||
| - CUBESTORE_REMOTE_DIR=/cube/data | ||
| volumes: | ||
| - .cubestore:/cube/data | ||
| depends_on: | ||
| - cubestore_worker_1 | ||
| - cubestore_worker_2 | ||
| - cubestore_worker_3 | ||
| - cubestore_worker_4 | ||
|
|
||
| cubestore_worker_1: | ||
| image: cubejs/cubestore:latest | ||
| environment: | ||
| - CUBESTORE_WORKER_PORT=10001 | ||
| - CUBESTORE_REMOTE_DIR=/cube/data | ||
| volumes: | ||
| - .cubestore:/cube/data | ||
|
|
||
| cubestore_worker_2: | ||
| image: cubejs/cubestore:latest | ||
| environment: | ||
| - CUBESTORE_WORKER_PORT=10002 | ||
| - CUBESTORE_REMOTE_DIR=/cube/data | ||
| volumes: | ||
| - .cubestore:/cube/data | ||
|
|
||
| cubestore_worker_3: | ||
| image: cubejs/cubestore:latest | ||
| environment: | ||
| - CUBESTORE_WORKER_PORT=10003 | ||
| - CUBESTORE_REMOTE_DIR=/cube/data | ||
| volumes: | ||
| - .cubestore:/cube/data | ||
|
|
||
| cubestore_worker_4: | ||
| image: cubejs/cubestore:latest | ||
| environment: | ||
| - CUBESTORE_WORKER_PORT=10004 | ||
| - CUBESTORE_REMOTE_DIR=/cube/data | ||
| volumes: | ||
| - .cubestore:/cube/data | ||
|
|
||
| redis: | ||
| image: bitnami/redis:latest | ||
| environment: | ||
| - ALLOW_EMPTY_PASSWORD=yes | ||
| logging: | ||
| driver: none |
83 changes: 83 additions & 0 deletions
83
examples/cubestore-benchmark/cubejs-cubestore/schema/Mobility.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| cube(`Mobility`, { | ||
| sql: ` | ||
| SELECT * | ||
| FROM \`bigquery-public-data.covid19_google_mobility.mobility_report\` | ||
| `, | ||
|
|
||
| refreshKey: { | ||
| sql: ` | ||
| SELECT COUNT(*) | ||
| FROM \`bigquery-public-data.covid19_google_mobility.mobility_report\` | ||
| `, | ||
| }, | ||
|
|
||
| measures: { | ||
| grocery: { | ||
| sql: `grocery_and_pharmacy_percent_change_from_baseline`, | ||
| type: `max`, | ||
| format: 'percent', | ||
| }, | ||
|
|
||
| park: { | ||
| sql: `parks_percent_change_from_baseline`, | ||
| type: `max`, | ||
| format: 'percent', | ||
| }, | ||
|
|
||
| residential: { | ||
| sql: `residential_percent_change_from_baseline`, | ||
| type: `max`, | ||
| format: 'percent', | ||
| }, | ||
|
|
||
| retail: { | ||
| sql: `retail_and_recreation_percent_change_from_baseline`, | ||
| type: `max`, | ||
| format: 'percent', | ||
| }, | ||
|
|
||
| transit: { | ||
| sql: `transit_stations_percent_change_from_baseline`, | ||
| type: `max`, | ||
| format: 'percent', | ||
| }, | ||
|
|
||
| workplace: { | ||
| sql: `workplaces_percent_change_from_baseline`, | ||
| type: `max`, | ||
| format: 'percent', | ||
| }, | ||
| }, | ||
|
|
||
| dimensions: { | ||
| key: { | ||
| sql: `CONCAT(country_region, '-', sub_region_1, '-', sub_region_2, '-', ${Mobility}.date)`, | ||
| type: `string`, | ||
| primaryKey: true | ||
| }, | ||
|
|
||
| country: { | ||
| sql: `country_region`, | ||
| type: `string` | ||
| }, | ||
|
|
||
| date: { | ||
| sql: `TIMESTAMP(${Mobility}.date)`, | ||
| type: `time` | ||
| }, | ||
| }, | ||
|
|
||
| preAggregations: { | ||
| main: { | ||
| type: `rollup`, | ||
| external: true, | ||
| scheduledRefresh: true, | ||
| refreshKey: { every: '1 hour' }, | ||
| measureReferences: [ grocery, park, residential, retail, transit, workplace ], | ||
| dimensionReferences: [ country ], | ||
| timeDimensionReference: date, | ||
| granularity: 'day', | ||
| partitionGranularity: 'month', | ||
| }, | ||
| }, | ||
| }); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| module.exports = { | ||
| orchestratorOptions: { | ||
| rollupOnlyMode: !process.env.CUBEJS_SCHEDULED_REFRESH_TIMER, | ||
| queryCacheOptions: { | ||
| externalQueueOptions: { | ||
| concurrency: 20 | ||
| } | ||
| }, | ||
| preAggregationsOptions: { | ||
| externalRefresh: process.env.CUBEJS_SCHEDULED_REFRESH_TIMER !== 'true' | ||
| } | ||
| } | ||
| } |
73 changes: 73 additions & 0 deletions
73
examples/cubestore-benchmark/cubejs-postgres/docker-compose.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| version: '2.2' | ||
|
|
||
| services: | ||
| cube: | ||
| image: cubejs/cube:latest | ||
| ports: | ||
| - 4002:4000 | ||
| - 3002:3000 | ||
| environment: | ||
| - CUBEJS_DB_TYPE=bigquery | ||
| # TODO: Fill in the values below | ||
| - CUBEJS_DB_BQ_PROJECT_ID= | ||
| - CUBEJS_DB_BQ_CREDENTIALS= | ||
| - CUBEJS_DB_BQ_EXPORT_BUCKET= | ||
|
|
||
| - CUBEJS_EXT_DB_TYPE=postgres | ||
| - CUBEJS_EXT_DB_HOST=postgres | ||
| - CUBEJS_EXT_DB_PORT=5432 | ||
| - CUBEJS_EXT_DB_NAME=postgres | ||
| - CUBEJS_EXT_DB_USER=postgres | ||
| - CUBEJS_EXT_DB_PASS=postgres | ||
|
|
||
| - CUBEJS_REDIS_URL=redis://redis:6379 | ||
|
|
||
| - CUBEJS_API_SECRET=SECRET | ||
| volumes: | ||
| - .:/cube/conf | ||
| depends_on: | ||
| - refresh_worker | ||
| - postgres | ||
| - redis | ||
|
|
||
| refresh_worker: | ||
| image: cubejs/cube:latest | ||
| environment: | ||
| - CUBEJS_SCHEDULED_REFRESH_TIMER=true | ||
| - CUBEJS_DB_TYPE=bigquery | ||
| # TODO: Fill in the values below | ||
| - CUBEJS_DB_BQ_PROJECT_ID= | ||
| - CUBEJS_DB_BQ_CREDENTIALS= | ||
| - CUBEJS_DB_BQ_EXPORT_BUCKET= | ||
|
|
||
| - CUBEJS_EXT_DB_TYPE=postgres | ||
| - CUBEJS_EXT_DB_HOST=postgres | ||
| - CUBEJS_EXT_DB_PORT=5432 | ||
| - CUBEJS_EXT_DB_NAME=postgres | ||
| - CUBEJS_EXT_DB_USER=postgres | ||
| - CUBEJS_EXT_DB_PASS=postgres | ||
|
|
||
| - CUBEJS_REDIS_URL=redis://redis:6379 | ||
|
|
||
| - CUBEJS_API_SECRET=SECRET | ||
| volumes: | ||
| - .:/cube/conf | ||
| depends_on: | ||
| - postgres | ||
| - redis | ||
|
|
||
| postgres: | ||
| image: postgres:latest | ||
| environment: | ||
| POSTGRES_USER: postgres | ||
| POSTGRES_PASSWORD: postgres | ||
| PGDATA: /data/postgres | ||
| volumes: | ||
| - .postgres:/data/postgres | ||
|
|
||
| redis: | ||
| image: bitnami/redis:latest | ||
| environment: | ||
| - ALLOW_EMPTY_PASSWORD=yes | ||
| logging: | ||
| driver: none |
Oops, something went wrong.