Merge branch 'web-analytics' of github.com:statsbotco/cube.js into we…

…b-analytics * 'web-analytics' of github.com:statsbotco/cube.js: (94 commits) Update dependencies Update dependencies Update web-analytics v0.18.0 fix: Redis query queue locking redesign Update Subquery.md tests: Cleanup redis pool properly in tests (#458) Debug CI Update client core dependency for vue tests Fix vue tests and client build Fix tests race conditions fix: Handle multiple occurrences in the first event of a funnel: conversion percent discrepancies. feat: Redis connection pooling (#433) Thanks to @jcw! build(deps): bump moment-range from 4.0.1 to 4.0.2 (#442) feat: Add role parameter to driver options (#448) Thanks to @smbkr! build(deps-dev): bump request from 2.88.0 to 2.88.2 (#441) build(deps): bump recharts from 1.7.1 to 1.8.5 (#446) build(deps): bump sqlite3 from 4.1.0 to 4.1.1 (#445) build(deps): bump tslib from 1.9.3 to 1.11.0 (#440) build(deps-dev): bump http-server from 0.11.1 to 0.12.1 (#437) ...
cube-js · Mar 12, 2020 · f621607 · f621607
2 parents d9b657b + dbde04b
commit f621607
Show file tree

Hide file tree

Showing 147 changed files with 30,143 additions and 22,150 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -3,7 +3,7 @@ version: 2
 jobs:
   test:
     docker:
-      - image: circleci/node:10.15
+      - image: circleci/node:10.16
         environment:
           TEST_PG_USER: root
           TEST_CLICKHOUSE_HOST: localhost
@@ -47,7 +47,7 @@ jobs:
 
   lint:
     docker:
-      - image: circleci/node:10.15
+      - image: circleci/node:10.16
 
     working_directory: ~/repo
 

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,6 +3,83 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [0.18.0](https://github.com/cube-js/cube.js/compare/v0.17.10...v0.18.0) (2020-03-01)
+
+
+### Bug Fixes
+
+* Error: client.readOnly is not a function ([6069499](https://github.com/cube-js/cube.js/commit/6069499))
+* External rollup type conversions: cast double to decimal for postgres ([#421](https://github.com/cube-js/cube.js/issues/421)) Thanks to [@sandeepravi](https://github.com/sandeepravi)! ([a19410a](https://github.com/cube-js/cube.js/commit/a19410a))
+* **athena-driver:** Remove debug output ([f538135](https://github.com/cube-js/cube.js/commit/f538135))
+* Handle missing body-parser error ([b90dd89](https://github.com/cube-js/cube.js/commit/b90dd89))
+* Handle multiple occurrences in the first event of a funnel: conversion percent discrepancies. ([0989482](https://github.com/cube-js/cube.js/commit/0989482))
+* Handle primaryKey shown: false pitfall error ([5bbf5f0](https://github.com/cube-js/cube.js/commit/5bbf5f0))
+* Redis query queue locking redesign ([a2eb9b2](https://github.com/cube-js/cube.js/commit/a2eb9b2)), closes [#459](https://github.com/cube-js/cube.js/issues/459)
+* TypeError: Cannot read property 'queryKey' of null under load ([0c996d8](https://github.com/cube-js/cube.js/commit/0c996d8))
+
+
+### Features
+
+* Add role parameter to driver options ([#448](https://github.com/cube-js/cube.js/issues/448)) Thanks to [@smbkr](https://github.com/smbkr)! ([9bfb71d](https://github.com/cube-js/cube.js/commit/9bfb71d)), closes [#447](https://github.com/cube-js/cube.js/issues/447)
+* COMPILE_CONTEXT and async driverFactory support ([160f931](https://github.com/cube-js/cube.js/commit/160f931))
+* Redis connection pooling ([#433](https://github.com/cube-js/cube.js/issues/433)) Thanks to [@jcw](https://github.com/jcw)! ([cf133a9](https://github.com/cube-js/cube.js/commit/cf133a9)), closes [#104](https://github.com/cube-js/cube.js/issues/104)
+
+
+
+
+
+## [0.17.10](https://github.com/cube-js/cube.js/compare/v0.17.9...v0.17.10) (2020-02-20)
+
+
+### Bug Fixes
+
+* Revert "feat: Bump corejs ([#378](https://github.com/cube-js/cube.js/issues/378))" ([b21cbe6](https://github.com/cube-js/cube.js/commit/b21cbe6)), closes [#418](https://github.com/cube-js/cube.js/issues/418)
+* uuidv4 upgrade ([c46c721](https://github.com/cube-js/cube.js/commit/c46c721))
+
+
+### Features
+
+* **cubejs-cli:** Add node_modules to .gitignore ([207544b](https://github.com/cube-js/cube.js/commit/207544b))
+* Support external rollups from readonly source ([#395](https://github.com/cube-js/cube.js/issues/395)) ([b17e841](https://github.com/cube-js/cube.js/commit/b17e841))
+
+
+
+
+
+## [0.17.9](https://github.com/cube-js/cube.js/compare/v0.17.8...v0.17.9) (2020-02-18)
+
+
+### Features
+
+* Add .gitignore with .env content to templates.js ([#403](https://github.com/cube-js/cube.js/issues/403)) ([c0d1a76](https://github.com/cube-js/cube.js/commit/c0d1a76)), closes [#402](https://github.com/cube-js/cube.js/issues/402)
+* Bump corejs ([#378](https://github.com/cube-js/cube.js/issues/378)) ([cb8d51c](https://github.com/cube-js/cube.js/commit/cb8d51c))
+* Enhanced trace logging ([1fdd8e9](https://github.com/cube-js/cube.js/commit/1fdd8e9))
+* Extend meta response with aggregation type ([#394](https://github.com/cube-js/cube.js/issues/394)) Thanks to [@pyrooka](https://github.com/pyrooka)! ([06eed0b](https://github.com/cube-js/cube.js/commit/06eed0b))
+* Request id trace span ([880f65e](https://github.com/cube-js/cube.js/commit/880f65e))
+
+
+
+
+
+## [0.17.8](https://github.com/cube-js/cube.js/compare/v0.17.7...v0.17.8) (2020-02-14)
+
+
+### Bug Fixes
+
+* typings export ([#373](https://github.com/cube-js/cube.js/issues/373)) Thanks to [@lvauvillier](https://github.com/lvauvillier)! ([f4ea839](https://github.com/cube-js/cube.js/commit/f4ea839))
+* Wrong interval functions for BigQuery ([#367](https://github.com/cube-js/cube.js/issues/367)) Thanks to [@lvauvillier](https://github.com/lvauvillier)! ([0e09d4d](https://github.com/cube-js/cube.js/commit/0e09d4d))
+* **@cubejs-backend/oracle-driver:** a pre-built node-oracledb binary was not found for Node.js v12.16.0 ([#375](https://github.com/cube-js/cube.js/issues/375)) ([fd66bb6](https://github.com/cube-js/cube.js/commit/fd66bb6)), closes [#370](https://github.com/cube-js/cube.js/issues/370)
+* **@cubejs-client/core:** improve types ([#376](https://github.com/cube-js/cube.js/issues/376)) Thanks to [@hassankhan](https://github.com/hassankhan)! ([cfb65a2](https://github.com/cube-js/cube.js/commit/cfb65a2))
+
+
+### Features
+
+* Athena HLL support ([45c7b83](https://github.com/cube-js/cube.js/commit/45c7b83))
+
+
+
+
+
 ## [0.17.7](https://github.com/cube-js/cube.js/compare/v0.17.6...v0.17.7) (2020-02-12)
 
 

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -20,7 +20,7 @@ Cube.js works with Node.js 8+ and uses yarn as a package manager.
 ## Development Workflow
 ### Cube.js Client
 
-1. After cloning Cube.js repository run `$ yarn` in `packages/cubejs-client-core` and `packages/cubejs-react` to install dependencies.
+1. After cloning Cube.js repository run `$ yarn` in `packages/cubejs-client-core` and `packages/cubejs-client-react` to install dependencies.
 2. Use `$ yarn link` to add these packages to link registry.
 3. Perform required code changes.
 4. Use `$ yarn build` in the repository root to build CommonJS and UMD modules.
@@ -61,6 +61,17 @@ In case you need to tweak it a little bit please follow [Implementing Driver](#i
 In order to run tests in `cubejs-schema-compiler` package you need to have running [Docker](https://docs.docker.com/install/) on your machine.
 When it's up and running just use `$ npm test` in `packages/cubejs-schema-compiler` to execute tests.
 
+### Linking Server Core for Development
+
+It's convenient to link `@cubejs-backend/server-core` into your project for manual tests of changes of backend code.
+Cube.js uses `yarn` as package manager instead of `npm`.
+In order to link `@cubejs-backend/server-core`:
+
+1. Install yarn: `npm install -g yarn`.
+2. Link server-core package: `yarn link` inside `packages/cubejs-server-core`.
+3. Link all drivers and dependent packages where you make changes in `packages/cubejs-server-core`.
+4. Run `yarn build` in `packages/cubejs-playground`.
+
 ## Style guides
 
 We're passionate about what code can do rather how it's formatted.

diff --git a/docs/Cube.js-Backend/Connecting-to-the-Database.md b/docs/Cube.js-Backend/Connecting-to-the-Database.md
@@ -47,7 +47,7 @@ The table below shows which environment variables are used for different databas
 | AWS Athena | `CUBEJS_AWS_KEY`, `CUBEJS_AWS_SECRET`, `CUBEJS_AWS_REGION`, `CUBEJS_AWS_S3_OUTPUT_LOCATION` |
 | Google Bigquery | `CUBEJS_DB_BQ_PROJECT_ID`, `CUBEJS_DB_BQ_KEY_FILE or CUBEJS_DB_BQ_CREDENTIALS` |
 | MongoDB | `CUBEJS_DB_HOST`, `CUBEJS_DB_NAME`, `CUBEJS_DB_PORT`, `CUBEJS_DB_USER`, `CUBEJS_DB_PASS`, `CUBEJS_DB_SSL`, `CUBEJS_DB_SSL_CA`, `CUBEJS_DB_SSL_CERT`, `CUBEJS_DB_SSL_CIPHERS`, `CUBEJS_DB_SSL_PASSPHRASE` |
-| Snowflake | `CUBEJS_DB_SNOWFLAKE_ACCOUNT`, `CUBEJS_DB_SNOWFLAKE_REGION`, `CUBEJS_DB_SNOWFLAKE_WAREHOUSE`, `CUBEJS_DB_NAME`, `CUBEJS_DB_USER`, `CUBEJS_DB_PASS`|
+| Snowflake | `CUBEJS_DB_SNOWFLAKE_ACCOUNT`, `CUBEJS_DB_SNOWFLAKE_REGION`, `CUBEJS_DB_SNOWFLAKE_WAREHOUSE`, `CUBEJS_DB_SNOWFLAKE_ROLE`, `CUBEJS_DB_NAME`, `CUBEJS_DB_USER`, `CUBEJS_DB_PASS`|
 | Presto | `CUBEJS_DB_HOST`, `CUBEJS_DB_PORT`, `CUBEJS_DB_CATALOG`, `CUBEJS_DB_SCHEMA`, `CUBEJS_DB_USER`, `CUBEJS_DB_PASS` |
 
 ## Notes

diff --git a/docs/Cube.js-Backend/REST-API.md b/docs/Cube.js-Backend/REST-API.md
@@ -212,6 +212,7 @@ Example response:
           shortTitle:"Count",
           aliasName:"users.count",
           type:"number",
+          aggType:"count"
           drillMembers:[
             "Users.id",
             "Users.city",

diff --git a/docs/Cube.js-Frontend/Query-Format.md b/docs/Cube.js-Frontend/Query-Format.md
@@ -30,7 +30,7 @@ Query has the following properties:
     fields to order is based on the order of the keys in the object.
 - `timezone`: All time based calculations performed within Cube.js are timezone-aware. Using this property you can set your desired timezone in [TZ Database Name](https://en.wikipedia.org/wiki/Tz_database) format, e.g.: `America/Los_Angeles`. The default value is `UTC`.
 - `renewQuery`: If `renewQuery` is set to `true`, query will always refresh cache and return the latest data from the database. The default value is `false`.
-- `ungrouped`: If `ungrouped` is set to `true` no `GROUP BY` statement will be added to the query and raw results after filtering and joining will be returned. 
+- `ungrouped`: If `ungrouped` is set to `true` no `GROUP BY` statement will be added to the query and raw results after filtering and joining will be returned without grouping. 
 By default `ungrouped` query requires to pass primary key as a dimension of every cube involved in query for security purpose.
 To disable this behavior please see [allowUngroupedWithoutPrimaryKey](@cubejs-backend-server-core#options-reference-allow-ungrouped-without-primary-key) server option. 
 

diff --git a/docs/Guides/Subquery.md b/docs/Guides/Subquery.md
@@ -29,7 +29,6 @@ GROUPD BY 1
 Cube.js makes subqueries easy and efficient. Subqueries are defined as regular dimensions with the parameter `subQuery` set to true.
 
 ```javascript
-
 cube(`Deals`, {
   sql: `select * from deals`,
 
@@ -50,8 +49,21 @@ cube(`SalesManagers`, {
       sql: `${SalesManagers}.id = ${Deals}.sales_manager_id`
     }
   },
+
+  measures: {
+    averageDealAmount: {
+      sql: `${dealsAmount}`,
+      type: `avg`
+    }
+  },
 
   dimensions: {
+    id: {
+      sql: `id`,
+      type: `string`,
+      primaryKey: true
+    },
+
     dealsAmount: {
       sql: `${Deals.amount}`,
       type: `number`,
@@ -63,14 +75,44 @@ cube(`SalesManagers`, {
 You can **reference subquery dimensions in measures as usual dimensions**. The example below shows the definition of an average deal amount per sales manager:
 
 ```javascript
-
 cube(`SalesManagers`, {
-   measures: {
-      averageDealsAmount: {
-        sql: `${dealsAmount}`,
-        type: `avg`
-      }
-   }
+  measures: {
+    averageDealsAmount: {
+      sql: `${dealsAmount}`,
+      type: `avg`
+    }
+  },
+
+  dimensions: {
+    id: {
+      sql: `id`,
+      type: `string`,
+      primaryKey: true
+    }
+  }
 });
 ```
 
+## Under the hood
+
+Based on sub query dimension definition, Cube.js will create a query that will include primary key dimension of main cube and all measures and dimensions included in sql definition of sub query dimension.
+This query will be joined as a left join to the main SQL query.
+For example for `SalesManagers.dealsAmount` sub query dimension following query will be generated:
+
+```javascript
+{
+  measures: ['SalesManagers.dealsAmount'],
+  dimensions: ['SalesManagers.id']
+}
+```
+
+In case of `{ measures: ['SalesManagers.averageDealAmount'] }` query following SQL will be generated:
+
+```javascript
+SELECT avg(sales_managers__average_deal_amount) FROM sales_managers
+LEFT JOIN (
+  SELECT sales_managers.id sales_managers__id, sum(deals.amount) sales_managers__average_deal_amount FROM sales_managers
+  LEFT JOIN deals ON sales_managers.id = deals.sales_manager_id
+  GROUP BY 1
+) sales_managers__average_deal_amount_subquery ON sales_managers__average_deal_amount_subquery.sales_managers__id = sales_managers.id
+```
diff --git a/docs/Schema/pre-aggregations.md b/docs/Schema/pre-aggregations.md
@@ -97,7 +97,7 @@ cube(`Orders`, {
       sql: `amount`,
       type: `sum`
     },
-    
+
     averageRevenue: {
       sql: `${revenue} / ${count}`,
       type: `number`
@@ -271,6 +271,28 @@ cube(`Orders`, {
 In order to make external pre-aggregations work you should set
 [externalDriverFactory](@cubejs-backend-server-core#external-driver-factory) and [externalDbType](@cubejs-backend-server-core#external-db-type) params while creating your server instance.
 
+Note that by default, Cube.js materializes the pre-aggregration query results as new tables in the source database. For external pre-aggregations, these source tables are temporary - once downloaded and uploaded to the external database, they are cleaned-up.
+
+However, it may not be possible to stage pre-aggregation query results in materialized tables in the source database like this - for example, if the driver doesn't support it, or if your source database is read-only. To fallback to a strategy where the pre-aggreation query results are downloaded without first being materialized, set the `readOnly` param of [driverFactory](@cubejs-backend-server-core#driver-factory) in your configuration:
+
+```javascript
+const CubejsServer = require('@cubejs-backend/server');
+const PostgresDriver = require('@cubejs-backend/postgres-driver');
+
+const options = {
+  driverFactory: () => new PostgresDriver({
+    readOnly: true
+  }),
+  externalDbType: 'postgres',
+  externalDriverFactory: () => new PostgresDriver({
+    host: 'my_host',
+    database: 'my_db',
+    user: 'my_user',
+    password: 'my_pw'
+  })
+};
+```
+
 ## refreshKey
 
 Cube.js also takes care of keeping pre-aggregations up to date.
@@ -320,7 +342,7 @@ In case of partitioned rollups incremental `refreshKey` can be used as follows:
 ```javascript
 cube(`Orders`, {
   sql: `select * from orders`,
-  
+
   // ...
 
   preAggregations: {
@@ -358,7 +380,7 @@ Example usage:
 ```javascript
 cube(`Orders`, {
   sql: `select * from orders`,
-  
+
   // ...
 
   preAggregations: {
@@ -374,3 +396,30 @@ cube(`Orders`, {
 });
 ```
 
+## Indexes
+
+In case of pre-aggregation table has quite significant cardinality you might want to create indexes for such pre-aggregation in databases which support it.
+This is can be done as following:
+
+```javascript
+cube(`Orders`, {
+  sql: `select * from orders`,
+
+  // ...
+
+  preAggregations: {
+    categoryAndDate: {
+      type: `rollup`,
+      measureReferences: [Orders.count, revenue],
+      dimensionReferences: [category],
+      timeDimensionReference: createdAt,
+      granularity: `day`,
+      indexes: {
+        main: {
+          columns: [category]
+        }
+      }
+    }
+  }
+});
+```
diff --git a/docs/Schema/schema-execution-environment.md b/docs/Schema/schema-execution-environment.md
@@ -38,6 +38,10 @@ cube(`Users`, {
 });
 ```
 
+## Cube.js globals (cube and others)
+
+Cube.js defines `cube()`, `context()` and `asyncModule()` global variable functions in order to provide API for schema configuration which aren't normally accessible outside of Cube.js schema.
+
 ## Import / Export
 
 Data schema JavaScript files are transpiled to convert ES6 `import` and `export` expressions to corresponding Node.js calls.

diff --git a/examples/real-time-dashboard/package.json b/examples/real-time-dashboard/package.json
@@ -8,8 +8,8 @@
     "start": "node index.js"
   },
   "dependencies": {
-    "@cubejs-backend/mongobi-driver": "^0.13.0",
-    "@cubejs-backend/server": "^0.13.11",
+    "@cubejs-backend/mongobi-driver": "^0.17.5",
+    "@cubejs-backend/server": "^0.17.7",
     "moment": "^2.24.0",
     "mongodb": "^3.3.3",
     "serve-static": "^1.14.1"