From e430fc9b5c1097a95b49d0aa59ffea55f6776e79 Mon Sep 17 00:00:00 2001 From: alrocar Date: Mon, 15 Jul 2024 19:59:26 +0200 Subject: [PATCH 1/3] add default values --- alter_default_value_in_column/README.MD | 23 +++++++++++++++++++ .../datasources/analytics_events.datasource | 10 ++++---- 2 files changed, 28 insertions(+), 5 deletions(-) create mode 100644 alter_default_value_in_column/README.MD diff --git a/alter_default_value_in_column/README.MD b/alter_default_value_in_column/README.MD new file mode 100644 index 00000000..084f66d2 --- /dev/null +++ b/alter_default_value_in_column/README.MD @@ -0,0 +1,23 @@ +# Alter DEFAULT value in a column of the landing Data Source + +[Pull Request](https://github.com/tinybirdco/use-case-examples/pull/340/files) + +- Just a add the default values to the desired columns. If the event is not sending the value or it is null, the default value will be applied. + + +```diff +SCHEMA > ++ `timestamp` DateTime `json:$.timestamp` DEFAULT now(), ++ `session_id` String `json:$.session_id` DEFAULT '', ++ `action` LowCardinality(String) `json:$.action` DEFAULT 'None', ++ `version` LowCardinality(String) `json:$.version` DEFAULT '1.0', ++ `payload` String `json:$.payload` DEFAULT '{}', +- `timestamp` DateTime `json:$.timestamp`, +- `session_id` String `json:$.session_id`, +- `action` LowCardinality(String) `json:$.action`, +- `version` LowCardinality(String) `json:$.version`, +- `payload` String `json:$.payload` +``` + +- Create a PR with the change above, a new branch will be created as part of the CI process. You can double check the default value is what you expect by ingesteing some null value in the CI branch created. +- To deploy the change, merge to PR to you main branch, the CD job will run and will deploy the changes. diff --git a/alter_default_value_in_column/datasources/analytics_events.datasource b/alter_default_value_in_column/datasources/analytics_events.datasource index 241f6984..e96fbe7c 100644 --- a/alter_default_value_in_column/datasources/analytics_events.datasource +++ b/alter_default_value_in_column/datasources/analytics_events.datasource @@ -4,11 +4,11 @@ DESCRIPTION > Analytics events landing data source SCHEMA > - `timestamp` DateTime `json:$.timestamp`, - `session_id` String `json:$.session_id`, - `action` LowCardinality(String) `json:$.action`, - `version` LowCardinality(String) `json:$.version`, - `payload` String `json:$.payload` + `timestamp` DateTime `json:$.timestamp` DEFAULT now(), + `session_id` String `json:$.session_id` DEFAULT '', + `action` LowCardinality(String) `json:$.action` DEFAULT 'None', + `version` LowCardinality(String) `json:$.version` DEFAULT '1.0', + `payload` String `json:$.payload` DEFAULT '{}', ENGINE "MergeTree" ENGINE_PARTITION_KEY "toYYYYMM(timestamp)" From e0c6d30e685bb996c01b2dd37ec66690a2fc0ce8 Mon Sep 17 00:00:00 2001 From: alrocar Date: Mon, 15 Jul 2024 19:59:50 +0200 Subject: [PATCH 2/3] fix typo --- .../datasources/analytics_events.datasource | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/alter_default_value_in_column/datasources/analytics_events.datasource b/alter_default_value_in_column/datasources/analytics_events.datasource index e96fbe7c..085bac9d 100644 --- a/alter_default_value_in_column/datasources/analytics_events.datasource +++ b/alter_default_value_in_column/datasources/analytics_events.datasource @@ -8,7 +8,7 @@ SCHEMA > `session_id` String `json:$.session_id` DEFAULT '', `action` LowCardinality(String) `json:$.action` DEFAULT 'None', `version` LowCardinality(String) `json:$.version` DEFAULT '1.0', - `payload` String `json:$.payload` DEFAULT '{}', + `payload` String `json:$.payload` DEFAULT '{}' ENGINE "MergeTree" ENGINE_PARTITION_KEY "toYYYYMM(timestamp)" From 045f5ffe5d0fbfd72cba1f1eaa7fe9dc860b0b95 Mon Sep 17 00:00:00 2001 From: alrocar Date: Mon, 15 Jul 2024 20:07:43 +0200 Subject: [PATCH 3/3] update readme --- alter_default_value_in_column/README.MD | 92 ++++++++++++++++++++++++- 1 file changed, 91 insertions(+), 1 deletion(-) diff --git a/alter_default_value_in_column/README.MD b/alter_default_value_in_column/README.MD index 084f66d2..68175b47 100644 --- a/alter_default_value_in_column/README.MD +++ b/alter_default_value_in_column/README.MD @@ -19,5 +19,95 @@ SCHEMA > - `payload` String `json:$.payload` ``` -- Create a PR with the change above, a new branch will be created as part of the CI process. You can double check the default value is what you expect by ingesteing some null value in the CI branch created. +- Create a PR with the change above, a new branch will be created as part of the CI process. You can double check the default value is what you expect by ingesting some null value in the CI branch created. + +For instance, in this case ingest an empty event using the Tinybird branch token: + +```sh +curl \ + -X POST 'https://api.tinybird.co/v0/events?name=analytics_events' \ + -H "Authorization: Bearer $BRANCH_TOKEN" \ + -d $'{}' +{"successful_rows":1,"quarantined_rows":0}% +``` + +Check the content of the Data Source: + +``` +curl https://api.tinybird.co/v0/sql?q=SELECT%20*%20FROM%20analytics_events%20LIMIT%20100%20%0A%20FORMAT%20JSON&token=$BRANCH_TOKEN + +{ + "meta": + [ + { + "name": "timestamp", + "type": "DateTime" + }, + { + "name": "session_id", + "type": "String" + }, + { + "name": "action", + "type": "LowCardinality(String)" + }, + { + "name": "version", + "type": "LowCardinality(String)" + }, + { + "name": "payload", + "type": "String" + } + ], + + "data": + [ + { + "timestamp": "2024-07-15 18:04:45", + "session_id": "", + "action": "None", + "version": "1.0", + "payload": "{}" + } + ], + + "rows": 1, + + "rows_before_limit_at_least": 1, + + "statistics": + { + "elapsed": 0.00113733, + "rows_read": 1, + "bytes_read": 26 + } +} +``` + +You can see all columns have their default value. + + - To deploy the change, merge to PR to you main branch, the CD job will run and will deploy the changes. + +``` +... +** Diffs from current commit '7457be4ad47c137aff1537ae91626b9722c4a89d' and new 'e0c6d30e685bb996c01b2dd37ec66690a2fc0ce8': + modified: alter_default_value_in_column/datasources/analytics_events.datasource +** Preparing commit ... +** Processing ./datasources/analytics_events.datasource +** Building dependencies +** [DRY RUN] Deploying commit ... +** [DRY RUN] Running 'analytics_events' +** Deploying commit ... +** Running 'analytics_events' +** The description or schema of 'analytics_events' has changed. +** - MODIFY COLUMN `timestamp` DEFAULT now() +** - MODIFY COLUMN `session_id` DEFAULT '' +** - MODIFY COLUMN `action` DEFAULT 'None' +** - MODIFY COLUMN `version` DEFAULT '1.0' +** - MODIFY COLUMN `payload` DEFAULT '{}' +** The Data Source has been correctly updated. +** 'analytics_events' created +... +```