Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Input formats: save unknown fields to a special column instead of skipping #11775

Open
ksverdlov opened this issue Jun 18, 2020 · 0 comments
Open
Labels
comp-formats Input / output formats feature

Comments

@ksverdlov
Copy link

Use case
If input_format_skip_unknown_fields = 1 (skipping is enabled), ClickHouse doesn’t insert extra data. I need to save extra data in origin format to a special column (String-type) for further processing.

Describe the solution you'd like
Some additional option for format, in which I can pass a column name. In that column Clickhouse will save unknown/skipped fields in origin format.

For example,

INSERT INTO UserActivity FORMAT JSONEachRow(`skipped`) {"PageViews":5, "UserID":"4324182021466249494", "Duration":146,"Sign":-1,"SomeUnknownField":"StringValue"} {"UserID":"4324182021466249494","PageViews":6,"Duration":185,"Sign":1,"SomeOtherUnknownField":123}

Describe alternatives you've considered
Insert all JSON object to one String column with short TTL for latter processing. Unfortunately Clickhouse doesn`t have any functions to modify JSONs (to cut some fields, for example).

@filimonov filimonov added the comp-formats Input / output formats label Jun 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp-formats Input / output formats feature
Projects
None yet
Development

No branches or pull requests

2 participants