-
Notifications
You must be signed in to change notification settings - Fork 6.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add 'union' mode for schema inference #55892
Conversation
This is an automated comment for commit 3b1ffbd with description of existing statuses. It's updated for the latest CI running ❌ Click here to open a full report in a separate page Successful checks
|
src/Core/SettingsChangesHistory.h
Outdated
@@ -80,6 +80,9 @@ namespace SettingsChangesHistory | |||
/// It's used to implement `compatibility` setting (see https://github.com/ClickHouse/ClickHouse/issues/35972) | |||
static std::map<ClickHouseVersion, SettingsChangesHistory::SettingsChanges> settings_changes_history = | |||
{ | |||
{"23.10", {{"input_format_parquet_allow_missing_columns", false, true, "Allow missing columns in Parquet files by default"}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
{"23.10", {{"input_format_parquet_allow_missing_columns", false, true, "Allow missing columns in Parquet files by default"}, | |
{"23.11", {{"input_format_parquet_allow_missing_columns", false, true, "Allow missing columns in Parquet files by default"}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Implementations of IReadBufferIterator
start to look very heavy and a bit repetitive. Maybe we can do something to keep the common code separated? A common base class? But maybe it is okay, at least it is super straightforward.
Anyways, PR looks good to me.
|
Actually I have a dream to refactor reading from S3/File/HDFS/Azure/URL storages to make some base interfaces, because there are a lot of copying, but don't have time right now :( |
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add 'union' mode for schema inference. In this mode the resulting table schema is the union of all files schemas (so schema is inferred from each file). The mode of schema inference is controlled by a setting
schema_inference_mode
with 2 possible values -default
andunion
. Closes #55428Documentation entry for user-facing changes