0.4.1-prerelease

Improvements

feat(clickhousech@m-11932130): ProfileEvent for prewhere
feat(clickhousech@m-14198773): push implicit column to prewhere
feat(clickhousech@m-17104648): send progress in optimizer mode
feat(clickhousech@m-3000207505): limit the size of or expressions to pushdown to prewhere
feat(clickhousech@m-3000999997): 【CP】refine log
feat(clickhousech@m-3011045309): [CP to 2.1]CnchMergeTree support bytecool part move parser
feat(clickhousech@m-3011597342): [cp]add function sparkHashSimple to align with Spark
feat(clickhousech@m-3015142595):[CP TO CHCH2.1]Adjust local and remote fs buffer size
feat(clickhousech@m-3032987723): set default bsp max_disk_bytes to 500GB, cp to 2.1
feat(clickhousech@m-3086283635): [cp cnch-2.1] introduce sample consuming function for CnchKafka
feat(clickhousech@m-3156389534): filter delete bitmap with bucket number.
feat(clickhousech@m-3265377520): do not show information for system tables without rbac check
feat(clickhousech@m-3346761095): unique table support append mode
feat(clickhousech@m-3346761604): Separate user name for mysql protocol
feat(clickhousech@m-3347623183): add udaf fastPrevAuc2 to handle the miduim results for fastAuc2
feat(clickhousech@m-3348298112): Ingest with bucket
feat(clickhousech@m-3404807508): [CNCH] memory control on joined block
feat(clickhousech@m-3460318692): make max_bytes_to_read_local respect max_bytes_to_read_leaf
feat(clickhousech@m-3460336453): [cp]support BitMap64 type with Parquet format
feat(clickhousech@m-3672686692): [cp-cnch2.1] 1.small keys support agg two level 2.countD agg state memory size estimation
feat(clickhousech@m-5301115): rpc client base brpc timeout should read config
feat(optimizer@m-3000286965): remove redundant cast

Bug fixes

fix(clickhousech@m-10100323): use nnproxy access to replace nameservice style hdfs url
fix(clickhousech@m-12459087):Suppress otel header parse exception[cp]
fix(clickhousech@m-17322645): leader election improve
fix(clickhousech@m-17441741): fix serveral mv with same target table insert in worker side report error
fix(clickhousech@m-3000662130): prevent from acquire RWlock recursively
fix(clickhousech@m-3000736999): close optimize_rewrite_sum_if_to_count_if
fix(clickhousech@m-3000927742): fix lowCardinalityIsNoneEncoded in filter
fix(clickhousech@m-3001047433): Fixed Unexpected Integer conversion
fix(clickhousech@m-3011112771): Fix bug when prefetching data into CachedCompressedReadBuffer
fix(clickhousech@m-3011577263): make delete bitmap cache immutable.
fix(clickhousech@m-3018479783): Allow extra argument for toString
fix(clickhousech@m-3033432511): fix prewhere selection when containing bitmap index functions
fix(clickhousech@m-3033450748): [CP to 2.1]fix metrics2 not init and update hdfs to support more metrics
fix(clickhousech@m-3047443070): fix too deep recursion cnch 2.1
fix(clickhousech@m-3105053835): push down aggregation op to metastore.
fix(clickhousech@m-3106204787): [cp] add setting storage_dialect_type and use it for key expressions
fix(clickhousech@m-3156347174):[TO CNCH2.1] Reduce S3 API invoke in merge process
fix(clickhousech@m-3158769174): [cp cnch-dev] change the order of loading index to avoid unnecessary loading
fix(clickhousech@m-3158825186): [cp] skip bad ttl when doing partition gc
fix(clickhousech@m-3260650011): [cp] fix cache inconsistency issues
fix(clickhousech@m-3284060938): refactor move comment from storage to astcreate[cp]
fix(clickhousech@m-3284084166): not do constant folding for complex literals cnch 2.1
fix(clickhousech@m-3298001072): add obsolete setting
fix(clickhousech@m-3298014142): fix the wrong logic when parallel size is worker number + 1
fix(clickhousech@m-3323373157): fix duplicate create cloud table for s3/hdfs table
fix(clickhousech@m-3347546554): Merge branch 'fix-bsp-scheduler-hung' into 'cnch-2.1'
fix(clickhousech@m-3347547143): apply settings via mysql client [cp]
fix(clickhousech@m-3347641016): enhancement for MySQL DDL and DQL 2.1
fix(clickhousech@m-3347646011): fix instance id operator<
fix(clickhousech@m-3347651074): disable bsp_shuffle_reduce_locality_enabled for now
fix(clickhousech@m-3348176173): fix connection reuse
fix(clickhousech@m-3348189669): add node info when read data part failed
fix(clickhousech@m-3348198742): support mysql dual table
fix(clickhousech@m-3348236069): [cp]lock bugs
fix(clickhousech@m-3359728725): Fix bug with LowCardinality in short-curcuit function evaluation Closes #1301
fix(clickhousech@m-3386615999): add aggregate function bitmap* to push partial blacklist cnch-dev
fix(clickhousech@m-3386703027): [cp] fix perf of detach partition and hdfs disk
fix(clickhousech@m-3388077045): [cp] cnch 2.1 remove unnecessary getChecksums calls in loadDataParts
fix(clickhousech@m-3388089557): [cp] Does not change server_vw_name for IStorage::renameInMemory to avoid affected by QueryAnalyzer etc
fix(clickhousech@m-3459432705): [cp] fix rollback of detach partition
fix(clickhousech@m-3460250027): not select partition key as prewhere cnch 2.1
fix(clickhousech@m-3460279495): [cp cnch-2.1]minor fix edge case for PB consuming: Invalid number of rows in Chunk
fix(clickhousech@m-3460291004): move setUser to async thread
fix(clickhousech@m-3460298109): gc scan in a round-robin way.
fix(clickhousech@m-3594280570): [cp]lock manager bug and support system commands
fix(clickhousech@m-3595066568): fix arrayjoin performance slow with optimizer
fix(clickhousech@m-3595103553): [manual cp] fix toCNCHDataPart lost commit time and add some catalog debug log
fix(clickhousech@m-3595136086): skip access storage when enable_skip_non_cnch_tables_for_cnch_trash_items is enabled.
fix(clickhousech@m-3595158575): fix report root cause
fix(clickhousech@m-3643636185): [cp] Make check table happens at worker side
fix(clickhousech@m-3651696533): fix toDateOrNull core cnch 2.1
fix(clickhousech@m-3671827753): [cp] Make sure cluster_status set to 0 after modify cluster by
fix(clickhousech@m-3671852025): forwarding once
fix(clickhousech@m-3671861263): fixed conjunct the where and prewhere in getFilterFromQueryInfo
fix(clickhousech@m-3671907062): [cp] Fix cleanUndoBuffers logic for DaemonJobTxnGC to let it works well with write_undo_buffer_new_key
fix(clickhousech@m-3782687661): fix mybatis connection[cp]
fix(clickhousech@m-3798655921): Remove constraint unique from unique key
fix(clickhousech@m-3813624009): Rbac notify fix 2.1
fix(clickhousech@m-3813624009): block concurrent RBAC accesses for the same user
fix(clickhousech@m-3856683587): Mysql Default Text Case Roll Back 2.1
fix(optimizer@m-14637742): optimizer remove sample check
fix(optimizer@m-15069680): optimizer support final sample
fix(optimizer@m-16951867): fix distinct to agg
fix(optimizer@m-3000537417): fix bitmap index not pushed down
fix(optimizer@m-3000935223): fix pushdown projection symbol mapping cnch-dev
fix(optimizer@m-3159498150): fix memory leak in statistics
fix(optimizer@m-3179121605): Refactor sql binding
fix(optimizer@m-3265473722):fix predicate pushdown ineffective bug
fix(optimizer@m-3265514516): close optimizer fallback
fix(optimizer@m-3310530537): fix cte property explore with broadcast cnch-2.1
fix(optimizer@m-3347557629): fix runtime filter core dump in runtime for date type date cnch-2.1
fix(optimizer@m-3348236068): Fix bitmap index check
fix(optimizer@m-3348274178): fix explain error info
fix(optimizer@m-3386699280): make offloading_with_query_plan take effect in cascades
fix(optimizer@m-3459430016): delete legacy histogram when re-collect stats without histogram
fix(optimizer@m-3645281049): fix rm redundant distinct
fix(optimizer@m-3652546032): In func partition prune
fix(optimizer@m-3695172707): fix offloading_with_query_plan don't set shuffle_with_order cnch_2.1

ByConity 0.4.0 Release

Changelog

All notable changes to this project will be documented in this file.

[0.4.0] - 2024-04-15

🚀 Features

• Support MySQL Array Functions (FINAL)
• MySQL BI compatibility improvements
• proactively create table meta when creating table.
• support distributed outfile in worker
• Bucket size check && support alias column
• Add hdfs lock to concurrent outfile to the same file
• Close Lazy seek for hdfs
• Dedup worker only get wanted bucket numbers' parts
• KV Rbac performance optimization for granting lots of query to one user. Also fix a coredump and dead lock
• enable write part parallel
• make columns nullable by default under mysql
• fix array_remove and reverse function
• support state function no shuffle
• support additional-services for fulltext search.
• support delete bitmap cache in catalog
• json functions support Object(JSON)
• Support parts splitting for table scan
• support nullable date type for TTL
• upgrade aggregate function attributionAnalysis
• optimize query system tables/databases when tenant is set
• complete storage io trace and total query path
• enable microseconds event_time/query_start_time in worker query log
• Handle overflow date under MySQL dialect
• Disk partition writer size limit
• enhance skip index
• support distributed_aggregation_memory_efficient in optimizer
• Embed the query profiler into cnch binary
• statistics support implicit columns
• impl NVM load/store in PrimaryIndexCache
• support outfile to TOS
• add local disk fd cache

🐛 Bug Fixes

• remove validate during create , fix view rewrite , remove sync mv path
• Undo buffer use reversed transaction id to avoid write increment keys to bytekv
• fix isMonitoredCnchQuery core
• init lock request before put in map
• remove useless get delete bitmap request for non-unique table
• Fix for visiting the removed entry in remove operation
• fix insert select can not parallel on non-unique table
• when mv target table with materialized view throw exception
• get consul http host from multi env
• fix rbac check with wrong database name
• tenant should not see DBs create by default user
• fix global_disk_written_bytes leakage
• destroy task_record after phase-2 finished for merge task
• fix drop table meta and view tables core
• fix mv version part meta cache weight
• fix hdfsCloseFile() can be called twice if encounter error
• fix non partition based mv refresh
• need_invalid_cache also controls bitmaps cache.
• fix reading partition col without row count
• Fix table_definition_hash and make it backward compatible
• fix refresh materialized view submit async
• fix refresh empty database, vw settings, tenant id
• fix QueryAnalyzer process asterisk fail for INSERT query cnch dev
• fix assert failure in part gc thread
• fix file not found and add dlb inspector
• add supportsPrewhere check in pushFilterIntoStorage
• fix for materialized view union rewrite
• fix some functions can not push into prewhere
• enable single character of M-D h:m:s under mysql dialect
• fix unsafe request consul server
• enable runtime filter in bsp mode
• keep grace hash join when exchange type is broadcast
• fix write rows/bytes in query log
• catch finish senders in DiskExchangeManager::submitReadTask
• force UnifyNullableVisitor visit all plan nodes
• Fix arrayElement for map column under MySQL
• fix materialized view rewrite for unqiue table & view
• enable single character of M-D h:m:s under mysql dialect
• fix unsafe request consul server
• enable runtime filter in bsp mode
• add enable_add_projection_to_pruning setting
• keep grace hash join when exchange type is broadcast
• catch finish senders in DiskExchangeManager::submitReadTask
• force UnifyNullableVisitor visit all plan nodes
• Fix TableFinishStep overwrite partition when there is view rewrite
• Leader callback exception will be caught in storage elector
• add partition info to KV when seeing new staged part.
• Restrict tenanted users to whitelisted settings
• fix validateAndSyncBaseTablePartitions for query rewrite
• Fix hive table access and add enable_async_mv_debug setting
• unique table make sure parts and delete bitmap metas are same version
• Send dict table resources in beginning
• fix materialized view rewrite for order by columns
• enable implicit type convert and case insensitive by default under mysql dialect
• fix bug when grace hash join rehash
• fix core create table as using unique key
• update log level and ignore unlock exception
• fix bitmap index function pattern
• "X.Y.Z" format column query support in multi-tenant enviornment
• fix temporary_data_merge_threads set when enable optimizer
• make union input/output stream nullable data type consistent
• fix bound ratio incorrect merge cnch dev
• fix adddate for implicit type convert
• SingleDistinctAggregationToGroupBy support multiple input arguments
• support mark disk cache load data when disk cache write data
• support metrics2 for profile event
• disk cache support global setting
• getDeleteBitmap method need to be thread-safe
• optimizer handle non deterministic function better
• fix wrong write buffer usage in part dumper
• fix parse datetime best effort more than 20 fractional digits
• fix disk cache only cached partial data
• fix external agg spills too many small files
• reduce useless scatter when grace hash join
• Fix timediff function under mysql mode
• add lru cache for invert index to avoid oom AND disable String to Array(XXX) CAST at ddl modify column Scenario
• fix explain pipeline insert into select
• add table-level mutate settings
• touch active ts for all type tx type
• dialect type not set join use nulls cnch dev
• minor fix for quitting core bg threads
• fix first filter is const case in combineFilterEqualSize
• compatible settings format_csv_write_utf8_with_bom
• check disk partition writer finish result
• Split scheduler classes into files
• add untrack memroy for global memory tracker
• Add undo resource for fs lock
• remove path in fd_cache when DiskCacheLRU evict segments
• Use MultiWrite for writeUndoBuffer
• Cnch table's alter only acquire lock for DDL
• Allow BIT_AND and similar bitwise aggregation functions to operate on signed integers
• Only EnableLogging when send_logs_level is larger than information
• Fix MaxIOProfileEvents can't be logged if read from S3 without cache
• avoid return value from illegal address
• Fix S3 DeleteObjects exceed 1000 with error message MalformedXML
• fix report exception,See merge request dp/ClickHouse!19029
• fix lc fallback cut and compact error
• wait until finish for IO-bound tasks
• improve parts_info's accuracy for unique table.
• fix unclean drop in S3 storage policy
• fix RemoveRedundantSort with filling step
• narrow down the locality of scheduling in bsp mode, decouple scheduling trigger for bsp & mpp
• throw exception when get sample key if table not sample key
• fix cancel disk partition writer coredump
• Enable interactive transaction support when use optimizer

🧪 Testing

• Temporary disable 00715_bounding_ratio_ck233.sql
• Improve 50010_direct_insert test
• Testing remote function to community clickhouse
• Reenable 00715_bounding_ratio_ck233.sql
• Fix mysql integration test docker compose
• Add health check for mysql db in integration test
• Increase start period for healthcheck for mysql in integration test
• Debug compose integration
• Make mysql db init idempotent
• Remove debugging sleep
• Nightly integration test
• Fix nightly integration test
• Github workflow for nightly integration test
• Nightly test docker compose
• Nightly test github work flow
• Mysql integration test docker compose

⚙️ Miscellaneous Tasks

• Fix bin dir inside the docker
• Add script to create pr automatically
• Fix create_pr script
• Add split_pr script
• Add README and fix ci
• Fix util script
• Fix 50010_direct_insert
• Fix broken link in README.md
• Fix utils script
• Add action for auto release.
• Release disk space after pushing docker.

Testing release process

Testing

0.3.3-testonly.2

ci: add action for auto release.

0.3.2

Fix some issue:

• fix: make tso fallback configurable.
• fix: fix daemon manager unable to fetch status from servers
• fix: make select forwarding configurable [cp]
• fix: reportError variable lifetime
• fix: [cp] clear commit time for published part after dedup for unique table
• fix: set a valid enum status in parts_info_local.
• fix: Make show create cnch system tables works
• fix: unlock before call doRoundRobinPick
• fix: enable parallel Las files fetching
• fix: [cp] check_cancel for merge task & [cp] mark a 0 rows merged part as deleted when merging parts
• fix: [cp] limit bytes by the whole part chain
• fix: fix getMapKeys called multi times
• fix: remove unused code
• fix: fix UnifyNullableType don't handle cte
• fix: skip unstable test case for now
• fix: cover random join order
• fix: set enable_optimizer default true, set enable_auto_query_forwarding default true
• fix: fix wrong mapping in storage cache
• fix: fix txn coordinator NPE during server shutdown
• fix: fix txn coordinator NPE during server shutdown
• fix: fix wrong mapping in storage cache
• fix: fix join order by sort ordered_base_nodes stably.
• fix: Fix grouping set no merging
• fix: [cp cnch-ce-merge] minor fix preload for local table without uuid on worker side
• fix: fix not grouping key error
• fix: fix getMapKeys called multi times
• fix: ban optimizer when sql contains getMapKeys function
• fix: fix inner query execute when analyzing
• fix: try fix tealimit order by
• fix: fix the wrong bool expression.
• fix: fix undo buffer key conflict when dump parts parallel in multi workers [cp]
• fix: Fix attach parts with delta in s3
• fix: fix unstable testcase 10120
• fix: cherry pick from cnch1.4
• fix: cherry pick rewrite unknown left join identifier
• fix: remove unnecessary code [cp]
• fix: report real exception before execute
• fix: fix cnchhive alter cnch_server_vw
• fix: disable ut of bucket.eviction_expired
• fix: try fix random join order
• fix: Write empty object when no data
• fix: close outdated session
• fix: pick funcs/fixs
• fix: fix bug for low_cardinality_allow_in_native_format
• fix: fix gc under multi-server env
• fix: fix null pointer access for controller
• fix: Fix StartOfInterval Monotonic
• fix: cp memory leak fix
• fix: fix a bug that disk cache segment file may be incomplete
• fix: Fix ParseDatetime Coredump
• fix: Use MultiWrite for writeUndoBuffer & add flinkFieldHash fnv1aHash
• fix: Las integ
• fix: Fix Window Limit Push Down CNCH
• fix: [cp cnch-2.0] restart consume task if storage schema change
• fix: Fix Server UUID CNCH
• fix: try fix random join order
• fix: Fix transaction related issue on multi server environment
• fix: Subquery results are accessed by name
• fix: fix empty header core dump when create stats
• fix: remove SUBMIT_THREADPOOL for removeWorkerResource
• fix: fix coverity scan
• fix: column type in the header is inconsistent after ColumnPruning
• fix: pick the lock related code to avoid hold recursive mutex and request brpc mutex
• fix: [cp] avoid dead threads from ThreadPool
• fix: Fix Nullable Array Join for CNCH-2.0
• fix: refactor BitMap64 parsing
• fix: Prevent crash when passing wrong aggregation states to groupBitmap
• fix: parser ECNRYPT and SECURITY flags
• fix: add hive cnch_vw_read settings for back compatibility
• fix: optimizer uses cpu time to determine timeout
• fix: cherry-pick tso fix from 1.4
• fix: Fix function podName/getEnv
• fix: prefetch task core dump
• fix: Set default value for std::atomic to solve…
• fix: Add transaction clean tasks system tables and minor fix
• fix: fix failed to gc tables after drop database
• fix: fix optimizer join 1 equals 1
• fix: [cp] Fix aarch64 build for libunwind-old
• fix: update connection and retry when unable to access tso
• fix: rework query cache for optimizer
• fix: plan cache add current database check
• fix: [cp] Use old libunwind for jeprof backtrace
• fix: [cp] cnch-ce-merge temporary allow nullable type in map value
• fix: fix compatibility issue in TTL expression
• fix: wait for all threads finish before return to avoid use after free.
• fix: Update CMakeLists.txt to speed up compilation
• fix: union single property
• fix: fix a bug that tombstone may be trashed before covered parts
• fix: fix deadlock when shutdown context
• fix: [cp] cnch-ce-merge fix map parse null value
• fix: rewrite arrayElement to mapElement in optimizer

Add some feature:

• feat: merge-tree multi-stages reader
• feat: add information schema
• feat: remove duplicate or predicate
• feat: delay DiskExchangeSource read files op to generate
• feat: delay DiskExchangeSource read files op to generate
• feat:support json type
• feat: ReadBufferFromS3 Support SEEK_CUR mode
• feat: support aggregated funciton funnelPathSplit
• feat: refine topn filtering
• feat: improve executing PlanSemgent performace
• feat: Add metrics for updateTSO thread stop functioning
• feat: add plan segment query log
• feat: add send plan segment timeout
• feat: compatible Byte map (KV) with community map
• feat: Cnch ck23p3 alignment
• feat: [cp cnch-2.0] support to skip kafka offsets hole
• feat: complete AVRO format and support for CnchKafka in cnch-2.0
• feat: move insert_null_as_default from logical plan to physical plan
• feat: enable detach/drop staged part/partition command
• feat:[TO cnch-2.0] Support OUTFILE INTO OSS
• feat: [cp] set max merge concurrency based on number of workers & allow aggressive/final merge for small partition
• feat: Move simple reorder join into cascades
• feat: add optimizer profile events
• feat: fully support for insert_null_as_default
• feat: send bsp upstream exception to coordinator
• feat: Add configurable endpoint and timeout for vetos config

And add some optimization

0.3.1

Add some features and fix some bugs happen in 0.3.0:

fix: make global thread pool size same as before
fix: add PrimaryIndexCache back
fix:[breaking change]make the input & output of query exchange log right
feat: implement left (anti, semi, outer) , right(anti, semi, outer), inner join with unequal conditions
fix: use right fields of disk partition writer in query exchange log
fix: detach bsp cleanup from brpc worker
fix: fix index col cache miss
feat: control all bg threads
fix: temporary fix connection lifetime
fix: fix undo buffer clean in dm
feat: Use Read Buffer to replace Read Ahead when Infile from S3
feat: Use prefetch to optimize read from S3 external table performance
fix: skip adaptive load when index_granularities is empty & disable merge prefetcher for adaptive_granularity
feat: Add support for infile compression
fix: use replicated-join as default join shuffle, when statistics not exists
fix: fix potential coredump in getChecksums
fix: fix perf regression of sendResources
fix: fix unexpected while-loop when request empty s3 url
fix: fix create view bugs
fix: check close in BroadcastSenderProxy
fix: ensure worker clients in worker group are always ok
fix: disable rename column with bitmap index
feat: Use VW pool as the only way for getting clients (#126)
fix: cnch-2.0 fix map parse error pop back
fix: disable fsync for IndexFileWriter
fix: fix ALTER UPDATE
feat: Support multiple instances of same plan segment on same node in bsp mode
feat: refactor table scan step dyanmic cast
feat: Normalize the representation of nullable types in ANSI mode
fix: Remove alias checking in ANSI mode
fix: explain add verbose setting
fix: fix array join with lambda
fix: fix in database.table is not supported by optimizer cnch-ce-merge
fix: fix settings in global context changed after startup
fix: [cp cnch-ce-merge] add some setting params to compatible with 1.4
fix: Fix write to gin without any term
fix: some fix & opitmize for bitmap index
fix: after cnch-2.0.3 release branch
fix: fix forwarding logic
fix: fix performance regression
fix: disk partition writer enable sender metrics by default
fix: enable bsp_mode in explain analyze
fix: sync recent fixs to cnch-2.0 from cnch-1.4
fix: ban rewrite_map_col in optimizer link
feat: show bytehouse logo when connect to client
fix: refactor commit parts and txn commit logic
fix: fix refreshWhere getTable twice
feat: Change the default value of use_environment_credentials
fix: plan cache add stats check
feat: Use IRSA / credential files for S3 Disk
fix: nerver set false to partition_metrics_loaded.
feat: generate actively prewhere by statistics
fix: fix cmake copy bug
fix: fix core for dedup worker
feat: Disk partition writer exchange log
feat: Cherry pick any_value CNCH
feat: disable enable_auto_query_forwarding
fix: fix aws log level can't be set
fix: add optimizer unsupported setting list
fix: async send for bsp register
fix: fix ctes in view
fix: cnch 2.0 fix read virtual column _map_column_keys
fix: fix distinct to agg
fix: fix client display progress protocol compatibility with 2.0 itself
feat: add disk_cache_read_bytes stats to query log for totally query path

0.3.0

Add some features and optimisations. Fixing bugs and refactor code.

New features:

• Using prefetch and async read to optimize remote disk read
• Attempt to add memory tracking to CRoaring
• explain metadata add fucntion_const_arguments field
• optimize PlanSegmentProcessList mutex
• remove add exchange
• integrate nvm_cache into checksums_cache
• refactor table scan step dyanmic cast
• background gc for bsp
• Fix RBAC Entity to SQL Conversion Logic
• swap join order plan hint
• Support setting to convert MergeTree to CnchMergeTree
• add dry_run for InterpretSelectQuery
• update croaring version to 0.9.9
• add brpc_data_parts_timeout_ms parameters to control part data sending timeout
• add brpc_data_parts_timeout_ms parameters to control part data sending timeout
• fulltext search cnch ce merge pick
• optimizer support create select
• Support Division By Zero
• Analyzer constant folding
• (with the fix) Extend the support of interpreting "WindowFunction + OrderBy"
• Support DB/Table Snapshots
• RBAC
• use threadpool for worker service
• support bitmap index in cnch2.0
• 【CP】add some compatible settings for 2.0(alpha、beta)
• nvm_cache persist & recover
• Optimization for log and yield in leader election component
• add hybrid allocation table settings for backward compatibility
• To cnch-2.0] Support configurable virtual_hosted_style for s3 infile
• use real pk fk info in join estimator
• nvm_cache impl and integrate with mark_cache & uncompressed_cache
• [cherry]S3 Storage support
• [cp] support reload log level from config file
• Update aws-cpp-sdk and submodules, Using CrtHttpClient
• update statistics binary for plan check cnch-ce-merge
• common expression share
• support protobuf for QueryPlan, skip input_streams for JoinStep hash_equals
• explain analyze pipeline
• fix errors in debug mode
• staged execution and disk based shuffle
• clean up all warnings and treat warning as error
• workload dump to client
• CnchPartsInfo and cnchTrashItemsInfo with snapshots and recalculation.
• support explain json in full format
• Log segment profiles into cnch_query_log and query_log
• DELETE Mutation
• cherry pick push partition key condition down to storage
• cherry pick Add ability to pause/resume clustering task
• fix range filter inclusive bug
• rm unnecessary copy of plan segment
• CnchMaterializingMySQL 2.0 support
• 【CP】window function parallel evaluation cherry from community
• 【CP】replace exchange_timeout with max_execution_time
• API handler for resource reporting
• use new function register API
• Write graphviz dot strings into query_log when print_graph=1
• [CP]support split disk cache into meta&data for cnch-2.0
• Add partition to the system part_log and merge table.
• cp backup vw to cnch2.0
• Add support for DWARF-5
• introduce StorageDictCloudMergeTree
• Add Decimal 256 Support CNCH
• bitengine basic create table, write and read
• optimizer support offset
• Support agg window keys powerset in property
• unique table support attach/detach commands when using S3 storage
• create view check column names
• fix checkSubQueryReturnSingleRow nullptr core
• unique table support bucket lock & fix modify cluster by key bug
• unique table support bucket lock & fix modify cluster by key bug
• Export metric to prometheus
• cherry-pick workload tuning advisor to cnch

v0.2.2-fix.1

Fix get machine ip incorrectly in CentOS

0.2.2

New Features

1. Users now could access data managed by hive and glue without creating external tables. (#686)
2. The optimizer could collect statistics automatically.

Bug Fix

1. Fixed rename failure issue (#676).
2. Fixed slowness in merge and kafka write.
3. Fixed unfunctional bloom filter (#694).
4. Fixed transaction-releated issues in (#657 and #624)

0.2.0

New Feature

• Add multi-workers local cluster
• Add scripts for packaging
• StorageCnchMergeTree properly supports skipping indices
• Enable data preload write through
• support krb5 for hdfs
• cnchhive support read ORC format (preview)
• add certificate test suite
• add sql test suite
• enable multi-worker in ci
• auto forward query to host server when having multiple servers
• patch S3 storage basic
• add hms with kerberos
• add timeout for FoundationDB transactions
• use optimizer for join by default
• Fix unstable test cases
• support expressions with window functions
• add hive docker compose
• ByConity Projection
• add docker image build for centos
• improve ingest column query
• support query cache
• Add S3 rename tool
• support query result cache
• support key value store for RBAC

Bug Fix

• Fix read without cache
• Remove buffer size assertion during loading marks
• Fix attach related issue
• Sync from internal - fix optimizer arrayjoin with limit issue
• Sync from internal: Statistics related fix
• Fix txn leak when exception happening
• Core fix for querying system tables
• Fix thread map related issue
• Directly remove part from catalog if part don't exists on disk
• Fix fmt cannot write to file
• add calculate column sizes
• Fix mark count missmatch during loading index
• Fix partition prunning with ttl when ttl expression doesn't exactly match partition expression
• rpc service use ipv6
• Fix implicit map data loss after merges
• dns service discovery mode not work in k8s
• rpc service use ipv6
• update config file to support ipv6
• Explicitly disable optimizer for insert select
• fix test case 00921_datetime64_compatibility_long
• fix hang when shutdown
• CertificateStateful not reporting failure
• fix test case in sql suite
• fix some test cases
• fix some join test case
• fix bad a test case
• fix listen_try logic on brpc
• fix show create database query
• Support to read Hive table without partition key
• Complete the full support of positional arguments for Analyzer
• support listen_host and listen try for TSO, DM, RM
• Update Dockerfile
• fix forward query to worker
• Fix Analyzer over window functions + order by
• fix unstable test case
• fix Fdb scanning timeout issue
• fix failed to get exchange port
• fix executable wrapper script
• fix submodule
• fix submodule cassandra
• move setings to CI
• fix document
• fix insert data to VersionedCollapsingMergeTree get final error
• add rm for executable wrapper
• fix dictionary failed to update and tenant id
• fix query cache
• fix coredump related to query cache result
• fix query forwarding

Performance

• Performance optimization for key condition analysis
• Performance optimization + part removal fix
• Performance optimization for count distinct
• Performance optimization: trivial count
• Increased parallelism of group by
• Reduce syscalls and catalog calls
• Remove inefficient call to database catalog
• Precalculate values inside HashTableGrower
• In-memory cache for primary key index
• Predict hash table size for aggregation
• Optimize optimizer static cost

Enhancement

• bypass db lock when creating table by default
• Use VW pool as the only way for getting clients
• Update local deployment
• Update docker image for executable wrapper
• Improve CAS operation
• Remove use test from test cases
• Support hive table on external Hdfs
• support get worker id info from config
• old kernel hdfs support
• Added the first version of release_note generation script
• add annotations for FunctionsConversion
• hive schema inference
• Export s3's virtual host style config

Test

• Update clickhouse-test
• Testing test case client
• add ci cases

Documentation

• Update Readme.md
• Discord and community section added to ReadMe
• Clean up the deployment READMEs
• Remove outdated content
• Update README.md with new contribution guideline