materialize scannode slot for case 'select count* from select xxxx fr… #39
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stdpain
pushed a commit
to stdpain/incubator-doris
that referenced
this pull request
Jul 5, 2021
HappenLee
pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Jul 15, 2021
HappenLee
pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Sep 7, 2021
3 tasks
3 tasks
3 tasks
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jun 20, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <platonekosama@gmail.com>
Date: Mon Jun 20 15:37:21 2022 +0800
temporarily avoid src/io conflicts with doris/master
commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 17 20:59:41 2022 +0800
(cold_on_s3) Report remote data size in tablets (apache#195)
* be report tablet remote size
* update TTabletStat
* fe report tablet remote data size
commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 16:06:23 2022 +0800
(cold_on_s3) support modify partition add storage policy
commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 10:24:14 2022 +0800
(cold_on_s3) support alter table add storage policy
commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 16:01:02 2022 +0800
(cold_on_s3) directly upload rowset exceeded ttl (apache#186)
* directly upload rowset exceed ttl
* clang-format
commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 11:39:08 2022 +0800
(cold_on_s3) Implement and rewrite some filesystem methods (apache#183)
* remove temporary test code
* no need to send fs to LocalFileWriter ctor
* rewrite some fs methods
* fix opened fd leak
commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 18:07:52 2022 +0800
(cold_on_s3) fix log printing (apache#176)
commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 17:02:48 2022 +0800
(cold_on_s3) fix rowset write timestamp bug (apache#175)
commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 16:14:56 2022 +0800
(cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)
commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 11:17:23 2022 +0800
(cold_on_s3) fix compile error and thread pool core
commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon Jun 13 19:15:37 2022 +0800
(cold_on_s3) Revert FilePathDesc (apache#136)
* change cooldown pick rowset strategy; add/modify some log
* revert FilePathDesc
* return unsupport error to avoid undefined behavior
* patch to reduce path copy
* fix be ut
commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:36 2022 +0800
(cold_on_s3) Add ut for s3 read (apache#147)
commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:21 2022 +0800
(cold_on_s3) Add metrics for file system operation (apache#146)
* Add metrics for file system operation
* fix status code
commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <565620795@qq.com>
Date: Mon Jun 13 12:53:10 2022 +0800
(cold_on_s3) support show proc backends return remote used capacity
commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <565620795@qq.com>
Date: Fri Jun 10 16:08:09 2022 +0800
(cold_on_s3) support storage policy use default policy
commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <565620795@qq.com>
Date: Thu Jun 9 10:40:46 2022 +0800
(cold_on_s3) create table support different partition use their own storage policy
commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 10 22:28:40 2022 +0800
(cold_on_s3) Implement some fs methods (apache#163)
* implement some fs methods
* add more log info
commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <1091517373@qq.com>
Date: Thu Jun 9 23:12:58 2022 +0800
[fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)
apache#9986
commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 9 16:17:06 2022 +0800
[hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)
commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <814383175@qq.com>
Date: Wed Jun 8 16:09:35 2022 +0800
(cold_on_s3) Optimize cooldown policy (apache#113)
* Optimize cooldown policy
* fix ut
* remove useless log
* add ut
* add ut
* rename var
* fix typo and ut
commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed Jun 8 16:02:41 2022 +0800
provide quick_cooldown switch for regression test (apache#132)
commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <565620795@qq.com>
Date: Tue Jun 7 18:05:01 2022 +0800
(cold_on_s3) fix multiple fe synchronization resource storage policy exit
commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 11:38:46 2022 +0800
(cold_on_s3) fix move bug
commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <565620795@qq.com>
Date: Thu Jun 2 11:24:16 2022 +0800
(cold_on_s3) fix clang report '_exec_env' is not used
commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 00:05:42 2022 +0800
(cold_on_s3) reformat be code to make ci happy
commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 23:44:56 2022 +0800
(cold_on_s3) fix be ut
commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 22:36:37 2022 +0800
(cold_on_s3) fix fe compile error
commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu May 26 11:15:31 2022 +0800
[Enhancement](cold_on_s3) refresh fs map (apache#67)
* refresh fs map
* fetch cooldown ttl from storage policy
commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 21:28:06 2022 +0800
(cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)
* time from the beginning of upload
* compact to appropriate state before upload
* only do compaction on local rowsets
commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 19:22:25 2022 +0800
(cold_on_s3) use noncopy iostream for s3 get object (apache#54)
commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 18:42:09 2022 +0800
(cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)
commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <565620795@qq.com>
Date: Tue May 24 12:58:43 2022 +0800
(cold_on_s3) support storage policy change fix thrift merge redefine
commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <565620795@qq.com>
Date: Tue May 24 11:45:13 2022 +0800
(cold_on_s3) support storage policy change and create table add storage policy name (apache#39)
* [feature](cold-hot)support storage policy change and create table add storage policy name
* support storage policy change and create table add storage policy name, check resource policy exist when create table.
* when create table use storage policy check it exist
* support storage policy change and create table add storage policy name and when create table use storage policy check it exist
* support storage policy change fix some bug.
* fix some code style
commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <565620795@qq.com>
Date: Sat May 21 07:37:43 2022 -0700
(cold_on_s3) support storage policy (apache#28)
* [feature](cold-hot)support storage policy
* fix review
* fix review v1
commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri May 20 16:22:36 2022 +0800
(cold_on_s3) Batch upload to improve upload speed (apache#38)
* test upload duration
* use s3 transfer
* batch upload
* some fix in s3 filesystem
commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <814383175@qq.com>
Date: Wed May 18 15:08:40 2022 +0800
(cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)
* Implement periodic uploading of rowsets to remote storage
* add queue size check
* add test config for cooldwon
* modify read path
commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 17:22:19 2022 +0800
(cold_on_s3) fe distinguish drop partition (apache#26)
commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <565620795@qq.com>
Date: Mon May 16 21:02:51 2022 -0700
(cold_on_s3) support clone task don't download file in s3 (apache#15)
* apache#12 support clone task don't download file in s3
* apache#12 add clone ut
* fix code format
* fix commit apache#12
commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 12:01:32 2022 +0800
[Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24)
* disk capacity differentiates between local and remote
* fe drop table
commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon May 16 11:55:42 2022 +0800
[Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)
* update is_local logical
* remove remote rowsets when drop table/partition
commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <814383175@qq.com>
Date: Mon May 16 11:17:30 2022 +0800
(cold_on_s3) Implement reading rowset data from remote storage (apache#11)
* adjust io stack
* support s3
* refine code
* fix ut
* remove useless code
* rename rblock
* fix err msg
commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed May 11 16:29:54 2022 +0800
(cold_on_s3) Implement tablet cooldown (apache#10)
* tablet cooldown
* add global_local_filesystem to keep compatible
* test tablet cooldown
* revert engine_storage_migration_task_test.cpp
commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <platonekosama@gmail.com>
Date: Sun May 1 12:06:41 2022 +0800
(cold_on_s3) Rowset write path using fs (apache#7)
* replace WritableBlock with FileWriter
* local file system
* set fs when load dir
* fix be ut
commit 3585fe6
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue Jun 7 11:31:49 2022 +0800
[fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)
* fix tracker 0602
* fix format
commit 429a41d
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue May 31 19:12:42 2022 +0800
[bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate
commit 89d3a36
Author: Pxl <pxl290@qq.com>
Date: Thu May 26 16:51:01 2022 +0800
[Bug] [Vectorized] add padding when load char type data (apache#9734)
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jun 20, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <platonekosama@gmail.com>
Date: Mon Jun 20 15:37:21 2022 +0800
temporarily avoid src/io conflicts with doris/master
commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 17 20:59:41 2022 +0800
(cold_on_s3) Report remote data size in tablets (apache#195)
* be report tablet remote size
* update TTabletStat
* fe report tablet remote data size
commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 16:06:23 2022 +0800
(cold_on_s3) support modify partition add storage policy
commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 10:24:14 2022 +0800
(cold_on_s3) support alter table add storage policy
commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 16:01:02 2022 +0800
(cold_on_s3) directly upload rowset exceeded ttl (apache#186)
* directly upload rowset exceed ttl
* clang-format
commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 11:39:08 2022 +0800
(cold_on_s3) Implement and rewrite some filesystem methods (apache#183)
* remove temporary test code
* no need to send fs to LocalFileWriter ctor
* rewrite some fs methods
* fix opened fd leak
commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 18:07:52 2022 +0800
(cold_on_s3) fix log printing (apache#176)
commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 17:02:48 2022 +0800
(cold_on_s3) fix rowset write timestamp bug (apache#175)
commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 16:14:56 2022 +0800
(cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)
commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 11:17:23 2022 +0800
(cold_on_s3) fix compile error and thread pool core
commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon Jun 13 19:15:37 2022 +0800
(cold_on_s3) Revert FilePathDesc (apache#136)
* change cooldown pick rowset strategy; add/modify some log
* revert FilePathDesc
* return unsupport error to avoid undefined behavior
* patch to reduce path copy
* fix be ut
commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:36 2022 +0800
(cold_on_s3) Add ut for s3 read (apache#147)
commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:21 2022 +0800
(cold_on_s3) Add metrics for file system operation (apache#146)
* Add metrics for file system operation
* fix status code
commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <565620795@qq.com>
Date: Mon Jun 13 12:53:10 2022 +0800
(cold_on_s3) support show proc backends return remote used capacity
commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <565620795@qq.com>
Date: Fri Jun 10 16:08:09 2022 +0800
(cold_on_s3) support storage policy use default policy
commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <565620795@qq.com>
Date: Thu Jun 9 10:40:46 2022 +0800
(cold_on_s3) create table support different partition use their own storage policy
commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 10 22:28:40 2022 +0800
(cold_on_s3) Implement some fs methods (apache#163)
* implement some fs methods
* add more log info
commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <1091517373@qq.com>
Date: Thu Jun 9 23:12:58 2022 +0800
[fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)
apache#9986
commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 9 16:17:06 2022 +0800
[hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)
commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <814383175@qq.com>
Date: Wed Jun 8 16:09:35 2022 +0800
(cold_on_s3) Optimize cooldown policy (apache#113)
* Optimize cooldown policy
* fix ut
* remove useless log
* add ut
* add ut
* rename var
* fix typo and ut
commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed Jun 8 16:02:41 2022 +0800
provide quick_cooldown switch for regression test (apache#132)
commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <565620795@qq.com>
Date: Tue Jun 7 18:05:01 2022 +0800
(cold_on_s3) fix multiple fe synchronization resource storage policy exit
commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 11:38:46 2022 +0800
(cold_on_s3) fix move bug
commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <565620795@qq.com>
Date: Thu Jun 2 11:24:16 2022 +0800
(cold_on_s3) fix clang report '_exec_env' is not used
commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 00:05:42 2022 +0800
(cold_on_s3) reformat be code to make ci happy
commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 23:44:56 2022 +0800
(cold_on_s3) fix be ut
commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 22:36:37 2022 +0800
(cold_on_s3) fix fe compile error
commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu May 26 11:15:31 2022 +0800
[Enhancement](cold_on_s3) refresh fs map (apache#67)
* refresh fs map
* fetch cooldown ttl from storage policy
commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 21:28:06 2022 +0800
(cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)
* time from the beginning of upload
* compact to appropriate state before upload
* only do compaction on local rowsets
commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 19:22:25 2022 +0800
(cold_on_s3) use noncopy iostream for s3 get object (apache#54)
commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 18:42:09 2022 +0800
(cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)
commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <565620795@qq.com>
Date: Tue May 24 12:58:43 2022 +0800
(cold_on_s3) support storage policy change fix thrift merge redefine
commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <565620795@qq.com>
Date: Tue May 24 11:45:13 2022 +0800
(cold_on_s3) support storage policy change and create table add storage policy name (apache#39)
* [feature](cold-hot)support storage policy change and create table add storage policy name
* support storage policy change and create table add storage policy name, check resource policy exist when create table.
* when create table use storage policy check it exist
* support storage policy change and create table add storage policy name and when create table use storage policy check it exist
* support storage policy change fix some bug.
* fix some code style
commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <565620795@qq.com>
Date: Sat May 21 07:37:43 2022 -0700
(cold_on_s3) support storage policy (apache#28)
* [feature](cold-hot)support storage policy
* fix review
* fix review v1
commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri May 20 16:22:36 2022 +0800
(cold_on_s3) Batch upload to improve upload speed (apache#38)
* test upload duration
* use s3 transfer
* batch upload
* some fix in s3 filesystem
commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <814383175@qq.com>
Date: Wed May 18 15:08:40 2022 +0800
(cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)
* Implement periodic uploading of rowsets to remote storage
* add queue size check
* add test config for cooldwon
* modify read path
commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 17:22:19 2022 +0800
(cold_on_s3) fe distinguish drop partition (apache#26)
commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <565620795@qq.com>
Date: Mon May 16 21:02:51 2022 -0700
(cold_on_s3) support clone task don't download file in s3 (apache#15)
* apache#12 support clone task don't download file in s3
* apache#12 add clone ut
* fix code format
* fix commit apache#12
commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 12:01:32 2022 +0800
[Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24)
* disk capacity differentiates between local and remote
* fe drop table
commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon May 16 11:55:42 2022 +0800
[Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)
* update is_local logical
* remove remote rowsets when drop table/partition
commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <814383175@qq.com>
Date: Mon May 16 11:17:30 2022 +0800
(cold_on_s3) Implement reading rowset data from remote storage (apache#11)
* adjust io stack
* support s3
* refine code
* fix ut
* remove useless code
* rename rblock
* fix err msg
commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed May 11 16:29:54 2022 +0800
(cold_on_s3) Implement tablet cooldown (apache#10)
* tablet cooldown
* add global_local_filesystem to keep compatible
* test tablet cooldown
* revert engine_storage_migration_task_test.cpp
commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <platonekosama@gmail.com>
Date: Sun May 1 12:06:41 2022 +0800
(cold_on_s3) Rowset write path using fs (apache#7)
* replace WritableBlock with FileWriter
* local file system
* set fs when load dir
* fix be ut
commit 3585fe6
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue Jun 7 11:31:49 2022 +0800
[fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)
* fix tracker 0602
* fix format
commit 429a41d
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue May 31 19:12:42 2022 +0800
[bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate
commit 89d3a36
Author: Pxl <pxl290@qq.com>
Date: Thu May 26 16:51:01 2022 +0800
[Bug] [Vectorized] add padding when load char type data (apache#9734)
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jun 20, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <platonekosama@gmail.com>
Date: Mon Jun 20 15:37:21 2022 +0800
temporarily avoid src/io conflicts with doris/master
commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 17 20:59:41 2022 +0800
(cold_on_s3) Report remote data size in tablets (apache#195)
* be report tablet remote size
* update TTabletStat
* fe report tablet remote data size
commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 16:06:23 2022 +0800
(cold_on_s3) support modify partition add storage policy
commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 10:24:14 2022 +0800
(cold_on_s3) support alter table add storage policy
commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 16:01:02 2022 +0800
(cold_on_s3) directly upload rowset exceeded ttl (apache#186)
* directly upload rowset exceed ttl
* clang-format
commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 11:39:08 2022 +0800
(cold_on_s3) Implement and rewrite some filesystem methods (apache#183)
* remove temporary test code
* no need to send fs to LocalFileWriter ctor
* rewrite some fs methods
* fix opened fd leak
commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 18:07:52 2022 +0800
(cold_on_s3) fix log printing (apache#176)
commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 17:02:48 2022 +0800
(cold_on_s3) fix rowset write timestamp bug (apache#175)
commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 16:14:56 2022 +0800
(cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)
commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 11:17:23 2022 +0800
(cold_on_s3) fix compile error and thread pool core
commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon Jun 13 19:15:37 2022 +0800
(cold_on_s3) Revert FilePathDesc (apache#136)
* change cooldown pick rowset strategy; add/modify some log
* revert FilePathDesc
* return unsupport error to avoid undefined behavior
* patch to reduce path copy
* fix be ut
commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:36 2022 +0800
(cold_on_s3) Add ut for s3 read (apache#147)
commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:21 2022 +0800
(cold_on_s3) Add metrics for file system operation (apache#146)
* Add metrics for file system operation
* fix status code
commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <565620795@qq.com>
Date: Mon Jun 13 12:53:10 2022 +0800
(cold_on_s3) support show proc backends return remote used capacity
commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <565620795@qq.com>
Date: Fri Jun 10 16:08:09 2022 +0800
(cold_on_s3) support storage policy use default policy
commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <565620795@qq.com>
Date: Thu Jun 9 10:40:46 2022 +0800
(cold_on_s3) create table support different partition use their own storage policy
commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 10 22:28:40 2022 +0800
(cold_on_s3) Implement some fs methods (apache#163)
* implement some fs methods
* add more log info
commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <1091517373@qq.com>
Date: Thu Jun 9 23:12:58 2022 +0800
[fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)
apache#9986
commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 9 16:17:06 2022 +0800
[hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)
commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <814383175@qq.com>
Date: Wed Jun 8 16:09:35 2022 +0800
(cold_on_s3) Optimize cooldown policy (apache#113)
* Optimize cooldown policy
* fix ut
* remove useless log
* add ut
* add ut
* rename var
* fix typo and ut
commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed Jun 8 16:02:41 2022 +0800
provide quick_cooldown switch for regression test (apache#132)
commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <565620795@qq.com>
Date: Tue Jun 7 18:05:01 2022 +0800
(cold_on_s3) fix multiple fe synchronization resource storage policy exit
commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 11:38:46 2022 +0800
(cold_on_s3) fix move bug
commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <565620795@qq.com>
Date: Thu Jun 2 11:24:16 2022 +0800
(cold_on_s3) fix clang report '_exec_env' is not used
commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 00:05:42 2022 +0800
(cold_on_s3) reformat be code to make ci happy
commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 23:44:56 2022 +0800
(cold_on_s3) fix be ut
commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 22:36:37 2022 +0800
(cold_on_s3) fix fe compile error
commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu May 26 11:15:31 2022 +0800
[Enhancement](cold_on_s3) refresh fs map (apache#67)
* refresh fs map
* fetch cooldown ttl from storage policy
commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 21:28:06 2022 +0800
(cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)
* time from the beginning of upload
* compact to appropriate state before upload
* only do compaction on local rowsets
commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 19:22:25 2022 +0800
(cold_on_s3) use noncopy iostream for s3 get object (apache#54)
commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 18:42:09 2022 +0800
(cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)
commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <565620795@qq.com>
Date: Tue May 24 12:58:43 2022 +0800
(cold_on_s3) support storage policy change fix thrift merge redefine
commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <565620795@qq.com>
Date: Tue May 24 11:45:13 2022 +0800
(cold_on_s3) support storage policy change and create table add storage policy name (apache#39)
* [feature](cold-hot)support storage policy change and create table add storage policy name
* support storage policy change and create table add storage policy name, check resource policy exist when create table.
* when create table use storage policy check it exist
* support storage policy change and create table add storage policy name and when create table use storage policy check it exist
* support storage policy change fix some bug.
* fix some code style
commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <565620795@qq.com>
Date: Sat May 21 07:37:43 2022 -0700
(cold_on_s3) support storage policy (apache#28)
* [feature](cold-hot)support storage policy
* fix review
* fix review v1
commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri May 20 16:22:36 2022 +0800
(cold_on_s3) Batch upload to improve upload speed (apache#38)
* test upload duration
* use s3 transfer
* batch upload
* some fix in s3 filesystem
commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <814383175@qq.com>
Date: Wed May 18 15:08:40 2022 +0800
(cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)
* Implement periodic uploading of rowsets to remote storage
* add queue size check
* add test config for cooldwon
* modify read path
commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 17:22:19 2022 +0800
(cold_on_s3) fe distinguish drop partition (apache#26)
commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <565620795@qq.com>
Date: Mon May 16 21:02:51 2022 -0700
(cold_on_s3) support clone task don't download file in s3 (apache#15)
* apache#12 support clone task don't download file in s3
* apache#12 add clone ut
* fix code format
* fix commit apache#12
commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 12:01:32 2022 +0800
[Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24)
* disk capacity differentiates between local and remote
* fe drop table
commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon May 16 11:55:42 2022 +0800
[Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)
* update is_local logical
* remove remote rowsets when drop table/partition
commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <814383175@qq.com>
Date: Mon May 16 11:17:30 2022 +0800
(cold_on_s3) Implement reading rowset data from remote storage (apache#11)
* adjust io stack
* support s3
* refine code
* fix ut
* remove useless code
* rename rblock
* fix err msg
commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed May 11 16:29:54 2022 +0800
(cold_on_s3) Implement tablet cooldown (apache#10)
* tablet cooldown
* add global_local_filesystem to keep compatible
* test tablet cooldown
* revert engine_storage_migration_task_test.cpp
commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <platonekosama@gmail.com>
Date: Sun May 1 12:06:41 2022 +0800
(cold_on_s3) Rowset write path using fs (apache#7)
* replace WritableBlock with FileWriter
* local file system
* set fs when load dir
* fix be ut
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: deardeng <565620795@qq.com>
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jun 20, 2022
Co-authored-by: deardeng <565620795@qq.com>
Squashed commit of the following:
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <platonekosama@gmail.com>
Date: Mon Jun 20 15:37:21 2022 +0800
temporarily avoid src/io conflicts with doris/master
commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 17 20:59:41 2022 +0800
(cold_on_s3) Report remote data size in tablets (apache#195)
* be report tablet remote size
* update TTabletStat
* fe report tablet remote data size
commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 16:06:23 2022 +0800
(cold_on_s3) support modify partition add storage policy
commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 10:24:14 2022 +0800
(cold_on_s3) support alter table add storage policy
commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 16:01:02 2022 +0800
(cold_on_s3) directly upload rowset exceeded ttl (apache#186)
* directly upload rowset exceed ttl
* clang-format
commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 11:39:08 2022 +0800
(cold_on_s3) Implement and rewrite some filesystem methods (apache#183)
* remove temporary test code
* no need to send fs to LocalFileWriter ctor
* rewrite some fs methods
* fix opened fd leak
commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 18:07:52 2022 +0800
(cold_on_s3) fix log printing (apache#176)
commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 17:02:48 2022 +0800
(cold_on_s3) fix rowset write timestamp bug (apache#175)
commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 16:14:56 2022 +0800
(cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)
commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 11:17:23 2022 +0800
(cold_on_s3) fix compile error and thread pool core
commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon Jun 13 19:15:37 2022 +0800
(cold_on_s3) Revert FilePathDesc (apache#136)
* change cooldown pick rowset strategy; add/modify some log
* revert FilePathDesc
* return unsupport error to avoid undefined behavior
* patch to reduce path copy
* fix be ut
commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:36 2022 +0800
(cold_on_s3) Add ut for s3 read (apache#147)
commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:21 2022 +0800
(cold_on_s3) Add metrics for file system operation (apache#146)
* Add metrics for file system operation
* fix status code
commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <565620795@qq.com>
Date: Mon Jun 13 12:53:10 2022 +0800
(cold_on_s3) support show proc backends return remote used capacity
commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <565620795@qq.com>
Date: Fri Jun 10 16:08:09 2022 +0800
(cold_on_s3) support storage policy use default policy
commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <565620795@qq.com>
Date: Thu Jun 9 10:40:46 2022 +0800
(cold_on_s3) create table support different partition use their own storage policy
commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 10 22:28:40 2022 +0800
(cold_on_s3) Implement some fs methods (apache#163)
* implement some fs methods
* add more log info
commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <1091517373@qq.com>
Date: Thu Jun 9 23:12:58 2022 +0800
[fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)
apache#9986
commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 9 16:17:06 2022 +0800
[hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)
commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <814383175@qq.com>
Date: Wed Jun 8 16:09:35 2022 +0800
(cold_on_s3) Optimize cooldown policy (apache#113)
* Optimize cooldown policy
* fix ut
* remove useless log
* add ut
* add ut
* rename var
* fix typo and ut
commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed Jun 8 16:02:41 2022 +0800
provide quick_cooldown switch for regression test (apache#132)
commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <565620795@qq.com>
Date: Tue Jun 7 18:05:01 2022 +0800
(cold_on_s3) fix multiple fe synchronization resource storage policy exit
commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 11:38:46 2022 +0800
(cold_on_s3) fix move bug
commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <565620795@qq.com>
Date: Thu Jun 2 11:24:16 2022 +0800
(cold_on_s3) fix clang report '_exec_env' is not used
commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 00:05:42 2022 +0800
(cold_on_s3) reformat be code to make ci happy
commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 23:44:56 2022 +0800
(cold_on_s3) fix be ut
commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 22:36:37 2022 +0800
(cold_on_s3) fix fe compile error
commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu May 26 11:15:31 2022 +0800
[Enhancement](cold_on_s3) refresh fs map (apache#67)
* refresh fs map
* fetch cooldown ttl from storage policy
commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 21:28:06 2022 +0800
(cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)
* time from the beginning of upload
* compact to appropriate state before upload
* only do compaction on local rowsets
commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 19:22:25 2022 +0800
(cold_on_s3) use noncopy iostream for s3 get object (apache#54)
commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 18:42:09 2022 +0800
(cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)
commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <565620795@qq.com>
Date: Tue May 24 12:58:43 2022 +0800
(cold_on_s3) support storage policy change fix thrift merge redefine
commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <565620795@qq.com>
Date: Tue May 24 11:45:13 2022 +0800
(cold_on_s3) support storage policy change and create table add storage policy name (apache#39)
* [feature](cold-hot)support storage policy change and create table add storage policy name
* support storage policy change and create table add storage policy name, check resource policy exist when create table.
* when create table use storage policy check it exist
* support storage policy change and create table add storage policy name and when create table use storage policy check it exist
* support storage policy change fix some bug.
* fix some code style
commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <565620795@qq.com>
Date: Sat May 21 07:37:43 2022 -0700
(cold_on_s3) support storage policy (apache#28)
* [feature](cold-hot)support storage policy
* fix review
* fix review v1
commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri May 20 16:22:36 2022 +0800
(cold_on_s3) Batch upload to improve upload speed (apache#38)
* test upload duration
* use s3 transfer
* batch upload
* some fix in s3 filesystem
commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <814383175@qq.com>
Date: Wed May 18 15:08:40 2022 +0800
(cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)
* Implement periodic uploading of rowsets to remote storage
* add queue size check
* add test config for cooldwon
* modify read path
commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 17:22:19 2022 +0800
(cold_on_s3) fe distinguish drop partition (apache#26)
commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <565620795@qq.com>
Date: Mon May 16 21:02:51 2022 -0700
(cold_on_s3) support clone task don't download file in s3 (apache#15)
* apache#12 support clone task don't download file in s3
* apache#12 add clone ut
* fix code format
* fix commit apache#12
commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 12:01:32 2022 +0800
[Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24)
* disk capacity differentiates between local and remote
* fe drop table
commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon May 16 11:55:42 2022 +0800
[Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)
* update is_local logical
* remove remote rowsets when drop table/partition
commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <814383175@qq.com>
Date: Mon May 16 11:17:30 2022 +0800
(cold_on_s3) Implement reading rowset data from remote storage (apache#11)
* adjust io stack
* support s3
* refine code
* fix ut
* remove useless code
* rename rblock
* fix err msg
commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed May 11 16:29:54 2022 +0800
(cold_on_s3) Implement tablet cooldown (apache#10)
* tablet cooldown
* add global_local_filesystem to keep compatible
* test tablet cooldown
* revert engine_storage_migration_task_test.cpp
commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <platonekosama@gmail.com>
Date: Sun May 1 12:06:41 2022 +0800
(cold_on_s3) Rowset write path using fs (apache#7)
* replace WritableBlock with FileWriter
* local file system
* set fs when load dir
* fix be ut
commit 3585fe6
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue Jun 7 11:31:49 2022 +0800
[fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)
* fix tracker 0602
* fix format
commit 429a41d
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue May 31 19:12:42 2022 +0800
[bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate
commit 89d3a36
Author: Pxl <pxl290@qq.com>
Date: Thu May 26 16:51:01 2022 +0800
[Bug] [Vectorized] add padding when load char type data (apache#9734)
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jun 20, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <platonekosama@gmail.com>
Date: Mon Jun 20 15:37:21 2022 +0800
temporarily avoid src/io conflicts with doris/master
commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 17 20:59:41 2022 +0800
(cold_on_s3) Report remote data size in tablets (apache#195)
* be report tablet remote size
* update TTabletStat
* fe report tablet remote data size
commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 16:06:23 2022 +0800
(cold_on_s3) support modify partition add storage policy
commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 10:24:14 2022 +0800
(cold_on_s3) support alter table add storage policy
commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 16:01:02 2022 +0800
(cold_on_s3) directly upload rowset exceeded ttl (apache#186)
* directly upload rowset exceed ttl
* clang-format
commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 11:39:08 2022 +0800
(cold_on_s3) Implement and rewrite some filesystem methods (apache#183)
* remove temporary test code
* no need to send fs to LocalFileWriter ctor
* rewrite some fs methods
* fix opened fd leak
commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 18:07:52 2022 +0800
(cold_on_s3) fix log printing (apache#176)
commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 17:02:48 2022 +0800
(cold_on_s3) fix rowset write timestamp bug (apache#175)
commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 16:14:56 2022 +0800
(cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)
commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 11:17:23 2022 +0800
(cold_on_s3) fix compile error and thread pool core
commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon Jun 13 19:15:37 2022 +0800
(cold_on_s3) Revert FilePathDesc (apache#136)
* change cooldown pick rowset strategy; add/modify some log
* revert FilePathDesc
* return unsupport error to avoid undefined behavior
* patch to reduce path copy
* fix be ut
commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:36 2022 +0800
(cold_on_s3) Add ut for s3 read (apache#147)
commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:21 2022 +0800
(cold_on_s3) Add metrics for file system operation (apache#146)
* Add metrics for file system operation
* fix status code
commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <565620795@qq.com>
Date: Mon Jun 13 12:53:10 2022 +0800
(cold_on_s3) support show proc backends return remote used capacity
commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <565620795@qq.com>
Date: Fri Jun 10 16:08:09 2022 +0800
(cold_on_s3) support storage policy use default policy
commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <565620795@qq.com>
Date: Thu Jun 9 10:40:46 2022 +0800
(cold_on_s3) create table support different partition use their own storage policy
commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 10 22:28:40 2022 +0800
(cold_on_s3) Implement some fs methods (apache#163)
* implement some fs methods
* add more log info
commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <1091517373@qq.com>
Date: Thu Jun 9 23:12:58 2022 +0800
[fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)
apache#9986
commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 9 16:17:06 2022 +0800
[hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)
commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <814383175@qq.com>
Date: Wed Jun 8 16:09:35 2022 +0800
(cold_on_s3) Optimize cooldown policy (apache#113)
* Optimize cooldown policy
* fix ut
* remove useless log
* add ut
* add ut
* rename var
* fix typo and ut
commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed Jun 8 16:02:41 2022 +0800
provide quick_cooldown switch for regression test (apache#132)
commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <565620795@qq.com>
Date: Tue Jun 7 18:05:01 2022 +0800
(cold_on_s3) fix multiple fe synchronization resource storage policy exit
commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 11:38:46 2022 +0800
(cold_on_s3) fix move bug
commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <565620795@qq.com>
Date: Thu Jun 2 11:24:16 2022 +0800
(cold_on_s3) fix clang report '_exec_env' is not used
commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 00:05:42 2022 +0800
(cold_on_s3) reformat be code to make ci happy
commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 23:44:56 2022 +0800
(cold_on_s3) fix be ut
commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 22:36:37 2022 +0800
(cold_on_s3) fix fe compile error
commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu May 26 11:15:31 2022 +0800
[Enhancement](cold_on_s3) refresh fs map (apache#67)
* refresh fs map
* fetch cooldown ttl from storage policy
commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 21:28:06 2022 +0800
(cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)
* time from the beginning of upload
* compact to appropriate state before upload
* only do compaction on local rowsets
commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 19:22:25 2022 +0800
(cold_on_s3) use noncopy iostream for s3 get object (apache#54)
commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 18:42:09 2022 +0800
(cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)
commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <565620795@qq.com>
Date: Tue May 24 12:58:43 2022 +0800
(cold_on_s3) support storage policy change fix thrift merge redefine
commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <565620795@qq.com>
Date: Tue May 24 11:45:13 2022 +0800
(cold_on_s3) support storage policy change and create table add storage policy name (apache#39)
* [feature](cold-hot)support storage policy change and create table add storage policy name
* support storage policy change and create table add storage policy name, check resource policy exist when create table.
* when create table use storage policy check it exist
* support storage policy change and create table add storage policy name and when create table use storage policy check it exist
* support storage policy change fix some bug.
* fix some code style
commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <565620795@qq.com>
Date: Sat May 21 07:37:43 2022 -0700
(cold_on_s3) support storage policy (apache#28)
* [feature](cold-hot)support storage policy
* fix review
* fix review v1
commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri May 20 16:22:36 2022 +0800
(cold_on_s3) Batch upload to improve upload speed (apache#38)
* test upload duration
* use s3 transfer
* batch upload
* some fix in s3 filesystem
commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <814383175@qq.com>
Date: Wed May 18 15:08:40 2022 +0800
(cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)
* Implement periodic uploading of rowsets to remote storage
* add queue size check
* add test config for cooldwon
* modify read path
commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 17:22:19 2022 +0800
(cold_on_s3) fe distinguish drop partition (apache#26)
commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <565620795@qq.com>
Date: Mon May 16 21:02:51 2022 -0700
(cold_on_s3) support clone task don't download file in s3 (apache#15)
* apache#12 support clone task don't download file in s3
* apache#12 add clone ut
* fix code format
* fix commit apache#12
commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 12:01:32 2022 +0800
[Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24)
* disk capacity differentiates between local and remote
* fe drop table
commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon May 16 11:55:42 2022 +0800
[Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)
* update is_local logical
* remove remote rowsets when drop table/partition
commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <814383175@qq.com>
Date: Mon May 16 11:17:30 2022 +0800
(cold_on_s3) Implement reading rowset data from remote storage (apache#11)
* adjust io stack
* support s3
* refine code
* fix ut
* remove useless code
* rename rblock
* fix err msg
commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed May 11 16:29:54 2022 +0800
(cold_on_s3) Implement tablet cooldown (apache#10)
* tablet cooldown
* add global_local_filesystem to keep compatible
* test tablet cooldown
* revert engine_storage_migration_task_test.cpp
commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <platonekosama@gmail.com>
Date: Sun May 1 12:06:41 2022 +0800
(cold_on_s3) Rowset write path using fs (apache#7)
* replace WritableBlock with FileWriter
* local file system
* set fs when load dir
* fix be ut
commit 3585fe6
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue Jun 7 11:31:49 2022 +0800
[fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)
* fix tracker 0602
* fix format
commit 429a41d
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue May 31 19:12:42 2022 +0800
[bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate
commit 89d3a36
Author: Pxl <pxl290@qq.com>
Date: Thu May 26 16:51:01 2022 +0800
[Bug] [Vectorized] add padding when load char type data (apache#9734)
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: deardeng <565620795@qq.com>
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jul 6, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <platonekosama@gmail.com>
Date: Mon Jun 20 15:37:21 2022 +0800
temporarily avoid src/io conflicts with doris/master
commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 17 20:59:41 2022 +0800
(cold_on_s3) Report remote data size in tablets (apache#195)
* be report tablet remote size
* update TTabletStat
* fe report tablet remote data size
commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 16:06:23 2022 +0800
(cold_on_s3) support modify partition add storage policy
commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 10:24:14 2022 +0800
(cold_on_s3) support alter table add storage policy
commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 16:01:02 2022 +0800
(cold_on_s3) directly upload rowset exceeded ttl (apache#186)
* directly upload rowset exceed ttl
* clang-format
commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 11:39:08 2022 +0800
(cold_on_s3) Implement and rewrite some filesystem methods (apache#183)
* remove temporary test code
* no need to send fs to LocalFileWriter ctor
* rewrite some fs methods
* fix opened fd leak
commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 18:07:52 2022 +0800
(cold_on_s3) fix log printing (apache#176)
commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 17:02:48 2022 +0800
(cold_on_s3) fix rowset write timestamp bug (apache#175)
commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 16:14:56 2022 +0800
(cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)
commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 11:17:23 2022 +0800
(cold_on_s3) fix compile error and thread pool core
commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon Jun 13 19:15:37 2022 +0800
(cold_on_s3) Revert FilePathDesc (apache#136)
* change cooldown pick rowset strategy; add/modify some log
* revert FilePathDesc
* return unsupport error to avoid undefined behavior
* patch to reduce path copy
* fix be ut
commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:36 2022 +0800
(cold_on_s3) Add ut for s3 read (apache#147)
commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:21 2022 +0800
(cold_on_s3) Add metrics for file system operation (apache#146)
* Add metrics for file system operation
* fix status code
commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <565620795@qq.com>
Date: Mon Jun 13 12:53:10 2022 +0800
(cold_on_s3) support show proc backends return remote used capacity
commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <565620795@qq.com>
Date: Fri Jun 10 16:08:09 2022 +0800
(cold_on_s3) support storage policy use default policy
commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <565620795@qq.com>
Date: Thu Jun 9 10:40:46 2022 +0800
(cold_on_s3) create table support different partition use their own storage policy
commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 10 22:28:40 2022 +0800
(cold_on_s3) Implement some fs methods (apache#163)
* implement some fs methods
* add more log info
commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <1091517373@qq.com>
Date: Thu Jun 9 23:12:58 2022 +0800
[fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)
apache#9986
commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 9 16:17:06 2022 +0800
[hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)
commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <814383175@qq.com>
Date: Wed Jun 8 16:09:35 2022 +0800
(cold_on_s3) Optimize cooldown policy (apache#113)
* Optimize cooldown policy
* fix ut
* remove useless log
* add ut
* add ut
* rename var
* fix typo and ut
commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed Jun 8 16:02:41 2022 +0800
provide quick_cooldown switch for regression test (apache#132)
commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <565620795@qq.com>
Date: Tue Jun 7 18:05:01 2022 +0800
(cold_on_s3) fix multiple fe synchronization resource storage policy exit
commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 11:38:46 2022 +0800
(cold_on_s3) fix move bug
commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <565620795@qq.com>
Date: Thu Jun 2 11:24:16 2022 +0800
(cold_on_s3) fix clang report '_exec_env' is not used
commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 00:05:42 2022 +0800
(cold_on_s3) reformat be code to make ci happy
commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 23:44:56 2022 +0800
(cold_on_s3) fix be ut
commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 22:36:37 2022 +0800
(cold_on_s3) fix fe compile error
commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu May 26 11:15:31 2022 +0800
[Enhancement](cold_on_s3) refresh fs map (apache#67)
* refresh fs map
* fetch cooldown ttl from storage policy
commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 21:28:06 2022 +0800
(cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)
* time from the beginning of upload
* compact to appropriate state before upload
* only do compaction on local rowsets
commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 19:22:25 2022 +0800
(cold_on_s3) use noncopy iostream for s3 get object (apache#54)
commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 18:42:09 2022 +0800
(cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)
commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <565620795@qq.com>
Date: Tue May 24 12:58:43 2022 +0800
(cold_on_s3) support storage policy change fix thrift merge redefine
commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <565620795@qq.com>
Date: Tue May 24 11:45:13 2022 +0800
(cold_on_s3) support storage policy change and create table add storage policy name (apache#39)
* [feature](cold-hot)support storage policy change and create table add storage policy name
* support storage policy change and create table add storage policy name, check resource policy exist when create table.
* when create table use storage policy check it exist
* support storage policy change and create table add storage policy name and when create table use storage policy check it exist
* support storage policy change fix some bug.
* fix some code style
commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <565620795@qq.com>
Date: Sat May 21 07:37:43 2022 -0700
(cold_on_s3) support storage policy (apache#28)
* [feature](cold-hot)support storage policy
* fix review
* fix review v1
commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri May 20 16:22:36 2022 +0800
(cold_on_s3) Batch upload to improve upload speed (apache#38)
* test upload duration
* use s3 transfer
* batch upload
* some fix in s3 filesystem
commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <814383175@qq.com>
Date: Wed May 18 15:08:40 2022 +0800
(cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)
* Implement periodic uploading of rowsets to remote storage
* add queue size check
* add test config for cooldwon
* modify read path
commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 17:22:19 2022 +0800
(cold_on_s3) fe distinguish drop partition (apache#26)
commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <565620795@qq.com>
Date: Mon May 16 21:02:51 2022 -0700
(cold_on_s3) support clone task don't download file in s3 (apache#15)
* apache#12 support clone task don't download file in s3
* apache#12 add clone ut
* fix code format
* fix commit apache#12
commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 12:01:32 2022 +0800
[Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24)
* disk capacity differentiates between local and remote
* fe drop table
commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon May 16 11:55:42 2022 +0800
[Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)
* update is_local logical
* remove remote rowsets when drop table/partition
commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <814383175@qq.com>
Date: Mon May 16 11:17:30 2022 +0800
(cold_on_s3) Implement reading rowset data from remote storage (apache#11)
* adjust io stack
* support s3
* refine code
* fix ut
* remove useless code
* rename rblock
* fix err msg
commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed May 11 16:29:54 2022 +0800
(cold_on_s3) Implement tablet cooldown (apache#10)
* tablet cooldown
* add global_local_filesystem to keep compatible
* test tablet cooldown
* revert engine_storage_migration_task_test.cpp
commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <platonekosama@gmail.com>
Date: Sun May 1 12:06:41 2022 +0800
(cold_on_s3) Rowset write path using fs (apache#7)
* replace WritableBlock with FileWriter
* local file system
* set fs when load dir
* fix be ut
commit 3585fe6
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue Jun 7 11:31:49 2022 +0800
[fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)
* fix tracker 0602
* fix format
commit 429a41d
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue May 31 19:12:42 2022 +0800
[bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate
commit 89d3a36
Author: Pxl <pxl290@qq.com>
Date: Thu May 26 16:51:01 2022 +0800
[Bug] [Vectorized] add padding when load char type data (apache#9734)
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: deardeng <565620795@qq.com>
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jul 6, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <platonekosama@gmail.com>
Date: Mon Jun 20 15:37:21 2022 +0800
temporarily avoid src/io conflicts with doris/master
commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 17 20:59:41 2022 +0800
(cold_on_s3) Report remote data size in tablets (apache#195)
* be report tablet remote size
* update TTabletStat
* fe report tablet remote data size
commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 16:06:23 2022 +0800
(cold_on_s3) support modify partition add storage policy
commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 10:24:14 2022 +0800
(cold_on_s3) support alter table add storage policy
commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 16:01:02 2022 +0800
(cold_on_s3) directly upload rowset exceeded ttl (apache#186)
* directly upload rowset exceed ttl
* clang-format
commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 11:39:08 2022 +0800
(cold_on_s3) Implement and rewrite some filesystem methods (apache#183)
* remove temporary test code
* no need to send fs to LocalFileWriter ctor
* rewrite some fs methods
* fix opened fd leak
commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 18:07:52 2022 +0800
(cold_on_s3) fix log printing (apache#176)
commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 17:02:48 2022 +0800
(cold_on_s3) fix rowset write timestamp bug (apache#175)
commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 16:14:56 2022 +0800
(cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)
commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 11:17:23 2022 +0800
(cold_on_s3) fix compile error and thread pool core
commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon Jun 13 19:15:37 2022 +0800
(cold_on_s3) Revert FilePathDesc (apache#136)
* change cooldown pick rowset strategy; add/modify some log
* revert FilePathDesc
* return unsupport error to avoid undefined behavior
* patch to reduce path copy
* fix be ut
commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:36 2022 +0800
(cold_on_s3) Add ut for s3 read (apache#147)
commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:21 2022 +0800
(cold_on_s3) Add metrics for file system operation (apache#146)
* Add metrics for file system operation
* fix status code
commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <565620795@qq.com>
Date: Mon Jun 13 12:53:10 2022 +0800
(cold_on_s3) support show proc backends return remote used capacity
commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <565620795@qq.com>
Date: Fri Jun 10 16:08:09 2022 +0800
(cold_on_s3) support storage policy use default policy
commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <565620795@qq.com>
Date: Thu Jun 9 10:40:46 2022 +0800
(cold_on_s3) create table support different partition use their own storage policy
commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 10 22:28:40 2022 +0800
(cold_on_s3) Implement some fs methods (apache#163)
* implement some fs methods
* add more log info
commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <1091517373@qq.com>
Date: Thu Jun 9 23:12:58 2022 +0800
[fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)
apache#9986
commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 9 16:17:06 2022 +0800
[hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)
commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <814383175@qq.com>
Date: Wed Jun 8 16:09:35 2022 +0800
(cold_on_s3) Optimize cooldown policy (apache#113)
* Optimize cooldown policy
* fix ut
* remove useless log
* add ut
* add ut
* rename var
* fix typo and ut
commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed Jun 8 16:02:41 2022 +0800
provide quick_cooldown switch for regression test (apache#132)
commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <565620795@qq.com>
Date: Tue Jun 7 18:05:01 2022 +0800
(cold_on_s3) fix multiple fe synchronization resource storage policy exit
commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 11:38:46 2022 +0800
(cold_on_s3) fix move bug
commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <565620795@qq.com>
Date: Thu Jun 2 11:24:16 2022 +0800
(cold_on_s3) fix clang report '_exec_env' is not used
commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 00:05:42 2022 +0800
(cold_on_s3) reformat be code to make ci happy
commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 23:44:56 2022 +0800
(cold_on_s3) fix be ut
commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 22:36:37 2022 +0800
(cold_on_s3) fix fe compile error
commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu May 26 11:15:31 2022 +0800
[Enhancement](cold_on_s3) refresh fs map (apache#67)
* refresh fs map
* fetch cooldown ttl from storage policy
commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 21:28:06 2022 +0800
(cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)
* time from the beginning of upload
* compact to appropriate state before upload
* only do compaction on local rowsets
commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 19:22:25 2022 +0800
(cold_on_s3) use noncopy iostream for s3 get object (apache#54)
commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 18:42:09 2022 +0800
(cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)
commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <565620795@qq.com>
Date: Tue May 24 12:58:43 2022 +0800
(cold_on_s3) support storage policy change fix thrift merge redefine
commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <565620795@qq.com>
Date: Tue May 24 11:45:13 2022 +0800
(cold_on_s3) support storage policy change and create table add storage policy name (apache#39)
* [feature](cold-hot)support storage policy change and create table add storage policy name
* support storage policy change and create table add storage policy name, check resource policy exist when create table.
* when create table use storage policy check it exist
* support storage policy change and create table add storage policy name and when create table use storage policy check it exist
* support storage policy change fix some bug.
* fix some code style
commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <565620795@qq.com>
Date: Sat May 21 07:37:43 2022 -0700
(cold_on_s3) support storage policy (apache#28)
* [feature](cold-hot)support storage policy
* fix review
* fix review v1
commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri May 20 16:22:36 2022 +0800
(cold_on_s3) Batch upload to improve upload speed (apache#38)
* test upload duration
* use s3 transfer
* batch upload
* some fix in s3 filesystem
commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <814383175@qq.com>
Date: Wed May 18 15:08:40 2022 +0800
(cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)
* Implement periodic uploading of rowsets to remote storage
* add queue size check
* add test config for cooldwon
* modify read path
commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 17:22:19 2022 +0800
(cold_on_s3) fe distinguish drop partition (apache#26)
commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <565620795@qq.com>
Date: Mon May 16 21:02:51 2022 -0700
(cold_on_s3) support clone task don't download file in s3 (apache#15)
* apache#12 support clone task don't download file in s3
* apache#12 add clone ut
* fix code format
* fix commit apache#12
commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 12:01:32 2022 +0800
[Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24)
* disk capacity differentiates between local and remote
* fe drop table
commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon May 16 11:55:42 2022 +0800
[Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)
* update is_local logical
* remove remote rowsets when drop table/partition
commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <814383175@qq.com>
Date: Mon May 16 11:17:30 2022 +0800
(cold_on_s3) Implement reading rowset data from remote storage (apache#11)
* adjust io stack
* support s3
* refine code
* fix ut
* remove useless code
* rename rblock
* fix err msg
commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed May 11 16:29:54 2022 +0800
(cold_on_s3) Implement tablet cooldown (apache#10)
* tablet cooldown
* add global_local_filesystem to keep compatible
* test tablet cooldown
* revert engine_storage_migration_task_test.cpp
commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <platonekosama@gmail.com>
Date: Sun May 1 12:06:41 2022 +0800
(cold_on_s3) Rowset write path using fs (apache#7)
* replace WritableBlock with FileWriter
* local file system
* set fs when load dir
* fix be ut
commit 3585fe6
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue Jun 7 11:31:49 2022 +0800
[fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)
* fix tracker 0602
* fix format
commit 429a41d
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue May 31 19:12:42 2022 +0800
[bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate
commit 89d3a36
Author: Pxl <pxl290@qq.com>
Date: Thu May 26 16:51:01 2022 +0800
[Bug] [Vectorized] add padding when load char type data (apache#9734)
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: deardeng <565620795@qq.com>
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jul 6, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <platonekosama@gmail.com>
Date: Mon Jun 20 15:37:21 2022 +0800
temporarily avoid src/io conflicts with doris/master
commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 17 20:59:41 2022 +0800
(cold_on_s3) Report remote data size in tablets (apache#195)
* be report tablet remote size
* update TTabletStat
* fe report tablet remote data size
commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 16:06:23 2022 +0800
(cold_on_s3) support modify partition add storage policy
commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 10:24:14 2022 +0800
(cold_on_s3) support alter table add storage policy
commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 16:01:02 2022 +0800
(cold_on_s3) directly upload rowset exceeded ttl (apache#186)
* directly upload rowset exceed ttl
* clang-format
commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 11:39:08 2022 +0800
(cold_on_s3) Implement and rewrite some filesystem methods (apache#183)
* remove temporary test code
* no need to send fs to LocalFileWriter ctor
* rewrite some fs methods
* fix opened fd leak
commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 18:07:52 2022 +0800
(cold_on_s3) fix log printing (apache#176)
commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 17:02:48 2022 +0800
(cold_on_s3) fix rowset write timestamp bug (apache#175)
commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 16:14:56 2022 +0800
(cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)
commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 11:17:23 2022 +0800
(cold_on_s3) fix compile error and thread pool core
commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon Jun 13 19:15:37 2022 +0800
(cold_on_s3) Revert FilePathDesc (apache#136)
* change cooldown pick rowset strategy; add/modify some log
* revert FilePathDesc
* return unsupport error to avoid undefined behavior
* patch to reduce path copy
* fix be ut
commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:36 2022 +0800
(cold_on_s3) Add ut for s3 read (apache#147)
commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:21 2022 +0800
(cold_on_s3) Add metrics for file system operation (apache#146)
* Add metrics for file system operation
* fix status code
commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <565620795@qq.com>
Date: Mon Jun 13 12:53:10 2022 +0800
(cold_on_s3) support show proc backends return remote used capacity
commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <565620795@qq.com>
Date: Fri Jun 10 16:08:09 2022 +0800
(cold_on_s3) support storage policy use default policy
commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <565620795@qq.com>
Date: Thu Jun 9 10:40:46 2022 +0800
(cold_on_s3) create table support different partition use their own storage policy
commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 10 22:28:40 2022 +0800
(cold_on_s3) Implement some fs methods (apache#163)
* implement some fs methods
* add more log info
commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <1091517373@qq.com>
Date: Thu Jun 9 23:12:58 2022 +0800
[fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)
apache#9986
commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 9 16:17:06 2022 +0800
[hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)
commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <814383175@qq.com>
Date: Wed Jun 8 16:09:35 2022 +0800
(cold_on_s3) Optimize cooldown policy (apache#113)
* Optimize cooldown policy
* fix ut
* remove useless log
* add ut
* add ut
* rename var
* fix typo and ut
commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed Jun 8 16:02:41 2022 +0800
provide quick_cooldown switch for regression test (apache#132)
commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <565620795@qq.com>
Date: Tue Jun 7 18:05:01 2022 +0800
(cold_on_s3) fix multiple fe synchronization resource storage policy exit
commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 11:38:46 2022 +0800
(cold_on_s3) fix move bug
commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <565620795@qq.com>
Date: Thu Jun 2 11:24:16 2022 +0800
(cold_on_s3) fix clang report '_exec_env' is not used
commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 00:05:42 2022 +0800
(cold_on_s3) reformat be code to make ci happy
commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 23:44:56 2022 +0800
(cold_on_s3) fix be ut
commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 22:36:37 2022 +0800
(cold_on_s3) fix fe compile error
commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu May 26 11:15:31 2022 +0800
[Enhancement](cold_on_s3) refresh fs map (apache#67)
* refresh fs map
* fetch cooldown ttl from storage policy
commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 21:28:06 2022 +0800
(cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)
* time from the beginning of upload
* compact to appropriate state before upload
* only do compaction on local rowsets
commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 19:22:25 2022 +0800
(cold_on_s3) use noncopy iostream for s3 get object (apache#54)
commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 18:42:09 2022 +0800
(cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)
commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <565620795@qq.com>
Date: Tue May 24 12:58:43 2022 +0800
(cold_on_s3) support storage policy change fix thrift merge redefine
commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <565620795@qq.com>
Date: Tue May 24 11:45:13 2022 +0800
(cold_on_s3) support storage policy change and create table add storage policy name (apache#39)
* [feature](cold-hot)support storage policy change and create table add storage policy name
* support storage policy change and create table add storage policy name, check resource policy exist when create table.
* when create table use storage policy check it exist
* support storage policy change and create table add storage policy name and when create table use storage policy check it exist
* support storage policy change fix some bug.
* fix some code style
commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <565620795@qq.com>
Date: Sat May 21 07:37:43 2022 -0700
(cold_on_s3) support storage policy (apache#28)
* [feature](cold-hot)support storage policy
* fix review
* fix review v1
commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri May 20 16:22:36 2022 +0800
(cold_on_s3) Batch upload to improve upload speed (apache#38)
* test upload duration
* use s3 transfer
* batch upload
* some fix in s3 filesystem
commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <814383175@qq.com>
Date: Wed May 18 15:08:40 2022 +0800
(cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)
* Implement periodic uploading of rowsets to remote storage
* add queue size check
* add test config for cooldwon
* modify read path
commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 17:22:19 2022 +0800
(cold_on_s3) fe distinguish drop partition (apache#26)
commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <565620795@qq.com>
Date: Mon May 16 21:02:51 2022 -0700
(cold_on_s3) support clone task don't download file in s3 (apache#15)
* apache#12 support clone task don't download file in s3
* apache#12 add clone ut
* fix code format
* fix commit apache#12
commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 12:01:32 2022 +0800
[Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24)
* disk capacity differentiates between local and remote
* fe drop table
commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon May 16 11:55:42 2022 +0800
[Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)
* update is_local logical
* remove remote rowsets when drop table/partition
commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <814383175@qq.com>
Date: Mon May 16 11:17:30 2022 +0800
(cold_on_s3) Implement reading rowset data from remote storage (apache#11)
* adjust io stack
* support s3
* refine code
* fix ut
* remove useless code
* rename rblock
* fix err msg
commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed May 11 16:29:54 2022 +0800
(cold_on_s3) Implement tablet cooldown (apache#10)
* tablet cooldown
* add global_local_filesystem to keep compatible
* test tablet cooldown
* revert engine_storage_migration_task_test.cpp
commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <platonekosama@gmail.com>
Date: Sun May 1 12:06:41 2022 +0800
(cold_on_s3) Rowset write path using fs (apache#7)
* replace WritableBlock with FileWriter
* local file system
* set fs when load dir
* fix be ut
commit 3585fe6
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue Jun 7 11:31:49 2022 +0800
[fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)
* fix tracker 0602
* fix format
commit 429a41d
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue May 31 19:12:42 2022 +0800
[bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate
commit 89d3a36
Author: Pxl <pxl290@qq.com>
Date: Thu May 26 16:51:01 2022 +0800
[Bug] [Vectorized] add padding when load char type data (apache#9734)
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: deardeng <565620795@qq.com>
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jul 7, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <platonekosama@gmail.com>
Date: Mon Jun 20 15:37:21 2022 +0800
temporarily avoid src/io conflicts with doris/master
commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 17 20:59:41 2022 +0800
(cold_on_s3) Report remote data size in tablets (apache#195)
* be report tablet remote size
* update TTabletStat
* fe report tablet remote data size
commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 16:06:23 2022 +0800
(cold_on_s3) support modify partition add storage policy
commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <565620795@qq.com>
Date: Fri Jun 17 10:24:14 2022 +0800
(cold_on_s3) support alter table add storage policy
commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 16:01:02 2022 +0800
(cold_on_s3) directly upload rowset exceeded ttl (apache#186)
* directly upload rowset exceed ttl
* clang-format
commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 16 11:39:08 2022 +0800
(cold_on_s3) Implement and rewrite some filesystem methods (apache#183)
* remove temporary test code
* no need to send fs to LocalFileWriter ctor
* rewrite some fs methods
* fix opened fd leak
commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 18:07:52 2022 +0800
(cold_on_s3) fix log printing (apache#176)
commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <814383175@qq.com>
Date: Tue Jun 14 17:02:48 2022 +0800
(cold_on_s3) fix rowset write timestamp bug (apache#175)
commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 16:14:56 2022 +0800
(cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)
commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <565620795@qq.com>
Date: Tue Jun 14 11:17:23 2022 +0800
(cold_on_s3) fix compile error and thread pool core
commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon Jun 13 19:15:37 2022 +0800
(cold_on_s3) Revert FilePathDesc (apache#136)
* change cooldown pick rowset strategy; add/modify some log
* revert FilePathDesc
* return unsupport error to avoid undefined behavior
* patch to reduce path copy
* fix be ut
commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:36 2022 +0800
(cold_on_s3) Add ut for s3 read (apache#147)
commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <814383175@qq.com>
Date: Mon Jun 13 16:07:21 2022 +0800
(cold_on_s3) Add metrics for file system operation (apache#146)
* Add metrics for file system operation
* fix status code
commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <565620795@qq.com>
Date: Mon Jun 13 12:53:10 2022 +0800
(cold_on_s3) support show proc backends return remote used capacity
commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <565620795@qq.com>
Date: Fri Jun 10 16:08:09 2022 +0800
(cold_on_s3) support storage policy use default policy
commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <565620795@qq.com>
Date: Thu Jun 9 10:40:46 2022 +0800
(cold_on_s3) create table support different partition use their own storage policy
commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri Jun 10 22:28:40 2022 +0800
(cold_on_s3) Implement some fs methods (apache#163)
* implement some fs methods
* add more log info
commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <1091517373@qq.com>
Date: Thu Jun 9 23:12:58 2022 +0800
[fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)
apache#9986
commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu Jun 9 16:17:06 2022 +0800
[hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)
commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <814383175@qq.com>
Date: Wed Jun 8 16:09:35 2022 +0800
(cold_on_s3) Optimize cooldown policy (apache#113)
* Optimize cooldown policy
* fix ut
* remove useless log
* add ut
* add ut
* rename var
* fix typo and ut
commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed Jun 8 16:02:41 2022 +0800
provide quick_cooldown switch for regression test (apache#132)
commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <565620795@qq.com>
Date: Tue Jun 7 18:05:01 2022 +0800
(cold_on_s3) fix multiple fe synchronization resource storage policy exit
commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 11:38:46 2022 +0800
(cold_on_s3) fix move bug
commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <565620795@qq.com>
Date: Thu Jun 2 11:24:16 2022 +0800
(cold_on_s3) fix clang report '_exec_env' is not used
commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <platonekosama@gmail.com>
Date: Thu Jun 2 00:05:42 2022 +0800
(cold_on_s3) reformat be code to make ci happy
commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 23:44:56 2022 +0800
(cold_on_s3) fix be ut
commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <platonekosama@gmail.com>
Date: Wed Jun 1 22:36:37 2022 +0800
(cold_on_s3) fix fe compile error
commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Thu May 26 11:15:31 2022 +0800
[Enhancement](cold_on_s3) refresh fs map (apache#67)
* refresh fs map
* fetch cooldown ttl from storage policy
commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 21:28:06 2022 +0800
(cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)
* time from the beginning of upload
* compact to appropriate state before upload
* only do compaction on local rowsets
commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 19:22:25 2022 +0800
(cold_on_s3) use noncopy iostream for s3 get object (apache#54)
commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 24 18:42:09 2022 +0800
(cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)
commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <565620795@qq.com>
Date: Tue May 24 12:58:43 2022 +0800
(cold_on_s3) support storage policy change fix thrift merge redefine
commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <565620795@qq.com>
Date: Tue May 24 11:45:13 2022 +0800
(cold_on_s3) support storage policy change and create table add storage policy name (apache#39)
* [feature](cold-hot)support storage policy change and create table add storage policy name
* support storage policy change and create table add storage policy name, check resource policy exist when create table.
* when create table use storage policy check it exist
* support storage policy change and create table add storage policy name and when create table use storage policy check it exist
* support storage policy change fix some bug.
* fix some code style
commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <565620795@qq.com>
Date: Sat May 21 07:37:43 2022 -0700
(cold_on_s3) support storage policy (apache#28)
* [feature](cold-hot)support storage policy
* fix review
* fix review v1
commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Fri May 20 16:22:36 2022 +0800
(cold_on_s3) Batch upload to improve upload speed (apache#38)
* test upload duration
* use s3 transfer
* batch upload
* some fix in s3 filesystem
commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <814383175@qq.com>
Date: Wed May 18 15:08:40 2022 +0800
(cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)
* Implement periodic uploading of rowsets to remote storage
* add queue size check
* add test config for cooldwon
* modify read path
commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 17:22:19 2022 +0800
(cold_on_s3) fe distinguish drop partition (apache#26)
commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <565620795@qq.com>
Date: Mon May 16 21:02:51 2022 -0700
(cold_on_s3) support clone task don't download file in s3 (apache#15)
* apache#12 support clone task don't download file in s3
* apache#12 add clone ut
* fix code format
* fix commit apache#12
commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Tue May 17 12:01:32 2022 +0800
[Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24)
* disk capacity differentiates between local and remote
* fe drop table
commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Mon May 16 11:55:42 2022 +0800
[Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)
* update is_local logical
* remove remote rowsets when drop table/partition
commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <814383175@qq.com>
Date: Mon May 16 11:17:30 2022 +0800
(cold_on_s3) Implement reading rowset data from remote storage (apache#11)
* adjust io stack
* support s3
* refine code
* fix ut
* remove useless code
* rename rblock
* fix err msg
commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <36853835+platoneko@users.noreply.github.com>
Date: Wed May 11 16:29:54 2022 +0800
(cold_on_s3) Implement tablet cooldown (apache#10)
* tablet cooldown
* add global_local_filesystem to keep compatible
* test tablet cooldown
* revert engine_storage_migration_task_test.cpp
commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <platonekosama@gmail.com>
Date: Sun May 1 12:06:41 2022 +0800
(cold_on_s3) Rowset write path using fs (apache#7)
* replace WritableBlock with FileWriter
* local file system
* set fs when load dir
* fix be ut
commit 3585fe6
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue Jun 7 11:31:49 2022 +0800
[fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)
* fix tracker 0602
* fix format
commit 429a41d
Author: Xinyi Zou <zouxinyi02@gmail.com>
Date: Tue May 31 19:12:42 2022 +0800
[bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate
commit 89d3a36
Author: Pxl <pxl290@qq.com>
Date: Thu May 26 16:51:01 2022 +0800
[Bug] [Vectorized] add padding when load char type data (apache#9734)
Co-authored-by: Luwei <814383175@qq.com>
Co-authored-by: deardeng <565620795@qq.com>
3 tasks
3 tasks
starocean999
pushed a commit
to starocean999/incubator-doris
that referenced
this pull request
Jul 26, 2023
morrySnow
pushed a commit
that referenced
this pull request
Apr 21, 2024
…is not enough to provide all the data for the query (#33800) When the materialized view is not enough to provide all the data for the query, if the materialized view is increment update by partition. we can union materialized view and origin query to reponse the query. this depends on #33362 such as materialized view def is as following: > CREATE MATERIALIZED VIEW mv_10086 > BUILD IMMEDIATE REFRESH AUTO ON MANUAL > partition by(l_shipdate) > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ('replication_num' = '1') > AS > select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total > from lineitem > left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate > group by > l_shipdate, > o_orderdate, > l_partkey, > l_suppkey; the materialized view data is as following: +------------+-------------+-----------+-----------+-----------+ | l_shipdate | o_orderdate | l_partkey | l_suppkey | sum_total | +------------+-------------+-----------+-----------+-----------+ | 2023-10-18 | 2023-10-18 | 2 | 3 | 109.20 | | 2023-10-17 | 2023-10-17 | 2 | 3 | 99.50 | | 2023-10-19 | 2023-10-19 | 2 | 3 | 99.50 | +------------+-------------+-----------+-----------+-----------+ when we insert data to partition `2023-10-17`, if we run query as following ``` select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total from lineitem left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate group by l_shipdate, o_orderdate, l_partkey, l_suppkey; ``` query rewrite by materialzied view will fail with message `Check partition query used validation fail` if we turn on the switch `SET enable_materialized_view_union_rewrite = true;` default true we run the query above again, it will success and will use union all materialized view and origin query to response the query correctly. the plan is as following: ``` | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | l_shipdate[#52] | | o_orderdate[#53] | | l_partkey[#54] | | l_suppkey[#55] | | sum_total[#56] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 11:VEXCHANGE | | offset: 0 | | distribute expr lists: | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 11 | | UNPARTITIONED | | | | 10:VUNION(756) | | | | | |----9:VAGGREGATE (merge finalize)(753) | | | | output: sum(partial_sum(o_totalprice)[#46])[#51] | | | | group by: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | cardinality=2 | | | | distribute expr lists: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | | | | 8:VEXCHANGE | | | offset: 0 | | | distribute expr lists: l_shipdate[#42] | | | | | 1:VEXCHANGE | | offset: 0 | | distribute expr lists: | | | | PLAN FRAGMENT 2 | | | | PARTITION: HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 08 | | HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | 7:VAGGREGATE (update serialize)(747) | | | STREAMING | | | output: partial_sum(o_totalprice[#41])[#46] | | | group by: l_shipdate[#37], o_orderdate[#38], l_partkey[#39], l_suppkey[#40] | | | cardinality=2 | | | distribute expr lists: l_shipdate[#37] | | | | | 6:VHASH JOIN(741) | | | join op: RIGHT OUTER JOIN(PARTITIONED)[] | | | equal join conjunct: (o_orderkey[#21] = l_orderkey[#5]) | | | equal join conjunct: (o_orderdate[#25] = l_shipdate[#15]) | | | runtime filters: RF000[min_max] <- l_orderkey[#5](2/2/2048), RF001[bloom] <- l_orderkey[#5](2/2/2048), RF002[min_max] <- l_shipdate[#15](1/1/2048), RF003[bloom] <- l_shipdate[#15](1/1/2048) | | | cardinality=2 | | | vec output tuple id: 4 | | | output tuple id: 4 | | | vIntermediate tuple ids: 3 | | | hash output slot ids: 6 7 24 25 15 | | | final projections: l_shipdate[#36], o_orderdate[#32], l_partkey[#34], l_suppkey[#35], o_totalprice[#31] | | | final project output tuple id: 4 | | | distribute expr lists: o_orderkey[#21], o_orderdate[#25] | | | distribute expr lists: l_orderkey[#5], l_shipdate[#15] | | | | | |----3:VEXCHANGE | | | offset: 0 | | | distribute expr lists: l_orderkey[#5] | | | | | 5:VEXCHANGE | | offset: 0 | | distribute expr lists: | | | | PLAN FRAGMENT 3 | | | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 05 | | HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25] | | | | 4:VOlapScanNode(722) | | TABLE: union_db.orders(orders), PREAGGREGATION: ON | | runtime filters: RF000[min_max] -> o_orderkey[#21], RF001[bloom] -> o_orderkey[#21], RF002[min_max] -> o_orderdate[#25], RF003[bloom] -> o_orderdate[#25] | | partitions=3/3 (p_20231017,p_20231018,p_20231019), tablets=9/9, tabletList=161188,161190,161192 ... | | cardinality=3, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | PLAN FRAGMENT 4 | | | | PARTITION: HASH_PARTITIONED: l_orderkey[#5] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 03 | | HASH_PARTITIONED: l_orderkey[#5], l_shipdate[#15] | | | | 2:VOlapScanNode(729) | | TABLE: union_db.lineitem(lineitem), PREAGGREGATION: ON | | PREDICATES: (l_shipdate[#15] >= '2023-10-17') AND (l_shipdate[#15] < '2023-10-18') | | partitions=1/3 (p_20231017), tablets=3/3, tabletList=161223,161225,161227 | | cardinality=2, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | PLAN FRAGMENT 5 | | | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 01 | | RANDOM | | | | 0:VOlapScanNode(718) | | TABLE: union_db.mv_10086(mv_10086), PREAGGREGATION: ON | | partitions=2/3 (p_20231018_20231019,p_20231019_20231020), tablets=4/4, tabletList=161251,161253,161265 ... | | cardinality=2, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | MaterializedView | | MaterializedViewRewriteSuccessAndChose: | | Names: mv_10086 | | MaterializedViewRewriteSuccessButNotChose: | | | | MaterializedViewRewriteFail: | +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ```
yiguolei
pushed a commit
that referenced
this pull request
Apr 21, 2024
…is not enough to provide all the data for the query (#33800) When the materialized view is not enough to provide all the data for the query, if the materialized view is increment update by partition. we can union materialized view and origin query to reponse the query. this depends on #33362 such as materialized view def is as following: > CREATE MATERIALIZED VIEW mv_10086 > BUILD IMMEDIATE REFRESH AUTO ON MANUAL > partition by(l_shipdate) > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ('replication_num' = '1') > AS > select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total > from lineitem > left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate > group by > l_shipdate, > o_orderdate, > l_partkey, > l_suppkey; the materialized view data is as following: +------------+-------------+-----------+-----------+-----------+ | l_shipdate | o_orderdate | l_partkey | l_suppkey | sum_total | +------------+-------------+-----------+-----------+-----------+ | 2023-10-18 | 2023-10-18 | 2 | 3 | 109.20 | | 2023-10-17 | 2023-10-17 | 2 | 3 | 99.50 | | 2023-10-19 | 2023-10-19 | 2 | 3 | 99.50 | +------------+-------------+-----------+-----------+-----------+ when we insert data to partition `2023-10-17`, if we run query as following ``` select l_shipdate, o_orderdate, l_partkey, l_suppkey, sum(o_totalprice) as sum_total from lineitem left join orders on lineitem.l_orderkey = orders.o_orderkey and l_shipdate = o_orderdate group by l_shipdate, o_orderdate, l_partkey, l_suppkey; ``` query rewrite by materialzied view will fail with message `Check partition query used validation fail` if we turn on the switch `SET enable_materialized_view_union_rewrite = true;` default true we run the query above again, it will success and will use union all materialized view and origin query to response the query correctly. the plan is as following: ``` | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | l_shipdate[#52] | | o_orderdate[#53] | | l_partkey[#54] | | l_suppkey[#55] | | sum_total[#56] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 11:VEXCHANGE | | offset: 0 | | distribute expr lists: | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 11 | | UNPARTITIONED | | | | 10:VUNION(756) | | | | | |----9:VAGGREGATE (merge finalize)(753) | | | | output: sum(partial_sum(o_totalprice)[#46])[#51] | | | | group by: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | cardinality=2 | | | | distribute expr lists: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | | | | 8:VEXCHANGE | | | offset: 0 | | | distribute expr lists: l_shipdate[#42] | | | | | 1:VEXCHANGE | | offset: 0 | | distribute expr lists: | | | | PLAN FRAGMENT 2 | | | | PARTITION: HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 08 | | HASH_PARTITIONED: l_shipdate[#42], o_orderdate[#43], l_partkey[#44], l_suppkey[#45] | | | | 7:VAGGREGATE (update serialize)(747) | | | STREAMING | | | output: partial_sum(o_totalprice[#41])[#46] | | | group by: l_shipdate[#37], o_orderdate[#38], l_partkey[#39], l_suppkey[#40] | | | cardinality=2 | | | distribute expr lists: l_shipdate[#37] | | | | | 6:VHASH JOIN(741) | | | join op: RIGHT OUTER JOIN(PARTITIONED)[] | | | equal join conjunct: (o_orderkey[#21] = l_orderkey[#5]) | | | equal join conjunct: (o_orderdate[#25] = l_shipdate[#15]) | | | runtime filters: RF000[min_max] <- l_orderkey[#5](2/2/2048), RF001[bloom] <- l_orderkey[#5](2/2/2048), RF002[min_max] <- l_shipdate[#15](1/1/2048), RF003[bloom] <- l_shipdate[#15](1/1/2048) | | | cardinality=2 | | | vec output tuple id: 4 | | | output tuple id: 4 | | | vIntermediate tuple ids: 3 | | | hash output slot ids: 6 7 24 25 15 | | | final projections: l_shipdate[#36], o_orderdate[#32], l_partkey[#34], l_suppkey[#35], o_totalprice[#31] | | | final project output tuple id: 4 | | | distribute expr lists: o_orderkey[#21], o_orderdate[#25] | | | distribute expr lists: l_orderkey[#5], l_shipdate[#15] | | | | | |----3:VEXCHANGE | | | offset: 0 | | | distribute expr lists: l_orderkey[#5] | | | | | 5:VEXCHANGE | | offset: 0 | | distribute expr lists: | | | | PLAN FRAGMENT 3 | | | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 05 | | HASH_PARTITIONED: o_orderkey[#21], o_orderdate[#25] | | | | 4:VOlapScanNode(722) | | TABLE: union_db.orders(orders), PREAGGREGATION: ON | | runtime filters: RF000[min_max] -> o_orderkey[#21], RF001[bloom] -> o_orderkey[#21], RF002[min_max] -> o_orderdate[#25], RF003[bloom] -> o_orderdate[#25] | | partitions=3/3 (p_20231017,p_20231018,p_20231019), tablets=9/9, tabletList=161188,161190,161192 ... | | cardinality=3, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | PLAN FRAGMENT 4 | | | | PARTITION: HASH_PARTITIONED: l_orderkey[#5] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 03 | | HASH_PARTITIONED: l_orderkey[#5], l_shipdate[#15] | | | | 2:VOlapScanNode(729) | | TABLE: union_db.lineitem(lineitem), PREAGGREGATION: ON | | PREDICATES: (l_shipdate[#15] >= '2023-10-17') AND (l_shipdate[#15] < '2023-10-18') | | partitions=1/3 (p_20231017), tablets=3/3, tabletList=161223,161225,161227 | | cardinality=2, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | PLAN FRAGMENT 5 | | | | PARTITION: RANDOM | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 01 | | RANDOM | | | | 0:VOlapScanNode(718) | | TABLE: union_db.mv_10086(mv_10086), PREAGGREGATION: ON | | partitions=2/3 (p_20231018_20231019,p_20231019_20231020), tablets=4/4, tabletList=161251,161253,161265 ... | | cardinality=2, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | MaterializedView | | MaterializedViewRewriteSuccessAndChose: | | Names: mv_10086 | | MaterializedViewRewriteSuccessButNotChose: | | | | MaterializedViewRewriteFail: | +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ```
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…om xxxx'