S3 zero copy replication #16240

ianton-ru · 2020-10-21T16:00:30Z

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Zero-copy replication for ReplicatedMergeTree over S3 storage

Detailed description / Documentation draft:

Zero-copy replication over S3 storage

filimonov · 2020-10-23T18:36:28Z

S3ZeroCopyReplication.md

+Приемник в ответ отсылает куку send_s3_metadata=1 в случае, если идут метаданные. В остальных случаях отсылаются данные, как и прежде.
+
+Применик перед запросом смотрит, будет ли хранить данные в S3. Проверка сейчас кривая - если в сторадже есть S3, то считаем, что будет S3.
+Если да S3, то отсылает в запросе send_s3_metadata=1.


Строго говоря не исключена ситуация когда и приёмник и передатчик на s3 но банкеты разные. Наверное надо адрес бакета отправлять или какой-то хэш от него.

Адрес может быть разным в случае например нахождения нод в разных ДЦ и использования разных прокси для доступа к S3. В качестве страховки приемник после получения метаданных проверяет доступность (фактически - наличие в S3 первого объекта от первого файла, не скачивая, в просто через list), и если данные не доступны, то сваливается в старый механизм с полноценной копией.

Бонусом дешево получилась возможность работы с несколькими разными S3, когда у разных нод разная очередность (перебирает все S3 в сторадже в поисках нужного). Но непонятно, кому это может пригодиться. :)

src/Storages/MergeTree/DataPartsExchange.cpp

Sallery-X · 2021-03-08T03:14:50Z

How to solve the cache consistency problem of different replicas?

In Clickhouse data is "append only". Sometimes entry part can be deleted. So could you please describe where you see a problem?

If multiple replicas share the same storage, how to resolve the share file confliction when the diffrent replicas to excute insert and merge?

Sallery-X · 2021-03-09T02:47:37Z

I think need to consider the ddl worker, no need to do the ddl on the all replicas.

ianton-ru · 2021-03-09T09:15:36Z

How to solve the cache consistency problem of different replicas?

In Clickhouse data is "append only". Sometimes entry part can be deleted. So could you please describe where you see a problem?

If multiple replicas share the same storage, how to resolve the share file confliction when the diffrent replicas to excute insert and merge?

On insert one replica creates all files of part, after that files never changes.
On merge replica makes a lock in zookeeper, other replicas wait for end and get merged part.

…o s3_zero_copy_replication

alesapin

Ok, but need to remove suspicious if.

src/Storages/StorageReplicatedMergeTree.cpp

alesapin · 2021-03-12T08:06:53Z

Yarrr! Conflict on ErrorCode.

…o s3_zero_copy_replication

boqu · 2021-04-13T07:56:32Z

Can I know whether this change will be merged to 21.3-lts? If yes, do we have any timeline? Thanks!

alexey-milovidov · 2021-05-25T03:29:44Z

src/Disks/DiskType.h

+namespace DB
+{
+
+struct DiskType


This looks like antipattern.
And there is zero comments in this file 😭

What does it mean "disk type" and why do we need to discriminate them?
Maybe remove this file and all its usages?

ianton-ru added 7 commits October 8, 2020 18:45

S3 zero copy replication proof of concept

acf8656

S3 zero copy replication proof of concept - description

a4adb39

Merge master

766dbfd

Move Zookeeper lock for S3 shared part in IMergeTreeDataPart

9272ed0

Zero copy replication over S3: check s3 storage

b877459

Zero copy replication over S3: fetch instead of merge

14a78f8

Zero copy replication over S3: base tests

fb178ef

robot-clickhouse added doc-alert pr-feature Pull request with new product feature labels Oct 21, 2020

Merge branch 'master' into s3_zero_copy_replication

f833501

ianton-ru force-pushed the s3_zero_copy_replication branch 2 times, most recently from 123025c to d5bc7ad Compare October 22, 2020 11:26

Fix style, fix build

652c56e

ianton-ru force-pushed the s3_zero_copy_replication branch from d5bc7ad to 652c56e Compare October 22, 2020 13:07

S3 zero copy replication: fix tests

1ffe0b1

ianton-ru force-pushed the s3_zero_copy_replication branch from 698e68c to 4f7065e Compare October 23, 2020 12:02

S3 zero copy replication: fix virtual method default parameter

e3879af

ianton-ru force-pushed the s3_zero_copy_replication branch from 4f7065e to e3879af Compare October 23, 2020 13:28

filimonov reviewed Oct 23, 2020

View reviewed changes

ianton-ru commented Nov 2, 2020

View reviewed changes

src/Storages/MergeTree/DataPartsExchange.cpp Outdated Show resolved Hide resolved

ianton-ru added 2 commits November 3, 2020 12:20

S3 zero copy replication: more simple s3 check

7802171

Merge master

1742fb3

ianton-ru force-pushed the s3_zero_copy_replication branch from 0899321 to 1742fb3 Compare November 3, 2020 10:41

alex-zaitsev mentioned this pull request Nov 5, 2020

Is replicated engine still useful if DiskS3 is used? #16706

Closed

alexey-milovidov mentioned this pull request Nov 30, 2020

Roadmap 2021 (discussion) #17623

Closed

ianton-ru added 2 commits December 16, 2020 18:31

Merge master

0f0500c

Zero copy replication over S3: Hybrid storage support

eba98b0

ianton-ru force-pushed the s3_zero_copy_replication branch from b8f9f57 to eba98b0 Compare January 18, 2021 16:17

ianton-ru changed the title ~~[WIP] Prototype/MVP/Proof-of-concept etc. S3 zero copy replication~~ S3 zero copy replication Jan 18, 2021

Merge master

357d98e

Sallery-X approved these changes Mar 9, 2021

View reviewed changes

alesapin added 2 commits March 9, 2021 10:59

Merge branch 'master' into ianton-ru-s3_zero_copy_replication

c00c8c3

Slightly better

d80c2ce

ianton-ru force-pushed the s3_zero_copy_replication branch from 885ae9e to 18ff577 Compare March 9, 2021 14:56

ianton-ru added 2 commits March 9, 2021 20:39

Use 'merge on single replica' option instead of zookeeper lock

265d293

Make method StorageReplicatedMergeTree::fetchExistsPart

aff13c0

ianton-ru force-pushed the s3_zero_copy_replication branch from 18ff577 to aff13c0 Compare March 9, 2021 17:50

ianton-ru added 3 commits March 10, 2021 14:08

Fix refreshState logic

deb0b83

Merge branch 'master' of https://github.com/ClickHouse/ClickHouse int…

cef0dda

…o s3_zero_copy_replication

Fix PVS detected errors

6f7800e

alesapin approved these changes Mar 11, 2021

View reviewed changes

src/Storages/StorageReplicatedMergeTree.cpp Outdated Show resolved Hide resolved

Remove debug lines

109cb63

alesapin and others added 3 commits March 12, 2021 11:07

Merge branch 'master' into s3_zero_copy_replication

b98e27b

Merge branch 'master' of https://github.com/ClickHouse/ClickHouse int…

ace628d

…o s3_zero_copy_replication

Merge branch 'master' into s3_zero_copy_replication

06eb2d8

alesapin merged commit dad5238 into ClickHouse:master Mar 14, 2021

godliness mentioned this pull request Mar 19, 2021

Build clickhouse in docker(On MacOS) failed due to case sensitive APFS. #21900

Closed

godliness referenced this pull request in BohuTANG/ClickHouse Mar 23, 2021

demo

f67d98e

godliness mentioned this pull request Mar 24, 2021

Clickhouse计算与存储分离调研 cloudnativecube/octopus#48

Open

alexey-milovidov reviewed May 25, 2021

View reviewed changes

filimonov mentioned this pull request May 25, 2021

question about execute_merges_on_single_replica_time_threshold feature #23300

Closed

UnamedRus mentioned this pull request May 29, 2021

Can multi ClickHouse share same S3 storage path, one rw, others ro #24752

Closed

zhaojintaozhao mentioned this pull request Jul 27, 2021

21.8 release checklist #26148

Closed

olgarev mentioned this pull request Aug 1, 2021

DOCSUP-12003: S3 and HDFS zero copy replication #27055

Merged

marising mentioned this pull request Mar 29, 2022

Are there any plans to support zero copy replication for AWS EFS? #35708

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3 zero copy replication #16240

S3 zero copy replication #16240

ianton-ru commented Oct 21, 2020 •

edited by alesapin

filimonov Oct 23, 2020

ianton-ru Oct 26, 2020

ianton-ru Oct 26, 2020

Sallery-X commented Mar 8, 2021

Sallery-X commented Mar 9, 2021

ianton-ru commented Mar 9, 2021 •

edited

alesapin left a comment

alesapin commented Mar 12, 2021

boqu commented Apr 13, 2021

alexey-milovidov May 25, 2021

S3 zero copy replication #16240

S3 zero copy replication #16240

Conversation

ianton-ru commented Oct 21, 2020 • edited by alesapin

filimonov Oct 23, 2020

Choose a reason for hiding this comment

ianton-ru Oct 26, 2020

Choose a reason for hiding this comment

ianton-ru Oct 26, 2020

Choose a reason for hiding this comment

Sallery-X commented Mar 8, 2021

Sallery-X commented Mar 9, 2021

ianton-ru commented Mar 9, 2021 • edited

alesapin left a comment

Choose a reason for hiding this comment

alesapin commented Mar 12, 2021

boqu commented Apr 13, 2021

alexey-milovidov May 25, 2021

Choose a reason for hiding this comment

ianton-ru commented Oct 21, 2020 •

edited by alesapin

ianton-ru commented Mar 9, 2021 •

edited