stream-nb-25-1: Backup encryption feature backport #19625

UgnineSirdis · 2025-06-12T06:25:33Z

Changelog entry

...

Changelog category

Not for changelog (changelog entry is not required)

Description for reviewers

...

…s of checksums, compression and encryption (ydb-platform#14862) (cherry picked from commit 499b85e)

(cherry picked from commit 00996cd)

…ns (ydb-platform#15116) (cherry picked from commit c5b2980)

(cherry picked from commit c09cf0f)

(cherry picked from commit 22d0ecf)

(cherry picked from commit b7e42a2)

…ashard code (ydb-platform#16055) (cherry picked from commit 064d1ba)

(cherry picked from commit f1d47d6)

(cherry picked from commit aae5680)

….h, implement fallback class for export metadata uploader (ydb-platform#16343) (cherry picked from commit 5f9dbe8)

….cpp (ydb-platform#16366) (cherry picked from commit 9fb5600)

(cherry picked from commit 4ae39ac)

… -> schemeshard_import_getters.h (ydb-platform#16729) (cherry picked from commit c95801c)

…pp -> schemeshard_import_getters.cpp (ydb-platform#16749) (cherry picked from commit d550d6c)

(cherry picked from commit bf5ada7)

…latform#17461) (cherry picked from commit 63874fb)

(cherry picked from commit f0122e1)

(cherry picked from commit 7ee4040)

(cherry picked from commit d18fee7)

…adata files (ydb-platform#17812) (cherry picked from commit 822094a)

(cherry picked from commit aff4b61)

…() implementation to cpp (ydb-platform#18030) (cherry picked from commit 4e56d1c)

(cherry picked from commit 6dde11e)

(cherry picked from commit 19d52ae)

ydb-platform#18384) (cherry picked from commit 1f4ccf3)

(cherry picked from commit b875f79)

(cherry picked from commit d8d0d97)

…ass ExportInfo by ref in functions (ydb-platform#18425) (cherry picked from commit d6e49c5)

(cherry picked from commit 5891045)

…tform#18454) (cherry picked from commit ddb9664)

…b-platform#18709) (cherry picked from commit 3506e58)

(cherry picked from commit 1329bba)

(cherry picked from commit 49e39df)

…rm#19331) (cherry picked from commit e1b310a)

"Fix topic export encryption IV (ydb-platform#19458)" (there were small changes for views, topics are not in this branch) (from commit fd2db0f)

…-platform#19390) (cherry picked from commit 6bf3311)

github-actions · 2025-06-12T06:28:48Z

⚪ 2025-06-12 06:28:47 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1c9266d has started.
⚪ 2025-06-12 06:29:00 UTC Artifacts will be uploaded here
⚪ 2025-06-12 06:32:20 UTC ya make is running...
🟡 2025-06-12 08:17:13 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
28937	26407	0	1	2467	62

⚪ 2025-06-12 08:19:40 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-06-12 08:39:56 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
135 (only retried tests)	75	0	0	1	59

🟢 2025-06-12 08:40:03 UTC Build successful.
🔴 2025-06-12 08:40:24 UTC ydbd size 2.1 GiB changed* by +2.3 MiB, which is >= 2.0 MiB vs stream-nb-25-1: Alert

ydbd size dash	stream-nb-25-1: `6c291f2`	merge: `1c9266d`	diff	diff %
ydbd size	2 258 225 816 Bytes	2 260 627 952 Bytes	+2.3 MiB	+0.106%
ydbd stripped size	478 008 824 Bytes	478 442 136 Bytes	+423.2 KiB	+0.091%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-06-12T06:29:04Z

⚪ 2025-06-12 06:29:03 UTC Pre-commit check linux-x86_64-release-asan for 1c9266d has started.
⚪ 2025-06-12 06:29:14 UTC Artifacts will be uploaded here
⚪ 2025-06-12 06:32:29 UTC ya make is running...
🟡 2025-06-12 08:22:49 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14943	14807	0	32	66	38

⚪ 2025-06-12 08:24:07 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-06-12 08:44:18 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
179 (only retried tests)	83	0	8	54	34

⚪ 2025-06-12 08:44:26 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-06-12 09:04:11 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
126 (only retried tests)	55	0	8	29	34

🟢 2025-06-12 09:04:18 UTC Build successful.
🔴 2025-06-12 09:04:46 UTC ydbd size 3.7 GiB changed* by +4.3 MiB, which is >= 2.0 MiB vs stream-nb-25-1: Alert

ydbd size dash	stream-nb-25-1: `6c291f2`	merge: `1c9266d`	diff	diff %
ydbd size	3 928 315 328 Bytes	3 932 864 888 Bytes	+4.3 MiB	+0.116%
ydbd stripped size	1 373 617 296 Bytes	1 375 191 120 Bytes	+1.5 MiB	+0.115%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

UgnineSirdis added 30 commits June 11, 2025 11:22

Refactor S3Buffer for export so that it can use different combination…

583c4fb

…s of checksums, compression and encryption (ydb-platform#14862) (cherry picked from commit 499b85e)

Serializer/deserializer for encrypted backup files (ydb-platform#14912)

136c04d

(cherry picked from commit 00996cd)

Rename EncryptFile/DecryptFile to avoid conflict with windows functio…

895dd47

…ns (ydb-platform#15116) (cherry picked from commit c5b2980)

Add missing recurse (ydb-platform#15141)

b83dfb8

(cherry picked from commit c09cf0f)

Rename forgotten DecryptFile to DecryptFullFile (ydb-platform#15927)

a20022a

(cherry picked from commit 22d0ecf)

Backup encryption API (ydb-platform#15975)

26d421e

(cherry picked from commit b7e42a2)

Read controller for encrypted file deserializer in import from S3 dat…

46d6c0a

…ashard code (ydb-platform#16055) (cherry picked from commit 064d1ba)

Implement writing SchemaMapping in export (ydb-platform#16190)

3b80ab1

(cherry picked from commit f1d47d6)

Support encryption for tables in export (ydb-platform#16320)

2ddbd59

(cherry picked from commit aae5680)

Refactoring after PR 16190: rename schemeshard_export_scheme_uploader…

4da9720

….h, implement fallback class for export metadata uploader (ydb-platform#16343) (cherry picked from commit 5f9dbe8)

Refactoring after PR 16190: rename schemeshard_export_scheme_uploader…

4f192bb

….cpp (ydb-platform#16366) (cherry picked from commit 9fb5600)

Implement encrypted import (ydb-platform#16434)

c3346ce

(cherry picked from commit 4ae39ac)

Refactoring after PR 16434: rename schemeshard_import_scheme_getter.h…

55fe325

… -> schemeshard_import_getters.h (ydb-platform#16729) (cherry picked from commit c95801c)

Refactoring after PR 16434: rename schemeshard_import_scheme_getter.c…

846288e

…pp -> schemeshard_import_getters.cpp (ydb-platform#16749) (cherry picked from commit d550d6c)

Fix starting of S3 recipe (ydb-platform#16933)

994dee4

(cherry picked from commit bf5ada7)

Refactor ut_export: move local scope test variables to fixture (ydb-p…

4e43f8c

…latform#17461) (cherry picked from commit 63874fb)

Fix S3 buffer min bytes setting (ydb-platform#17556)

255df88

(cherry picked from commit f0122e1)

Support zero data encrypted files in import (ydb-platform#17620)

fdcafd1

(cherry picked from commit 7ee4040)

Fix leak in test ExportS3BufferTest::MinBufferSize (ydb-platform#17773)

499aec6

(cherry picked from commit d18fee7)

Fix backup progress state for the case when we have SchemaMapping met…

2cdfb22

…adata files (ydb-platform#17812) (cherry picked from commit 822094a)

Export/import encryption parameters in YDB SDK (ydb-platform#17798)

9c23d4a

(cherry picked from commit aff4b61)

Refactoring after PR ydb-platform#16320: Move TS3Export::CreateBuffer…

ff58985

…() implementation to cpp (ydb-platform#18030) (cherry picked from commit 4e56d1c)

Recursive export and validation (ydb-platform#18024)

d7c2480

(cherry picked from commit 6dde11e)

ListObjectsInS3Export API (ydb-platform#18035)

92b5780

(cherry picked from commit 19d52ae)

Make lazy server start for backup tests in order to quickly list tests (

7619fe5

ydb-platform#18384) (cherry picked from commit 1f4ccf3)

Fix use after free in ExportFromS3 rpc (ydb-platform#18368)

19b0878

(cherry picked from commit b875f79)

TImportClient::ListObjectsInS3Export in C++ SDK (ydb-platform#18366)

ea6fd00

(cherry picked from commit d8d0d97)

Fix after PR ydb-platform#16190: make PersistExportMetadata method, p…

fad3da9

…ass ExportInfo by ref in functions (ydb-platform#18425) (cherry picked from commit d6e49c5)

Refactoring: pass ImportInfo by reference (ydb-platform#18508)

8b8131e

(cherry picked from commit 5891045)

Implement ImportService.ListObjectsInS3Export on server side (ydb-pla…

a762e67

…tform#18454) (cherry picked from commit ddb9664)

UgnineSirdis added 7 commits June 11, 2025 15:16

Fix test ParallelBackupWholeDatabase for stream-nb-25-1

15f9d6b

Support checksums for export metadata json and for schema mapping (yd…

5b08cf2

…b-platform#18709) (cherry picked from commit 3506e58)

Support export encryption for changefeeds (ydb-platform#19203)

91b763e

(cherry picked from commit 1329bba)

Support export encryption for views/topics (ydb-platform#19248)

1b897fa

(cherry picked from commit 49e39df)

Make permissions required in import if they were exported (ydb-platfo…

19ba580

…rm#19331) (cherry picked from commit e1b310a)

Ported by hands something similar to

a116ff9

"Fix topic export encryption IV (ydb-platform#19458)" (there were small changes for views, topics are not in this branch) (from commit fd2db0f)

Universal test that checks common export encryption requirements (ydb…

55de569

…-platform#19390) (cherry picked from commit 6bf3311)

UgnineSirdis requested review from mvgorbunov, dcherednik, va-kuznecov and alexv-smirnov as code owners June 12, 2025 06:25

va-kuznecov approved these changes Jun 16, 2025

View reviewed changes

UgnineSirdis merged commit 2884e25 into ydb-platform:stream-nb-25-1 Jun 16, 2025
13 checks passed

UgnineSirdis deleted the backup-encryption-feature-backport branch June 29, 2025 09:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

stream-nb-25-1: Backup encryption feature backport #19625

stream-nb-25-1: Backup encryption feature backport #19625

Uh oh!

UgnineSirdis commented Jun 12, 2025

Uh oh!

github-actions bot commented Jun 12, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jun 12, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

stream-nb-25-1: Backup encryption feature backport #19625

stream-nb-25-1: Backup encryption feature backport #19625

Uh oh!

Conversation

UgnineSirdis commented Jun 12, 2025

Changelog entry

Changelog category

Description for reviewers

Uh oh!

github-actions bot commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 12, 2025 •

edited

Loading

github-actions bot commented Jun 12, 2025 •

edited

Loading