Skip to content

Handle new prefixes in prefixed vector index update #25505

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Oct 2, 2025
Merged

Handle new prefixes in prefixed vector index update #25505

merged 4 commits into from
Oct 2, 2025

Conversation

@vitalif
Copy link
Collaborator

@vitalif vitalif commented Sep 22, 2025 •
edited
Loading

Changelog category

  • Not for changelog (changelog entry is not required)

Description for reviewers

Для поддержки вставки новых префиксов:

  • В алгоритм поиска по префиксному векторному индексу добавлен пропуск таблицы level в том случае, когда ID кластера в таблице prefix является сразу листовым. Производительность проверил с помощью workload - накладные расходы на это, вроде бы, минимальны, в пределах погрешности (условно 1690 tps vs 1720 tps в 10 потоков).
  • В алгоритм построения префиксного векторного индекса добавлено создание sequence в таблице prefix и его привязка к колонке __ydb_id.
  • В алгоритм обновления добавлена обработка новых префиксов, их прогон через KqpCnSequencer для генерации ID и их вставка в prefix таблицу.
  • В VectorResolveActor добавлен пропуск вычисления кластеров, когда на входе для префиксного индекса в качестве корневого кластера передан листовой ID.
  • Для старых префиксных индексов без sequence новый алгоритм пропускается и всё работает как раньше, без обработки неизвестных префиксов.

Таким образом, даже при вставке новых префиксов в индекс таблица level всегда остаётся по-настоящему иммутабельной.

@vitalif vitalif requested review from a team as code owners September 22, 2025 11:09
@vitalif vitalif linked an issue Sep 22, 2025 that may be closed by this pull request
Prefixed vector index update #22913
Closed
@github-actions
Copy link

github-actions bot commented Sep 22, 2025 •
edited
Loading

🟢 2025-09-22 11:13:45 UTC The validation of the Pull Request description is successful.

@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
kungasc
kungasc reviewed Sep 22, 2025
View reviewed changes
ydb/core/kqp/opt/physical/effects/kqp_opt_phy_vector_index.cpp
fullPrefixColumns.push_back(NTableIndex::NKMeans::IdColumn);
auto fullPrefixColumnList = BuildColumnsList(fullPrefixColumns, pos, ctx);

auto cnSequencer = Build<TKqpCnSequencer>(ctx, pos)
Copy link
Contributor

@kungasc kungasc Sep 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

что будет при обновлении индекса созданного ранее (без sequencer?)

Copy link
Collaborator Author

@vitalif vitalif Sep 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

посмотри в kqp_opt_phy_insert_index и kqp_opt_phy_upsert_index - там смотрится, если у колонки __ydb_id в префиксной таблице нет DEFAULT_FROM_SEQUENCE, то используется старый алгоритм, который игнорит неизвестные префиксы. точнее я это только что ещё поправил т.к. оказалось что LookupMode=Join в StreamLookup то же самое что SemiJoin т.е. сам он строки с null не фильтрует. я добавил фильтр)

в тесты добавить это было бы интересно, но это видимо в тесты на совместимость как раз т.к. сейчас создать так индексную таблицу не получится по идее.

Copy link
Contributor

@kungasc kungasc Sep 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

конкретно этот сценарий (и другие про обновление во время построения) чуть проще чем на совместимость, и можно прямо юнит-тестом написать через BlockEvents

а так да, надо бы

Copy link
Collaborator Author

@vitalif vitalif Sep 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Тут не делал, т.к. сходу пока хз, как такое написать)

kungasc
kungasc reviewed Sep 22, 2025
View reviewed changes
kungasc
kungasc reviewed Sep 22, 2025
View reviewed changes
kungasc
kungasc reviewed Sep 22, 2025
View reviewed changes
kungasc
kungasc reviewed Sep 22, 2025
View reviewed changes
kungasc
kungasc reviewed Sep 22, 2025
View reviewed changes
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
kungasc
kungasc reviewed Sep 22, 2025
View reviewed changes
@github-actions

This comment was marked as outdated.

Sign in to view
@vitalif vitalif force-pushed the prefixed-vector-index-new-prefix-update branch from f3b876b to 58ae4a2 Compare September 30, 2025 07:57
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
@vitalif vitalif force-pushed the prefixed-vector-index-new-prefix-update branch from 58ae4a2 to 3d7b628 Compare September 30, 2025 15:23
@vitalif
Copy link
Collaborator Author

vitalif commented Sep 30, 2025

Всё, #26046 смержился, тут осталось только KQP

@github-actions
Copy link

github-actions bot commented Sep 30, 2025 •
edited
Loading

⚪ 2025-09-30 15:43:36 UTC Pre-commit check linux-x86_64-release-asan for 4978f78 has started.
⚪ 2025-09-30 15:43:43 UTC Artifacts will be uploaded here
⚪ 2025-09-30 15:47:26 UTC ya make is running...
🟡 2025-09-30 18:10:54 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17567 17099 0 174 266 28

🟢 2025-09-30 18:10:59 UTC Build successful.
🔴 2025-09-30 18:11:22 UTC ydbd size 3.7 GiB changed* by +4.2 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 64ee86c merge: 4978f78 diff diff %
ydbd size 3 987 861 456 Bytes 3 992 226 256 Bytes +4.2 MiB +0.109%
ydbd stripped size 1 486 406 048 Bytes 1 488 093 152 Bytes +1.6 MiB +0.114%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 30, 2025 •
edited
Loading

⚪ 2025-09-30 15:45:27 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4978f78 has started.
⚪ 2025-09-30 15:45:59 UTC Artifacts will be uploaded here
⚪ 2025-09-30 15:50:16 UTC ya make is running...
🟡 2025-09-30 17:42:16 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40508 37596 0 4 2870 38

⚪ 2025-09-30 17:42:25 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-09-30 17:56:53 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
727 (only retried tests) 706 0 0 1 20

🟢 2025-09-30 17:56:57 UTC Build successful.
🟡 2025-09-30 17:57:13 UTC ydbd size 2.2 GiB changed* by +1.8 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 440fe54 merge: 4978f78 diff diff %
ydbd size 2 397 694 880 Bytes 2 399 557 176 Bytes +1.8 MiB +0.078%
ydbd stripped size 512 631 368 Bytes 512 970 408 Bytes +331.1 KiB +0.066%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@vitalif vitalif requested a review from gridnevvvit October 1, 2025 08:39
@vitalif vitalif merged commit b1e3f5a into ydb-platform:main Oct 2, 2025
12 checks passed
vitalif added a commit to vitalif/ydb that referenced this pull request Oct 6, 2025
@vitalif
Handle new prefixes in prefixed vector index update (ydb-platform#25505)
db3695c
@vitalif vitalif mentioned this pull request Oct 6, 2025
Merged
vitalif added a commit to vitalif/ydb that referenced this pull request Oct 6, 2025
@vitalif
Handle new prefixes in prefixed vector index update (ydb-platform#25505)
4cca497
@vitalif vitalif mentioned this pull request Oct 6, 2025
Closed
vitalif added a commit to vitalif/ydb that referenced this pull request Oct 6, 2025
@vitalif
Handle new prefixes in prefixed vector index update (ydb-platform#25505)
d0edea0
vitalif added a commit to vitalif/ydb that referenced this pull request Oct 6, 2025
@vitalif
Handle new prefixes in prefixed vector index update (ydb-platform#25505)
814fad2
maVovk pushed a commit to maVovk/ydb that referenced this pull request Oct 6, 2025
@vitalif @maVovk
Handle new prefixes in prefixed vector index update (ydb-platform#25505)
1c3ff9f
vitalif added a commit to vitalif/ydb that referenced this pull request Oct 6, 2025
@vitalif
Handle new prefixes in prefixed vector index update (ydb-platform#25505)
19789f1
vitalif added a commit to vitalif/ydb that referenced this pull request Oct 6, 2025
@vitalif
Handle new prefixes in prefixed vector index update (ydb-platform#25505)
1e1435d
neyrox pushed a commit to neyrox/ydb that referenced this pull request Oct 7, 2025
@vitalif @neyrox
Handle new prefixes in prefixed vector index update (ydb-platform#25505)
c92195f
neyrox pushed a commit to neyrox/ydb that referenced this pull request Oct 10, 2025
@vitalif @neyrox
Handle new prefixes in prefixed vector index update (ydb-platform#25505)
c6c3e11
vitalif added a commit to vitalif/ydb that referenced this pull request Nov 1, 2025
@vitalif
Handle new prefixes in prefixed vector index update (ydb-platform#25505)
8168138
@vitalif vitalif mentioned this pull request Nov 1, 2025
Open
vitalif added a commit to vitalif/ydb that referenced this pull request Nov 1, 2025
@vitalif
Handle new prefixes in prefixed vector index update (ydb-platform#25505)
5259bd4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gridnevvvit gridnevvvit gridnevvvit approved these changes

@CyberROFL CyberROFL CyberROFL left review comments

+1 more reviewer

@kungasc kungasc kungasc left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

not-for-changelog

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Prefixed vector index update

4 participants

@vitalif @kungasc @CyberROFL @gridnevvvit