Skip to content

Conversation

azevaykin
Copy link
Collaborator

@azevaykin azevaykin commented May 29, 2024

Changelog entry

Support for vector index in SchemeShard.
Note! There are several index implementation tables now.

Base table:

CREATE TABLE vectors (
    id Uint32,  -- an id of a vector
    embedding String, -- a vector embedding
    covered String, -- a covered column for index
    other String, -- another column
    PRIMARY KEY (id)
)

Vector index tables:

CREATE TABLE indexImplLevelTable(
    '-parent' Uint32, -- a parent id of the cluster
    '-id' Uint32,  -- an id of the cluster
    '-embedding'String, -- a coordinate of the cluster
    PRIMARY KEY ('-parent', '-id')
)

CREATE TABLE indexImplPostingTable(
    '-parent' Uint32,	-- a parent id of the cluster
    id Uint32, -- a PK of the base table
    covered String, -- a covered column
    PRIMARY KEY ('-parent', id)
)

Changelog category

  • Not for changelog (changelog entry is not required)

Additional information

...

Copy link

github-actions bot commented May 29, 2024

2024-05-29 10:52:15 UTC Pre-commit check for 73ba2af has started.
2024-05-29 10:55:06 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-29 11:38:30 UTC Build successful.
2024-05-29 11:38:42 UTC Tests are running...
🔴 2024-05-29 13:37:55 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10858 10786 0 32 29 11

Copy link

github-actions bot commented May 29, 2024

2024-05-29 10:52:35 UTC Pre-commit check for 73ba2af has started.
2024-05-29 10:55:32 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-29 11:38:10 UTC Build successful.
2024-05-29 11:44:07 UTC Tests are running...
🔴 2024-05-29 13:23:39 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38400 33612 0 1 4782 5

Copy link

github-actions bot commented May 29, 2024

2024-05-29 10:52:40 UTC Pre-commit check for 73ba2af has started.
2024-05-29 10:55:42 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-29 11:35:17 UTC Build successful.

@azevaykin azevaykin requested a review from ijon May 29, 2024 13:46
@azevaykin azevaykin force-pushed the SchemeShardVectorIndex branch from 311e293 to cc8d456 Compare May 30, 2024 06:21
Copy link

github-actions bot commented May 30, 2024

2024-05-30 06:24:45 UTC Pre-commit check for d3cd618 has started.
2024-05-30 06:27:37 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-30 07:06:09 UTC Build successful.

Copy link

github-actions bot commented May 30, 2024

2024-05-30 06:24:47 UTC Pre-commit check for d3cd618 has started.
2024-05-30 06:27:36 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-30 07:07:55 UTC Build successful.
2024-05-30 07:08:09 UTC Tests are running...
🔴 2024-05-30 08:47:21 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10774 10687 0 41 34 12

Copy link

github-actions bot commented May 30, 2024

2024-05-30 06:24:58 UTC Pre-commit check for d3cd618 has started.
2024-05-30 06:27:48 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-30 07:06:28 UTC Build successful.
2024-05-30 07:06:42 UTC Tests are running...
🔴 2024-05-30 08:34:56 UTC Some tests failed, follow the links below.

Test history

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38356 33551 0 12 4787 6

@azevaykin azevaykin requested a review from CyberROFL May 30, 2024 06:39
@azevaykin
Copy link
Collaborator Author

Комментарий от ijon:

  • нет поддержки нового типа индекса в public api и с++ sdk — но это пока ожидаемо, полагаю, ты этого пока и не делал (там помимо прочего должна быть конверсия в table_description.cpp, и, например, расширение enum в ytable_enum.h)

  • расширение EPathSubType как способ указать, что таблица не обычная пользовательская, а реализует что-то специфическое

    • выдача подтипа в результате describe
    • и сокрытие такой спец.таблицы от глаз пользователя на уровне scheme-cache — индексные таблицы у нас от пользователя прячутся

@azevaykin azevaykin marked this pull request as draft June 11, 2024 06:10
@azevaykin azevaykin force-pushed the SchemeShardVectorIndex branch from cc8d456 to 88cb424 Compare June 13, 2024 05:33
Copy link

github-actions bot commented Jun 13, 2024

2024-06-13 05:36:53 UTC Pre-commit check for 113cc6c has started.
2024-06-13 05:39:54 UTC Build linux-x86_64-release-clang14 is running...
2024-06-13 06:06:18 UTC Check cancelled

Copy link

github-actions bot commented Jun 13, 2024

2024-06-13 05:37:06 UTC Pre-commit check for 113cc6c has started.
2024-06-13 05:40:09 UTC Build linux-x86_64-relwithdebinfo is running...
2024-06-13 06:06:18 UTC Check cancelled

Copy link

github-actions bot commented Jun 13, 2024

2024-06-13 05:37:08 UTC Pre-commit check for 113cc6c has started.
2024-06-13 05:40:11 UTC Build linux-x86_64-release-asan is running...
2024-06-13 06:06:17 UTC Check cancelled

@azevaykin azevaykin force-pushed the SchemeShardVectorIndex branch from 88cb424 to 09e16ce Compare June 13, 2024 06:05
Copy link

github-actions bot commented Jun 13, 2024

2024-06-13 06:07:41 UTC Pre-commit check for 1800b49 has started.
2024-06-13 06:10:30 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-06-13 06:46:47 UTC Build successful.

Copy link

github-actions bot commented Jun 13, 2024

2024-06-13 06:07:42 UTC Pre-commit check for 1800b49 has started.
2024-06-13 06:10:36 UTC Build linux-x86_64-release-asan is running...
🟢 2024-06-13 06:52:14 UTC Build successful.
2024-06-13 06:52:31 UTC Tests are running...
🔴 2024-06-13 08:50:12 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15386 15284 0 41 48 13

Copy link

github-actions bot commented Jun 13, 2024

2024-06-13 06:10:15 UTC Pre-commit check for 1800b49 has started.
2024-06-13 06:13:10 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-06-13 06:53:29 UTC Build successful.
2024-06-13 06:53:43 UTC Tests are running...
🔴 2024-06-13 09:07:38 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
75042 62226 0 1 12804 11

Copy link

github-actions bot commented Jun 20, 2024

2024-06-20 13:28:35 UTC Pre-commit check for 3196274 has started.
2024-06-20 13:31:37 UTC Build linux-x86_64-release-asan is running...
🟢 2024-06-20 14:01:30 UTC Build successful.
2024-06-20 14:01:39 UTC Tests are running...
🔴 2024-06-20 15:56:25 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14791 14642 0 75 54 20

🟡 2024-06-20 15:57:11 UTC ydbd size 5.7 GiB changed by +252.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 0531706 merge: 3196274 diff diff %
ydbd size 6 149 731 584 Bytes 6 149 990 040 Bytes +252.4 KiB +0.004%
ydbd stripped size 1 527 571 776 Bytes 1 527 634 912 Bytes +61.7 KiB +0.004%

Copy link

github-actions bot commented Jun 20, 2024

2024-06-20 13:47:39 UTC Pre-commit check for 3196274 has started.
2024-06-20 13:50:30 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-06-20 14:21:14 UTC Build successful.
2024-06-20 14:21:24 UTC Tests are running...
🔴 2024-06-20 16:44:05 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
75345 61990 0 46 13291 18

🟡 2024-06-20 16:44:47 UTC ydbd size 8.6 GiB changed by +298.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 0531706 merge: 3196274 diff diff %
ydbd size 9 215 542 624 Bytes 9 215 847 784 Bytes +298.0 KiB +0.003%
ydbd stripped size 711 334 640 Bytes 711 366 416 Bytes +31.0 KiB +0.004%

Copy link

github-actions bot commented Jun 20, 2024

2024-06-20 13:48:04 UTC Pre-commit check for 3196274 has started.
2024-06-20 13:50:56 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-06-20 14:30:57 UTC Build successful.

@azevaykin azevaykin force-pushed the SchemeShardVectorIndex branch 2 times, most recently from 3d3d6ec to 590842a Compare June 27, 2024 07:54
Copy link

github-actions bot commented Jun 27, 2024

2024-06-27 07:57:41 UTC Pre-commit check for 10222bd has started.
2024-06-27 08:00:39 UTC Build linux-x86_64-release-clang14 is running...
🔴 2024-06-27 08:39:11 UTC Build failed. see the build logs.

Copy link

github-actions bot commented Jun 27, 2024

2024-06-27 07:57:45 UTC Pre-commit check for 10222bd has started.
2024-06-27 08:00:54 UTC Build linux-x86_64-release-asan is running...
🔴 2024-06-27 08:34:18 UTC Build failed. see the build logs.

Copy link

github-actions bot commented Jun 27, 2024

2024-06-27 07:57:55 UTC Pre-commit check for 10222bd has started.
2024-06-27 08:00:57 UTC Build linux-x86_64-relwithdebinfo is running...
🔴 2024-06-27 08:35:19 UTC Build failed. see the build logs.

@azevaykin azevaykin force-pushed the SchemeShardVectorIndex branch from 590842a to ba7ffd4 Compare July 1, 2024 10:02
Copy link

github-actions bot commented Jul 12, 2024

2024-07-12 07:05:14 UTC Pre-commit check for c542ec4 has started.
2024-07-12 07:08:29 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-12 09:45:55 UTC Build successful.
🔴 2024-07-12 09:50:18 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15299 14701 0 85 337 176

🟡 2024-07-12 09:51:08 UTC ydbd size 5.2 GiB changed* by +584.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: d8c88f9 merge: c542ec4 diff diff %
ydbd size 5 609 029 976 Bytes 5 609 628 888 Bytes +584.9 KiB +0.011%
ydbd stripped size 1 207 403 312 Bytes 1 207 502 512 Bytes +96.9 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 12, 2024

2024-07-12 07:05:18 UTC Pre-commit check for c542ec4 has started.
2024-07-12 07:08:37 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-12 07:47:47 UTC Build successful.

Copy link

github-actions bot commented Jul 12, 2024

2024-07-12 07:05:19 UTC Pre-commit check for c542ec4 has started.
2024-07-12 07:08:31 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-12 09:49:46 UTC Build successful.
🔴 2024-07-12 10:03:12 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
76368 62879 0 40 13432 17

🟡 2024-07-12 10:04:09 UTC ydbd size 8.1 GiB changed* by +968.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: d8c88f9 merge: c542ec4 diff diff %
ydbd size 8 719 374 656 Bytes 8 720 366 208 Bytes +968.3 KiB +0.011%
ydbd stripped size 475 751 088 Bytes 475 783 376 Bytes +31.5 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

ijon
ijon previously approved these changes Jul 12, 2024
Copy link

github-actions bot commented Jul 12, 2024

2024-07-12 12:20:31 UTC Pre-commit check for 78a1ffc has started.
2024-07-12 12:23:57 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-12 12:52:45 UTC Build successful.

Copy link

github-actions bot commented Jul 12, 2024

2024-07-12 12:22:57 UTC Pre-commit check for 78a1ffc has started.
2024-07-12 12:26:07 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-12 14:49:52 UTC Build successful.
🔴 2024-07-12 15:01:30 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
76370 62913 0 4 13432 21

🟡 2024-07-12 15:02:25 UTC ydbd size 8.1 GiB changed* by +966.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: d0aff76 merge: 78a1ffc diff diff %
ydbd size 8 718 800 608 Bytes 8 719 789 800 Bytes +966.0 KiB +0.011%
ydbd stripped size 475 668 912 Bytes 475 700 816 Bytes +31.2 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 12, 2024

2024-07-12 12:33:47 UTC Pre-commit check for 78a1ffc has started.
2024-07-12 12:37:23 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-12 14:59:15 UTC Build successful.
🔴 2024-07-12 15:03:06 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15297 14702 0 85 334 176

🟡 2024-07-12 15:03:55 UTC ydbd size 5.2 GiB changed* by +578.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: d0aff76 merge: 78a1ffc diff diff %
ydbd size 5 608 995 032 Bytes 5 609 587 768 Bytes +578.8 KiB +0.011%
ydbd stripped size 1 207 400 496 Bytes 1 207 498 416 Bytes +95.6 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@azevaykin azevaykin requested a review from ijon July 13, 2024 09:05
Copy link

github-actions bot commented Jul 13, 2024

2024-07-13 11:21:01 UTC Pre-commit check for 7871b5a has started.
2024-07-13 11:24:43 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-13 14:01:02 UTC Build successful.
🔴 2024-07-13 14:05:08 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15312 14719 0 86 331 176

🟡 2024-07-13 14:05:56 UTC ydbd size 5.2 GiB changed* by +642.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 6ee94d0 merge: 7871b5a diff diff %
ydbd size 5 574 428 104 Bytes 5 575 085 992 Bytes +642.5 KiB +0.012%
ydbd stripped size 1 201 350 864 Bytes 1 201 460 080 Bytes +106.7 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 13, 2024

2024-07-13 11:21:03 UTC Pre-commit check for 7871b5a has started.
2024-07-13 11:24:13 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-13 12:03:25 UTC Build successful.

Copy link

github-actions bot commented Jul 13, 2024

2024-07-13 11:22:21 UTC Pre-commit check for 7871b5a has started.
2024-07-13 11:26:20 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-13 14:12:34 UTC Build successful.
🔴 2024-07-13 14:24:45 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
76383 62932 0 4 13429 18

🟡 2024-07-13 14:25:39 UTC ydbd size 8.1 GiB changed* by +1.1 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 6ee94d0 merge: 7871b5a diff diff %
ydbd size 8 691 591 640 Bytes 8 692 759 368 Bytes +1.1 MiB +0.013%
ydbd stripped size 473 712 304 Bytes 473 746 064 Bytes +33.0 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants