Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use sharding keys to calculate bucket id #181

Merged
merged 8 commits into from
Nov 27, 2021
Merged

Use sharding keys to calculate bucket id #181

merged 8 commits into from
Nov 27, 2021

Conversation

ligurio
Copy link
Member

@ligurio ligurio commented Jul 6, 2021 •
edited
Loading

What has been done? Why? What problem is being solved?

I didn't forget about

  • Tests
  • Changelog
  • Documentation

Closes #166

@ligurio ligurio force-pushed the ligurio/gh-166-sharding-keys branch from e2331a6 to 173d313 Compare July 6, 2021 16:40
@ligurio ligurio marked this pull request as draft July 6, 2021 16:40
@ligurio ligurio force-pushed the ligurio/gh-166-sharding-keys branch 8 times, most recently from bf843a1 to 0fc03c7 Compare July 9, 2021 16:07
@artur-barsegyan artur-barsegyan mentioned this pull request Jul 13, 2021
Closed
@ligurio ligurio force-pushed the ligurio/gh-166-sharding-keys branch 4 times, most recently from 11a77fc to b791fc5 Compare July 16, 2021 06:56
@ligurio ligurio requested a review from olegrok July 16, 2021 07:01
@ligurio ligurio marked this pull request as ready for review July 16, 2021 07:01
@ligurio ligurio force-pushed the ligurio/gh-166-sharding-keys branch from b791fc5 to e898d0e Compare July 16, 2021 07:08
olegrok
olegrok reviewed Jul 16, 2021
View reviewed changes
Copy link
Contributor

@olegrok olegrok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your patch. See my comments below.

crud/common/sharding.lua Outdated Show resolved Hide resolved
crud/common/sharding.lua Outdated Show resolved Hide resolved
crud/common/utils.lua Outdated Show resolved Hide resolved
crud/common/utils.lua Outdated Show resolved Hide resolved
crud/common/utils.lua Outdated Show resolved Hide resolved
crud/delete.lua Outdated Show resolved Hide resolved
crud/delete.lua Outdated Show resolved Hide resolved
crud/delete.lua Outdated Show resolved Hide resolved
crud/get.lua Outdated Show resolved Hide resolved
test/entrypoint/srv_ddl.lua Outdated Show resolved Hide resolved
mRrvz
mRrvz reviewed Jul 26, 2021
View reviewed changes
crud/common/sharding.lua Outdated Show resolved Hide resolved
mRrvz
mRrvz reviewed Jul 26, 2021
View reviewed changes
crud/common/utils.lua Outdated Show resolved Hide resolved
crud/common/utils.lua Outdated Show resolved Hide resolved
crud/delete.lua Outdated Show resolved Hide resolved
crud/delete.lua Outdated Show resolved Hide resolved
crud/get.lua Outdated Show resolved Hide resolved
@ligurio ligurio force-pushed the ligurio/gh-166-sharding-keys branch from e898d0e to ca5eced Compare July 28, 2021 07:08
@ligurio ligurio changed the title Use sharding keys to calculate bucket id (WIP) Use sharding keys to calculate bucket id Jul 28, 2021
olegrok
olegrok reviewed Jul 28, 2021
View reviewed changes
crud/common/utils.lua Outdated Show resolved Hide resolved
@ligurio ligurio force-pushed the ligurio/gh-166-sharding-keys branch 2 times, most recently from 3dccab0 to 0b9e6b7 Compare July 30, 2021 07:42
rosik
rosik reviewed Jul 30, 2021
View reviewed changes
Copy link
Contributor

@rosik rosik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try this, it should make the test compatible with both cartridge 2.5.1 and 2.6.0

test/integration/ddl_sharding_key_test.lua Outdated Show resolved Hide resolved
test/integration/ddl_sharding_key_test.lua Outdated Show resolved Hide resolved
@ligurio ligurio force-pushed the ligurio/gh-166-sharding-keys branch from fef89bf to 9a8dab1 Compare July 30, 2021 18:10
olegrok
olegrok reviewed Jul 30, 2021
View reviewed changes
crud/select/plan.lua Outdated Show resolved Hide resolved
@ligurio @DifferentialOrange
Doc: use custom sharding key to calculate bucket id
0c5776d 
Describe functionality and current limitations (#212, #213, #219, #243)
with custom sharding key in CHANGELOG and README.

Thanks to Oleg Babin (@olegrok) and Alexander Turenko (@Totktonada) for
help with feature implementation.

Closes #166

Reviewed-by: Oleg Babin <babinoleg@mail.ru>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
Co-authored-by: Georgy Moiseev <Georgy.moiseev@corp.mail.ru>
@ligurio ligurio merged commit cc2e0e8 into master Nov 27, 2021
@ligurio ligurio deleted the ligurio/gh-166-sharding-keys branch November 27, 2021 11:37
ligurio added a commit that referenced this pull request Nov 27, 2021
@ligurio
Support custom sharding key in select
5abe1c1 
NOTE: Prior to this patch CRUD assumes an index is unique. It was true
for the primary key, but it is not guaranteed for a sharding key. Patch
adds a tests with select() for non-unique index that failed due to
assumption regarding uniq index in crud/select/plan.lua. Seems we
can remove this condition and fix tests that relies on
total_tuple_count == 1. See also related discussion in [1].

1. #181 (comment)

Part of #166

Reviewed-by: Oleg Babin <babinoleg@mail.ru>
Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
DifferentialOrange added a commit that referenced this pull request Nov 29, 2021
@DifferentialOrange
Extract sharding key from conditions
f8a1c3c 
PR #181 introduced support of DDL sharding keys. But if sharding key
hasn't got a separate index in schema, select with equal conditions
for all required sharding key fields still led to map-reduce instead of
a single storage call. This patch introduces impoved support of
sharding keys extraction and fixes the issue.

Closes #213
DifferentialOrange added a commit that referenced this pull request Nov 29, 2021
@DifferentialOrange
Extract sharding key from conditions
3ca66dd 
PR #181 introduced support of DDL sharding keys. But if sharding key
hasn't got a separate index in schema, select with equal conditions
for all required sharding key fields still led to map-reduce instead of
a single storage call. This patch introduces impoved support of
sharding keys extraction and fixes the issue.

Closes #213
DifferentialOrange added a commit that referenced this pull request Nov 30, 2021
@DifferentialOrange
Extract sharding key from conditions
40846c6 
PR #181 introduced support of DDL sharding keys. But if sharding key
hasn't got a separate index in schema, select with equal conditions
for all required sharding key fields still led to map-reduce instead of
a single storage call. This patch introduces impoved support of
sharding keys extraction and fixes the issue.

Closes #213
DifferentialOrange added a commit that referenced this pull request Nov 30, 2021
@DifferentialOrange
Extract sharding key from conditions
1bd6e85 
PR #181 introduced support of DDL sharding keys. But if sharding key
hasn't got a separate index in schema, select with equal conditions
for all required sharding key fields still led to map-reduce instead of
a single storage call. This patch introduces impoved support of
sharding keys extraction and fixes the issue.

Closes #213
DifferentialOrange added a commit that referenced this pull request Nov 30, 2021
@DifferentialOrange
Extract sharding key from conditions
5cac02b 
PR #181 introduced support of DDL sharding keys. But if sharding key
hasn't got a separate index in schema, select with equal conditions
for all required sharding key fields still led to map-reduce instead of
a single storage call. This patch introduces impoved support of
sharding keys extraction and fixes the issue.

Closes #213
DifferentialOrange added a commit that referenced this pull request Nov 30, 2021
@DifferentialOrange
Extract sharding key from conditions
0638d5a 
PR #181 introduced support of DDL sharding keys. But if sharding key
hasn't got a separate index in schema, select with equal conditions
for all required sharding key fields still led to map-reduce instead of
a single storage call. This patch introduces impoved support of
sharding keys extraction and fixes the issue.

Closes #213
DifferentialOrange added a commit that referenced this pull request Dec 1, 2021
@DifferentialOrange
Extract sharding key from conditions
b2b485f 
PR #181 introduced support of DDL sharding keys. But if sharding key
hasn't got a separate index in schema, select with equal conditions
for all required sharding key fields still led to map-reduce instead of
a single storage call. This patch introduces impoved support of
sharding keys extraction and fixes the issue.

Closes #213
DifferentialOrange added a commit that referenced this pull request Dec 1, 2021
@DifferentialOrange
Extract sharding key from conditions
5c395fe 
PR #181 introduced support of DDL sharding keys. But if sharding key
hasn't got a separate index in schema, select with equal conditions
for all required sharding key fields still led to map-reduce instead of
a single storage call. This patch introduces impoved support of
sharding keys extraction and fixes the issue.

Closes #213
DifferentialOrange added a commit that referenced this pull request Dec 1, 2021
@DifferentialOrange
Extract sharding key from conditions
e7a21b2 
PR #181 introduced support of DDL sharding keys. But if sharding key
hasn't got a separate index in schema, select with equal conditions
for all required sharding key fields still led to map-reduce instead of
a single storage call. This patch introduces impoved support of
sharding keys extraction and fixes the issue.

Closes #213
Totktonada pushed a commit that referenced this pull request Dec 1, 2021
@DifferentialOrange @Totktonada
Extract sharding key from conditions
dc1cdc4 
PR #181 introduced support of DDL sharding keys. But if sharding key
hasn't got a separate index in schema, select with equal conditions
for all required sharding key fields still led to map-reduce instead of
a single storage call. This patch introduces impoved support of
sharding keys extraction and fixes the issue.

Closes #213
@ligurio ligurio mentioned this pull request Dec 13, 2021
Merged
AnaNek added a commit that referenced this pull request Jan 12, 2022
@AnaNek
Copy the content of sharding_key.lua file to `sharding_metadata.lua…
ca650da 
…` file

PR #181 introduced support of DDL sharding keys. Implementation of
sharding keys support contains methods that are common to support
sharding keys and sharding functions. That's why a separate file
`sharding_metadata.lua` was created to contain common methods.
In this commit content of `sharding_key.lua` file is coppied to
`sharding_metadata.lua` file to simplify a reviewer's life
and display the history of changes relative to PR #181 in
the following commits.

Part of #237
AnaNek added a commit that referenced this pull request Jan 27, 2022
@AnaNek
Copy the content of sharding_key.lua file to `sharding_metadata.lua…
be073b3 
…` file

PR #181 introduced support of DDL sharding keys. Implementation of
sharding keys support contains methods that are common to support
sharding keys and sharding functions. That's why a separate file
`sharding_metadata.lua` was created to contain common methods.
In this commit content of `sharding_key.lua` file is coppied to
`sharding_metadata.lua` file to simplify a reviewer's life
and display the history of changes relative to PR #181 in
the following commits.

Part of #237
AnaNek added a commit that referenced this pull request Jan 28, 2022
@AnaNek
Copy the content of sharding_key.lua file to `sharding_metadata.lua…
98d555b 
…` file

PR #181 introduced support of DDL sharding keys. Implementation of
sharding keys support contains methods that are common to support
sharding keys and sharding functions. That's why a separate file
`sharding_metadata.lua` was created to contain common methods.
In this commit content of `sharding_key.lua` file is coppied to
`sharding_metadata.lua` file to simplify a reviewer's life
and display the history of changes relative to PR #181 in
the following commits.

Part of #237
AnaNek added a commit that referenced this pull request Jan 28, 2022
@AnaNek
Copy the content of sharding_key.lua file to `sharding_metadata.lua…
05cd72d 
…` file

PR #181 introduced support of DDL sharding keys. Implementation of
sharding keys support contains methods that are common to support
sharding keys and sharding functions. That's why a separate file
`sharding_metadata.lua` was created to contain common methods.
In this commit content of `sharding_key.lua` file is coppied to
`sharding_metadata.lua` file to simplify a reviewer's life
and display the history of changes relative to PR #181 in
the following commits.

Part of #237
AnaNek added a commit that referenced this pull request Jan 31, 2022
@AnaNek
Copy the content of sharding_key.lua file to `sharding_metadata.lua…
1441a86 
…` file

PR #181 introduced support of DDL sharding keys. Implementation of
sharding keys support contains methods that are common to support
sharding keys and sharding functions. That's why a separate file
`sharding_metadata.lua` was created to contain common methods.
In this commit content of `sharding_key.lua` file is coppied to
`sharding_metadata.lua` file to simplify a reviewer's life
and display the history of changes relative to PR #181 in
the following commits.

Part of #237
AnaNek added a commit that referenced this pull request Jan 31, 2022
@AnaNek
Copy the content of sharding_key.lua file to `sharding_metadata.lua…
7862406 
…` file

PR #181 introduced support of DDL sharding keys. Implementation of
sharding keys support contains methods that are common to support
sharding keys and sharding functions. That's why a separate file
`sharding_metadata.lua` was created to contain common methods.
In this commit content of `sharding_key.lua` file is coppied to
`sharding_metadata.lua` file to simplify a reviewer's life
and display the history of changes relative to PR #181 in
the following commits.

Part of #237
AnaNek added a commit that referenced this pull request Jan 31, 2022
@AnaNek
Copy the content of sharding_key.lua file to `sharding_metadata.lua…
14876bb 
…` file

PR #181 introduced support of DDL sharding keys. Implementation of
sharding keys support contains methods that are common to support
sharding keys and sharding functions. That's why a separate file
`sharding_metadata.lua` was created to contain common methods.
This commit relocates functions for fetching sharding key from
`sharding_key.lua` file to `sharding_metadata.lua` file
to simplify a reviewer's life and display the history of
changes relative to PR #181 in the following commits.

Part of #237
AnaNek added a commit that referenced this pull request Feb 1, 2022
@AnaNek
Copy the content of sharding_key.lua file to `sharding_metadata.lua…
1789f43 
…` file

PR #181 introduced support of DDL sharding keys. Implementation of
sharding keys support contains methods that are common to support
sharding keys and sharding functions. That's why a separate file
`sharding_metadata.lua` was created to contain common methods.
This commit relocates functions for fetching sharding key from
`sharding_key.lua` file to `sharding_metadata.lua` file
to simplify a reviewer's life and display the history of
changes relative to PR #181 in the following commits.

Part of #237
AnaNek added a commit that referenced this pull request Feb 1, 2022
@AnaNek
Copy the content of sharding_key.lua file to `sharding_metadata.lua…
eba3bd7 
…` file

PR #181 introduced support of DDL sharding keys. Implementation of
sharding keys support contains methods that are common to support
sharding keys and sharding functions. That's why a separate file
`sharding_metadata.lua` was created to contain common methods.
This commit relocates functions for fetching sharding key from
`sharding_key.lua` file to `sharding_metadata.lua` file
to simplify a reviewer's life and display the history of
changes relative to PR #181 in the following commits.

Part of #237
AnaNek added a commit that referenced this pull request Feb 1, 2022
@AnaNek
Copy the content of sharding_key.lua file to `sharding_metadata.lua…
9016ed6 
…` file

PR #181 introduced support of DDL sharding keys. Implementation of
sharding keys support contains methods that are common to support
sharding keys and sharding functions. That's why a separate file
`sharding_metadata.lua` was created to contain common methods.
This commit relocates functions for fetching sharding key from
`sharding_key.lua` file to `sharding_metadata.lua` file
to simplify a reviewer's life and display the history of
changes relative to PR #181 in the following commits.

Part of #237
AnaNek added a commit that referenced this pull request Feb 7, 2022
@AnaNek
Copy the content of sharding_key.lua file to `sharding_metadata.lua…
6e0fcc7 
…` file

PR #181 introduced support of DDL sharding keys. Implementation of
sharding keys support contains methods that are common to support
sharding keys and sharding functions. That's why a separate file
`sharding_metadata.lua` was created to contain common methods.
This commit relocates functions for fetching sharding key from
`sharding_key.lua` file to `sharding_metadata.lua` file
to simplify a reviewer's life and display the history of
changes relative to PR #181 in the following commits.

Part of #237
AnaNek added a commit that referenced this pull request Feb 21, 2022
@AnaNek
Copy the content of sharding_key.lua file to `sharding_metadata.lua…
cc69113 
…` file

PR #181 introduced support of DDL sharding keys. Implementation of
sharding keys support contains methods that are common to support
sharding keys and sharding functions. That's why a separate file
`sharding_metadata.lua` was created to contain common methods.
This commit relocates functions for fetching sharding key from
`sharding_key.lua` file to `sharding_metadata.lua` file
to simplify a reviewer's life and display the history of
changes relative to PR #181 in the following commits.

Part of #237
Totktonada pushed a commit that referenced this pull request Feb 21, 2022
@AnaNek @Totktonada
Copy the content of sharding_key.lua file to `sharding_metadata.lua…
c48b96f 
…` file

PR #181 introduced support of DDL sharding keys. Implementation of
sharding keys support contains methods that are common to support
sharding keys and sharding functions. That's why a separate file
`sharding_metadata.lua` was created to contain common methods.
This commit relocates functions for fetching sharding key from
`sharding_key.lua` file to `sharding_metadata.lua` file
to simplify a reviewer's life and display the history of
changes relative to PR #181 in the following commits.

Part of #237
@Totktonada Totktonada mentioned this pull request May 21, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AnaNek AnaNek AnaNek left review comments

@mRrvz mRrvz mRrvz left review comments

@DifferentialOrange DifferentialOrange DifferentialOrange left review comments

@olegrok olegrok olegrok left review comments

@rosik rosik rosik left review comments

@Totktonada Totktonada Totktonada approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support calculating bucket_id based on ddl.sharding_key
7 participants
@ligurio @Totktonada @rosik @olegrok @DifferentialOrange @mRrvz @AnaNek