Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix shard weight. Now we can set it to zero #1192

Merged
merged 4 commits into from
Nov 22, 2023

Conversation

maxistua
Copy link
Contributor

I can't set the shard weight to zero
We need it for some reasons...

Hopefully this MR will fix that

@UnamedRus
Copy link

It's problem with other settings as well AFAIK.
If value is 0, it will not be materialized in xml config by operator

Copy link
Member

@alex-zaitsev alex-zaitsev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please provide a usage example. Ideally a test, but even example would be helpful.

@maxistua
Copy link
Contributor Author

Yes, imagine a system where historical data is filled from shard to shard.

And now, when a new shard is added, we need to limit writing to the previous ones.

We can write data directly to the node (which is more efficient in terms of resources), but we can also write it to a distributed table.

So, in order to limit writing to already filled shards, we can set their weight to 0.

Currently, it is not possible to disable writing completely. The minimum weight that can be set is 1.

An example of the desired configuration will be below:

configuration:
  clusters:
    - name: "data-lake"
      secret:
        auto: "True"
      layout:
        shards:
          - name: "1"
            weight: 0 <--
            replicasCount: 1
            ...
          - name: "2"
            weight: 0
            replicasCount: 1
            ...
          - name: "3"
            weight: 0
            replicasCount: 1
            ...
          - name: "4"
            weight: 1
            replicasCount: 1
            ...
          - name: "5"
            weight: 1
            replicasCount: 1
            ...
...

This example will not work because a weight of 0 will be considered an empty value and will not be included in the final configuration.

In this case, the weight of all shards will be equal to 1.

@Slach Slach changed the base branch from 0.21.3 to 0.23.0 November 21, 2023 10:26
@Slach
Copy link
Collaborator

Slach commented Nov 21, 2023

@maxistua thanks for explanation, could you resolve conflicts?

@antip00 could you write test for this use case? which change weight to 0 and insert data into distributed table and check data not inserted into zero-weighted shard?

@sunsingerus
Copy link
Collaborator

I'll take a look on this PR

@maxistua
Copy link
Contributor Author

@maxistua thanks for explanation, could you resolve conflicts?

@antip00 could you write test for this use case? which change weight to 0 and insert data into distributed table and check data not inserted into zero-weighted shard?

@Slach All conflicts have been resolved

@sunsingerus sunsingerus merged commit 5033f9e into Altinity:0.23.0 Nov 22, 2023
sunsingerus pushed a commit that referenced this pull request Nov 29, 2023
* Shard weight can be set to zero

(cherry picked from commit 5033f9e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants