Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement]: Avoid assign too much segment/channels to new querynode #34095

Closed
1 task done
weiliu1031 opened this issue Jun 24, 2024 · 2 comments
Closed
1 task done

[Enhancement]: Avoid assign too much segment/channels to new querynode #34095

weiliu1031 opened this issue Jun 24, 2024 · 2 comments
Assignees
@weiliu1031
Labels
kind/enhancement Issues or changes related to enhancement

Comments

@weiliu1031
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

What would you like to be added?

When a new query node comes online, the segment_checker, channel_checker, and balance_checker simultaneously attempt to allocate segments to it. If this occurs during the execution of a load task and the distribution of the new query node hasn't been updated, the query coordinator may mistakenly view the new query node as empty. As a result, it assigns segments or channels to it, potentially overloading the new query node with more segments or channels than expected.

Why is this needed?

No response

Anything else?

No response
@weiliu1031 weiliu1031 added the kind/enhancement Issues or changes related to enhancement label Jun 24, 2024
@weiliu1031
Copy link
Contributor Author

/assign

@weiliu1031 weiliu1031 mentioned this issue Jun 24, 2024
Merged
@xiaofan-luan
Copy link
Contributor

for each balance step, we should set a limit(see only 5 segments can be balanced at the same time)

sre-ci-robot pushed a commit that referenced this issue Jun 27, 2024
@weiliu1031
enhance: Avoid assign too much segment/channels to new querynode (#34096
8123bea 
)

issue: #34095

When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.

This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
weiliu1031 added a commit to weiliu1031/milvus that referenced this issue Jun 27, 2024
@weiliu1031
enhance: Avoid assign too much segment/channels to new querynode (mil…
ffc3cc3 
…vus-io#34096)

issue: milvus-io#34095

When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.

This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
@weiliu1031 weiliu1031 mentioned this issue Jun 27, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Jul 1, 2024
@weiliu1031
enhance: Avoid assign too much segment/channels to new querynode (#34096
b18de95 
) (#34245)

issue: #34095
pr: #34096

When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.

This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
yellow-shine pushed a commit to yellow-shine/milvus that referenced this issue Jul 2, 2024
@weiliu1031 @yellow-shine
enhance: Avoid assign too much segment/channels to new querynode (mil…
b0a7d0a 
…vus-io#34096)

issue: milvus-io#34095

When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.

This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
weiliu1031 added a commit to weiliu1031/milvus that referenced this issue Jul 5, 2024
@weiliu1031
enhance: Avoid assign too much segment/channels to new querynode (mil…
0753532 
…vus-io#34096)

issue: milvus-io#34095

When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.

This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
@weiliu1031 weiliu1031 mentioned this issue Jul 5, 2024
Merged
weiliu1031 added a commit to weiliu1031/milvus that referenced this issue Jul 6, 2024
@weiliu1031
enhance: Avoid assign too much segment/channels to new querynode (mil…
ba08192 
…vus-io#34096)

issue: milvus-io#34095

When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.

This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
weiliu1031 added a commit to weiliu1031/milvus that referenced this issue Jul 8, 2024
@weiliu1031
enhance: Avoid assign too much segment/channels to new querynode (mil…
fec9c45 
…vus-io#34096)

issue: milvus-io#34095

When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.

This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Jul 10, 2024
@weiliu1031
enhance: Avoid assign too much segment/channels to new querynode (#34096
d3d1920 
) (#34461)

issue: #34095
pr: #34096

When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.

This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@weiliu1031 weiliu1031

Labels
kind/enhancement Issues or changes related to enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@xiaofan-luan @weiliu1031