Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [Nightly]Search failed reporting 'channel xx is not available in any replica, err=LackSegment' #21667

Closed
binbinlv opened this issue Jan 12, 2023 · 7 comments
Closed

[Bug]: [Nightly]Search failed reporting 'channel xx is not available in any replica, err=LackSegment' #21667

binbinlv opened this issue Jan 12, 2023 · 7 comments
Assignees
@yah01 @binbinlv
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.2.3

Comments

@binbinlv
Copy link
Contributor

Environment

  • Milvus version:04e595d
  • Deployment mode(standalone or cluster):standalone
  • MQ type(rocksmq, pulsar or kafka): rocksmq
  • SDK version(e.g. pymilvus v2.0.0rc2):2.2.1.dev5
  • OS(Ubuntu or CentOS):
  • CPU/Memory:
  • GPU:
  • Others:

Current Behavior:

Search failed reporting 'channel xx is not available in any replica, err=LackSegment'.

[2023-01-11T21:23:59.403Z] [2023-01-11 20:09:57 - DEBUG - ci_test]: (api_request)  : [Collection.search] args: [[[0.3075515340498379, 0.5043181252648048, 0.22726466740039653, 0.30972667693707223, 0.2594818093884532, 0.24384080544801257, 0.6972569059363569, 0.15206223592125911, 0.53188441717009, 0.4191003444515854, 0.3723171426058829, 0.2817927118118889, 0.9542171557641594, 0.0620067113971039, 0.7747235244060......, kwargs: {} (api_request.py:56)
[2023-01-11T21:23:59.403Z] [2023-01-11 20:10:17 - ERROR - pymilvus.decorators]: RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=All attempts results:
[2023-01-11T21:23:59.403Z] attempt #1:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_56_438689392754393991v1 is not available in any replica, err=LackSegment(segmentID=438689392754596628)
[2023-01-11T21:23:59.403Z] attempt #2:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_55_438689392754393991v0 is not available in any replica, err=LackSegment(segmentID=438689392754596630)
[2023-01-11T21:23:59.403Z] attempt #3:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_55_438689392754393991v0 is not available in any replica, err=LackSegment(segmentID=438689392754596630)
[2023-01-11T21:23:59.403Z] attempt #4:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_55_438689392754393991v0 is not available in any replica, err=LackSegment(segmentID=438689392754596630)
[2023-01-11T21:23:59.403Z] attempt #5:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_55_438689392754393991v0 is not available in any replica, err=LackSegment(segmentID=438689392754596630)
[2023-01-11T21:23:59.403Z] attempt #6:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_55_438689392754393991v0 is not available in any replica, err=LackSegment(segmentID=438689392754596630)
[2023-01-11T21:23:59.403Z] attempt #7:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_55_438689392754393991v0 is not available in any replica, err=LackSegment(segmentID=438689392754596630)
[2023-01-11T21:23:59.403Z] attempt #8:context deadline exceeded
[2023-01-11T21:23:59.403Z] )>, <Time:{'RPC start': '2023-01-11 20:09:57.273919', 'RPC error': '2023-01-11 20:10:17.277969'}> (decorators.py:108)
[2023-01-11T21:23:59.403Z] [2023-01-11 20:10:17 - ERROR - ci_test]: Traceback (most recent call last):
[2023-01-11T21:23:59.403Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 26, in inner_wrapper
[2023-01-11T21:23:59.403Z]     res = func(*args, **_kwargs)
[2023-01-11T21:23:59.403Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 57, in api_request
[2023-01-11T21:23:59.403Z]     return func(*arg, **kwargs)
[2023-01-11T21:23:59.403Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 609, in search
[2023-01-11T21:23:59.403Z]     res = conn.search(self._name, data, anns_field, param, limit, expr,
[2023-01-11T21:23:59.404Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
[2023-01-11T21:23:59.404Z]     raise e
[2023-01-11T21:23:59.404Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler
[2023-01-11T21:23:59.404Z]     return func(*args, **kwargs)
[2023-01-11T21:23:59.404Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2023-01-11T21:23:59.404Z]     ret = func(self, *args, **kwargs)
[2023-01-11T21:23:59.404Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 85, in handler
[2023-01-11T21:23:59.404Z]     raise e
[2023-01-11T21:23:59.404Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
[2023-01-11T21:23:59.404Z]     return func(self, *args, **kwargs)
[2023-01-11T21:23:59.404Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 470, in search
[2023-01-11T21:23:59.404Z]     return self._execute_search_requests(requests, timeout, round_decimal=round_decimal, auto_id=auto_id, **kwargs)
[2023-01-11T21:23:59.404Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 439, in _execute_search_requests
[2023-01-11T21:23:59.404Z]     raise pre_err
[2023-01-11T21:23:59.404Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 430, in _execute_search_requests
[2023-01-11T21:23:59.404Z]     raise MilvusException(response.status.error_code, response.status.reason)
[2023-01-11T21:23:59.404Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to search on all shard leaders, err=All attempts results:
[2023-01-11T21:23:59.404Z] attempt #1:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_56_438689392754393991v1 is not available in any replica, err=LackSegment(segmentID=438689392754596628)
[2023-01-11T21:23:59.404Z] attempt #2:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_55_438689392754393991v0 is not available in any replica, err=LackSegment(segmentID=438689392754596630)
[2023-01-11T21:23:59.404Z] attempt #3:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_55_438689392754393991v0 is not available in any replica, err=LackSegment(segmentID=438689392754596630)
[2023-01-11T21:23:59.404Z] attempt #4:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_55_438689392754393991v0 is not available in any replica, err=LackSegment(segmentID=438689392754596630)
[2023-01-11T21:23:59.404Z] attempt #5:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_55_438689392754393991v0 is not available in any replica, err=LackSegment(segmentID=438689392754596630)
[2023-01-11T21:23:59.404Z] attempt #6:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_55_438689392754393991v0 is not available in any replica, err=LackSegment(segmentID=438689392754596630)
[2023-01-11T21:23:59.404Z] attempt #7:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_55_438689392754393991v0 is not available in any replica, err=LackSegment(segmentID=438689392754596630)
[2023-01-11T21:23:59.404Z] attempt #8:context deadline exceeded
[2023-01-11T21:23:59.404Z] )>
[2023-01-11T21:23:59.404Z]  (api_request.py:39)
[2023-01-11T21:23:59.404Z] [2023-01-11 20:10:17 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=fail to search on all shard leaders, err=All attempts results:
[2023-01-11T21:23:59.404Z] attempt #1:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_56_438689392754393991v1 is not available in any replica, err=LackSegment(segmentID=438689392754596628)
[2023-01-11T21:23:59.404Z] attempt #2:fail ...... (api_request.py:40)

Expected Behavior:

pass

Report

https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.2.0/61/pipeline/124/
@binbinlv binbinlv added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 12, 2023
@binbinlv binbinlv added this to the 2.2.3 milestone Jan 12, 2023
@NicoYuan1986
Copy link
Contributor

  1. log: artifacts-milvus-standalone-nightly-61-pymilvus-e2e-logs.tar.gz
  2. collection name: index_qpRtboPM
  3. failed time: [2023-01-11T20:10:17.593Z] [gw1] [ 24%] FAILED testcases/test_index.py::TestNewIndexBase::test_index_collection_with_after_load

@yanliang567
Copy link
Contributor

/assign @jiaoew1991
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 14, 2023
@jiaoew1991
Copy link
Contributor

/assign @yah01
/unassign

@sre-ci-robot sre-ci-robot assigned yah01 and unassigned jiaoew1991 Jan 16, 2023
@yah01 yah01 mentioned this issue Jan 16, 2023
Merged
@yah01
Copy link
Member

yah01 commented Jan 17, 2023

fixed with #21742 #21762
/assign @binbinlv

@binbinlv
Copy link
Contributor Author

@NicoYuan1986

Whether this issue still exists in the nightly tests last night?

@NicoYuan1986
Copy link
Contributor

@NicoYuan1986

Whether this issue still exists in the nightly tests last night?

Not reproduced for these days.

@binbinlv
Copy link
Contributor Author

OK, close this issue first, will reopen it if it occurs in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yah01 yah01

@binbinlv binbinlv

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.2.3
Development

No branches or pull requests
5 participants
@jiaoew1991 @yah01 @yanliang567 @binbinlv @NicoYuan1986