Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: query failed with error fail to Query on QueryNode 14: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_xxxv1] after querynode pod failure chaos test #31560

Closed
1 task done
zhuwenxing opened this issue Mar 25, 2024 · 3 comments
Closed
1 task done

[Bug]: query failed with error fail to Query on QueryNode 14: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_xxxv1] after querynode pod failure chaos test #31560

zhuwenxing opened this issue Mar 25, 2024 · 3 comments
Assignees
@tedxu @zhuwenxing @congqixia
Labels
kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. test/chaos chaos test triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@zhuwenxing
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:2.4-20240324-1e0bf5ac-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:02 - DEBUG - ci_test]: (api_request)  : [Connections.connect] args: ['default', '', '', ''], kwargs: {'host': '10.255.123.232', 'port': 19530} (api_request.py:62)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:02 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:02 - DEBUG - ci_test]: (api_request)  : [Connections.has_connection] args: ['default'], kwargs: {} (api_request.py:62)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:02 - DEBUG - ci_test]: (api_response) : True  (api_request.py:37)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:02 - DEBUG - ci_test]: (api_request)  : [Collection] args: ['Checker__UJ6hPI2k', {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}, {'name': 'varchar', 'description': '', 'type': <DataType......, kwargs: {'consistency_level': 'Strong'} (api_request.py:62)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:02 - DEBUG - ci_test]: (api_response) : <Collection>:

[2024-03-25T01:48:20.900Z] -------------

[2024-03-25T01:48:20.900Z] <name>: Checker__UJ6hPI2k

[2024-03-25T01:48:20.900Z] <description>: 

[2024-03-25T01:48:20.900Z] <schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}......  (api_request.py:37)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:02 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:03 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:03 - DEBUG - ci_test]: (api_request)  : [Collection.compact] args: [180], kwargs: {} (api_request.py:62)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:03 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:40 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:40 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:40 - INFO - ci_test]: assert create collection: 0.018023252487182617, init_entities: 686573 (test_all_collections_after_chaos.py:49)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:41 - DEBUG - ci_test]: (api_request)  : [Collection.insert] args: [[[-3000, -2999, -2998, -2997, -2996, -2995, -2994, -2993, -2992, -2991, -2990, -2989, -2988, -2987, -2986, -2985, -2984, -2983, -2982, -2981, -2980, -2979, -2978, -2977, -2976, -2975, -2974, -2973, -2972, -2971, -2970, -2969, -2968, -2967, -2966, -2965, -2964, -2963, -2962, -2961, -2960, -2959, -29......, kwargs: {'timeout': 180} (api_request.py:62)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:41 - DEBUG - ci_test]: (api_response) : (insert count: 2000, delete count: 0, upsert count: 0, timestamp: 448615222109011970, success count: 2000, err count: 0)  (api_request.py:37)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:41 - INFO - ci_test]: assert insert: 0.3872401714324951 (test_all_collections_after_chaos.py:61)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:41 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:44 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:44 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:45 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:45 - INFO - ci_test]: assert flush: 3.6483564376831055, entities: 688573 (test_all_collections_after_chaos.py:71)

[2024-03-25T01:48:20.900Z] [2024-03-25 01:47:45 - INFO - ci_test]: index info: [{'collection': 'Checker__UJ6hPI2k', 'field': 'float_vector', 'index_name': 'vec_index', 'index_param': {'params': {'M': 48, 'efConstruction': 500}, 'index_type': 'HNSW', 'metric_type': 'L2'}}] (test_all_collections_after_chaos.py:87)

[2024-03-25T01:48:20.901Z] [2024-03-25 01:47:45 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)

[2024-03-25T01:48:20.901Z] [2024-03-25 01:47:45 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-25T01:48:20.901Z] [2024-03-25 01:47:45 - DEBUG - ci_test]: (api_request)  : [Collection.search] args: [[[0.1550505819093183, 0.04455169997032458, 0.002402624803400971, 0.09715286707307286, 0.060910072235617195, 0.010995898225257148, 0.05795158910328207, 0.12071754089701908, 0.1261115160771701, 0.012795545020214676, 0.1525373419677744, 0.05604552236176172, 0.1150000718111701, 0.1315209332544024, 0.08......, kwargs: {} (api_request.py:62)

[2024-03-25T01:48:20.901Z] [2024-03-25 01:47:46 - DEBUG - ci_test]: (api_response) : ["['id: 1711330802754340, distance: 22.673309326171875, entity: {}']"]  (api_request.py:37)

[2024-03-25T01:48:20.901Z] [2024-03-25 01:47:46 - INFO - ci_test]: assert search: 0.6403112411499023 (test_all_collections_after_chaos.py:100)

[2024-03-25T01:48:20.901Z] [2024-03-25 01:47:46 - DEBUG - ci_test]: (api_request)  : [Collection.query] args: ['int64 in [-3000, -2999, -2998, -2997, -2996, -2995, -2994, -2993, -2992, -2991, -2990, -2989, -2988, -2987, -2986, -2985, -2984, -2983, -2982, -2981, -2980, -2979, -2978, -2977, -2976, -2975, -2974, -2973, -2972, -2971, -2970, -2969, -2968, -2967, -2966, -2965, -2964, -2963, -2962, -2961, -2960, -......, kwargs: {} (api_request.py:62)

[2024-03-25T01:48:20.901Z] [2024-03-25 01:47:47 - ERROR - pymilvus.decorators]: RPC error: [query], <MilvusException: (code=503, message=fail to Query on QueryNode 14: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_14_448614674856488642v1])>, <Time:{'RPC start': '2024-03-25 01:47:46.035545', 'RPC error': '2024-03-25 01:47:47.035588'}> (decorators.py:139)

[2024-03-25T01:48:20.901Z] [2024-03-25 01:47:47 - ERROR - ci_test]: Traceback (most recent call last):

[2024-03-25T01:48:20.901Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper

[2024-03-25T01:48:20.901Z]     res = func(*args, **_kwargs)

[2024-03-25T01:48:20.901Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request

[2024-03-25T01:48:20.901Z]     return func(*arg, **kwargs)

[2024-03-25T01:48:20.901Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 925, in query

[2024-03-25T01:48:20.901Z]     return conn.query(

[2024-03-25T01:48:20.901Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 140, in handler

[2024-03-25T01:48:20.901Z]     raise e from e

[2024-03-25T01:48:20.901Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler

[2024-03-25T01:48:20.901Z]     return func(*args, **kwargs)

[2024-03-25T01:48:20.901Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 175, in handler

[2024-03-25T01:48:20.901Z]     return func(self, *args, **kwargs)

[2024-03-25T01:48:20.901Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 115, in handler

[2024-03-25T01:48:20.901Z]     raise e from e

[2024-03-25T01:48:20.901Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 86, in handler

[2024-03-25T01:48:20.901Z]     return func(*args, **kwargs)

[2024-03-25T01:48:20.901Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 1386, in query

[2024-03-25T01:48:20.901Z]     check_status(response.status)

[2024-03-25T01:48:20.901Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/utils.py", line 54, in check_status

[2024-03-25T01:48:20.901Z]     raise MilvusException(status.code, status.reason, status.error_code)

[2024-03-25T01:48:20.901Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=503, message=fail to Query on QueryNode 14: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_14_448614674856488642v1])>

[2024-03-25T01:48:20.901Z]  (api_request.py:45)

[2024-03-25T01:48:20.901Z] [2024-03-25 01:47:47 - ERROR - ci_test]: (api_response) : <MilvusException: (code=503, message=fail to Query on QueryNode 14: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_14_448614674856488642v1])> (api_request.py:46)

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-kafka-for-release-cron/detail/chaos-test-kafka-for-release-cron/11861/pipeline

log:
artifacts-querynode-pod-failure-11861-server-logs.tar.gz

Anything else?

may related to #31375
@zhuwenxing zhuwenxing added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 25, 2024
@zhuwenxing
Copy link
Contributor Author

search also failed with this error
failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-for-release-cron/detail/chaos-test-for-release-cron/11369/pipeline
log:
artifacts-querynode-pod-failure-11369-server-logs.tar.gz

[2024-03-24T23:19:38.262Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_request)  : [Connections.connect] args: ['default', '', '', ''], kwargs: {'host': '10.255.151.122', 'port': 19530} (api_request.py:62)

[2024-03-24T23:19:38.262Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-24T23:19:38.262Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_request)  : [Connections.has_connection] args: ['default'], kwargs: {} (api_request.py:62)

[2024-03-24T23:19:38.262Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_response) : True  (api_request.py:37)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_request)  : [Collection] args: ['Checker__ZYZ0pfxV', {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}, {'name': 'varchar', 'description': '', 'type': <DataType......, kwargs: {'consistency_level': 'Strong'} (api_request.py:62)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_response) : <Collection>:

[2024-03-24T23:19:38.263Z] -------------

[2024-03-24T23:19:38.263Z] <name>: Checker__ZYZ0pfxV

[2024-03-24T23:19:38.263Z] <description>: 

[2024-03-24T23:19:38.263Z] <schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}......  (api_request.py:37)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:13:54 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:13:54 - DEBUG - ci_test]: (api_request)  : [Collection.compact] args: [180], kwargs: {} (api_request.py:62)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:13:54 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:54 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:55 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:55 - INFO - ci_test]: assert create collection: 0.03525233268737793, init_entities: 745955 (test_all_collections_after_chaos.py:49)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:55 - DEBUG - ci_test]: (api_request)  : [Collection.insert] args: [[[-3000, -2999, -2998, -2997, -2996, -2995, -2994, -2993, -2992, -2991, -2990, -2989, -2988, -2987, -2986, -2985, -2984, -2983, -2982, -2981, -2980, -2979, -2978, -2977, -2976, -2975, -2974, -2973, -2972, -2971, -2970, -2969, -2968, -2967, -2966, -2965, -2964, -2963, -2962, -2961, -2960, -2959, -29......, kwargs: {'timeout': 180} (api_request.py:62)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:55 - DEBUG - ci_test]: (api_response) : (insert count: 2000, delete count: 0, upsert count: 0, timestamp: 448612819296583683, success count: 2000, err count: 0)  (api_request.py:37)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:55 - INFO - ci_test]: assert insert: 0.5283844470977783 (test_all_collections_after_chaos.py:61)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:55 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - INFO - ci_test]: assert flush: 3.569885492324829, entities: 747955 (test_all_collections_after_chaos.py:71)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - INFO - ci_test]: index info: [{'collection': 'Checker__ZYZ0pfxV', 'field': 'float_vector', 'index_name': 'vec_index', 'index_param': {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 48, 'efConstruction': 500}}}] (test_all_collections_after_chaos.py:87)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - DEBUG - ci_test]: (api_request)  : [Collection.search] args: [[[0.1394604147661045, 0.10148974298142811, 0.09137981234031903, 0.03233836942294032, 0.09185298370886213, 0.08251813531741621, 0.058100080704232296, 0.007344297647441485, 0.08989386617268842, 0.09960945674128009, 0.058880340721370904, 0.03205196773015912, 0.10427486666482212, 0.10932439188056996, 0......, kwargs: {} (api_request.py:62)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:15:00 - ERROR - pymilvus.decorators]: RPC error: [search], <MilvusException: (code=503, message=fail to search on QueryNode 14: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_11_448612279430753456v0])>, <Time:{'RPC start': '2024-03-24 23:14:59.857512', 'RPC error': '2024-03-24 23:15:00.755138'}> (decorators.py:139)

[2024-03-24T23:19:38.263Z] [2024-03-24 23:15:00 - ERROR - ci_test]: Traceback (most recent call last):

[2024-03-24T23:19:38.263Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper

[2024-03-24T23:19:38.263Z]     res = func(*args, **_kwargs)

[2024-03-24T23:19:38.263Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request

[2024-03-24T23:19:38.263Z]     return func(*arg, **kwargs)

[2024-03-24T23:19:38.263Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 792, in search

[2024-03-24T23:19:38.263Z]     resp = conn.search(

[2024-03-24T23:19:38.263Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 140, in handler

[2024-03-24T23:19:38.263Z]     raise e from e

[2024-03-24T23:19:38.263Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler

[2024-03-24T23:19:38.263Z]     return func(*args, **kwargs)

[2024-03-24T23:19:38.263Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 175, in handler

[2024-03-24T23:19:38.263Z]     return func(self, *args, **kwargs)

[2024-03-24T23:19:38.263Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 115, in handler

[2024-03-24T23:19:38.263Z]     raise e from e

[2024-03-24T23:19:38.263Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 86, in handler

[2024-03-24T23:19:38.263Z]     return func(*args, **kwargs)

[2024-03-24T23:19:38.263Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 775, in search

[2024-03-24T23:19:38.263Z]     return self._execute_search(request, timeout, round_decimal=round_decimal, **kwargs)

[2024-03-24T23:19:38.263Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 735, in _execute_search

[2024-03-24T23:19:38.263Z]     raise e from e

[2024-03-24T23:19:38.263Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 728, in _execute_search

[2024-03-24T23:19:38.263Z]     check_status(response.status)

[2024-03-24T23:19:38.263Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/utils.py", line 54, in check_status

[2024-03-24T23:19:38.263Z]     raise MilvusException(status.code, status.reason, status.error_code)

[2024-03-24T23:19:38.263Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=503, message=fail to search on QueryNode 14: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_11_448612279430753456v0])>

@zhuwenxing zhuwenxing added test/chaos chaos test priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. labels Mar 25, 2024
@xiaofan-luan
Copy link
Contributor

/assign @congqixia @tedxu
We don't really need the "distribution is not servcieable" everything should be just find the correct shard delegator and let the delegator decide whether to host or wait until recovery done.

Currently design is too error prone and there is no easy way to sync complex state between node

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 27, 2024
@yanliang567 yanliang567 removed their assignment Mar 27, 2024
@congqixia
Copy link
Contributor

Segment lack error could be related to #31584 #31602
it looks like this error does not exist in recent chaos test
/assign @zhuwenxing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tedxu tedxu

@zhuwenxing zhuwenxing

@congqixia congqixia

Labels
kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. test/chaos chaos test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@tedxu @zhuwenxing @yanliang567 @xiaofan-luan @congqixia