Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: standalone panic with error failed to seek, error please subcribe the channel, channel name =by-dev-rootcoord-dml during test #33893

Closed
1 task done
zhuwenxing opened this issue Jun 17, 2024 · 2 comments
Closed
1 task done

[Bug]: standalone panic with error failed to seek, error please subcribe the channel, channel name =by-dev-rootcoord-dml during test #33893

zhuwenxing opened this issue Jun 17, 2024 · 2 comments
Assignees
@weiliu1031
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.4.6

Comments

@zhuwenxing
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:2.4-20240614-fd1c7b1a-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka): kafka   
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[2024/06/16 06:46:46.810 +00:00] [INFO] [indexnode/task.go:516] ["index params are ready"] [buildID=450499640327638097] ["index params"="{\"dim\":\"128\",\"index_type\":\"IVF_PQ\",\"m\":\"16\",\"metric_type\":\"L2\",\"nbits\":\"8\",\"nlist\":\"128\"}"]
I20240616 06:46:46.810719  6071 MinioChunkManager.cpp:225] [SERVER][PreCheck][milvus] start to precheck chunk manager with configuration: [address=kafka-standalone-reinstall-2517-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=warn, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
I20240616 06:46:46.813861  6071 ChunkManager.cpp:112] [SERVER][AwsChunkManager][milvus] init AwsChunkManager with parameter[endpoint=kafka-standalone-reinstall-2517-minio:9000][bucket_name=milvus-bucket][root_path=file][use_secure=false]
I20240616 06:46:46.813952  6071 index_factory.cc:52] [KNOWHERE][Create][milvus] use key IVF_PQ_fp32 to create knowhere index IVF_PQ with version 4
[2024/06/16 06:46:46.825 +00:00] [DEBUG] [datacoord/server.go:810] ["Updating segment number of rows"] [segmentID=450499640328244453] ["old value"=3028] ["new value"=3028]
[2024/06/16 06:46:46.827 +00:00] [DEBUG] [datacoord/server.go:810] ["Updating segment number of rows"] [segmentID=450499640328244455] ["old value"=0] ["new value"=2972]
[2024/06/16 06:46:46.827 +00:00] [DEBUG] [datanode/binlog_io.go:47] ["down load"] [path="[file/insert_log/450499550930224499/450499550930224500/450499640327638393/108/450499640325835589,file/insert_log/450499550930224499/450499550930224500/450499640327638393/100/450499640325835597,file/insert_log/450499550930224499/450499550930224500/450499640327638393/102/450499640325835591,file/insert_log/450499550930224499/450499550930224500/450499640327638393/111/450499640325835588,file/insert_log/450499550930224499/450499550930224500/450499640327638393/0/450499640325835595,file/insert_log/450499550930224499/450499550930224500/450499640327638393/112/450499640325835594,file/insert_log/450499550930224499/450499550930224500/450499640327638393/109/450499640325835590,file/insert_log/450499550930224499/450499550930224500/450499640327638393/107/450499640325835602,file/insert_log/450499550930224499/450499550930224500/450499640327638393/103/450499640325835601,file/insert_log/450499550930224499/450499550930224500/450499640327638393/104/450499640325835592,file/insert_log/450499550930224499/450499550930224500/450499640327638393/105/450499640325835593,file/insert_log/450499550930224499/450499550930224500/450499640327638393/1/450499640325835596,file/insert_log/450499550930224499/450499550930224500/450499640327638393/106/450499640325835599,file/insert_log/450499550930224499/450499550930224500/450499640327638393/101/450499640325835598,file/insert_log/450499550930224499/450499550930224500/450499640327638393/110/450499640325835600]"]
[2024/06/16 06:46:46.827 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/110/450499640325835600]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/107/450499640325835602]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/103/450499640325835601]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/111/450499640325835588]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/100/450499640325835597]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/0/450499640325835595]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/104/450499640325835592]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/109/450499640325835590]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/102/450499640325835591]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/101/450499640325835598]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/108/450499640325835589]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/1/450499640325835596]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/106/450499640325835599]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/112/450499640325835594]
[2024/06/16 06:46:46.828 +00:00] [DEBUG] [io/binlog_io.go:61] ["BinlogIO download"] [path=file/insert_log/450499550930224499/450499550930224500/450499640327638393/105/450499640325835593]
[2024/06/16 06:46:46.845 +00:00] [ERROR] [msgdispatcher/manager.go:241] ["split failed"] [role=datanode] [nodeID=2] [vchannel=by-dev-rootcoord-dml_8_450499550930024446v1] [error="failed to seek, error please subcribe the channel, channel name =by-dev-rootcoord-dml"] [stack="github.com/milvus-io/milvus/pkg/mq/msgdispatcher.(*dispatcherManager).split\n\t/go/src/github.com/milvus-io/milvus/pkg/mq/msgdispatcher/manager.go:241\ngithub.com/milvus-io/milvus/pkg/mq/msgdispatcher.(*dispatcherManager).Run.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/mq/msgdispatcher/manager.go:171\ngithub.com/milvus-io/milvus/pkg/util/typeutil.(*ConcurrentMap[...]).Range.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/typeutil/map.go:54\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:476\ngithub.com/milvus-io/milvus/pkg/util/typeutil.(*ConcurrentMap[...]).Range\n\t/go/src/github.com/milvus-io/milvus/pkg/util/typeutil/map.go:51\ngithub.com/milvus-io/milvus/pkg/mq/msgdispatcher.(*dispatcherManager).Run\n\t/go/src/github.com/milvus-io/milvus/pkg/mq/msgdispatcher/manager.go:170"]
panic: failed to seek, error please subcribe the channel, channel name =by-dev-rootcoord-dml

goroutine 8460 [running]:
panic({0x4e42760, 0xc02bb19f60})
	/usr/local/go/src/runtime/panic.go:987 +0x3bb fp=0xc00aff5af0 sp=0xc00aff5a30 pc=0x1c2cf1b
github.com/milvus-io/milvus/pkg/mq/msgdispatcher.(*dispatcherManager).split(0xc004a16750, 0xc008d33c40)
	/go/src/github.com/milvus-io/milvus/pkg/mq/msgdispatcher/manager.go:242 +0x5bd fp=0xc00aff5cd8 sp=0xc00aff5af0 pc=0x42bbffd
github.com/milvus-io/milvus/pkg/mq/msgdispatcher.(*dispatcherManager).Run.func1({0xc00529a360, 0x2b}, 0xc030084820?)
	/go/src/github.com/milvus-io/milvus/pkg/mq/msgdispatcher/manager.go:171 +0x32 fp=0xc00aff5d10 sp=0xc00aff5cd8 pc=0x42bb032
github.com/milvus-io/milvus/pkg/util/typeutil.(*ConcurrentMap[...]).Range.func1({0x4f210c0?, 0xc008d33c40?})
	/go/src/github.com/milvus-io/milvus/pkg/util/typeutil/map.go:54 +0x51 fp=0xc00aff5d38 sp=0xc00aff5d10 pc=0x2accad1
sync.(*Map).Range(0xc004e8f110, 0xc00aff5de0)
	/usr/local/go/src/sync/map.go:476 +0x228 fp=0xc00aff5dd0 sp=0xc00aff5d38 pc=0x1c87868
github.com/milvus-io/milvus/pkg/util/typeutil.(*ConcurrentMap[...]).Range(0xc005c57e40?, 0x0?)
	/go/src/github.com/milvus-io/milvus/pkg/util/typeutil/map.go:51 +0x45 fp=0xc00aff5e08 sp=0xc00aff5dd0 pc=0x2acd625
github.com/milvus-io/milvus/pkg/mq/msgdispatcher.(*dispatcherManager).Run(0xc004a16750)
	/go/src/github.com/milvus-io/milvus/pkg/mq/msgdispatcher/manager.go:170 +0x457 fp=0xc00aff5fc8 sp=0xc00aff5e08 pc=0x42baf57
github.com/milvus-io/milvus/pkg/mq/msgdispatcher.(*client).Register.func2()
	/go/src/github.com/milvus-io/milvus/pkg/mq/msgdispatcher/client.go:74 +0x2b fp=0xc00aff5fe0 sp=0xc00aff5fc8 pc=0x42b606b
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00aff5fe8 sp=0xc00aff5fe0 pc=0x1c66ec1
created by github.com/milvus-io/milvus/pkg/mq/msgdispatcher.(*client).Register
	/go/src/github.com/milvus-io/milvus/pkg/mq/msgdispatcher/client.go:74 +0x495

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_kafka_for_release_cron/detail/deploy_test_kafka_for_release_cron/2517/pipeline
log:
Kafka Standalone Reinstall Logs.tar.gz

cluster: 4am
namespace: chaos-testing
pod info


[2024-06-16T06:52:37.435Z] + kubectl get pods -o wide

[2024-06-16T06:52:37.436Z] + grep kafka-standalone-reinstall-2517

[2024-06-16T06:52:37.436Z] kafka-standalone-reinstall-2517-0                                 2/2     Running             1 (23m ago)       24m     10.104.34.148   4am-node37   <none>           <none>

[2024-06-16T06:52:37.436Z] kafka-standalone-reinstall-2517-1                                 2/2     Running             0                 24m     10.104.20.103   4am-node22   <none>           <none>

[2024-06-16T06:52:37.436Z] kafka-standalone-reinstall-2517-2                                 2/2     Running             0                 24m     10.104.26.34    4am-node32   <none>           <none>

[2024-06-16T06:52:37.436Z] kafka-standalone-reinstall-2517-etcd-0                            1/1     Running             0                 24m     10.104.34.151   4am-node37   <none>           <none>

[2024-06-16T06:52:37.436Z] kafka-standalone-reinstall-2517-etcd-1                            1/1     Running             0                 24m     10.104.20.101   4am-node22   <none>           <none>

[2024-06-16T06:52:37.436Z] kafka-standalone-reinstall-2517-etcd-2                            1/1     Running             0                 24m     10.104.26.35    4am-node32   <none>           <none>

[2024-06-16T06:52:37.436Z] kafka-standalone-reinstall-2517-exporter-67757654c4-6bg4c         1/1     Running             3 (23m ago)       24m     10.104.16.200   4am-node21   <none>           <none>

[2024-06-16T06:52:37.436Z] kafka-standalone-reinstall-2517-milvus-standalone-655568f9nqqdz   1/1     Running             5 (5m50s ago)     24m     10.104.16.202   4am-node21   <none>           <none>

[2024-06-16T06:52:37.436Z] kafka-standalone-reinstall-2517-minio-65bdbf565d-gp77r            1/1     Running             0                 24m     10.104.34.149   4am-node37   <none>           <none>

[2024-06-16T06:52:37.436Z] kafka-standalone-reinstall-2517-zookeeper-0                       1/1     Running             0                 24m     10.104.34.150   4am-node37   <none>           <none>

[2024-06-16T06:52:37.437Z] kafka-standalone-reinstall-2517-zookeeper-1                       1/1     Running             0                 24m     10.104.26.32    4am-node32   <none>           <none>

[2024-06-16T06:52:37.437Z] kafka-standalone-reinstall-2517-zookeeper-2                       1/1     Running             0                 24m     10.104.20.104   4am-node22   <none>           <none>

Anything else?

It looks like an unstable issue.

with the same image tag, the previous test was successful
job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_kafka_for_release_cron/detail/deploy_test_kafka_for_release_cron/2513/pipeline
@zhuwenxing zhuwenxing added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jun 17, 2024
@zhuwenxing zhuwenxing added this to the 2.4.5 milestone Jun 17, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jun 17, 2024
@yanliang567
Copy link
Contributor

/assign @weiliu1031
/unassign

@yanliang567 yanliang567 modified the milestones: 2.4.5, 2.4.6 Jun 26, 2024
@zhuwenxing
Copy link
Contributor Author

dup with #34221

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@weiliu1031 weiliu1031

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.4.6
Development

No branches or pull requests
3 participants
@zhuwenxing @yanliang567 @weiliu1031