This pod keeps restarting and reports the "CrashLoopBackOff" error. I really don't know how to solve it and am seeking help here. #49605
Replies: 1 comment
-
|
The logs indicate etcd instability / high latency, causing Milvus session lease expiration. Some etcd requests took several seconds, which is extremely slow for milvus. Normal latency per request in etcd should be less than 100ms. For this reason, milvus internal RPC failed with "rpc error: code = Canceled desc = context canceled". And then Milvus starts node-down handling. Milvus relies heavily on etcd for:
If etcd has multi-second letency, Milvus components may miss keepalive deadlines. Then etcd expires their leases, and Milvus treats the node as down. In standalone mode, several roles share the same server/pod, so one unhealthy process/pod can show lease expiry for datanode, streamingnode, and querynode together. Ideally, etcd should be deployed on a high-performance NVMe SSD. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
这个nkqx9一直重新启动然后报错CrashLoopBackOff。
日志：
kubectl describe pod my-milvus-standalone-5d76b85b98-nkqx9
Events:
Type Reason Age From Message
Warning Unhealthy 48m (x9 over 20h) kubelet Liveness probe failed: dial tcp 10.42.0.49:9091: i/o timeout
Warning NodeNotReady 48m (x7 over 20h) node-controller Node is not ready
Warning Unhealthy 48m kubelet Readiness probe failed: Get "http://10.42.0.49:9091/healthz": read tcp 10.42.0.1:45874->10.42.0.49:9091: read: connection reset by peer
Warning Unhealthy 13m (x149 over 23h) kubelet Readiness probe failed: Get "http://10.42.0.49:9091/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 4m29s (x107 over 25h) kubelet Readiness probe failed: Get "http://10.42.0.49:9091/healthz": dial tcp 10.42.0.49:9091: connect: connection refused
Warning BackOff 4m29s (x1577 over 22h) kubelet Back-off restarting failed container standalone in pod my-milvus-standalone-5d76b85b98-nkqx9_default(a053c2ce-bdf9-443c-933b-3a4798ed351f)
Normal Pulled 7s (x190 over 25h) kubelet Container image "milvusdb/milvus:v2.6.8" already present on machine
Normal Created 7s (x190 over 25h) kubelet Created container: standalone
kubectl logs my-milvus-standalone-5d76b85b98-nkqx9 --previous | tail -n 100
[2026/05/08 07:13:00.930 +00:00] [WARN] [task/executor.go:256] ["failed to load segment"] [taskID=1778224042229] [collectionID=464973051317556870] [replicaID=464973051343405060] [segmentID=465443805101243039] [node=221] [source=segment_checker] [shardLeader=221] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:571 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:585 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:102 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:163 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments\n/workspace/source/internal/querycoordv2/session/cluster.go:136 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments.func1\n/workspace/source/internal/querycoordv2/session/cluster.go:329 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n/workspace/source/internal/querycoordv2/session/cluster.go:133 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments\n/workspace/source/internal/querycoordv2/task/executor.go:253 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n/workspace/source/internal/querycoordv2/task/executor.go:179 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction: rpc error: code = Canceled desc = context canceled"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:571\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:585\n | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/querynode/client/client.go:102\n | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments\n | \t/workspace/source/internal/distributed/querynode/client/client.go:163\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments.func1\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:136\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:329\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:133\n | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n | \t/workspace/source/internal/querycoordv2/task/executor.go:253\n | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n | \t/workspace/source/internal/querycoordv2/task/executor.go:179\n | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n | \t/workspace/source/internal/querycoordv2/task/executor.go:139\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:571 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:585 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/querynode/client/client.go:102 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/querynode/client/client.go:163 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments\n | /workspace/source/internal/querycoordv2/session/cluster.go:136 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments.func1\n | /workspace/source/internal/querycoordv2/session/cluster.go:329 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n | /workspace/source/internal/querycoordv2/session/cluster.go:133 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments\n | /workspace/source/internal/querycoordv2/task/executor.go:253 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n | /workspace/source/internal/querycoordv2/task/executor.go:179 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[2026/05/08 07:13:00.929 +00:00] [WARN] [task/executor.go:256] ["failed to load segment"] [taskID=1778224042242] [collectionID=464973051317556870] [replicaID=464973051343405060] [segmentID=465443808122948423] [node=221] [source=segment_checker] [shardLeader=221] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:571 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:585 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:102 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:163 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments\n/workspace/source/internal/querycoordv2/session/cluster.go:136 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments.func1\n/workspace/source/internal/querycoordv2/session/cluster.go:329 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n/workspace/source/internal/querycoordv2/session/cluster.go:133 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments\n/workspace/source/internal/querycoordv2/task/executor.go:253 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n/workspace/source/internal/querycoordv2/task/executor.go:179 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction: rpc error: code = Canceled desc = context canceled"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:571\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:585\n | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/querynode/client/client.go:102\n | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments\n | \t/workspace/source/internal/distributed/querynode/client/client.go:163\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments.func1\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:136\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:329\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:133\n | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n | \t/workspace/source/internal/querycoordv2/task/executor.go:253\n | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n | \t/workspace/source/internal/querycoordv2/task/executor.go:179\n | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n | \t/workspace/source/internal/querycoordv2/task/executor.go:139\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:571 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:585 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/querynode/client/client.go:102 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/querynode/client/client.go:163 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments\n | /workspace/source/internal/querycoordv2/session/cluster.go:136 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments.func1\n | /workspace/source/internal/querycoordv2/session/cluster.go:329 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n | /workspace/source/internal/querycoordv2/session/cluster.go:133 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments\n | /workspace/source/internal/querycoordv2/task/executor.go:253 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n | /workspace/source/internal/querycoordv2/task/executor.go:179 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[2026/05/08 07:13:00.944 +00:00] [WARN] [resolver/watch_based_grpc_resolver.go:54] ["fail to update resolver state"] [prefix=by-dev/meta/session/streamingnode] [exclusive=false] [semver=">=2.6.0-dev"] [component=grpc-resolver] [scheme=milvus-session] [id=25] [state="Version: 6963284, Addrs: "] [error="bad resolver state"]
[2026/05/08 07:13:00.944 +00:00] [INFO] [resolver/resolver_with_discoverer.go:199] ["update resolver done"] [prefix=by-dev/meta/session/streamingnode] [exclusive=false] [semver=">=2.6.0-dev"] [component=grpc-resolver] [scheme=milvus-session]
[2026/05/08 07:13:00.945 +00:00] [INFO] [balancer/balancer_impl.go:419] ["start to balance"] [module=streamingcoord] [component=balancer] [policy=vchannelFair]
[2026/05/08 07:13:00.945 +00:00] [INFO] [balancer/balancer_impl.go:422] ["collect all status..."] [module=streamingcoord] [component=balancer] [policy=vchannelFair]
[2026/05/08 07:13:00.952 +00:00] [INFO] [meta/resource_manager.go:1007] ["unassign node to resource group"] [rgName=__default_resource_group] [node=221]
[2026/05/08 07:13:00.952 +00:00] [ERROR] [sessionutil/session_util.go:636] ["confirm the lease is expired, the session is expired without activing closing"] [component=service-registration] [role=datanode] [serverID=221] [address=10.42.0.49:21124] [] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).checkKeepaliveTTL\n\t/workspace/source/internal/util/sessionutil/session_util.go:636\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).processKeepAliveResponse\n\t/workspace/source/internal/util/sessionutil/session_util.go:588"]
[2026/05/08 07:13:00.952 +00:00] [ERROR] [sessionutil/session_util.go:636] ["confirm the lease is expired, the session is expired without activing closing"] [component=service-registration] [role=streamingnode] [serverID=221] [address=10.42.0.49:22222] [] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).checkKeepaliveTTL\n\t/workspace/source/internal/util/sessionutil/session_util.go:636\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).processKeepAliveResponse\n\t/workspace/source/internal/util/sessionutil/session_util.go:588"]
[2026/05/08 07:13:00.952 +00:00] [ERROR] [sessionutil/session_util.go:636] ["confirm the lease is expired, the session is expired without activing closing"] [component=service-registration] [role=querynode] [serverID=221] [address=10.42.0.49:21123] [] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).checkKeepaliveTTL\n\t/workspace/source/internal/util/sessionutil/session_util.go:636\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).processKeepAliveResponse\n\t/workspace/source/internal/util/sessionutil/session_util.go:588"]
[2026/05/08 07:13:00.952 +00:00] [INFO] [meta/resource_manager.go:571] ["HandleNodeDown: remove node from resource group"] [rgName=__default_resource_group] [node=221] []
kubectl logs my-milvus-etcd-0 --tail=50 查看了etcd 日，日志如下
[2026/05/08 07:13:00.930 +00:00] [WARN] [task/executor.go:256] ["failed to load segment"] [taskID=1778224042229] [collectionID=464973051317556870] [replicaID=464973051343405060] [segmentID=465443805101243039] [node=221] [source=segment_checker] [shardLeader=221] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:571 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:585 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:102 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:163 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments\n/workspace/source/internal/querycoordv2/session/cluster.go:136 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments.func1\n/workspace/source/internal/querycoordv2/session/cluster.go:329 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n/workspace/source/internal/querycoordv2/session/cluster.go:133 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments\n/workspace/source/internal/querycoordv2/task/executor.go:253 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n/workspace/source/internal/querycoordv2/task/executor.go:179 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction: rpc error: code = Canceled desc = context canceled"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:571\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:585\n | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/querynode/client/client.go:102\n | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments\n | \t/workspace/source/internal/distributed/querynode/client/client.go:163\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments.func1\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:136\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:329\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:133\n | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n | \t/workspace/source/internal/querycoordv2/task/executor.go:253\n | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n | \t/workspace/source/internal/querycoordv2/task/executor.go:179\n | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n | \t/workspace/source/internal/querycoordv2/task/executor.go:139\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:571 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:585 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/querynode/client/client.go:102 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/querynode/client/client.go:163 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments\n | /workspace/source/internal/querycoordv2/session/cluster.go:136 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments.func1\n | /workspace/source/internal/querycoordv2/session/cluster.go:329 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n | /workspace/source/internal/querycoordv2/session/cluster.go:133 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments\n | /workspace/source/internal/querycoordv2/task/executor.go:253 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n | /workspace/source/internal/querycoordv2/task/executor.go:179 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[2026/05/08 07:13:00.929 +00:00] [WARN] [task/executor.go:256] ["failed to load segment"] [taskID=1778224042242] [collectionID=464973051317556870] [replicaID=464973051343405060] [segmentID=465443808122948423] [node=221] [source=segment_checker] [shardLeader=221] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:571 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:585 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:102 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:163 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments\n/workspace/source/internal/querycoordv2/session/cluster.go:136 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments.func1\n/workspace/source/internal/querycoordv2/session/cluster.go:329 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n/workspace/source/internal/querycoordv2/session/cluster.go:133 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments\n/workspace/source/internal/querycoordv2/task/executor.go:253 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n/workspace/source/internal/querycoordv2/task/executor.go:179 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction: rpc error: code = Canceled desc = context canceled"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace: rpc error: code = Canceled desc = context canceled\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:571\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:585\n | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/querynode/client/client.go:102\n | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments\n | \t/workspace/source/internal/distributed/querynode/client/client.go:163\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments.func1\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:136\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:329\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:133\n | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n | \t/workspace/source/internal/querycoordv2/task/executor.go:253\n | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n | \t/workspace/source/internal/querycoordv2/task/executor.go:179\n | github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n | \t/workspace/source/internal/querycoordv2/task/executor.go:139\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/v2/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:571 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:585 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/querynode/client/client.go:102 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/querynode/client/client.go:163 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadSegments\n | /workspace/source/internal/querycoordv2/session/cluster.go:136 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments.func1\n | /workspace/source/internal/querycoordv2/session/cluster.go:329 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n | /workspace/source/internal/querycoordv2/session/cluster.go:133 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadSegments\n | /workspace/source/internal/querycoordv2/task/executor.go:253 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n | /workspace/source/internal/querycoordv2/task/executor.go:179 github.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\nWraps: (3) rpc error: code = Canceled desc = context canceled\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[2026/05/08 07:13:00.944 +00:00] [WARN] [resolver/watch_based_grpc_resolver.go:54] ["fail to update resolver state"] [prefix=by-dev/meta/session/streamingnode] [exclusive=false] [semver=">=2.6.0-dev"] [component=grpc-resolver] [scheme=milvus-session] [id=25] [state="Version: 6963284, Addrs: "] [error="bad resolver state"]
[2026/05/08 07:13:00.944 +00:00] [INFO] [resolver/resolver_with_discoverer.go:199] ["update resolver done"] [prefix=by-dev/meta/session/streamingnode] [exclusive=false] [semver=">=2.6.0-dev"] [component=grpc-resolver] [scheme=milvus-session]
[2026/05/08 07:13:00.945 +00:00] [INFO] [balancer/balancer_impl.go:419] ["start to balance"] [module=streamingcoord] [component=balancer] [policy=vchannelFair]
[2026/05/08 07:13:00.945 +00:00] [INFO] [balancer/balancer_impl.go:422] ["collect all status..."] [module=streamingcoord] [component=balancer] [policy=vchannelFair]
[2026/05/08 07:13:00.952 +00:00] [INFO] [meta/resource_manager.go:1007] ["unassign node to resource group"] [rgName=__default_resource_group] [node=221]
[2026/05/08 07:13:00.952 +00:00] [ERROR] [sessionutil/session_util.go:636] ["confirm the lease is expired, the session is expired without activing closing"] [component=service-registration] [role=datanode] [serverID=221] [address=10.42.0.49:21124] [] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).checkKeepaliveTTL\n\t/workspace/source/internal/util/sessionutil/session_util.go:636\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).processKeepAliveResponse\n\t/workspace/source/internal/util/sessionutil/session_util.go:588"]
[2026/05/08 07:13:00.952 +00:00] [ERROR] [sessionutil/session_util.go:636] ["confirm the lease is expired, the session is expired without activing closing"] [component=service-registration] [role=streamingnode] [serverID=221] [address=10.42.0.49:22222] [] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).checkKeepaliveTTL\n\t/workspace/source/internal/util/sessionutil/session_util.go:636\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).processKeepAliveResponse\n\t/workspace/source/internal/util/sessionutil/session_util.go:588"]
[2026/05/08 07:13:00.952 +00:00] [ERROR] [sessionutil/session_util.go:636] ["confirm the lease is expired, the session is expired without activing closing"] [component=service-registration] [role=querynode] [serverID=221] [address=10.42.0.49:21123] [] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).checkKeepaliveTTL\n\t/workspace/source/internal/util/sessionutil/session_util.go:636\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).processKeepAliveResponse\n\t/workspace/source/internal/util/sessionutil/session_util.go:588"]
[2026/05/08 07:13:00.952 +00:00] [INFO] [meta/resource_manager.go:571] ["HandleNodeDown: remove node from resource group"] [rgName=__default_resource_group] [node=221] []
(base) root@database:/home/trizhi# kubectl logs my-milvus-etcd-0 --tail=50
{"level":"info","ts":"2026-05-08T07:24:08.690475Z","caller":"v3compactor/revision.go:86","msg":"starting auto revision compaction","revision":6962798,"revision-compaction-retention":1000}
{"level":"info","ts":"2026-05-08T07:24:08.729250Z","caller":"v3compactor/revision.go:94","msg":"completed auto revision compaction","revision":6962798,"revision-compaction-retention":1000,"took":"38.775873ms"}
{"level":"info","ts":"2026-05-08T07:24:08.729288Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":6962798}
{"level":"info","ts":"2026-05-08T07:24:08.984054Z","caller":"mvcc/kvstore_compaction.go:72","msg":"finished scheduled compaction","compact-revision":6962798,"took":"252.325966ms","hash":3236343478,"current-db-size-bytes":43945984,"current-db-size":"44 MB","current-db-size-in-use-bytes":2220032,"current-db-size-in-use":"2.2 MB"}
{"level":"info","ts":"2026-05-08T07:24:08.984103Z","caller":"mvcc/hash.go:151","msg":"storing new hash","hash":3236343478,"revision":6962798,"compact-revision":6962573}
{"level":"info","ts":"2026-05-08T07:24:21.919569Z","caller":"traceutil/trace.go:171","msg":"trace[197190430] put","detail":"{key:by-dev/kv/gid/timestamp; req_size:35; response_revision:6963803; }","duration":"106.101478ms","start":"2026-05-08T07:24:21.813442Z","end":"2026-05-08T07:24:21.919543Z","steps":["trace[197190430] 'process raft request' (duration: 105.969605ms)"],"step_count":1}
{"level":"warn","ts":"2026-05-08T07:25:07.317870Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"142.231384ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"by-dev/config" range_end:"by-dev/confih" serializable:true ","response":"range_response_count:0 size:7"}
{"level":"info","ts":"2026-05-08T07:25:07.317956Z","caller":"traceutil/trace.go:171","msg":"trace[1865940083] range","detail":"{range_begin:by-dev/config; range_end:by-dev/confih; response_count:0; response_revision:6963830; }","duration":"142.337227ms","start":"2026-05-08T07:25:07.175601Z","end":"2026-05-08T07:25:07.317938Z","steps":["trace[1865940083] 'range keys from in-memory index tree' (duration: 142.213306ms)"],"step_count":1}
{"level":"info","ts":"2026-05-08T07:25:36.929546Z","caller":"traceutil/trace.go:171","msg":"trace[1760640101] put","detail":"{key:by-dev/kv/gid/timestamp; req_size:35; response_revision:6963852; }","duration":"113.517904ms","start":"2026-05-08T07:25:36.816001Z","end":"2026-05-08T07:25:36.929519Z","steps":["trace[1760640101] 'process raft request' (duration: 39.93424ms)","trace[1760640101] 'store kv pair into bolt db' (duration: 73.507154ms)"],"step_count":2}
{"level":"warn","ts":"2026-05-08T07:26:46.765058Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"177.157018ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"by-dev/config" range_end:"by-dev/confih" serializable:true ","response":"range_response_count:0 size:7"}
{"level":"info","ts":"2026-05-08T07:26:46.914666Z","caller":"traceutil/trace.go:171","msg":"trace[269814837] range","detail":"{range_begin:by-dev/config; range_end:by-dev/confih; response_count:0; response_revision:6963884; }","duration":"403.192195ms","start":"2026-05-08T07:26:46.499083Z","end":"2026-05-08T07:26:46.902275Z","steps":["trace[269814837] 'get authentication metadata' (duration: 121.648645ms)","trace[269814837] 'range keys from in-memory index tree' (duration: 29.162973ms)"],"step_count":2}
{"level":"warn","ts":"2026-05-08T07:26:47.608470Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2026-05-08T07:26:46.309381Z","time spent":"621.296412ms","remote":"10.42.0.49:33322","response type":"/etcdserverpb.KV/Range","request count":0,"request size":32,"response count":0,"response size":30,"request content":"key:"by-dev/config" range_end:"by-dev/confih" serializable:true "}
{"level":"warn","ts":"2026-05-08T07:26:47.779932Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"811.54162ms","expected-duration":"100ms","prefix":"","request":"header:<ID:4549653772102273622 > put:<key:"by-dev/kv/gid/timestamp" value_size:8 >","response":"size:7"}
{"level":"info","ts":"2026-05-08T07:26:47.845342Z","caller":"traceutil/trace.go:171","msg":"trace[951704211] put","detail":"{key:by-dev/kv/gid/timestamp; req_size:35; response_revision:6963885; }","duration":"1.486967548s","start":"2026-05-08T07:26:46.358316Z","end":"2026-05-08T07:26:47.845283Z","steps":["trace[951704211] 'process raft request' (duration: 492.308706ms)","trace[951704211] 'marshal mvccpb.KeyValue' (duration: 20.948952ms)","trace[951704211] 'store kv pair into bolt db' (duration: 411.473722ms)"],"step_count":3}
{"level":"warn","ts":"2026-05-08T07:26:47.845458Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2026-05-08T07:26:46.301327Z","time spent":"1.544101345s","remote":"10.42.0.49:33336","response type":"/etcdserverpb.KV/Put","request count":1,"request size":35,"response count":0,"response size":30,"request content":"key:"by-dev/kv/gid/timestamp" value_size:8 "}
{"level":"info","ts":"2026-05-08T07:26:52.939966Z","caller":"traceutil/trace.go:171","msg":"trace[1103327561] put","detail":"{key:by-dev/kv/gid/timestamp; req_size:35; response_revision:6963887; }","duration":"190.520828ms","start":"2026-05-08T07:26:52.749416Z","end":"2026-05-08T07:26:52.939937Z","steps":["trace[1103327561] 'process raft request' (duration: 136.492887ms)","trace[1103327561] 'store kv pair into bolt db' (duration: 46.251312ms)"],"step_count":2}
{"level":"warn","ts":"2026-05-08T07:26:53.004431Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2026-05-08T07:26:52.686464Z","time spent":"309.24078ms","remote":"10.42.0.49:33336","response type":"/etcdserverpb.KV/Put","request count":1,"request size":35,"response count":0,"response size":30,"request content":"key:"by-dev/kv/gid/timestamp" value_size:8 "}
{"level":"warn","ts":"2026-05-08T07:27:14.091021Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"457.335566ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"by-dev/config" range_end:"by-dev/confih" serializable:true ","response":"range_response_count:0 size:7"}
{"level":"info","ts":"2026-05-08T07:27:14.821801Z","caller":"traceutil/trace.go:171","msg":"trace[985501403] range","detail":"{range_begin:by-dev/config; range_end:by-dev/confih; response_count:0; response_revision:6963892; }","duration":"1.411331179s","start":"2026-05-08T07:27:13.090894Z","end":"2026-05-08T07:27:14.502226Z","steps":["trace[985501403] 'get authentication metadata' (duration: 187.916401ms)","trace[985501403] 'range keys from in-memory index tree' (duration: 234.220372ms)","trace[985501403] 'assemble the response' (duration: 21.011765ms)"],"step_count":3}
{"level":"warn","ts":"2026-05-08T07:27:16.237119Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2026-05-08T07:27:14.184517Z","time spent":"417.9445ms","remote":"10.42.0.49:33322","response type":"/etcdserverpb.KV/Range","request count":0,"request size":32,"response count":0,"response size":30,"request content":"key:"by-dev/config" range_end:"by-dev/confih" serializable:true "}
{"level":"warn","ts":"2026-05-08T07:27:17.102696Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"1.777299766s","expected-duration":"100ms","prefix":"","request":"header:<ID:4549653772102273634 > put:<key:"by-dev/kv/gid/timestamp" value_size:8 >","response":"size:7"}
{"level":"warn","ts":"2026-05-08T07:27:17.090356Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2026-05-08T07:27:12.168728Z","time spent":"4.229065258s","remote":"10.42.0.49:33322","response type":"/etcdserverpb.KV/Range","request count":0,"request size":32,"response count":0,"response size":30,"request content":"key:"by-dev/config" range_end:"by-dev/confih" serializable:true "}
{"level":"info","ts":"2026-05-08T07:27:21.120615Z","caller":"traceutil/trace.go:171","msg":"trace[374410748] put","detail":"{key:by-dev/kv/gid/timestamp; req_size:35; response_revision:6963893; }","duration":"7.606323297s","start":"2026-05-08T07:27:13.080240Z","end":"2026-05-08T07:27:20.686563Z","steps":["trace[374410748] 'process raft request' (duration: 1.152158782s)","trace[374410748] 'get key's previous created_revision and leaseID' (duration: 71.831103ms)","trace[374410748] 'marshal mvccpb.KeyValue' (duration: 50.150117ms)","trace[374410748] 'store kv pair into bolt db' (duration: 298.362785ms)"],"step_count":4}
{"level":"warn","ts":"2026-05-08T07:27:25.628259Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2026-05-08T07:27:12.168730Z","time spent":"10.517454209s","remote":"10.42.0.49:33336","response type":"/etcdserverpb.KV/Put","request count":1,"request size":35,"response count":0,"response size":30,"request content":"key:"by-dev/kv/gid/timestamp" value_size:8 "}
{"level":"warn","ts":"2026-05-08T07:27:25.739939Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.719459047s","expected-duration":"100ms","prefix":"read-only range ","request":"key:"by-dev/config" range_end:"by-dev/confih" serializable:true ","response":"range_response_count:0 size:7"}
{"level":"warn","ts":"2026-05-08T07:27:25.749827Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"3.398917573s","expected-duration":"100ms","prefix":"","request":"header:<ID:4549653772102273638 > lease_revoke:id:3f239e00292c0926","response":"size:30"}
{"level":"info","ts":"2026-05-08T07:27:25.750404Z","caller":"traceutil/trace.go:171","msg":"trace[790387215] range","detail":"{range_begin:by-dev/config; range_end:by-dev/confih; response_count:0; response_revision:6963894; }","duration":"2.819970007s","start":"2026-05-08T07:27:22.930411Z","end":"2026-05-08T07:27:25.750381Z","steps":["trace[790387215] 'get authentication metadata' (duration: 144.26728ms)","trace[790387215] 'range keys from in-memory index tree' (duration: 2.57518157s)"],"step_count":2}
{"level":"warn","ts":"2026-05-08T07:27:26.388597Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2026-05-08T07:27:22.786876Z","time spent":"2.963585284s","remote":"10.42.0.49:33322","response type":"/etcdserverpb.KV/Range","request count":0,"request size":32,"response count":0,"response size":30,"request content":"key:"by-dev/config" range_end:"by-dev/confih" serializable:true "}
{"level":"warn","ts":"2026-05-08T07:27:27.143703Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2026-05-08T07:27:25.528659Z","time spent":"1.315120465s","remote":"10.42.0.49:33330","response type":"/etcdserverpb.Lease/LeaseTimeToLive","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
{"level":"warn","ts":"2026-05-08T07:27:27.143715Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2026-05-08T07:27:25.528678Z","time spent":"1.31508598s","remote":"10.42.0.49:33330","response type":"/etcdserverpb.Lease/LeaseTimeToLive","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
{"level":"warn","ts":"2026-05-08T07:27:27.143702Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2026-05-08T07:27:25.528613Z","time spent":"1.31515459s","remote":"10.42.0.49:33330","response type":"/etcdserverpb.Lease/LeaseTimeToLive","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
{"level":"warn","ts":"2026-05-08T07:27:27.143711Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2026-05-08T07:27:25.528611Z","time spent":"1.315171427s","remote":"10.42.0.49:33330","response type":"/etcdserverpb.Lease/LeaseTimeToLive","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
{"level":"warn","ts":"2026-05-08T07:27:27.431801Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"1.446805237s","expected-duration":"100ms","prefix":"","request":"header:<ID:4549653772102273636 > lease_revoke:id:3f239e00292c0933","response":"size:30"}
{"level":"warn","ts":"2026-05-08T07:27:27.742950Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"153.310436ms","expected-duration":"100ms","prefix":"","request":"header:<ID:4549653772102273635 > lease_revoke:id:3f239e00292c0943","response":"size:30"}
{"level":"info","ts":"2026-05-08T07:27:27.831732Z","caller":"traceutil/trace.go:171","msg":"trace[66943133] put","detail":"{key:by-dev/kv/gid/timestamp; req_size:35; response_revision:6963898; }","duration":"2.082199284s","start":"2026-05-08T07:27:25.743012Z","end":"2026-05-08T07:27:27.825212Z","steps":["trace[66943133] 'process raft request' (duration: 2.08210306s)"],"step_count":1}
{"level":"warn","ts":"2026-05-08T07:27:27.837791Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2026-05-08T07:27:25.736675Z","time spent":"2.10108979s","remote":"10.42.0.49:33336","response type":"/etcdserverpb.KV/Put","request count":1,"request size":35,"response count":0,"response size":30,"request content":"key:"by-dev/kv/gid/timestamp" value_size:8 "}
{"level":"info","ts":"2026-05-08T07:29:08.729426Z","caller":"v3compactor/revision.go:86","msg":"starting auto revision compaction","revision":6962901,"revision-compaction-retention":1000}
{"level":"info","ts":"2026-05-08T07:29:08.730780Z","caller":"v3compactor/revision.go:94","msg":"completed auto revision compaction","revision":6962901,"revision-compaction-retention":1000,"took":"1.355189ms"}
{"level":"info","ts":"2026-05-08T07:29:08.730863Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":6962901}
{"level":"info","ts":"2026-05-08T07:29:08.833302Z","caller":"mvcc/kvstore_compaction.go:72","msg":"finished scheduled compaction","compact-revision":6962901,"took":"99.900686ms","hash":1410222138,"current-db-size-bytes":43945984,"current-db-size":"44 MB","current-db-size-in-use-bytes":2191360,"current-db-size-in-use":"2.2 MB"}
{"level":"info","ts":"2026-05-08T07:29:08.833357Z","caller":"mvcc/hash.go:151","msg":"storing new hash","hash":1410222138,"revision":6962901,"compact-revision":6962798}
{"level":"info","ts":"2026-05-08T07:30:35.942724Z","caller":"traceutil/trace.go:171","msg":"trace[1971862311] transaction","detail":"{read_only:false; response_revision:6963992; number_of_response:1; }","duration":"154.329112ms","start":"2026-05-08T07:30:35.788367Z","end":"2026-05-08T07:30:35.942696Z","steps":["trace[1971862311] 'process raft request' (duration: 66.030747ms)","trace[1971862311] 'compare' (duration: 88.141742ms)"],"step_count":2}
{"level":"info","ts":"2026-05-08T07:30:45.859912Z","caller":"traceutil/trace.go:171","msg":"trace[1396843695] put","detail":"{key:by-dev/meta/streamingnode-meta/wal/by-dev-rootcoord-dml_14/consume-checkpoint; req_size:111; response_revision:6964013; }","duration":"133.164405ms","start":"2026-05-08T07:30:45.726726Z","end":"2026-05-08T07:30:45.859890Z","steps":["trace[1396843695] 'process raft request' (duration: 133.091791ms)"],"step_count":1}
{"level":"info","ts":"2026-05-08T07:30:45.859946Z","caller":"traceutil/trace.go:171","msg":"trace[1924173119] put","detail":"{key:by-dev/meta/streamingnode-meta/wal/by-dev-rootcoord-dml_6/consume-checkpoint; req_size:110; response_revision:6964014; }","duration":"133.152005ms","start":"2026-05-08T07:30:45.726769Z","end":"2026-05-08T07:30:45.859921Z","steps":["trace[1924173119] 'process raft request' (duration: 133.073998ms)"],"step_count":1}
{"level":"info","ts":"2026-05-08T07:30:45.859991Z","caller":"traceutil/trace.go:171","msg":"trace[1293882412] put","detail":"{key:by-dev/meta/streamingnode-meta/wal/by-dev-rootcoord-dml_13/consume-checkpoint; req_size:111; response_revision:6964017; }","duration":"132.312082ms","start":"2026-05-08T07:30:45.727661Z","end":"2026-05-08T07:30:45.859973Z","steps":["trace[1293882412] 'process raft request' (duration: 132.238777ms)"],"step_count":1}
{"level":"info","ts":"2026-05-08T07:30:45.860019Z","caller":"traceutil/trace.go:171","msg":"trace[760373649] put","detail":"{key:by-dev/meta/streamingnode-meta/wal/by-dev-rootcoord-dml_10/consume-checkpoint; req_size:111; response_revision:6964015; }","duration":"133.018546ms","start":"2026-05-08T07:30:45.726982Z","end":"2026-05-08T07:30:45.860000Z","steps":["trace[760373649] 'process raft request' (duration: 132.874424ms)"],"step_count":1}
{"level":"info","ts":"2026-05-08T07:30:45.860065Z","caller":"traceutil/trace.go:171","msg":"trace[828358075] put","detail":"{key:by-dev/meta/streamingnode-meta/wal/by-dev-rootcoord-dml_2/consume-checkpoint; req_size:110; response_revision:6964016; }","duration":"132.483403ms","start":"2026-05-08T07:30:45.727570Z","end":"2026-05-08T07:30:45.860053Z","steps":["trace[828358075] 'process raft request' (duration: 132.302373ms)"],"step_count":1}
{"level":"info","ts":"2026-05-08T07:30:45.860102Z","caller":"traceutil/trace.go:171","msg":"trace[2046273564] put","detail":"{key:by-dev/meta/streamingnode-meta/wal/by-dev-rootcoord-dml_0/consume-checkpoint; req_size:110; response_revision:6964018; }","duration":"132.356913ms","start":"2026-05-08T07:30:45.727728Z","end":"2026-05-08T07:30:45.860085Z","steps":["trace[2046273564] 'process raft request' (duration: 132.184907ms)"],"step_count":1}
{"level":"info","ts":"2026-05-08T07:30:45.865685Z","caller":"traceutil/trace.go:171","msg":"trace[1773907907] put","detail":"{key:by-dev/meta/streamingnode-meta/wal/by-dev-rootcoord-dml_7/consume-checkpoint; req_size:110; response_revision:6964012; }","duration":"134.547355ms","start":"2026-05-08T07:30:45.725420Z","end":"2026-05-08T07:30:45.859967Z","steps":["trace[1773907907] 'process raft request' (duration: 72.162366ms)","trace[1773907907] 'get key's previous created_revision and leaseID' (duration: 62.097007ms)"],"step_count":2}
{"level":"info","ts":"2026-05-08T07:30:46.226345Z","caller":"traceutil/trace.go:171","msg":"trace[217509332] transaction","detail":"{read_only:false; response_revision:6964034; number_of_response:1; }","duration":"120.98904ms","start":"2026-05-08T07:30:46.105327Z","end":"2026-05-08T07:30:46.226316Z","steps":["trace[217509332] 'process raft request' (duration: 66.373825ms)","trace[217509332] 'compare' (duration: 54.473496ms)"],"step_count":2}
Beta Was this translation helpful? Give feedback.
All reactions