Why does performance degradation and collection loading & release failure occur due to entity upsert?

[pipi-olo]

Hi.
I am running Milvus v2.4.23 on kubernetes using helm chart.

When using entity upsert, Empty segments (L0) are created in the collection.

As a resut, Timeout occur during Milvus collection loading and release.
As the same time, The performance of Milvus search degrades for all collections including othres.

DataCoord Logs

[WARN] [datacoord/index_service.go:87] ["segment is level zero, skip create indexes"] [segmentID=456429702303976373]
[WARN] [datacoord/index_service.go:87] ["segment is level zero, skip create indexes"] [segmentID=456429702303976353]
[WARN] [datacoord/index_service.go:87] ["segment is level zero, skip create indexes"] [segmentID=456429702303976283]
[WARN] [datacoord/index_service.go:87] ["segment is level zero, skip create indexes"] [segmentID=456429702303976303]
[WARN] [datacoord/index_service.go:87] ["segment is level zero, skip create indexes"] [segmentID=456429702303976333]
[WARN] [datacoord/index_service.go:87] ["segment is level zero, skip create indexes"] [segmentID=456429702303976383]
[WARN] [datacoord/index_service.go:87] ["segment is level zero, skip create indexes"] [segmentID=456429702303976363]
[WARN] [datacoord/index_service.go:87] ["segment is level zero, skip create indexes"] [segmentID=456429702303976293]
[WARN] [datacoord/index_service.go:87] ["segment is level zero, skip create indexes"] [segmentID=456429702303976263]
[WARN] [datacoord/index_service.go:87] ["segment is level zero, skip create indexes"] [segmentID=456429702303976323]
[WARN] [datacoord/index_service.go:87] ["segment is level zero, skip create indexes"] [segmentID=456429702303976343]
[WARN] [datacoord/index_service.go:87] ["segment is level zero, skip create indexes"] [segmentID=456429702303976273]
[WARN] [datacoord/index_service.go:87] ["segment is level zero, skip create indexes"] [segmentID=456429702303976313]
[INFO] [datacoord/compaction_trigger.go:657] [generatePlans] [collectionID=0] [plan_num=0]
[INFO] [datacoord/compaction_trigger.go:657] [generatePlans] [collectionID=0] [plan_num=0]
[INFO] [datacoord/compaction_trigger.go:657] [generatePlans] [collectionID=0] [plan_num=0]
[INFO] [datacoord/compaction_trigger.go:657] [generatePlans] [collectionID=0] [plan_num=0]
[INFO] [datacoord/compaction_trigger.go:657] [generatePlans] [collectionID=0] [plan_num=0]
[INFO] [datacoord/compaction_trigger.go:657] [generatePlans] [collectionID=0] [plan_num=0]

QueryCoord Logs

[WARN] [grpcclient/client.go:404] ["call received grpc error"] [clientRole=querynode-484] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"]
[WARN] [retry/retry.go:130] ["retry func failed"] [retried=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"]
[WARN] [grpcclient/client.go:556] ["ClientBase Call grpc call get error"] [role=querynode-484] [address=10.64.63.211:21123] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:319 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).SyncDistribution\n/workspace/source/internal/querycoordv2/session/cluster.go:239 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).SyncDistribution.func1\n/workspace/source/internal/querycoordv2/session/cluster.go:272 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n/workspace/source/internal/querycoordv2/session/cluster.go:236 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).SyncDistribution\n/workspace/source/internal/querycoordv2/observers/target_observer.go:494 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).sync\n/workspace/source/internal/querycoordv2/observers/target_observer.go:452 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).shouldUpdateCurrentTarget: rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).SyncDistribution\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:319\n  | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).SyncDistribution.func1\n  | \t/workspace/source/internal/querycoordv2/session/cluster.go:239\n  | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n  | \t/workspace/source/internal/querycoordv2/session/cluster.go:272\n  | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).SyncDistribution\n  | \t/workspace/source/internal/querycoordv2/session/cluster.go:236\n  | github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).sync\n  | \t/workspace/source/internal/querycoordv2/observers/target_observer.go:494\n  | github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).shouldUpdateCurrentTarget\n  | \t/workspace/source/internal/querycoordv2/observers/target_observer.go:452\n  | github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).check\n  | \t/workspace/source/internal/querycoordv2/observers/target_observer.go:261\n  | github.com/milvus-io/milvus/internal/querycoordv2/observers.(*taskDispatcher[...]).schedule.func1.1\n  | \t/workspace/source/internal/querycoordv2/observers/task_dispatcher.go:101\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:319 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).SyncDistribution\n  | /workspace/source/internal/querycoordv2/session/cluster.go:239 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).SyncDistribution.func1\n  | /workspace/source/internal/querycoordv2/session/cluster.go:272 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n  | /workspace/source/internal/querycoordv2/session/cluster.go:236 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).SyncDistribution\n  | /workspace/source/internal/querycoordv2/observers/target_observer.go:494 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).sync\n  | /workspace/source/internal/querycoordv2/observers/target_observer.go:452 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).shouldUpdateCurrentTarget\nWraps: (3) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[WARN] [observers/target_observer.go:496] ["failed to sync distribution"] [leaderID=484] [collectionID=456431470361743692] [channel=by-dev-rootcoord-dml_15_456431470361743692v0] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:319 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).SyncDistribution\n/workspace/source/internal/querycoordv2/session/cluster.go:239 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).SyncDistribution.func1\n/workspace/source/internal/querycoordv2/session/cluster.go:272 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n/workspace/source/internal/querycoordv2/session/cluster.go:236 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).SyncDistribution\n/workspace/source/internal/querycoordv2/observers/target_observer.go:494 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).sync\n/workspace/source/internal/querycoordv2/observers/target_observer.go:452 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).shouldUpdateCurrentTarget: rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:555\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n  | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).SyncDistribution\n  | \t/workspace/source/internal/distributed/querynode/client/client.go:319\n  | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).SyncDistribution.func1\n  | \t/workspace/source/internal/querycoordv2/session/cluster.go:239\n  | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n  | \t/workspace/source/internal/querycoordv2/session/cluster.go:272\n  | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).SyncDistribution\n  | \t/workspace/source/internal/querycoordv2/session/cluster.go:236\n  | github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).sync\n  | \t/workspace/source/internal/querycoordv2/observers/target_observer.go:494\n  | github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).shouldUpdateCurrentTarget\n  | \t/workspace/source/internal/querycoordv2/observers/target_observer.go:452\n  | github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).check\n  | \t/workspace/source/internal/querycoordv2/observers/target_observer.go:261\n  | github.com/milvus-io/milvus/internal/querycoordv2/observers.(*taskDispatcher[...]).schedule.func1.1\n  | \t/workspace/source/internal/querycoordv2/observers/task_dispatcher.go:101\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:82\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n  | /workspace/source/internal/distributed/querynode/client/client.go:319 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).SyncDistribution\n  | /workspace/source/internal/querycoordv2/session/cluster.go:239 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).SyncDistribution.func1\n  | /workspace/source/internal/querycoordv2/session/cluster.go:272 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n  | /workspace/source/internal/querycoordv2/session/cluster.go:236 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).SyncDistribution\n  | /workspace/source/internal/querycoordv2/observers/target_observer.go:494 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).sync\n  | /workspace/source/internal/querycoordv2/observers/target_observer.go:452 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).shouldUpdateCurrentTarget\nWraps: (3) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[INFO] [balance/score_based_balancer.go:78] ["node workload status"] [collectionID=456431470359385826] [nodes="[\"{NodeID: 470, AssignedScore: 474423.000000, CurrentScore: 452162.000000, Priority: -22261}\",\"{NodeID: 492, AssignedScore: 474423.000000, CurrentScore: 579760.000000, Priority: 105337}\",\"{NodeID: 483, AssignedScore: 474423.000000, CurrentScore: 482004.300000, Priority: 7582}\",\"{NodeID: 472, AssignedScore: 474423.000000, CurrentScore: 431208.000000, Priority: -43215}\"]"]

Birdwatcher

Milvus(by-dev) > show replica
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x117ad56]
 
goroutine 1 [running]:
github.com/milvus-io/birdwatcher/states/etcd/show.(*Replicas).PrintAs(0xc000153340, 0x153340?)
	/home/runner/work/birdwatcher/birdwatcher/states/etcd/show/replica.go:63 +0x196
github.com/milvus-io/birdwatcher/common.parseMethod.func1(0xc000ff9200?, {0x323cf00, 0x0, 0x0})
	/home/runner/work/birdwatcher/birdwatcher/common/state.go:289 +0x699
github.com/spf13/cobra.(*Command).execute(0xc001708280, {0x323cf00, 0x0, 0x0})
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876 +0x691
github.com/spf13/cobra.(*Command).ExecuteC(0xc0005b4a00)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990 +0x38d
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918
github.com/milvus-io/birdwatcher/common.(*CmdState).Process(0xc000ea82c0, {0xc0008fe644?, 0x1b72760?})
	/home/runner/work/birdwatcher/birdwatcher/common/state.go:119 +0x20a
github.com/milvus-io/birdwatcher/bapps.(*PromptApp).promptExecute(0xc000995000, {0xc0008fe644?, 0x4?})
	/home/runner/work/birdwatcher/birdwatcher/bapps/go_prompt.go:137 +0x3bf
github.com/c-bata/go-prompt.(*Prompt).Run(0xc000ad5950)
	/home/runner/go/pkg/mod/github.com/c-bata/go-prompt@v0.2.6/prompt.go:84 +0x73f
github.com/milvus-io/birdwatcher/bapps.(*PromptApp).Run(0xc000153120?, {0x214bf68?, 0xc000e44300?})
	/home/runner/work/birdwatcher/birdwatcher/bapps/go_prompt.go:76 +0x45
main.main()
	/home/runner/work/birdwatcher/birdwatcher/cmd/birdwatcher/main.go:66 +0x282

Milvus(by-dev) > show segment
===============================CollectionID: 456431470361743692===========================
SegmentID: 456429702303976263 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976273 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976283 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976293 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976303 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976313 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976323 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976333 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976343 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976353 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976363 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976373 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976383 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976393 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976403 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976405 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976423 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976433 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976443 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456429702303976453 PartitionID: 456431470361743693 State: Flushed, Level: L0, Row Count:0,  PartitionStatsVersion:0, IsSorted: false 
SegmentID: 456431470361743851 PartitionID: 456431470361743693 State: Growing, Level: L1, Row Count:124,  PartitionStatsVersion:0, IsSorted: false
 
--- Growing: 1, Sealed: 0, Flushed: 29, Dropped: 0
--- Small Segments: 0, row count: 0	 Other Segments: 29, row count: 1725397
--- Total Segments: 30, row count: 1725521

cf. I have removed the logs for the other collections.

Q1. Why does empty segments (L0) appear when performing an entity upsert?
Q2. When are empty segments (L0) removed?
Q3. How can i solve this issue? I need a help..

[yhmo]

upsert = delete + insert
Each time you upsert an entity, milvus will trigger a delete action to delete by the entity's id and an insert action to insert a new entity.
All the deleted IDs are records in L0 segment.
With more upsert/delete actions, you will see more L0 segments generated.

Datanode triggers compaction actions to erase entities by the IDs from L0 segment. The empty L0 segments mean their IDs have been processed. Empty L0 segments will be remove by GC(garbage collection) of milvus. GC action is triggered with an interval of 1 hour.

Intensive/continually upsert action will downgrade search performance because it is a heavier operation than normal insert.

The log doesn't show the root cause why load() timeout.
Use this script to fetch the full logs: https://github.com/milvus-io/milvus/tree/master/deployments/export-log

[pipi-olo]

Thanks.

Does upsert not check whether the data exists? Does the delete operation always occur?

Do empty L0 segments occur due to the mandatory delete operations?

[yhmo]

upsert doesn't check data existence, because with more and more data inserted, the time cost to check existence becomes longer and longer, and eventually unbearable.
Each upsert action always puts an ID into L0 segment, the L0 segment is processed by compaction asynchronously.

The birdwatcher doesn't count the real row count of L0 segment, so you see the "Row Count" of L0 is 0. In fact, each L0 segment contains lots of IDs for the deleted entities.

[pipi-olo]

Is there a way to accurately check the actual row count of the L0 segment? I suspect that there is no data in that segment.

When performing a delete operation on non-existent data, the returned delete count value is 1. Does this cause empty L0 segments to be created?

[yhmo]

show segment --format=table --collection=collection_id --detail

You will see details of all segments, including the L0 segment.

[yhmo]

Even delete non-existent data, the "entries" of L0 segment is not zero.