The loading progress is 0 #22303

ingale726 · 2023-02-20T12:05:09Z

ingale726
Feb 20, 2023

I created an empty collection, created the index, inserted the data, and loaded, waiting an hour, but with zero progress
Create index code:

connections.connect("default", host="172.16.75.171", port="19530")
index_params = {
  "metric_type":"IP",
  "index_type":"HNSW",
  "params":{"M":32,"efConstruction":128},
}
start_time = time.time()
collection = Collection("query_test_256")      # Get an existing collection.
collection.create_index(
  field_name="embeddings",
  index_params=index_params
)

Create collection code:

connections.connect("default", host="172.16.75.171", port="19530")

fields = [
    FieldSchema(name="index", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="count", dtype=DataType.INT64, is_primary=False, auto_id=False),
    FieldSchema(name="string", dtype=DataType.VARCHAR,max_length=128),
    FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=256)
]
schema = CollectionSchema(fields, "hello_milvus is the simplest demo to introduce the APIs")
hello_milvus = Collection("query_test_256", schema)

yhmo · 2023-02-21T02:24:46Z

yhmo
Feb 21, 2023
Collaborator

Might be similar to this issue: #20769
@xiao12mm Could you search "NoSuchKey" in your query node log?

0 replies

ingale726 · 2023-02-21T03:16:29Z

ingale726
Feb 21, 2023
Author

[2023/02/20 01:53:43.867 +00:00] [WARN] [querynode/segment_loader.go:436] ["failed to load binlog"] [filePath=files/insert_log/439564969120897863/439564969120897864/439564969191352647/102/439564969191352650] [error="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191352647/102/439564969191352650)"]
[2023/02/20 01:53:43.867 +00:00] [WARN] [querynode/segment_loader.go:436] ["failed to load binlog"] [filePath=files/insert_log/439564969120897863/439564969120897864/439564969191352647/0/439564969191352652] [error="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191352647/0/439564969191352652)"]
[2023/02/20 01:53:43.867 +00:00] [WARN] [querynode/segment_loader.go:436] ["failed to load binlog"] [filePath=files/insert_log/439564969120897863/439564969120897864/439564969191352647/100/439564969191352648] [error="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191352647/100/439564969191352648)"]
[2023/02/20 01:53:43.875 +00:00] [WARN] [querynode/segment_loader.go:436] ["failed to load binlog"] [filePath=files/insert_log/439564969120897863/439564969120897864/439564969191552794/0/439564969191552816] [error="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191552794/0/439564969191552816)"]
[2023/02/20 01:53:43.901 +00:00] [ERROR] [querynode/segment_loader.go:176] ["load segment failed when load data into memory"] [collectionID=439564969120897863] [segmentType=Sealed] [partitionID=439564969120897864] [segmentID=439564969191352647] [error="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191352647/102/439564969191352650)"] [stack="github.com/milvus-io/milvus/internal/querynode.(*segmentLoader).LoadSegment.func3\n\t/go/src/github.com/milvus-io/milvus/internal/querynode/segment_loader.go:176\ngithub.com/milvus-io/milvus/internal/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/parallel.go:83"]
[2023/02/20 01:53:43.901 +00:00] [ERROR] [funcutil/parallel.go:85] [loadSegmentFunc] [error="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191352647/102/439564969191352650)"] [idx=1] [stack="github.com/milvus-io/milvus/internal/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/parallel.go:85"]
[2023/02/20 01:53:43.923 +00:00] [ERROR] [querynode/segment_loader.go:176] ["load segment failed when load data into memory"] [collectionID=439564969120897863] [segmentType=Sealed] [partitionID=439564969120897864] [segmentID=439564969191552794] [error="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191552794/0/439564969191552816)"] [stack="github.com/milvus-io/milvus/internal/querynode.(*segmentLoader).LoadSegment.func3\n\t/go/src/github.com/milvus-io/milvus/internal/querynode/segment_loader.go:176\ngithub.com/milvus-io/milvus/internal/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/parallel.go:83"]
[2023/02/20 01:53:43.923 +00:00] [ERROR] [funcutil/parallel.go:85] [loadSegmentFunc] [error="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191552794/0/439564969191552816)"] [idx=7] [stack="github.com/milvus-io/milvus/internal/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/parallel.go:85"]
[2023/02/20 01:54:08.792 +00:00] [WARN] [querynode/load_segment_task.go:125] ["failed to load segment"] [collectionID=439564969120897863] [replicaID=439564974428258308] [error="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191352647/102/439564969191352650)"]
[2023/02/20 01:54:08.792 +00:00] [WARN] [querynode/task_scheduler.go:111] ["NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191352647/102/439564969191352650)"]
[2023/02/20 01:54:08.792 +00:00] [WARN] [querynode/impl.go:521] ["NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191352647/102/439564969191352650)"]
[2023/02/20 01:54:08.792 +00:00] [WARN] [querynode/shard_cluster.go:657] ["follower load segment failed"] [collectionID=439564969120897863] [channel=by-dev-rootcoord-dml_21_439564969120897863v1] [replicaID=439564974428258308] [dstNodeID=110] [segmentIDs="[439564969230582234,439564969191352647,439564969225178246,439564969192153172,439564969201560431,439564969213769677,439564969193754443,439564969191552794,439564969215170710]"] [reason="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191352647/102/439564969191352650)"]
[2023/02/20 01:54:08.792 +00:00] [WARN] [querynode/impl_utils.go:50] ["shard cluster failed to load segments"] [traceID=4ca0e33fe4f8dfd6] [shard=by-dev-rootcoord-dml_21_439564969120897863v1] [segmentIDs="[439564969230582234,439564969191352647,439564969225178246,439564969192153172,439564969201560431,439564969213769677,439564969193754443,439564969191552794,439564969215170710]"] [error="follower 110 failed to load segment, reason NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191352647/102/439564969191352650)"]
[2023/02/20 01:56:21.671 +00:00] [WARN] [querynode/segment_loader.go:436] ["failed to load binlog"] [filePath=files/insert_log/439564969120897863/439564969120897864/439564969191552824/102/439564969191552844] [error="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191552824/102/439564969191552844)"]
[2023/02/20 01:56:21.707 +00:00] [ERROR] [querynode/segment_loader.go:176] ["load segment failed when load data into memory"] [collectionID=439564969120897863] [segmentType=Sealed] [partitionID=439564969120897864] [segmentID=439564969191552824] [error="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191552824/102/439564969191552844)"] [stack="github.com/milvus-io/milvus/internal/querynode.(*segmentLoader).LoadSegment.func3\n\t/go/src/github.com/milvus-io/milvus/internal/querynode/segment_loader.go:176\ngithub.com/milvus-io/milvus/internal/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/parallel.go:83"]
[2023/02/20 01:56:21.707 +00:00] [ERROR] [funcutil/parallel.go:85] [loadSegmentFunc] [error="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191552824/102/439564969191552844)"] [idx=4] [stack="github.com/milvus-io/milvus/internal/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/parallel.go:85"]
[2023/02/20 01:56:44.965 +00:00] [WARN] [querynode/load_segment_task.go:125] ["failed to load segment"] [collectionID=439564969120897863] [replicaID=439564974428258308] [error="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191552824/102/439564969191552844)"]
[2023/02/20 01:56:44.965 +00:00] [WARN] [querynode/task_scheduler.go:111] ["NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191552824/102/439564969191552844)"]
[2023/02/20 01:56:44.965 +00:00] [WARN] [querynode/impl.go:521] ["NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191552824/102/439564969191552844)"]
[2023/02/20 01:56:44.965 +00:00] [WARN] [querynode/shard_cluster.go:657] ["follower load segment failed"] [collectionID=439564969120897863] [channel=by-dev-rootcoord-dml_21_439564969120897863v1] [replicaID=439564974428258308] [dstNodeID=110] [segmentIDs="[439564969236386627,439564969232183488,439564969229581526,439564969237387421,439564969191552824,439564969195555860,439564969198558147,439564969233384341,439564969232583836,439564969233784748,439564969219774108,439564969191752924,439564969208165301,439564969202561136]"] [reason="NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191552824/102/439564969191552844)"]
[2023/02/20 01:56:44.965 +00:00] [WARN] [querynode/impl_utils.go:50] ["shard cluster failed to load segments"] [traceID=7a4093f7f4c33665] [shard=by-dev-rootcoord-dml_21_439564969120897863v1] [segmentIDs="[439564969236386627,439564969232183488,439564969229581526,439564969237387421,439564969191552824,439564969195555860,439564969198558147,439564969233384341,439564969232583836,439564969233784748,439564969219774108,439564969191752924,439564969208165301,439564969202561136]"] [error="follower 110 failed to load segment, reason NoSuchKey(key=files/insert_log/439564969120897863/439564969120897864/439564969191552824/102/439564969191552844)"]
[2023/02/20 02:01:31.333 +00:00] [ERROR] [querynode/segment_loader.go:176] ["load segment failed when load data into memory"] [collectionID=439564969120897863] [segmentType=Sealed] [partitionID=439564969120897864] [segmentID=439564969191152454] [error="NoSuchKey(key=files/index_files/439564969191152486/1/439564969120897864/439564969191152454/indexParams)"] [stack="github.com/milvus-io/milvus/internal/querynode.(*segmentLoader).LoadSegment.func3\n\t/go/src/github.com/milvus-io/milvus/internal/querynode/segment_loader.go:176\ngithub.com/milvus-io/milvus/internal/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/parallel.go:83"]
[2023/02/20 02:01:31.333 +00:00] [ERROR] [funcutil/parallel.go:85] [loadSegmentFunc] [error="NoSuchKey(key=files/index_files/439564969191152486/1/439564969120897864/439564969191152454/indexParams)"] [idx=7] [stack="github.com/milvus-io/milvus/internal/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/parallel.go:85"]
[2023/02/20 02:01:52.992 +00:00] [WARN] [querynode/load_segment_task.go:125] ["failed to load segment"] [collectionID=439564969120897863] [replicaID=439564974428258308] [error="NoSuchKey(key=files/index_files/439564969191152486/1/439564969120897864/439564969191152454/indexParams)"]
[2023/02/20 02:01:52.992 +00:00] [WARN] [querynode/task_scheduler.go:111] ["NoSuchKey(key=files/index_files/439564969191152486/1/439564969120897864/439564969191152454/indexParams)"]
[2023/02/20 02:01:52.992 +00:00] [WARN] [querynode/impl.go:521] ["NoSuchKey(key=files/index_files/439564969191152486/1/439564969120897864/439564969191152454/indexParams)"]
[2023/02/20 02:01:52.992 +00:00] [WARN] [querynode/shard_cluster.go:657] ["follower load segment failed"] [collectionID=439564969120897863] [channel=by-dev-rootcoord-dml_20_439564969120897863v0] [replicaID=439564974428258308] [dstNodeID=110] [segmentIDs="[439564969219974314,439564969200359421,439564969210967463,439564969196756737,439564969205363189,439564969218973575,439564969213569409,439564969191152454]"] [reason="NoSuchKey(key=files/index_files/439564969191152486/1/439564969120897864/439564969191152454/indexParams)"]
[2023/02/20 02:01:52.992 +00:00] [WARN] [querynode/impl_utils.go:50] ["shard cluster failed to load segments"] [traceID=14ab5c4d1f42beb7] [shard=by-dev-rootcoord-dml_20_439564969120897863v0] [segmentIDs="[439564969219974314,439564969200359421,439564969210967463,439564969196756737,439564969205363189,439564969218973575,439564969213569409,439564969191152454]"] [error="follower 110 failed to load segment, reason NoSuchKey(key=files/index_files/439564969191152486/1/439564969120897864/439564969191152454/indexParams)"]
[2023/02/20 02:04:27.911 +00:00] [ERROR] [querynode/segment_loader.go:176] ["load segment failed when load data into memory"] [collectionID=439564969120897863] [segmentType=Sealed] [partitionID=439564969120897864] [segmentID=439564969191352687] [error="NoSuchKey(key=files/index_files/439564969191352722/1/439564969120897864/439564969191352687/indexParams)"] [stack="github.com/milvus-io/milvus/internal/querynode.(*segmentLoader).LoadSegment.func3\n\t/go/src/github.com/milvus-io/milvus/internal/querynode/segment_loader.go:176\ngithub.com/milvus-io/milvus/internal/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/parallel.go:83"]
[2023/02/20 02:04:27.911 +00:00] [ERROR] [funcutil/parallel.go:85] [loadSegmentFunc] [error="NoSuchKey(key=files/index_files/439564969191352722/1/439564969120897864/439564969191352687/indexParams)"] [idx=1] [stack="github.com/milvus-io/milvus/internal/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/parallel.go:85"]
[2023/02/20 02:04:48.907 +00:00] [WARN] [querynode/load_segment_task.go:125] ["failed to load segment"] [collectionID=439564969120897863] [replicaID=439564974428258308] [error="NoSuchKey(key=files/index_files/439564969191352722/1/439564969120897864/439564969191352687/indexParams)"]
[2023/02/20 02:04:48.907 +00:00] [WARN] [querynode/task_scheduler.go:111] ["NoSuchKey(key=files/index_files/439564969191352722/1/439564969120897864/439564969191352687/indexParams)"]
[2023/02/20 02:04:48.907 +00:00] [WARN] [querynode/impl.go:521] ["NoSuchKey(key=files/index_files/439564969191352722/1/439564969120897864/439564969191352687/indexParams)"]
[2023/02/20 02:04:48.907 +00:00] [WARN] [querynode/shard_cluster.go:657] ["follower load segment failed"] [collectionID=439564969120897863] [channel=by-dev-rootcoord-dml_21_439564969120897863v1] [replicaID=439564974428258308] [dstNodeID=110] [segmentIDs="[439564969194154696,439564969191352687,439564969215370838,439564969217772640,439564969197157060]"] [reason="NoSuchKey(key=files/index_files/439564969191352722/1/439564969120897864/439564969191352687/indexParams)"]
[2023/02/20 02:04:48.908 +00:00] [WARN] [querynode/impl_utils.go:50] ["shard cluster failed to load segments"] [traceID=25bb690e66035e61] [shard=by-dev-rootcoord-dml_21_439564969120897863v1] [segmentIDs="[439564969194154696,439564969191352687,439564969215370838,439564969217772640,439564969197157060]"] [error="follower 110 failed to load segment, reason NoSuchKey(key=files/index_files/439564969191352722/1/439564969120897864/439564969191352687/indexParams)"]

3 replies

ingale726 Feb 21, 2023
Author

When I finished inserting data, I could not load the data, the progress was always 0, the CPU and memory resources were idle, at the same time, some large data could be loaded normally, such as the inserted data of 100 million, but the newly inserted data of 10 million could not be loaded

ingale726 Feb 21, 2023
Author

Some of my own attempts:

Create collection----create index-----load----release-------insert data,----load Successful loading
Create collection----create index----insert data,----load 0%..........waiting...................

In the first case, when the data volume increased to 8 million, 0% of the data also appeared

ingale726 Feb 21, 2023
Author

Some of my special operations with clustering

I've taken multiple nodes and turned them into single nodes

congqixia · 2023-02-23T02:23:51Z

congqixia
Feb 23, 2023
Maintainer

@xiao12mm quick question. Do your have multiple milvus cluster sharing same object storage service? (s3/minio)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The loading progress is 0 #22303

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 3 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

The loading progress is 0 #22303

Uh oh!

Uh oh!

ingale726 Feb 20, 2023

Replies: 3 comments · 3 replies

Uh oh!

yhmo Feb 21, 2023 Collaborator

Uh oh!

ingale726 Feb 21, 2023 Author

Uh oh!

ingale726 Feb 21, 2023 Author

Uh oh!

ingale726 Feb 21, 2023 Author

Uh oh!

ingale726 Feb 21, 2023 Author

Uh oh!

congqixia Feb 23, 2023 Maintainer

ingale726
Feb 20, 2023

Replies: 3 comments 3 replies

yhmo
Feb 21, 2023
Collaborator

ingale726
Feb 21, 2023
Author

ingale726 Feb 21, 2023
Author

ingale726 Feb 21, 2023
Author

ingale726 Feb 21, 2023
Author

congqixia
Feb 23, 2023
Maintainer