[Bug]: error="IO error: While open a file for random read: /data/milvus/rdb_data/030164.sst: Too many open files #25798
Labels
kind/bug
Issues or changes related a bug
stale
indicates no udpates for 30 days
triage/needs-information
Indicates an issue needs more information in order to work on it.
Is there an existing issue for this?
Environment
Current Behavior
Milvus is not working after running for around 2 days, the errors are
[2023/07/20 14:06:33.401 +08:00] [ERROR] [rootcoord/dml_channels.go:235] ["Broadcast failed"] [error="IO error: While open a file for random read: /data/milvus/rdb_data/030164.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_5] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusrpm-2.1/milvus/internal/rootcoord/dml_channels.go:235\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusrpm-2.1/milvus/internal/rootcoord/timeticksync.go:351\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusrpm-2.1/milvus/internal/rootcoord/timeticksync.go:312"]
[2023/07/20 14:06:33.401 +08:00] [WARN] [rootcoord/timeticksync.go:313] ["SendTimeTickToChannel fail"] [error="IO error: While open a file for random read: /data/milvus/rdb_data/030164.sst: Too many open files"]
Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
2023/07/20 14:06:33.401 +08:00] [ERROR] [rootcoord/dml_channels.go:235] ["Broadcast failed"] [error="IO error: While open a file for random read: /data/milvus/rdb_data/030164.sst: Too many open files"] [chanName=by-dev-rootcoord-dml_5] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*dmlChannels).broadcast\n\t/root/milvusrpm-2.1/milvus/internal/rootcoord/dml_channels.go:235\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).sendTimeTickToChannel\n\t/root/milvusrpm-2.1/milvus/internal/rootcoord/timeticksync.go:351\ngithub.com/milvus-io/milvus/internal/rootcoord.(*timetickSync).startWatch.func1\n\t/root/milvusrpm-2.1/milvus/internal/rootcoord/timeticksync.go:312"]
[2023/07/20 14:06:33.401 +08:00] [WARN] [rootcoord/timeticksync.go:313] ["SendTimeTickToChannel fail"] [error="IO error: While open a file for random read: /data/milvus/rdb_data/030164.sst: Too many open files"]
Anything else?
when this error happens, Milvus is just not responding
The text was updated successfully, but these errors were encountered: