Milvus hangs during multi-thread concurrent search

[stqf]

使用docker基于milvusdb/milvus:0.10.0-cpu-d061620-5f3c00镜像部署一个milvus服务,创建pics和faces两个集合,索引如下:

同时插入5w数据, 使用Jmeter压测[50线程并发],情况如下:
一. 单独压测 http://192.168.0.100:19121/collections/pics/vectors 接口时, 请求数达到 270W左右,milvus出现假死现象
二. 同时压测 http://192.168.0.100:19121/collections/pics/vectors 和 http://192.168.0.100:19121/collections/faces/vectors 接口时, 请求量达到 28w左右出现假死现象
备注: 出现假死时, 查看milvus服务未见error级别日志;且情况一单独压测时,OPS约为1200 情况二同时压测时, OPS仅为320 性能下降至三分之一左右; 测试数据为-1到1之间的Float类型

[JinHai-CN]

We will try to reproduce the issue. Please provide following information:

1. Hardware configuration.
2. Any changes on Milvus configuration.
3. Can this issue be reproduced each time?
4. What is the insertion batch size? 50K?

[stqf]

1. Hardware configuration:

1.1 CPU:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                6
On-line CPU(s) list:   0-5
Thread(s) per core:    1
Core(s) per socket:    6
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 158
Model name:            Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz
Stepping:              10
CPU MHz:               800.172
BogoMIPS:              6000.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              9216K
NUMA node0 CPU(s):     0-5

1.2 Mem:

Handle 0x003C, DMI type 16, 23 bytes
Physical Memory Array
        Location: System Board Or Motherboard
        Use: System Memory
        Error Correction Type: None
        Maximum Capacity: 32 GB
        Error Information Handle: Not Provided
        Number Of Devices: 4

Handle 0x003D, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x003C
        Error Information Handle: Not Provided
        Total Width: 64 bits
        Data Width: 64 bits
        Size: 16384 MB
        Form Factor: DIMM
        Set: None
        Locator: ChannelA-DIMM1
        Bank Locator: BANK 0
        Type: DDR4
        Type Detail: Synchronous
        Speed: 2400 MHz
        Manufacturer: Kingston
        Serial Number: 8E12A77C
        Asset Tag: 9876543210
        Part Number: 9905701-060.A00G
        Rank: 2
        Configured Clock Speed: 2400 MHz
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: 1.2 V

Handle 0x003E, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x003C
        Error Information Handle: Not Provided
        Total Width: 64 bits
        Data Width: 64 bits
        Size: 16384 MB
        Form Factor: DIMM
        Set: None
        Locator: ChannelA-DIMM2
        Bank Locator: BANK 1
        Type: DDR4
        Type Detail: Synchronous
        Speed: 2400 MHz
        Manufacturer: Kingston
        Serial Number: 93127156
        Asset Tag: 9876543210
        Part Number: 9905701-060.A00G
        Rank: 2
        Configured Clock Speed: 2400 MHz
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: 1.2 V

Handle 0x003F, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x003C
        Error Information Handle: Not Provided
        Total Width: 64 bits
        Data Width: 64 bits
        Size: 16384 MB
        Form Factor: DIMM
        Set: None
        Locator: ChannelB-DIMM1
        Bank Locator: BANK 2
        Type: DDR4
        Type Detail: Synchronous
        Speed: 2400 MHz
        Manufacturer: Kingston
        Serial Number: 93291A3D
        Asset Tag: 9876543210
        Part Number: 9905701-003.A00G
        Rank: 2
        Configured Clock Speed: 2400 MHz
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: 1.2 V

Handle 0x0040, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x003C
        Error Information Handle: Not Provided
        Total Width: 64 bits
        Data Width: 64 bits
        Size: 16384 MB
        Form Factor: DIMM
        Set: None
        Locator: ChannelB-DIMM2
        Bank Locator: BANK 3
        Type: DDR4
        Type Detail: Synchronous
        Speed: 2400 MHz
        Manufacturer: Kingston
        Serial Number: 8D2887BF
        Asset Tag: 9876543210
        Part Number: 9905701-003.A00G
        Rank: 2
        Configured Clock Speed: 2400 MHz
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: 1.2 V

2. Any changes on Milvus configuration:
   cache:
   cache_size: 32GB
   insert_buffer_size: 2GB
   preload_collection:

3. Can this issue be reproduced each time
   Yes, the number of requests is about 300w
   Case 2 The number of requests appears around 30w

4. What is the insertion batch size? 50K?
   A total of 5w, each time insert 5K, completed in 10 times, part of the source code is as follows:

IntStream.range(0, 10).parallel().forEach(indexItem -> {
	List<List<Float>> vectors = IntStream.range(0, 5_000).mapToObj(index -> feature()).collect(Collectors.toList());
	List<Long> vectorIds = mlvTemplate.insertFloatVector(table, vectors, null);
	log.info("InsertIndexItem: {} ... {}", indexItem, vectorIds.size());
})

public List<Long> insertFloatVector(String table, List<List<Float>> vectors, String part) {
    InsertParam.Builder builder = new InsertParam.Builder(table).withFloatVectors(vectors);

    boolean isBlank = Objects.isNull(part) || part.isEmpty();

    InsertParam ip = isBlank ? builder.build() : builder.withPartitionTag(part).build();

    return insertImpl(table, ip);
}

private List<Long> insertImpl(String table, InsertParam ip) {
    double start = System.currentTimeMillis();
    InsertResponse resp = client.insert(ip);
    double finish = System.currentTimeMillis();
    log.info("insert time: {}s", (finish - start) / 1000.0);
    if (!resp.ok()) {
        log.warn("insert failure:[{}] ... {}", table, resp.getResponse().getMessage());
        return new ArrayList<>();
    }

    return resp.getVectorIds();
}

[tinkerlin]

The number of requests is approximately 300w, each request inserts 5000 vectors, and the total number of inserted vectors is 3000000 * 5000?
when do you call create_index()? before insert vectors or some other time?
when server is not responding to requests (dump~) the actually disk usage is ?
@stqf

[stqf]

首先创建pics集合并创建索引,再想pics集合中插入5W条向量数据(每次5K 分10次), 然后通过rest接口做向量搜索请求, 当请求300W次左右 milvus服务出现假死, 不再有响应; 实际磁盘如下:

备注: collection中的总向量数只有5w 300W是向量搜索请求次数
@tinkerlin

[tinkerlin]

What is the query batch size? topk? or completely random?
Use gpu or not ?
@stqf

[stqf]

请求体JSON如下:

{
	"search": {
		"topk": 2,
		"params": {
			"nprobe": 1024
		},
		"vectors": [
			[-0.5802692174911499, -0.5646722912788391, 0.4531806409358978, 0.08557803928852081, -0.4402010440826416, 0.2709364891052246, -0.7719730138778687, -0.14499036967754364, -0.09719487279653549, -0.0036130195949226618, -0.5226969718933105, 0.8400065898895264, -0.8583701848983765, -0.3879886865615845, -0.2778930962085724, -0.31258130073547363, 0.8414964079856873, -0.7085063457489014, -0.5712656378746033, 0.7783883810043335, 0.47111982107162476, -0.6562513709068298, 0.3767057955265045, -0.2049490064382553, 0.7266510128974915, 0.8209235668182373, -0.6246964335441589, 0.31937703490257263, 0.10063290596008301, -0.3633731007575989, 0.43725624680519104, -0.462925523519516, -0.23716890811920166, 0.9463497996330261, -0.7100421786308289, 0.07691611349582672, -0.16159147024154663, 0.11053886264562607, -0.24092192947864532, -0.42069897055625916, -0.7422774434089661, -0.7086830735206604, 0.8417866230010986, 0.6110500693321228, -0.5910291075706482, -0.593100368976593, 0.9151867628097534, -0.1335265189409256, 0.9049851298332214, 0.7089880704879761, -0.517574667930603, -0.430858850479126, 0.5370334386825562, 0.8930017352104187, 0.11423490941524506, -0.7696517109870911, 0.5811673998832703, -0.6848098635673523, -0.5418833494186401, 0.5268110036849976, 0.32519644498825073, 0.5805152654647827, 0.448812335729599, -0.718492865562439, 0.9573961496353149, -0.3536747694015503, 0.12579141557216644, 0.10316690802574158, -0.633239209651947, -0.4395522177219391, -0.9640424847602844, 0.01888122409582138, -0.13872437179088593, -0.8083512783050537, -0.9238179326057434, 0.5568611025810242, -0.5416377186775208, -0.18357157707214355, -0.1481717973947525, -0.04642442241311073, -0.2655268609523773, 0.8922093510627747, -0.932272732257843, 0.5560694932937622, -0.6714724898338318, -0.5013505220413208, 0.5557528734207153, 0.8590866327285767, -0.4372205138206482, 0.46313080191612244, -0.2561188042163849, -0.06376451998949051, -0.7623206377029419, -0.5736212134361267, -0.4860469102859497, 0.9659932851791382, -0.8474448919296265, -0.2200624942779541, -0.7368857264518738, 0.15599150955677032, -0.9177006483078003, -0.4972485601902008, -0.5697733163833618, 0.483780175447464, -0.8086426854133606, -0.2481212168931961, -0.6788434386253357, 0.27200886607170105, 0.014394654892385006, -0.8878976106643677, 0.7362627387046814, -0.1558084934949875, -0.36653512716293335, -0.434876412153244, -0.42328986525535583, 0.6872494220733643, -0.16378965973854065, 0.7203178405761719, -0.3370395004749298, -0.24447406828403473, 0.8285366296768188, 0.03888529911637306, 0.6054043769836426, -0.004965880885720253, -0.6658512949943542, 0.8393322229385376, 0.18178555369377136, -0.11370836198329926, 0.6652790307998657, 0.09641111642122269, -0.25429072976112366, -0.8405077457427979, -0.28098034858703613, 0.3934052288532257, 0.1900872439146042, 0.28701967000961304, 0.5758180022239685, -0.43187862634658813, -0.3976157009601593, 0.5550856590270996, -0.17623749375343323, -0.9683277606964111, 0.5418764352798462, 0.04030992463231087, -0.9056122303009033, -0.473236083984375, 0.396583616733551, 0.3537599742412567, 0.5857648849487305, -0.4573036730289459, -0.8031067252159119, -0.6664934754371643, 0.40979889035224915, 0.06929250806570053, -0.9101993441581726, 0.6875059604644775, 0.5049436092376709, 0.8541622757911682, -0.24938060343265533, -0.4256136119365692, 0.8294170498847961, 0.8481770753860474, 0.8935773372650146, -0.2381872832775116, -0.2407456487417221, -0.3460588753223419, 0.10843987762928009, 0.2195085883140564, -0.03458099067211151, 0.005283867008984089, -0.5036336183547974, 0.8358609676361084, -0.22829149663448334, 0.21722376346588135, 0.30754709243774414, -0.3685295581817627, 0.8088196516036987, 0.24168819189071655, -0.8135301470756531, 0.2557373642921448, 0.5273621678352356, 0.41783738136291504, 0.15432117879390717, -0.6924462914466858, 0.6875506043434143, 0.48062795400619507, 0.8380681276321411, 0.02721494622528553, 0.2234557718038559, -0.6277190446853638, -0.5841697454452515, 0.9554637670516968, 0.18899153172969818, 0.8901127576828003, 0.9910117387771606, -0.47940099239349365, -0.90804123878479, 0.8467073440551758, 0.4961995482444763, 0.4688986837863922, 0.6035038828849792, -0.49731189012527466, 0.8032960891723633, -0.9407550692558289, 0.3199566900730133, -0.895663321018219, -0.6999837160110474, -0.8243597149848938, -0.976417601108551, 0.9190296530723572, 0.9390631914138794, 0.3222237825393677, 0.029675155878067017, 0.5401469469070435, -0.6343795657157898, 0.46346399188041687, 0.9427164793014526, -0.37415939569473267, -0.690217137336731, 0.007180290296673775, -0.4603918492794037, 0.8117997646331787, 0.9846315979957581, 0.9176233410835266, -0.7309967875480652, -0.5146957635879517, 0.005642379634082317, 0.18577633798122406, -0.1208275631070137, -0.6989251375198364, -0.9027764201164246, 0.9277678728103638, -0.4559471607208252, 0.9579063057899475, 0.5722951889038086, -0.3374440670013428, 0.11804976314306259, -0.14616407454013824, -0.9260150790214539, -0.009498657658696175, 0.5735844969749451, 0.4522196352481842, -0.2250933200120926, -0.6980719566345215, -0.23678353428840637, 0.7498269081115723, 0.9248574376106262, -0.8687958121299744, 0.7277564406394958, -0.9895543456077576, 0.20653346180915833, -0.927474856376648, 0.49436143040657043, 0.24250707030296326, -0.5298587679862976, -0.5463703274726868, 0.7390366792678833, -0.9629313349723816, 0.739808976650238, 0.9816550612449646, 0.8253657221794128, -0.6770109534263611, 0.7737691402435303, 0.7269609570503235, 0.6500208973884583, -0.5524323582649231, 0.6533978581428528, -0.8140605688095093, -0.6949333548545837, -0.6322082877159119, 0.054747603833675385, 0.8856616616249084, -0.016234425827860832, -0.908508837223053, -0.48216497898101807, 0.35955071449279785, -0.43919649720191956, 0.8834197521209717, -0.3465859591960907, 0.4805333614349365, 0.620410680770874, -0.5753769874572754, 0.6618231534957886, 0.5260798931121826, -0.3089563250541687, -0.8654187917709351, 0.6779069900512695, -0.2864350974559784, 0.3652554154396057, -0.3059545159339905, -0.6266201138496399, -0.699410080909729, 0.8190678358078003, -0.9208468794822693, 0.6542474627494812, -0.7277490496635437, -0.9648785591125488, -0.030335545539855957, -0.9744188785552979, -0.399209201335907, -0.2679421603679657, 0.28552642464637756, -0.7054983973503113, -0.048347294330596924, 0.2328016310930252, -0.8532477021217346, -0.3789350688457489, 0.4410324990749359, -0.2667071521282196, 0.010407530702650547, 0.4325996935367584, -0.0768912062048912, 0.2647496461868286, -0.40976864099502563, 0.8981539607048035, -0.06847252696752548, 0.6046262383460999, 0.9977343678474426, 0.6704595685005188, 0.4324131906032562, -0.4634430408477783, 0.20153912901878357, -0.851120114326477, -0.7918227314949036, -0.16967418789863586, 0.7931814193725586, -0.5260264277458191, -0.9354722499847412, -0.22800569236278534, -0.8953110575675964, 0.6282052397727966, -0.9744784235954285, 0.5308476686477661, -0.8985101580619812, 0.6873780488967896, -0.3318616449832916, 0.1798291951417923, -0.30549174547195435, -0.34715536236763, 0.014899814501404762, -0.8114772439002991, 0.6133894324302673, 0.912028431892395, -0.8417788743972778, 0.2108279913663864, -0.6035017371177673, -0.14879398047924042, 0.6023770570755005, 0.3282032608985901, -0.938996434211731, 0.4816577136516571, 0.14663325250148773, -0.40876391530036926, -0.18116161227226257, -0.1749773919582367, -0.38344743847846985, -0.17721930146217346, 0.16184087097644806, 0.1865168809890747, -0.42454397678375244, 0.544784665107727, -0.1547149270772934, -0.4095836877822876, -0.6403114199638367, -0.23103387653827667, 0.7607139945030212, -0.5874499082565308, 0.27301737666130066, -0.4579830765724182, -0.7612122297286987, 0.5871062874794006, -0.17976348102092743, 0.08676920086145401, 0.6370627880096436, 0.3845151364803314, 0.2559140622615814, 0.12254075706005096, 0.9826904535293579, 0.3446374833583832, 0.6388607621192932, -0.43401604890823364, -0.2791590690612793, 0.033429451286792755, 0.8139166831970215, 0.0975160002708435, 0.30228471755981445, 0.1823045313358307, 0.3226332366466522, -0.2897540330886841, 0.5926466584205627, 0.9535812139511108, -0.7633916139602661, -0.702233076095581, 0.612419068813324, -0.19354134798049927, -0.4369349479675293, 0.9565731883049011, -0.7197149991989136, 0.9387908577919006, 0.9322077035903931, -0.9782812595367432, 0.27243292331695557, -0.8291919827461243, 0.18342852592468262, 0.7327075600624084, -0.7455843687057495, 0.942791759967804, 0.27234137058258057, -0.894711971282959, -0.19090595841407776, 0.06319013983011246, -0.5523369312286377, -0.3603779077529907, 0.00641194311901927, -0.6101706027984619, 0.21260206401348114, 0.9486654996871948, 0.3734285235404968, -0.6055628657341003, 0.6914786696434021, -0.8034792542457581, 0.8666284084320068, 0.7919008731842041, -0.6934971809387207, -0.5565019845962524, 0.014889316633343697, -0.9909121990203857, -0.5077853798866272, -0.9083706736564636, -0.9041187763214111, -0.9267338514328003, -0.4657386541366577, -0.3059654235839844, -0.5533277988433838, 0.8975578546524048, 0.8835640549659729, -0.46892738342285156, 0.4111923575401306, -0.21700596809387207, -0.19139552116394043, 0.204336479306221, -0.22808122634887695, -0.5131177306175232, -0.7312142848968506, -0.31982386112213135, -0.527597188949585, 0.8650709390640259, -0.1692405343055725, 0.591948926448822, -0.5466968417167664, 0.8461695313453674, -0.9332528710365295, -0.9284268021583557, -0.03637942671775818, -0.6348055005073547, -0.05056839808821678, 0.46627265214920044, 0.4340777099132538, 0.3119189441204071, 0.2528172433376312, -0.9655771851539612, -0.43954384326934814, 0.7299214601516724, 0.8596165180206299, 0.7198875546455383, 0.696862518787384, -0.3223423361778259, -0.3544929325580597, 0.10770189017057419, -0.12727031111717224, -0.3035784363746643, 0.5828432440757751, 0.6311406493186951, 0.00606207549571991, -0.20869065821170807, -0.814649760723114, 0.6410235166549683, 0.2944173216819763, -0.9381942749023438, -0.1679498553276062, 0.7710793018341064, 0.5951834917068481, -0.38966280221939087, -0.347500741481781, 0.716823399066925, 0.7163486480712891, 0.11393875628709793, -0.9159365296363831, 0.4709431529045105, -0.7734928131103516, -0.9017250537872314, -0.48709556460380554, -0.7881401181221008, 0.9459800124168396, 0.9070432186126709, -0.8550847172737122, -0.7528649568557739, 0.36155182123184204, -0.9918026328086853, 0.6453409790992737, -0.37476226687431335, -0.8607597947120667, -0.37205803394317627, -0.4648365080356598, -0.2651728093624115, 0.35235756635665894, 0.3521897494792938, -0.0859014019370079, 0.9058785438537598, 0.367563396692276, 0.7776911854743958, 0.4189821183681488
            ]
		]
	}
}

其中: vectors取值完全随机,但是范围在(-1,1)
@tinkerlin

[tinkerlin]

I get it, but Only one vector is searched per request? And the dimension is ?

[stqf]

是的,每次请求只是搜索一个向量, 但是向量是随机生成的 且 dimension = 512
@tinkerlin

[del-zhenwu]

no produce with the following scripts in 2h:

        IntStream.range(0, loops).parallel().forEach(index -> {
                        List<List<Float>> queryVectors = Utils.genVectors(nq, dimension, true);
                        String params = "{\"nprobe\":1024}";
                        SearchParam searchParam = new SearchParam.Builder(collectionName)
                                .withFloatVectors(queryVectors)
                                .withParamsInJson(params)
                                .withTopK(top_k).build();
                        SearchResponse res_search = client.search(searchParam);
                        assert (res_search.getResponse().ok());
                });

[root@1239541e27ec logs]# grep "2020-07-01 18" milvus-20-07-01-14:42-info.log* | grep Request | wc -l
6623416
[root@1239541e27ec logs]# grep "2020-07-01 17" milvus-20-07-01-14:42-info.log* | grep Request | wc -l
6862990

and no re-produced with usingpython-sdk, with 5m requests totally.
To be continued.

[stqf]

What is the client CPU core number
@del-zhenwu

[yhmo]

"单独压测时,OPS约为1200 情况二同时压测时, OPS仅为320 性能下降至三分之一左右", the reason is relate to search combine mechanism, there is an issue #2648 about this.

[yhmo]

The hangs issue will be fixed in pr2715 in 0.10.1 branch

[yhmo]

Hi @stqf
This issue has been fixed in 0.10.1 branch. You can try our daily-build milvus(version 0.10.1):
docker pull milvusdb/daily-build:0.10.1-ubuntu18.04-cpu-20200703

0.10.1 is completely compatible with 0.10.0, it will be released after one or two weeks.

[stqf]

OK, Thanks
@yhmo