SNS/SQS performance - delivery times depend on the number of subscriptions

[czerwonkabartosz]

Type of request: This is a ...

[x] bug report
[ ] feature request

Detailed description

We use Localstack with quite a big number of microservices where messages between are transported using SNS/SQS - one service pushes an event to SNS and this is broadcasted to a few queues (every service has a separate queue). We've noticed some problems related to delivery time (performance issue). The problem is related to the number of subscribed queues to one SNS topic.

Expected behavior

Events should be delivered faster in an asynchronous way.

Actual behavior

I've prepared some reports that present actual behavior. As you can see, events are delivered to subscribed queues in a synchronous way in contrast to AWS.

Case 1 - 5 topics, 1 queue per topic, 5 messages

Deliver times for topic-0-queue-0: 210, 350, 348, 348, 349
Deliver times for topic-1-queue-0: 224, 225, 343, 344, 343
Deliver times for topic-2-queue-0: 269, 340, 340, 339, 341
Deliver times for topic-3-queue-0: 208, 324, 324, 323, 422
Deliver times for topic-4-queue-0: 212, 214, 308, 308, 308

Case 2 - 1 topic, 5 queues per topic, 5 messages

Deliver times for topic-0-queue-0: 177, 426, 427, 428, 427
Deliver times for topic-0-queue-1: 682, 889, 890, 891, 1149
Deliver times for topic-0-queue-2: 982, 1212, 1212, 1213, 1211
Deliver times for topic-0-queue-3: 1399, 1600, 1601, 1600, 1599
Deliver times for topic-0-queue-4: 1706, 1881, 1881, 1882, 1880

Case 3 - 1 topic, 10 queues per topic, 5 messages

Deliver times for topic-0-queue-0: 333, 594, 592, 592, 926
Deliver times for topic-0-queue-1: 913, 1151, 1151, 1154, 1152
Deliver times for topic-0-queue-2: 1317, 1317, 1588, 1588, 1891
Deliver times for topic-0-queue-3: 1749, 2092, 2090, 2411, 2411
Deliver times for topic-0-queue-4: 2429, 2648, 2650, 2648, 2647
Deliver times for topic-0-queue-5: 2848, 3036, 3034, 3033, 3033
Deliver times for topic-0-queue-6: 3230, 3445, 3444, 3447, 3444
Deliver times for topic-0-queue-7: 3620, 3841, 3840, 3843, 3840
Deliver times for topic-0-queue-8: 3912, 4140, 4143, 4141, 4370
Deliver times for topic-0-queue-9: 4300, 4498, 4498, 4501, 4606

Case 4 - 1 topic, 15 queues per topic, 10 messages

Deliver times for topic-0-queue-0: 436, 760, 759, 760, 757, 759, 1255, 1253, 1253, 1253
Deliver times for topic-0-queue-1: 1279, 1672, 1669, 1671, 1672, 1670, 2069, 2070, 2068, 2068
Deliver times for topic-0-queue-2: 2020, 2449, 2451, 2452, 2452, 2452, 2774, 2774, 2775, 2774
Deliver times for topic-0-queue-3: 2762, 3144, 3145, 3145, 3142, 3145, 3635, 3635, 3635, 3636
Deliver times for topic-0-queue-4: 3624, 4000, 4001, 4000, 3999, 4001, 4429, 4432, 4430, 4430
Deliver times for topic-0-queue-5: 4394, 4803, 4804, 4804, 4803, 4801, 5346, 5348, 5346, 5346
Deliver times for topic-0-queue-6: 5180, 5511, 5511, 5513, 5511, 5512, 5907, 5907, 5906, 5904
Deliver times for topic-0-queue-7: 6031, 6451, 6453, 6451, 6452, 6451, 6860, 6857, 6860, 6859
Deliver times for topic-0-queue-8: 6742, 7196, 7195, 7196, 7196, 7194, 7611, 7610, 7610, 7609
Deliver times for topic-0-queue-9: 7490, 7887, 7885, 7886, 7884, 7885, 8255, 8256, 8257, 8256
Deliver times for topic-0-queue-10: 8257, 8258, 8740, 8741, 8743, 8742, 8741, 9114, 9113, 9113
Deliver times for topic-0-queue-11: 8995, 9418, 9420, 9418, 9417, 9419, 9916, 9918, 9919, 9918
Deliver times for topic-0-queue-12: 9931, 9932, 10444, 10442, 10441, 10443, 10445, 10855, 10853, 10855
Deliver times for topic-0-queue-13: 10782, 11248, 11248, 11251, 11249, 11249, 11723, 11721, 11720, 11723
Deliver times for topic-0-queue-14: 11720, 11983, 11983, 11986, 11982, 11984, 12112, 12110, 12111, 121121

Case 5 - 3 topics, 10 queues per topic, 5 messages

Deliver times for topic-0-queue-0: 266, 570, 569, 568, 569
Deliver times for topic-0-queue-1: 616, 990, 991, 990, 989
Deliver times for topic-0-queue-2: 1180, 1613, 1614, 1613, 1612
Deliver times for topic-0-queue-3: 1833, 2102, 2101, 2100, 2101
Deliver times for topic-0-queue-4: 2540, 2761, 2759, 2760, 2760
Deliver times for topic-0-queue-5: 3001, 3307, 3305, 3306, 3306
Deliver times for topic-0-queue-6: 3481, 3749, 3750, 3748, 3749
Deliver times for topic-0-queue-7: 3935, 4203, 4203, 4202, 4205
Deliver times for topic-0-queue-8: 4464, 4689, 4689, 4688, 4691
Deliver times for topic-0-queue-9: 4943, 5100, 5100, 5102, 5099
Deliver times for topic-1-queue-0: 284, 534, 535, 534, 533
Deliver times for topic-1-queue-1: 797, 1081, 1080, 1080, 1079
Deliver times for topic-1-queue-2: 1260, 1647, 1648, 1649, 1648
Deliver times for topic-1-queue-3: 2077, 2078, 2322, 2323, 2322
Deliver times for topic-1-queue-4: 2560, 2982, 2981, 2982, 2983
Deliver times for topic-1-queue-5: 3048, 3345, 3344, 3346, 3345
Deliver times for topic-1-queue-6: 3677, 3950, 3951, 3950, 3952
Deliver times for topic-1-queue-7: 4226, 4470, 4470, 4469, 4471
Deliver times for topic-1-queue-8: 4729, 4982, 4981, 4980, 4981
Deliver times for topic-1-queue-9: 5352, 5498, 5498, 5498, 5497
Deliver times for topic-2-queue-0: 318, 521, 522, 522, 521
Deliver times for topic-2-queue-1: 731, 909, 910, 909, 1085
Deliver times for topic-2-queue-2: 1220, 1513, 1514, 1514, 1513
Deliver times for topic-2-queue-3: 1570, 2017, 2016, 2016, 2017
Deliver times for topic-2-queue-4: 2227, 2226, 2490, 2491, 2492
Deliver times for topic-2-queue-5: 2602, 2867, 2867, 2868, 2867
Deliver times for topic-2-queue-6: 3153, 3359, 3359, 3360, 3359
Deliver times for topic-2-queue-7: 3678, 3858, 3858, 3858, 3859
Deliver times for topic-2-queue-8: 4198, 4390, 4390, 4390, 4391
Deliver times for topic-2-queue-9: 4666, 4815, 4815, 4815, 4816

Case 6 - 1 topic, 30 queues, 5 messages

Deliver times for topic-0-queue-0: 333, 563, 564, 562
Deliver times for topic-0-queue-1: 745, 950, 951, 951
Deliver times for topic-0-queue-2: 1128, 1330, 1329, 1679
Deliver times for topic-0-queue-3: 1534, 1855, 1853, 1855, 1854
Deliver times for topic-0-queue-4: 2391, 2645, 2643, 2645, 2644
Deliver times for topic-0-queue-5: 2938, 3474, 3472, 3474, 3473
Deliver times for topic-0-queue-6: 3663, 4155, 4156, 4154, 4573
Deliver times for topic-0-queue-7: 4524, 4816, 4817, 5100, 5098
Deliver times for topic-0-queue-8: 5090, 5324, 5326, 5325, 5326
Deliver times for topic-0-queue-9: 5518, 5780, 5781, 5779, 5781
Deliver times for topic-0-queue-10: 6149, 6337, 6338, 6338, 6336
Deliver times for topic-0-queue-11: 6489, 6490, 6819, 6818, 6820
Deliver times for topic-0-queue-12: 7074, 7073, 7290, 7291, 7291
Deliver times for topic-0-queue-13: 7451, 7721, 7722, 7722, 7720
Deliver times for topic-0-queue-14: 7978, 8261, 8262, 8262, 8260
Deliver times for topic-0-queue-15: 8420, 8727, 8726, 8725, 8727
Deliver times for topic-0-queue-16: 8947, 9201, 9202, 9202, 9200
Deliver times for topic-0-queue-17: 9411, 9660, 9661, 9659, 9991
Deliver times for topic-0-queue-18: 9952, 10147, 10149, 10148, 10149
Deliver times for topic-0-queue-19: 10491, 10688, 10689, 10690, 10690
Deliver times for topic-0-queue-20: 10868, 11114, 11112, 11113, 11114
Deliver times for topic-0-queue-21: 11443, 11629, 11629, 11630, 11628
Deliver times for topic-0-queue-22: 11904, 12047, 12046, 12046, 12045
Deliver times for topic-0-queue-23: 12373, 12574, 12574, 12573, 12575
Deliver times for topic-0-queue-24: 12704, 13018, 13020, 13019, 13019
Deliver times for topic-0-queue-25: 13241, 13563, 13564, 13562, 13563
Deliver times for topic-0-queue-26: 13731, 13967, 13966, 13968, 14232
Deliver times for topic-0-queue-27: 14330, 14531, 14530, 14530, 14531
Deliver times for topic-0-queue-28: 14813, 15063, 15064, 15063, 15064
Deliver times for topic-0-queue-29: 15451, 15450, 15452, 15451, 15588

Case 7 - use AWS instead of Localstack, 1 topic, 15 queues per topic, 10 messages

Deliver times for topic-0-queue-0: 333, 331, 328, 329, 327, 576, 572, 803, 1032, 1246
Deliver times for topic-0-queue-1: 321, 328, 327, 325, 331, 520, 599, 828, 832, 1072
Deliver times for topic-0-queue-2: 307, 312, 305, 310, 309, 405, 407, 588, 832, 1060
Deliver times for topic-0-queue-3: 274, 268, 269, 270, 264, 508, 502, 610, 607, 863
Deliver times for topic-0-queue-4: 331, 330, 336, 340, 334, 575, 799, 1032, 1028, 1242
Deliver times for topic-0-queue-5: 331, 329, 330, 335, 332, 558, 559, 782, 1020, 1273
Deliver times for topic-0-queue-6: 288, 295, 294, 290, 292, 567, 810, 808, 819, 1094
Deliver times for topic-0-queue-7: 339, 334, 578, 1245, 1247, 1242, 1251, 1504, 1500, 1718
Deliver times for topic-0-queue-8: 304, 300, 306, 302, 307, 555, 562, 809, 805, 1027
Deliver times for topic-0-queue-9: 299, 304, 301, 300, 298, 911, 912, 1136, 1347, 1593
Deliver times for topic-0-queue-10: 266, 256, 263, 261, 259, 540, 538, 801, 1011, 1221
Deliver times for topic-0-queue-11: 284, 287, 282, 291, 288, 548, 814, 1062, 1303, 1307
Deliver times for topic-0-queue-12: 314, 313, 312, 308, 315, 441, 438, 590, 819, 1072
Deliver times for topic-0-queue-13: 314, 317, 319, 316, 312, 481, 608, 603, 601, 612
Deliver times for topic-0-queue-14: 280, 273, 277, 275, 279, 541, 793, 789, 800, 1036

Case 8 - use AWS instead of Localstack, 1 topic, 30 queues, 5 messages

Deliver times for topic-0-queue-0: 347, 344, 345, 343, 342
Deliver times for topic-0-queue-1: 385, 390, 388, 387, 386
Deliver times for topic-0-queue-2: 304, 309, 307, 305, 306
Deliver times for topic-0-queue-3: 359, 355, 354, 357, 630
Deliver times for topic-0-queue-4: 334, 329, 332, 330, 512
Deliver times for topic-0-queue-5: 325, 320, 321, 322, 323
Deliver times for topic-0-queue-6: 347, 351, 346, 349, 348
Deliver times for topic-0-queue-7: 319, 317, 321, 316, 318
Deliver times for topic-0-queue-8: 394, 726, 731, 782, 801
Deliver times for topic-0-queue-9: 350, 352, 349, 597, 736
Deliver times for topic-0-queue-10: 367, 369, 370, 372, 368
Deliver times for topic-0-queue-11: 312, 309, 314, 310, 311
Deliver times for topic-0-queue-12: 339, 338, 341, 340, 343
Deliver times for topic-0-queue-13: 354, 352, 351, 615, 618
Deliver times for topic-0-queue-14: 379, 382, 380, 384, 707
Deliver times for topic-0-queue-15: 358, 359, 360, 361, 363
Deliver times for topic-0-queue-16: 371, 373, 375, 370, 701
Deliver times for topic-0-queue-17: 376, 378, 374, 375, 373
Deliver times for topic-0-queue-18: 284, 283, 288, 286, 285
Deliver times for topic-0-queue-19: 362, 364, 363, 361, 366
Deliver times for topic-0-queue-20: 325, 326, 330, 328, 327
Deliver times for topic-0-queue-21: 336, 340, 337, 335, 338
Deliver times for topic-0-queue-22: 367, 369, 365, 364, 645
Deliver times for topic-0-queue-23: 313, 314, 315, 317, 312
Deliver times for topic-0-queue-24: 385, 384, 383, 386, 388
Deliver times for topic-0-queue-25: 334, 333, 337, 332, 335
Deliver times for topic-0-queue-26: 379, 377, 376, 381, 378
Deliver times for topic-0-queue-27: 301, 300, 298, 299, 303
Deliver times for topic-0-queue-28: 277, 281, 279, 278, 276
Deliver times for topic-0-queue-29: 289, 291, 294, 292, 290

Delivery times are in milliseconds.

Steps to reproduce

I've prepared a demo to reproduce this problem: https://github.com/czerwonkabartosz/localstack-sqs-sns-performance

Command used to start LocalStack

https://github.com/czerwonkabartosz/localstack-sqs-sns-performance/blob/main/docker-compose.yml

Client code (AWS SDK code snippet, or sequence of "awslocal" commands)

https://github.com/czerwonkabartosz/localstack-sqs-sns-performance/blob/main/index.js

[whummer]

Thanks for reporting @czerwonkabartosz . We've pushed a fix that parallelizes the SNS notifications in #4103 . Can you please pull the latest Docker image and run your benchmark again? Please let us know if we are still seeing performance issues - we'll be happy to dig into them in more detail.. Thanks for your help!

[czerwonkabartosz]

I've tested it and can confirm that it works in an asynchronous way 👍 and it improves performance a little bit. However, IMO, performance is still the main problem. Processing 5 messages on 30 queues require almost 20 seconds on MBP (i7 2,2 GHz 6-core, 16 GB RAM). But for sure is better so 👏 for this. 

---

Case 1 - 5 topics, 1 queue per topic, 5 messages

Deliver times for topic-0-queue-0: 209, 339, 337, 338, 337
Deliver times for topic-1-queue-0: 291, 396, 396, 397, 395
Deliver times for topic-2-queue-0: 287, 439, 439, 438, 438
Deliver times for topic-3-queue-0: 196, 350, 350, 351, 453
Deliver times for topic-4-queue-0: 227, 310, 310, 309, 309

Case 2 - 1 topic, 5 queues per topic, 5 messages

Deliver times for topic-0-queue-0: 402, 669, 667, 668, 1911
Deliver times for topic-0-queue-1: 233, 235, 510, 928, 1570
Deliver times for topic-0-queue-2: 233, 614, 1057, 1058, 1547
Deliver times for topic-0-queue-3: 700, 1055, 1054, 1357, 1358
Deliver times for topic-0-queue-4: 171, 644, 645, 1061, 1564

Case 3 - 1 topic, 10 queues per topic, 5 messages

Deliver times for topic-0-queue-0: 1895, 2331, 3516, 3514, 3903
Deliver times for topic-0-queue-1: 446, 1693, 2097, 3175, 3570
Deliver times for topic-0-queue-2: 364, 874, 1378, 1732, 3915
Deliver times for topic-0-queue-3: 737, 1710, 2148, 2879, 3919
Deliver times for topic-0-queue-4: 1142, 2127, 2562, 3002, 3932
Deliver times for topic-0-queue-5: 559, 1032, 2559, 3543, 3936
Deliver times for topic-0-queue-6: 1010, 1399, 2349, 2955, 3401
Deliver times for topic-0-queue-7: 991, 1313, 1666, 2165, 2553
Deliver times for topic-0-queue-8: 554, 2542, 2953, 2954, 3363
Deliver times for topic-0-queue-9: 1022, 1665, 2774, 3143, 3651

Case 4 - 3 topics, 10 queues per topic, 5 messages

Deliver times for topic-0-queue-0: 1197, 1702, 2285, 2608, 3105
Deliver times for topic-0-queue-1: 195, 2837, 3166, 3731, 4100
Deliver times for topic-0-queue-2: 1347, 2479, 2896, 3724, 4113
Deliver times for topic-0-queue-3: 1984, 2398, 3327, 4120, 4303
Deliver times for topic-0-queue-4: 422, 1968, 2409, 2783, 3336
Deliver times for topic-0-queue-5: 407, 745, 1450, 3730, 4114
Deliver times for topic-0-queue-6: 2127, 2781, 3112, 3113, 3709
Deliver times for topic-0-queue-7: 419, 844, 1486, 3265, 3703
Deliver times for topic-0-queue-8: 438, 971, 971, 1801, 4045
Deliver times for topic-0-queue-9: 724, 1400, 1783, 1784, 2114
Deliver times for topic-1-queue-0: 1351, 1818, 1817, 5540, 5755
Deliver times for topic-1-queue-1: 475, 1288, 2753, 3355, 3355
Deliver times for topic-1-queue-2: 3043, 3832, 4597, 5157, 5156
Deliver times for topic-1-queue-3: 288, 792, 1233, 4615, 5170
Deliver times for topic-1-queue-4: 281, 1699, 2079, 2711, 5912
Deliver times for topic-1-queue-5: 1836, 2494, 3322, 3875, 4589
Deliver times for topic-1-queue-6: 482, 866, 4060, 4591, 5135
Deliver times for topic-1-queue-7: 932, 1323, 1840, 5174, 5542
Deliver times for topic-1-queue-8: 916, 1256, 2224, 2760, 3476
Deliver times for topic-1-queue-9: 381, 2479, 3867, 4473, 4474
Deliver times for topic-2-queue-0: 865, 2125, 2549, 4868, 5720
Deliver times for topic-2-queue-1: 596, 1007, 2073, 4843, 5615
Deliver times for topic-2-queue-2: 1628, 2914, 3839, 4565, 4564
Deliver times for topic-2-queue-3: 597, 1152, 1584, 2099, 5567
Deliver times for topic-2-queue-4: 1609, 2542, 2949, 3562, 4162
Deliver times for topic-2-queue-5: 569, 1101, 1406, 2366, 4174
Deliver times for topic-2-queue-6: 1316, 3266, 3995, 4938, 6037
Deliver times for topic-2-queue-7: 2791, 3273, 3272, 4005, 5702
Deliver times for topic-2-queue-8: 588, 1098, 2057, 2502, 3603
Deliver times for topic-2-queue-9: 1860, 2367, 4854, 5647, 6035

Case 5 - 1 topic, 30 queues, 5 messages

Deliver times for topic-0-queue-0: 1073, 2957, 9904, 10502, 16970
Deliver times for topic-0-queue-1: 736, 8946, 13202, 13905, 14414
Deliver times for topic-0-queue-2: 8311, 11555, 12854, 13978, 14472
Deliver times for topic-0-queue-3: 6485, 10327, 12298, 16161, 16794
Deliver times for topic-0-queue-4: 7987, 8946, 9436, 14677, 15276
Deliver times for topic-0-queue-5: 2247, 3981, 6733, 8972, 18691
Deliver times for topic-0-queue-6: 7619, 10609, 11730, 13166, 16348
Deliver times for topic-0-queue-7: 6214, 9370, 11997, 14200, 15979
Deliver times for topic-0-queue-8: 6940, 9566, 10950, 12876, 13304
Deliver times for topic-0-queue-9: 2225, 5759, 6519, 17225, 19036
Deliver times for topic-0-queue-10: 2038, 5025, 17205, 17967, 19019
Deliver times for topic-0-queue-11: 8422, 12844, 14397, 17866, 18765
Deliver times for topic-0-queue-12: 649, 3577, 4804, 5768, 15184
Deliver times for topic-0-queue-13: 290, 6105, 6730, 12272, 15201
Deliver times for topic-0-queue-14: 262, 5599, 9988, 10613, 11528
Deliver times for topic-0-queue-15: 2965, 4765, 6792, 13659, 18735
Deliver times for topic-0-queue-16: 4592, 9753, 11103, 12031, 16776
Deliver times for topic-0-queue-17: 7054, 13110, 15926, 15926, 16318
Deliver times for topic-0-queue-18: 1154, 2269, 5086, 9104, 12450
Deliver times for topic-0-queue-19: 4016, 7288, 12004, 18753, 19047
Deliver times for topic-0-queue-20: 7744, 8457, 9914, 10499, 13569
Deliver times for topic-0-queue-21: 3176, 4066, 5377, 11570, 17965
Deliver times for topic-0-queue-22: 322, 1185, 4312, 13485, 14126
Deliver times for topic-0-queue-23: 265, 2070, 8868, 15285, 17605
Deliver times for topic-0-queue-24: 2041, 7712, 10511, 12456, 14691
Deliver times for topic-0-queue-25: 3270, 9014, 12549, 15292, 18005
Deliver times for topic-0-queue-26: 5520, 7717, 14486, 15252, 16967
Deliver times for topic-0-queue-27: 3952, 4745, 5534, 10632, 12832
Deliver times for topic-0-queue-28: 5985, 9582, 15510, 16234, 16806
Deliver times for topic-0-queue-29: 6907, 11514, 12031, 12438, 13333