[CI] AzureBlobStoreRepositoryTests.testMultipleSnapshotAndRollback failed

[DaveCTurner]

This build failed:

org.elasticsearch.repositories.azure.AzureBlobStoreRepositoryTests > testMultipleSnapshotAndRollback FAILED
    java.lang.AssertionError: 
    Expected: <6>
         but: was <5>

    java.lang.AssertionError: Only index blobs should remain in repository but found [/container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__8roQqxXMTe-pw58Xv5xc3A, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__-4Vuos2SRtqAThZ5mxPulQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__b1UMRNdZSGOiA3_Uhc9frA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__H1wN9vLfQ6mLQ67vnVqrrw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__OknHHIDRT4eEgUrimhmTbw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__OPyrzrLyQaK4J6siWUaakQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__8psQ5GhSSE-G6YASvFXTDA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__FcvPf7tVQzufmvDZsIP5rA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__OQ_wlMTZSuOn_sQAw11AaA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/snap-jfZOCH_YSaqL5l9_1rXlfw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__VWlJ8w5-SpiIRuZjZY_LSw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__fYfl4VFXSqKut5feQnB1bw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/snap-j7ek137PQours01hscZesw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__gf7zwB4rT5CrD9vxs3yBRw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__yV_LXywDThKsTx1lLGO_wA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__i9-YbJ42RxSYOinXSpNBbA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__V8--qrlsRm6Xr2vLrHGj2A, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__0SKdwQe6S1S82OFQSYvGcA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__ywC1747QREe_HzqGAKw2Pg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/snap--sYVWWBLQI-3JM9cq7jjIg.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__X6t9C7moTSqD9rT6AtNgtQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__Uzju7k1xRBqAgiMaSgVE5g, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__SnMpNjZgRuuI6LbFN-3JHA, /container/meta--sYVWWBLQI-3JM9cq7jjIg.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__9eTnTyIuSSq7djLRlQhaRw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__cu7vx4OmS7aaKkKwXifJVg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__cIgUvWWQQ6S_Oqvr4Rbh6g, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/snap-jfZOCH_YSaqL5l9_1rXlfw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__xEusTOCpT7mOEzkoyT9otQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__a0Fnpm_XTkGgIs_fMacOJQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__rTZHJqqEQtO45FEv4NILBw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__RcKtBX8aRBKiSFVOF7Khpg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__YoQJ_gWXQkuFYFKFfKGpOg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__zfm81ohLSaGc03fLJ8TV8A, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__fa9ourGzRt6N9LQf7fDnEQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__Ydj-hW1gTbWb6NIMnY6dbQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__nsKZhdU5SoaKkhFNvYRD2Q, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__VKkW_vzyTzmMMJKOoGX8kg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__ZkiXhEvNSTmQETaAL_YtNQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__CE3vDpOZS7epctfepn-2sQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__8tS_V9tRSTy5pQ6RbN2wug, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__cGaB9UZ_Tqm_f9lW89MsSQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__OlI_izALRRKuCCk6kx0IaQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__cmyFQ8H-SUGQtPxJX66QdA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__U0pJPbnbTrmErsf3mu1oCA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/snap--sYVWWBLQI-3JM9cq7jjIg.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__Y5IrEkaQTEG2kklDCXJ1Lw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/snap--sYVWWBLQI-3JM9cq7jjIg.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__rfd-zxWlSF66VnUT0n07Nw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__7JDyurNcSdSFZxYyYuTiog, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__F4PP89A3S_qDMRx_yiQ8HQ, /container/snap--sYVWWBLQI-3JM9cq7jjIg.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__4IP0VBAdQnSo4EJnIlh6Mw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__1ESFtwMLT-inAYFoccBkYw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__ZI1MoYSiSDKHJDqesF4B4Q, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/___y0UWWu8SZuh0omkP8hBZw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__Cz70xZqwQXm3WJtRCTQZ0Q, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__4mMw-prxQF-TfomDYh8I5g, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__6JVwyn2qTIq6Yq4JaBdc1g, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/snap-j7ek137PQours01hscZesw.dat, /container/snap-j7ek137PQours01hscZesw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__SzphckeZRRSzt-NBGANJvg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__Dd5J7F9DSz-iBQ3zc_26Pg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__w54XOI-JRLu8Yz8Ocwcu8w, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__0Yw0Y8yzQ2ia7SyGfgOfEA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/snap-jfZOCH_YSaqL5l9_1rXlfw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__fZlNeFqGQIqi-UjUZUnaLA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__kqrPmLp4RF6QRP5j12etKw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__6qejc4xpRrSvRZ8LKW5Sew, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__UogFaLrmQnGWPlrNDUmgNQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__8hUdPVq-SWGftn1_WQ6lqg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__u0ANDbnmS_6x-oGkDMRCcg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__1YGldmqsTPSVnhSmU9NkgQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__h5G2cLKSTzeeX9MY0zW6sA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/snap--sYVWWBLQI-3JM9cq7jjIg.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__Z2ZERHf9SNe1iPVusP_I3w, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__0TA6hUKgToevXXygd0-5YA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/snap-jfZOCH_YSaqL5l9_1rXlfw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__biJzL-KcTzC9t5JsGp3Blg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__thFXsxXPRfu5x7LWTLQIzg, /container/meta-j7ek137PQours01hscZesw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__bfHw9aiMTau4Y_Jbl8uJng, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__CoyVB_5HTcOJnFGpA34F8Q, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__3E4oFjUSRy6akwVK8RLmXQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__f9wUMw0ORj-QwtkyU5aJRw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__FoQhapHATOugMx5DCdcKNw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__CgQpfCL_Rc2yw-GYk1BRCg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__uuCEaE1ERzKWdWHUcm49cQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/meta--sYVWWBLQI-3JM9cq7jjIg.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__sq1hLWXsSo-liE-wF1KYrw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__23wZyT0MTNucugGCjbwngg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__6GxcssGGRNWtekaiRYKeMA, /container/snap-jfZOCH_YSaqL5l9_1rXlfw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__ZHOzxNo2TRWwJlkb-UnCAA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__0BG6v7CNSni7HeiMJxu_Gg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__1hhs3vLbSGOrVwnb08ylYg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__ZNpOYS2UT5-GxSeZNoZ19Q, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__cmb7mNBfRQSI4DXLNCBjlw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/snap-j7ek137PQours01hscZesw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__-x9ZeaP6TLyfjuW2hiHhLQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__MPFbj7lsRPmwjBSCgtt4tA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__vf9m66saRQqEf4-TWv7iDQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__PVEjYYt9QdOE9CbLUg0DPQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__hb5ug9ntRWSJIE302vqkeg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__TIdDc-83QHKQcDSArHVcLw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__bhY_Qw-lRBipDC0rSmVigg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__VdUex-KkRQi8rCzXPp4F3Q, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__NM2GHY8tQAGtfX-3Uob5Dg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__nzb4k5ZOQTiE8XkgBvxVCg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__W7ijzDYGQ4eTrWPBCYxf-w, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/snap--sYVWWBLQI-3JM9cq7jjIg.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__NjRQkjIKRdOjlJAdodp09w, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/snap-jfZOCH_YSaqL5l9_1rXlfw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/___g7cGN0lR1a5BGvS-BuKvw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__cNBvKf06QK-mhgmve75Iyg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__dJUnOY_aSD-3FR3m9Bpfyg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__s6GWmYN7S3eyiHP0UGFQJg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__6jb0_B69TAyazd6pwNIlZg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__COzLVvzsSHSQdi2cUSUK-g, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__JJUMEYVXQL6qk1xGOoU2_w, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__dLryVW7iToSJAXBacbZRTA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__-Z6923tzRyuyGJBY37B6Zg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__utt_NGlrRTmF3g4wa9ofYA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__2bHhpIT7QdKrgHGyyVbJqw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__vx8oHADxRaGTrB_Xtz-FQg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__7MGY_4BUSIiK0DFaOj-X3w, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__rLNDmHZITyuuQLvDESDeag, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__1sOoGbU0SzuV3aDK6Zb2Dg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__QLTgY3QyS-uiaWJmMCqIcg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__0muX0W59Q7iY7v5gUw935w, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__QmlLyQKzQmGPmlt1Ut-bnA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/meta-j7ek137PQours01hscZesw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__5m8Bx-5dSEi619z9hKRAkw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__dx9Av0IOTvOspoUha0P4Rw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__NJt76biVR7GCtsx-obeXCA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__20XSkk6QQGumUnyMeS9fLw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__YxlFIwhHRKWRYxusboukxw, /container/meta-jfZOCH_YSaqL5l9_1rXlfw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__YT3kSL09SaevvvnTcSVIew, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__hKrE2vQ4Rq-uD0sx3N58zw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/snap--sYVWWBLQI-3JM9cq7jjIg.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__w5zBhV6ZQtyrzpALJ4e8nw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/meta-jfZOCH_YSaqL5l9_1rXlfw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__zhXUvvD6R1W1ne39X3btYQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__7RGzX2DqTOuWdM1Y8SLDsQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__iKMnIRoGT06RLy4vskAPoQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/snap-jfZOCH_YSaqL5l9_1rXlfw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__pznZbdIhReqmBHj-igUoow, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__Pt0B_qpXQnmcEzzlmDdg-Q, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__65_QtKGmTmOJ67G0qwtQqw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/snap-j7ek137PQours01hscZesw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__CmYh37brTteTopjfVlpxZw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__fkJahOhkTL20BXabSEyEMQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__VQnXMRkGQjaVdojB8tc6Ug, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/____z89ITdRhClq1Nbc1xyIg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/1/__dyJ_mNi8SN2IZA90XA3zng, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__CdqfxMH3Tt6m3EJ5zVsTZg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/snap-j7ek137PQours01hscZesw.dat, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__Npu-iARzTCqMd90VhJc7cA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__j6sGs_djTha3rQzSiKfR6Q, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__Sg8s8KVMSqWmBFJ_Z_OXIg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__0-NoNRkLRBm75LdOVH-pxw, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__yqYqaERdTxOlIxEI_3KnwA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__Z814otilTQS3RRZ7GUrOzA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__oz8gJFj4QOGBu1WtFrrBsQ, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/3/__thbmXT0STLa684QQXlXcvA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__thnM0eSTTBKs0vX7l1rtww, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/2/__X-xOw9hYRpaJU372RWtliA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/0/__gaj_h1IwT7urirgMYygmOg, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__fl1LCB_0Tx-cK3CF7jG2CA, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/5/__j0Fif4CXQiKzWUw3_j0z3g, /container/indices/N1rNYH9DQwOtIHYYjIDUcA/4/__If0TqTASR5666C_X3p3tqQ]
    Expected: a collection with size <0>
         but: collection size was <172>
REPRODUCE WITH: ./gradlew ':plugins:repository-azure:test' --tests "org.elasticsearch.repositories.azure.AzureBlobStoreRepositoryTests.testMultipleSnapshotAndRollback" -Dtests.seed=8C7B57C0B29203F9 -Dtests.security.manager=true -Dtests.locale=es-UY -Dtests.timezone=Africa/Lome -Dcompiler.java=12 -Druntime.java=8

The REPRODUCE WITH line does not reproduce locally for me.

Apologies if this duplicates #48978 -- this seems at least superficially to be a different issue but I have not dug deeply.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

[tlrx]

Thanks David. This assertion might just be the consequence of #48978 (in which some "blobs" are left around and not delete because the HTTP server stopped to respond) but we need to troubleshoot #48978 first.