Skip to content

xlab-uiuc/rainmaker

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

16 Commits
infra
infra
 
 
patches
patches
 
 
results
results
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 
bugs.md
bugs.md
 
 
requirements.txt
requirements.txt
 
 

Repository files navigation

Push-Button Reliability Testing for Cloud-Backed Applications with Rainmaker

This is the research artifact of "Push-Button Reliability Testing for Cloud-Backed Applications with Rainmaker" presented at NSDI '23.

Table of Contents

Goal

The instructions will reproduce the key results in the paper. That is, the following instructions will lead you to (1) reproduce the bugs found by Rainmaker and (2) how to generate test plans.

Getting Started Instructions

Pre-requisites

Environment setup

  • Rainmaker repository cloned in the home directory:

    cd ~
git clone https://github.com/xlab-uiuc/rainmaker.git
cd rainmaker

  • .NET: The .NET version is decided by the cloud application under test. To check your .NET SDK: Go to C:\Program Files\dotnet\sdk to view all .NET SDK editions. If you do not have .NET SDK, please go to Download .NET to download and install.

  • Java and Maven installed.

    Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)                   
Java version: 11.0.14.1, vendor: Amazon.com Inc., runtime: XXX\.jdks\corretto-11.0.14.1
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

    Java version: 11

  • Python3 installed and dependency packages installed: pip3 install -r requirements.txt

  • Azure Storage Emulator installed, and modify the ports configuration from 10000, 10001, 10002 to 20000, 20001, 20002 in file AzureStorageEmulator.exe.config under C:\Program Files (x86)\Microsoft SDKs\Azure\Storage Emulator directory. After modification, it will look like:

    <services>
<service name="Blob" url="http://127.0.0.1:20000/"/>
<service name="Queue" url="http://127.0.0.1:20001/"/>
<service name="Table" url="http://127.0.0.1:20002/"/>
</services>

    Search "Azure Storage Emulator" in the searching bar and then click it.

  • Dynamic instrumentation tool turned on: The instrumentation tool in our infra aims to capture the stack trace of the Azure Storage operations, and attach it to the corresponding request.

    1. Open a PowerShell with Administrator permission (right click the PowerShell icon in Start menu, and select "Run as Administrator")
    2. cd infra\torch-tool
    3. Run .\ProfOn.ps1 to turn on the dynamic instrumentation profiler
    4. Since the environment settings of a PowerShell is established at the beginning of its life cycle, you should open another PowerShell with Administrator permission, and then the instrumentation will work
    5. Note that you should start the emulator first and then turn on Torch. If the emulator cannot be started, please run .\ProfOff.ps1 to turn off Torch and then run .\ProfOn.ps1.

  • CosmosDB Emulator installed and run.

  • Download and build the cloud application repos (https://github.com/xlab-uiuc/rainmaker/tree/master/patches) under the path C:\Users\XX;

    • Clone the cloud app repo and checkout to the commit git checkout COMMIT_HASH;
    • Enter the application .NET solution folder (i.e., the folder with .sln file. For example, cd C:\Users\XX\Alpakka\src);
    • Build it (dotnet build).

Evaluation Instructions

Key results: Bug reproduction (Table 5)

cd ~\rainmaker\infra\rainmaker-proxy

Cloud-backed applications require different versions of .NET, so we reproduce the bugs separately according to the applications. For bug reproduction, we only run the selected tests to reproduce bugs we found instead of running the whole test suite. The running time here is measured in the VM provided for the artifact. Note that we had a much more powerful machine used during our evaluation, and the actual running time written in the paper is for the whole test suites.

If needed, please refer to this markdown file or Google Spreadsheet for detailed bug reports and how they are classified.

Applications backed by Azure Storage service

Alpakka (~5 min)

Run python driver.py -e alpakka

After the tests finish, there will be a folder ~\rainmaker\results\Alpakka-injection-round_XXXX.XX.XX.XX.XX.XX created with a bug_inspection.csv file inside.

The bug_inspection.csv contains the following two records indicating the two bugs founded, e.g.:

name	turn	URI	SDK	policy	outcome	time	excp_type
Akka.Streams.Azure.StorageQueue.Tests.QueueSinkSpec.A_QueueSink_should_skip_failing_messages_if_supervision_strategy_is_restart	5	127.0.0.1:10001POST/devstoreaccount1/testqueue/messages	C:\Users\yinfang\Alpakka\src\Azure\Akka.Streams.Azure.StorageQueue\QueueSink.cs:63:Azure.Storage.Queues;Azure.Storage.Queues.QueueClient.SendMessageAsync(string,System.Nullable`1[System.TimeSpan],System.Nullable`1[System.TimeSpan],System.Threading.CancellationToken):Akka.Streams.Azure.StorageQueue.QueueSink.Logic.TryAdd(String message)	timeout_blind	Failed	12.8200062s	Assertion			
Akka.Streams.Azure.StorageQueue.Tests.QueueSinkSpec.A_QueueSink_should_skip_failing_messages_if_supervision_strategy_is_restart	2	127.0.0.1:10001GET/devstoreaccount1/testqueue/messages	C:\Users\yinfang\Alpakka\src\Azure\Akka.Streams.Azure.StorageQueue.Tests\QueueSinkSpec.cs:88:Azure.Storage.Queues;Azure.Storage.Queues.QueueClient.ReceiveMessagesAsync():Akka.Streams.Azure.StorageQueue.Tests.QueueSinkSpec.A_QueueSink_should_skip_failing_messages_if_supervision_strategy_is_restart(?)	timeout_blind	Failed	12.8599295s	System.IndexOutOfRangeException

ServiceBus.AttachmentPlugin (Figure 7; ~5 min)

Run python driver.py -e attachmentplugin

After the tests finish, there will be a folder ~\rainmaker\results\ServiceBus.AttachmentPlugin-injection-round_XXXX.XX.XX.XX.XX.XX created with a bug_inspection.csv file inside.

The bug_inspection.csv contains the following two records indicating the two bugs founded, e.g.:

ServiceBus.AttachmentPlugin.Tests.When_sending_message_using_connection_string.Should_be_able_to_send_if_container_was_not_found	4	127.0.0.1:10000PUT/devstoreaccount1/attachments-that-didnt-exist/restype=container	C:\Users\yinfang\ServiceBus.AttachmentPlugin\src\ServiceBus.AttachmentPlugin\AzureStorageAttachment.cs:51:Microsoft.Azure.Storage.Blob;Microsoft.Azure.Storage.Blob.CloudBlobContainer.CreateIfNotExistsAsync():ServiceBus.AttachmentPlugin.AzureStorageAttachment.BeforeMessageSend(?)	request_block	Failed	41.8454642s	Microsoft.Azure.Storage.StorageException		
ServiceBus.AttachmentPlugin.Tests.When_sending_message_using_connection_string.Should_be_able_to_send_if_container_was_not_found	3	127.0.0.1:10000HEAD/devstoreaccount1/attachments-that-didnt-exist/restype=container	C:\Users\yinfang\ServiceBus.AttachmentPlugin\src\ServiceBus.AttachmentPlugin\AzureStorageAttachment.cs:49:Microsoft.Azure.Storage.Blob;Microsoft.Azure.Storage.Blob.CloudBlobContainer.ExistsAsync():ServiceBus.AttachmentPlugin.AzureStorageAttachment.BeforeMessageSend(?)	request_block	Failed	39.5126248s	Microsoft.Azure.Storage.StorageException

BotBuilder (Figure 3, 5; ~8 min)

Run python driver.py -e botbuilder

After the tests finish, there will be a folder ~\rainmaker\results\Botbuilder-dotnet-injection-round_XXXX.XX.XX.XX.XX.XX created with a bug_inspection.csv file inside.

The bug_inspection.csv contains the following four records indicating the four bugs founded, e.g.:

Microsoft.Bot.Builder.Azure.Tests.AzureBlobTranscriptStoreTests.LogActivities	5	127.0.0.1:10000PUT/devstoreaccount1/blobtranscriptlogactivities/restype=container	C:\Users\yinfang\botbuilder-dotnet\libraries\Microsoft.Bot.Builder.Azure\AzureBlobTranscriptStore.cs:71:Microsoft.Azure.Storage.Blob;Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.CreateIfNotExistsAsync():Microsoft.Bot.Builder.Azure.AzureBlobTranscriptStore.<>c__DisplayClass3_0.<.ctor>b__0()	request_block	Failed	58.3299953s	Assertion			
Microsoft.Bot.Builder.Azure.Tests.AzureBlobTranscriptStoreTests.LogActivities	4	127.0.0.1:10000PUT/devstoreaccount1/blobtranscriptlogactivities/test/6e3a47cdf9f84e94a8a67125a9320d18/8daf22a585369f9-1.json/comp=metadata	C:\Users\yinfang\botbuilder-dotnet\libraries\Microsoft.Bot.Builder.Azure\AzureBlobTranscriptStore.cs:344:Microsoft.Azure.Storage.Blob;Microsoft.WindowsAzure.Storage.Blob.CloudBlob.SetMetadataAsync():Microsoft.Bot.Builder.Azure.AzureBlobTranscriptStore.LogActivityAsync(?)	request_block	Failed	2m 17.4268634s	Assertion			
Microsoft.Bot.Builder.Azure.Tests.AzureBlobTranscriptStoreTests.LogActivities	1	127.0.0.1:10000PUT/devstoreaccount1/blobtranscriptlogactivities/test/3794227049634ae4a43d02855a0d207b/8daf229d3184fae-1.json/blockid=ZjdlNWI4NTM2NzdmNGYzYWIzYzk0YmU0NjJmNDRiNjMtMDAwMDAwcomp=block	C:\Users\yinfang\botbuilder-dotnet\libraries\Microsoft.Bot.Builder.Azure\AzureBlobTranscriptStore.cs:0:Microsoft.Azure.Storage.Blob;Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.Flush():Microsoft.Bot.Builder.Azure.AzureBlobTranscriptStore.LogActivityAsync(?)	request_block	Failed	2m 2.4576851s	Assertion			
Microsoft.Bot.Builder.Azure.Tests.AzureBlobTranscriptStoreTests.LogActivities	0	127.0.0.1:10000PUT/devstoreaccount1/blobtranscriptlogactivities/test/1d3a246fe3f74d2e8b5c7d7683b92895/8daf22989923013-1.json/comp=blocklist	C:\Users\yinfang\botbuilder-dotnet\libraries\Microsoft.Bot.Builder.Azure\AzureBlobTranscriptStore.cs:0:Microsoft.Azure.Storage.Blob;Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.Dispose(bool):Microsoft.Bot.Builder.Azure.AzureBlobTranscriptStore.LogActivityAsync(?)	request_block	Failed	2m 4.7391729s	Assertion

DistributedLock (~20 min)

Run python driver.py -e distributedlock

After the tests finish, there will be a folder ~\rainmaker\results\DistributedLock-injection-round_XXXX.XX.XX.XX.XX.XX created with a bug_inspection.csv file inside.

The bug_inspection.csv contains the following two records (may have more than two due to the test flakiness) indicating the two bugs founded, e.g.:

TestParallelism	6	127.0.0.1:10000PUT/devstoreaccount1/distributed-lock-netcoreapp3-1/parallel_test_Medallion.Threading.Tests.Azure.Core_AzureBlobLease_AzureBlobLeaseSynchronizationStrategyTest.TestParallelism_netcoreapp3.1	C:\Users\yinfang\DistributedLock\DistributedLock.Tests\Infrastructure\Azure\TestingAzureBlobLeaseDistributedLockProvider.cs:23:Azure.Storage.Blobs;Azure.Storage.Blobs.BlobClient.Upload(System.IO.Stream):Medallion.Threading.Tests.Azure.TestingAzureBlobLeaseDistributedLockProvider.CreateLockWithExactName(String name)	request_block	Failed	12.1261476s	Azure.RequestFailedException			
TestParallelism	3	127.0.0.1:10000PUT/devstoreaccount1/distributed-lock-netcoreapp3-1/parallel_test_Medallion.Threading.Tests.Azure.Core_AzureBlobLease_AzureBlobLeaseSynchronizationStrategyTest.TestParallelism_netcoreapp3.1/comp=lease	C:\Users\yinfang\DistributedLock\DistributedLock.Azure\BlobLeaseClientWrapper.cs:52:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobLeaseClient.ReleaseAsync(Azure.RequestConditions,System.Threading.CancellationToken):Medallion.Threading.Azure.BlobLeaseClientWrapper.ReleaseAsync()	request_block	Failed	48.6737332s	Assertion

Insights (~30 min)

You may need to specify your .NET version in the global.json file under Insights directory.

Run python driver.py -e insights

After the tests finish, there will be a folder ~\rainmaker\results\Insights-injection-round_XXXX.XX.XX.XX.XX.XX created with a bug_inspection.csv file inside.

The bug_inspection.csv contains the following 14 records indicating the ten bugs founded (there are multiple test rounds denoting the same bug), e.g.:

NuGet.Insights.TimerExecutionServiceTest+TheExecuteAsyncMethod.RunsATimerAgainIfTheFrequencyAllows	11	127.0.0.1:10002PUT/devstoreaccount1/t230110hy7sj55ir41t1(PartitionKey='',RowKey='08585282762496425577-e36frff2nxauroxmwy6n7cc4iy-a')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\Insights\src\Logic\Timers\SpecificTimerExecutionService.cs:150:Azure.Data.Tables;Azure.Data.Tables.TableClient.UpdateEntityAsync(!!0,Azure.ETag,Azure.Data.Tables.TableUpdateMode,System.Threading.CancellationToken):NuGet.Insights.SpecificTimerExecutionService.<>c__DisplayClass9_1.<ExecuteAsync>b__5()	timeout_blind	Failed	19.4852939s	Azure.RequestFailedException			
NuGet.Insights.TimerExecutionServiceTest+TheExecuteAsyncMethod.ExecutesTimerEnabledForStorage	10	127.0.0.1:10002PUT/devstoreaccount1/t230110tb6rf2h5ca1t1(PartitionKey='',RowKey='08585282765200412877-vmgzu4c3usbuhm5l6mxbkz5y2y-a')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\Insights\src\Logic\Timers\SpecificTimerExecutionService.cs:138:Azure.Data.Tables;Azure.Data.Tables.TableClient.UpdateEntityAsync(!!0,Azure.ETag,Azure.Data.Tables.TableUpdateMode,System.Threading.CancellationToken):NuGet.Insights.SpecificTimerExecutionService.<>c__DisplayClass9_1.<ExecuteAsync>b__4()	timeout_blind	Failed	18.4538217s	Azure.RequestFailedException			
NuGet.Insights.TimerExecutionServiceTest+TheExecuteNowAsyncMethod.IgnoresFrequency	7	127.0.0.1:10002PUT/devstoreaccount1/t230110wup4qm3dve1t1(PartitionKey='',RowKey='08585282767406807050-22wuuwu2n3vupg5twgkw2t3woq-a')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\Insights\src\Logic\Timers\SpecificTimerExecutionService.cs:126:Azure.Data.Tables;Azure.Data.Tables.TableClient.UpdateEntityAsync(!!0,Azure.ETag,Azure.Data.Tables.TableUpdateMode,System.Threading.CancellationToken):NuGet.Insights.SpecificTimerExecutionService.<>c__DisplayClass9_1.<ExecuteAsync>b__3()	timeout_blind	Failed	17.6257799s	Azure.RequestFailedException			
NuGet.Insights.TimerExecutionServiceTest+TheExecuteAsyncMethod.CanExecuteNewTimerByDefault	10	127.0.0.1:10002POST/devstoreaccount1/t2301104aimvxlsaa1t1/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\Insights\src\Logic\Timers\SpecificTimerExecutionService.cs:111:Azure.Data.Tables;Azure.Data.Tables.TableClient.AddEntityAsync(!!0,System.Threading.CancellationToken):NuGet.Insights.SpecificTimerExecutionService.<>c__DisplayClass9_1.<ExecuteAsync>b__2()	timeout_blind	Failed	19.7820841s	Azure.RequestFailedException			
NuGet.Insights.TimerExecutionServiceTest+TheExecuteAsyncMethod.RunsATimerAgainIfTheFrequencyAllows	10	127.0.0.1:10002POST/devstoreaccount1/t230110dqdy57xypm1t1/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\Insights\src\Logic\Timers\SpecificTimerExecutionService.cs:111:Azure.Data.Tables;Azure.Data.Tables.TableClient.AddEntityAsync(!!0,System.Threading.CancellationToken):NuGet.Insights.SpecificTimerExecutionService.<>c__DisplayClass9_1.<ExecuteAsync>b__2()	timeout_blind	Failed	17.641413s	Azure.RequestFailedException			
NuGet.Insights.TimerExecutionServiceTest+TheExecuteNowAsyncMethod.IgnoresFrequency	6	127.0.0.1:10002POST/devstoreaccount1/t230110xepd7sv2nm1t1/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\Insights\src\Logic\Timers\SpecificTimerExecutionService.cs:111:Azure.Data.Tables;Azure.Data.Tables.TableClient.AddEntityAsync(!!0,System.Threading.CancellationToken):NuGet.Insights.SpecificTimerExecutionService.<>c__DisplayClass9_1.<ExecuteAsync>b__2()	timeout_blind	Failed	16.1100058s	Azure.RequestFailedException			
NuGet.Insights.MutableTableTransactionalBatchTest+TheUpdateMergeEntityMethod.MergesSingleEntity	1	127.0.0.1:10002PATCH/devstoreaccount1/t230110vhlnlnaub41mttb1(PartitionKey='08585282771052055910-52cdvl5lmqweppb64pwacb3m2u',RowKey='a')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\Insights\src\Logic\Storage\MutableTableTransactionalBatch.cs:63:Azure.Data.Tables;Azure.Data.Tables.TableClient.UpdateEntityAsync(!!0,Azure.ETag,Azure.Data.Tables.TableUpdateMode,System.Threading.CancellationToken):NuGet.Insights.MutableTableTransactionalBatch.<>c__DisplayClass8_0`1.<UpdateEntity>b__0(TableClient table)	timeout_blind	Failed	14.4070088s	Azure.RequestFailedException			
NuGet.Insights.MutableTableTransactionalBatchTest+TheDeleteEntityMethod.DeletesSingleEntity	1	127.0.0.1:10002DELETE/devstoreaccount1/t230110s4pw2lxicq1mttb1(PartitionKey='08585282771352760836-uemaot5unlvuxe5jrssslh26oy',RowKey='a')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\Insights\src\Logic\Storage\MutableTableTransactionalBatch.cs:48:Azure.Data.Tables;Azure.Data.Tables.TableClient.DeleteEntityAsync(string,string,Azure.ETag,System.Threading.CancellationToken):NuGet.Insights.MutableTableTransactionalBatch.<>c__DisplayClass7_0.<DeleteEntity>b__0(TableClient table)	timeout_blind	Failed	14.3151447s	Azure.RequestFailedException			
NuGet.Insights.MutableTableTransactionalBatchTest+TheAddEntityMethod.AddsSingleEntity	1	127.0.0.1:10002POST/devstoreaccount1/t230110w4imhlde6m1mttb1/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\Insights\src\Logic\Storage\MutableTableTransactionalBatch.cs:39:Azure.Data.Tables;Azure.Data.Tables.TableClient.AddEntityAsync(!!0,System.Threading.CancellationToken):NuGet.Insights.MutableTableTransactionalBatch.<>c__DisplayClass6_0`1.<AddEntity>b__0(TableClient table)	timeout_blind	Failed	13.7591475s	Azure.RequestFailedException			
NuGet.Insights.MutableTableTransactionalBatchTest+TheAddEntityMethod.AddsMultipleEntities	1	127.0.0.1:10002POST/devstoreaccount1/$batch	C:\Users\yinfang\Insights\src\Logic\Storage\MutableTableTransactionalBatch.cs:117:Azure.Data.Tables;Azure.Data.Tables.TableClient.SubmitTransactionAsync(System.Collections.Generic.IEnumerable`1[Azure.Data.Tables.TableTransactionAction],System.Threading.CancellationToken):NuGet.Insights.MutableTableTransactionalBatch.SubmitBatchAsync(?)	timeout_blind	Failed	14.7036363s	Azure.Data.Tables.TableTransactionFailedException			
NuGet.Insights.StorageLeaseServiceTest+TryReleaseAsync.ReleasesLease	0	127.0.0.1:10000PUT/devstoreaccount1/t2301102vf75tlvvm1l1/some-lease/comp=lease	C:\Users\yinfang\Insights\src\Logic\Leasing\StorageLeaseService.cs:142:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobLeaseClient.ReleaseAsync(Azure.RequestConditions,System.Threading.CancellationToken):NuGet.Insights.StorageLeaseService.TryReleaseAsync(?)	timeout_blind	Failed	16.2985284s	Assertion			
NuGet.Insights.TimerExecutionServiceTest+TheExecuteAsyncMethod.CanExecuteNewTimerByDefault	2	127.0.0.1:10000PUT/devstoreaccount1/t2301102biukf2msy1l1/TimerExecutionService/comp=lease	C:\Users\yinfang\Insights\src\Logic\Leasing\StorageLeaseService.cs:142:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobLeaseClient.ReleaseAsync(Azure.RequestConditions,System.Threading.CancellationToken):NuGet.Insights.StorageLeaseService.TryReleaseAsync(?)	timeout_blind	Failed	17.4693758s	System.InvalidOperationException			
NuGet.Insights.TimerExecutionServiceTest+TheExecuteAsyncMethod.ExecutesTimerEnabledForStorage	2	127.0.0.1:10000PUT/devstoreaccount1/t2301104zo6lkwfsu1l1/TimerExecutionService/comp=lease	C:\Users\yinfang\Insights\src\Logic\Leasing\StorageLeaseService.cs:142:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobLeaseClient.ReleaseAsync(Azure.RequestConditions,System.Threading.CancellationToken):NuGet.Insights.StorageLeaseService.TryReleaseAsync(?)	timeout_blind	Failed	18.2664068s	System.InvalidOperationException			
NuGet.Insights.TimerExecutionServiceTest+TheExecuteAsyncMethod.RunsATimerAgainIfTheFrequencyAllows	2	127.0.0.1:10000PUT/devstoreaccount1/t230110i7gysdxmxa1l1/TimerExecutionService/comp=lease	C:\Users\yinfang\Insights\src\Logic\Leasing\StorageLeaseService.cs:142:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobLeaseClient.ReleaseAsync(Azure.RequestConditions,System.Threading.CancellationToken):NuGet.Insights.StorageLeaseService.TryReleaseAsync(?)	timeout_blind	Failed	17.5005331s	System.InvalidOperationException

IronPigeon (~3 min)

Run python driver.py -e ironpigeon

After the tests finish, there will be a folder ~\rainmaker\results\IronPigeon-injection-round_XXXX.XX.XX.XX.XX.XX created with a bug_inspection.csv file inside.

The bug_inspection.csv contains the following one record indicating the one bug founded, e.g.:

Providers.AzureBlobStorageSubdirectoryTests.PurgeBlobsExpiringBeforeAsync	1	127.0.0.1:10000DELETE/devstoreaccount1/unittests75b20167-2d15-462a-b5af-8346bc8eada7/subdir%2F2023.01.11%2Fh9f2hws6VAwOwQX	C:\Users\yinfang\IronPigeon\src\IronPigeon\Providers\AzureBlobStorage.cs:125:Azure.Storage.Blobs;Azure.Storage.Blobs.BlobContainerClient.DeleteBlobAsync(string,Azure.Storage.Blobs.Models.DeleteSnapshotsOption,Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):IronPigeon.Providers.AzureBlobStorage.<>c__DisplayClass9_0.<PurgeBlobsExpiringBeforeAsync>b__0(BlobItem blob)	timeout_blind	Failed	12.6565605s	Azure.RequestFailedException

Orleans (Figure 4; ~85 min)

Run python driver.py -e orleans

After the tests finish, there will be two folders ~\rainmaker\results\Orleans-injection-round_XXXX.XX.XX.XX.XX.XX created with bug_inspection.csv files inside.

One bug_inspection.csv contains seven records, and the other one contains 23 alarms using a different policy, e.g.:

UnitTests.RemindersTest.AzureRemindersTableTests.RemindersTable_Azure_RemindersParallelUpsert	1	127.0.0.1:10002POST/devstoreaccount1/$batch	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:483:Azure.Data.Tables;Azure.Data.Tables.TableClient.SubmitTransactionAsync(System.Collections.Generic.IEnumerable`1[Azure.Data.Tables.TableTransactionAction],System.Threading.CancellationToken):Orleans.Reminders.AzureStorage.AzureTableDataManager`1.DeleteTableEntriesAsync[](?)	timeout_blind	Failed	23.5009078s	Azure.Data.Tables.TableTransactionFailedException		
Tester.AzureUtils.AzureMembershipTableTests.MembershipTable_Azure_Init	2	127.0.0.1:10002POST/devstoreaccount1/$batch	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:483:Azure.Data.Tables;Azure.Data.Tables.TableClient.SubmitTransactionAsync(System.Collections.Generic.IEnumerable`1[Azure.Data.Tables.TableTransactionAction],System.Threading.CancellationToken):Orleans.Clustering.AzureStorage.AzureTableDataManager`1.DeleteTableEntriesAsync[](?)	timeout_blind	Failed	14.5630498s	System.AggregateException		
Tester.AzureUtils.TimerTests.ReminderTests_AzureTable.Rem_Azure_Basic	1	127.0.0.1:10002DELETE/devstoreaccount1/OrleansReminders(PartitionKey='e1929f9fcce741f29383b7e608674cea_1B636C75',RowKey='cmVtaW5kZXJ0ZXN0Z3JhaW4y_YWE0OGYxMzQ3ZmNiNGI5ZGJhOTlhMmNkZjFhOTQ1N2E%3D-DEFAULT_REMINDER')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:367:Azure.Data.Tables;Azure.Data.Tables.TableClient.DeleteEntityAsync(string,string,Azure.ETag,System.Threading.CancellationToken):Orleans.Reminders.AzureStorage.AzureTableDataManager`1.DeleteTableEntryAsync[](?)	timeout_blind	Failed	57.7991138s	Orleans.Runtime.ReminderException		
Tester.AzureUtils.AzureQueueDataManagerTests.AQ_Standalone_1	5	127.0.0.1:10001DELETE/devstoreaccount1/test-1-28ddbcac-68b6-4d87-81ea-8caf3ab5a1b1-aqdatamanagertests/messages/783962d9-744e-4384-b4d6-14e9b00b3cb2/popreceipt=sCvVYmby2ggBAAAA	C:\Users\yinfang\orleans\src\Azure\Orleans.Streaming.AzureStorage\Storage\AzureQueueDataManager.cs:308:Azure.Storage.Queues;Azure.Storage.Queues.QueueClient.DeleteMessageAsync(string,string,System.Threading.CancellationToken):Orleans.AzureUtils.AzureQueueDataManager.DeleteQueueMessage(?)	timeout_blind	Failed	14.0004896s	System.AggregateException		
Tester.AzureUtils.AzureQueueDataManagerTests.AQ_Standalone_1	4	127.0.0.1:10001GET/devstoreaccount1/test-1-fb307594-9182-49c1-b9fc-a9dec3fdcc09-aqdatamanagertests/messages/numofmessages=1	C:\Users\yinfang\orleans\src\Azure\Orleans.Streaming.AzureStorage\Storage\AzureQueueDataManager.cs:253:Azure.Storage.Queues;Azure.Storage.Queues.QueueClient.ReceiveMessagesAsync(System.Nullable`1[int],System.Nullable`1[System.TimeSpan],System.Threading.CancellationToken):Orleans.AzureUtils.AzureQueueDataManager.GetQueueMessage(?)	timeout_blind	Failed	13.1567371s	System.NullReferenceException		
Tester.AzureUtils.AzureQueueDataManagerTests.AQ_Standalone_1	2	127.0.0.1:10001POST/devstoreaccount1/test-1-392f621e-0bd6-48db-a572-8c93b9c70ddc-aqdatamanagertests/messages	C:\Users\yinfang\orleans\src\Azure\Orleans.Streaming.AzureStorage\Storage\AzureQueueDataManager.cs:202:Azure.Storage.Queues;Azure.Storage.Queues.QueueClient.SendMessageAsync(string):Orleans.AzureUtils.AzureQueueDataManager.AddQueueMessage(?)	timeout_blind	Failed	12.9849566s	Assertion		
Tester.AzureUtils.Lease.AzureBlobLeaseProviderTests.ProviderCanReleaseLeases	3	127.0.0.1:10000PUT/devstoreaccount1/test-blob-container-name/testleasecategory-6ae1c5a8-3fdc-45e7-a983-e09c82997c90.json/comp=lease	C:\Users\yinfang\orleans\src\Azure\Orleans.Streaming.AzureStorage\Providers\Lease\AzureBlobLeaseProvider.cs:97:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobLeaseClient.ReleaseAsync(Azure.RequestConditions,System.Threading.CancellationToken):Orleans.LeaseProviders.AzureBlobLeaseProvider.Release(String category, AcquiredLease acquiredLease)	timeout_blind	Failed	15.3912006s	Azure.RequestFailedException		

Tester.AzureUtils.TimerTests.ReminderTests_AzureTable.Rem_Azure_Basic_Restart	2	127.0.0.1:10002GET/devstoreaccount1/OrleansReminders()/$filter=(PartitionKey gt '2b2c58d2058548cdba2543f81462cc2f_') and (PartitionKey lt '2b2c58d2058548cdba2543f81462cc2f`')$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:517:Azure.Data.Tables;Azure.Data.Tables.TableRestClient.QueryEntitiesAsync(string,System.Nullable`1[int],string,string,Azure.Data.Tables.Models.QueryOptions,System.Threading.CancellationToken):Orleans.Reminders.AzureStorage.AzureTableDataManager`1.<>c__DisplayClass31_0.<ReadTableEntriesAndEtagsAsync>b__0[](?)	request_block	Failed	1m 12.9247938s	Assertion			
Tester.AzureUtils.Streaming.AQStreamingTests.AQ_11_ManySame_ManyProducerGrainsManyConsumerClients	7	127.0.0.1:10002GET/devstoreaccount1/OrleansGrainState(PartitionKey='517fb8836bac4168875a35a3b8ae99a4_cHVic3VicmVuZGV6dm91cw%3D%3D_QXp1cmVRdWV1ZVByb3ZpZGVyL251bGwvNzE5MDUwZjJlMDY5NDBiOWIzNjI3MDg0NjE4NjUxMTg%3D',RowKey='Orleans.Streams.PubSubRendezvousGrain')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:403:Azure.Data.Tables;Azure.Data.Tables.TableClient.GetEntityAsync(string,string,System.Collections.Generic.IEnumerable`1[string],System.Threading.CancellationToken):Orleans.Persistence.AzureStorage.AzureTableDataManager`1.ReadSingleTableEntryAsync[](?)	request_block	Failed	47.8451609s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Persistence.PersistenceGrainTests_AzureTableGrainStorage.Grain_AzureTableGrainStorage_Delete	4	127.0.0.1:10002GET/devstoreaccount1/OrleansGrainState(PartitionKey='6aae139371c24a838cfaab5f4acc4626_Z3JhaW5zdG9yYWdldGVzdA%3D%3D_NjYzNjc2YmMyNzQ3NDMwOThhMDNmOTdlZmRkOWJhY2Q%3D',RowKey='UnitTests.Grains.GrainStorageTestGrain')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:403:Azure.Data.Tables;Azure.Data.Tables.TableClient.GetEntityAsync(string,string,System.Collections.Generic.IEnumerable`1[string],System.Threading.CancellationToken):Orleans.Persistence.AzureStorage.AzureTableDataManager`1.ReadSingleTableEntryAsync[](?)	request_block	Failed	1m 43.8633905s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Persistence.PersistenceGrainTests_AzureTableGrainStorage.Grain_GuidKeyExtended_AzureTableGrainStorage_Read_Write	4	127.0.0.1:10002GET/devstoreaccount1/OrleansGrainState(PartitionKey='854bea6d216c4e6a8246e01bb7f892f4_Z3JhaW5zdG9yYWdldGVzdGdyYWluZXh0ZW5kZWRrZXk%3D_NmUwOGJmMTBlMjllNDRkYThkNDBjZjI3ZTJkMTAzMDYrMzY0MzYxMzYx',RowKey='UnitTests.Grains.GrainStorageTestGrainExtendedKey')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:403:Azure.Data.Tables;Azure.Data.Tables.TableClient.GetEntityAsync(string,string,System.Collections.Generic.IEnumerable`1[string],System.Threading.CancellationToken):Orleans.Persistence.AzureStorage.AzureTableDataManager`1.ReadSingleTableEntryAsync[](?)	request_block	Failed	1m 56.1135363s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Streaming.AQStreamingTests.AQ_06_ManyDifferent_ManyProducerGrainManyConsumerClients	7	127.0.0.1:10002GET/devstoreaccount1/OrleansGrainState(PartitionKey='b6ba1e68b1904965a1b460c7c83c2832_cHVic3VicmVuZGV6dm91cw%3D%3D_QXp1cmVRdWV1ZVByb3ZpZGVyL251bGwvZjdhMzZkMTExNDk3NDQ2NDhiMDgzYTQ3ODg4MWUxMjk%3D',RowKey='Orleans.Streams.PubSubRendezvousGrain')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:403:Azure.Data.Tables;Azure.Data.Tables.TableClient.GetEntityAsync(string,string,System.Collections.Generic.IEnumerable`1[string],System.Threading.CancellationToken):Orleans.Persistence.AzureStorage.AzureTableDataManager`1.ReadSingleTableEntryAsync[](?)	request_block	Failed	43.2513077s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.TimerTests.ReminderTests_AzureTable.Rem_Azure_Basic_Restart	1	127.0.0.1:10002DELETE/devstoreaccount1/OrleansReminders(PartitionKey='34a3448892bd4a17a62c481d16147b8c_468709C7',RowKey='cmVtaW5kZXJ0ZXN0Z3JhaW4y_NDVmNjgzYjRkMmYzNDIwOGE2MjIzOTk3MDY3ZjYwOWU%3D-DEFAULT_REMINDER')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:367:Azure.Data.Tables;Azure.Data.Tables.TableClient.DeleteEntityAsync(string,string,Azure.ETag,System.Threading.CancellationToken):Orleans.Reminders.AzureStorage.AzureTableDataManager`1.DeleteTableEntryAsync[](?)	request_block	Failed	1m 2.096186s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Persistence.PersistenceGrainTests_AzureTableGrainStorage.Grain_AzureTableGrainStorage_Delete	2	127.0.0.1:10002DELETE/devstoreaccount1/OrleansGrainState(PartitionKey='b61d13a2036245cf8c5404b7b920c1bb_Z3JhaW5zdG9yYWdldGVzdA%3D%3D_Zjc1YTg2MmVjNGNkNDhlMzhhYTY1OWVmYjA5OTY5OWU%3D',RowKey='UnitTests.Grains.GrainStorageTestGrain')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:367:Azure.Data.Tables;Azure.Data.Tables.TableClient.DeleteEntityAsync(string,string,Azure.ETag,System.Threading.CancellationToken):Orleans.Persistence.AzureStorage.AzureTableDataManager`1.DeleteTableEntryAsync[](?)	request_block	Failed	1m 50.3324845s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Persistence.PersistenceGrainTests_AzureTableGrainStorage.Grain_GuidKeyExtended_AzureTableGrainStorage_Read_Write	2	127.0.0.1:10002PUT/devstoreaccount1/OrleansGrainState(PartitionKey='2a7e25bfc46b4f4f9ef24209451121b9_Z3JhaW5zdG9yYWdldGVzdGdyYWluZXh0ZW5kZWRrZXk%3D_YzZmODEwNmNhY2U4NGY3ZGI4NDQ5ZmRkNjY1YmE2MWMrMzI4MTU4MTQy',RowKey='UnitTests.Grains.GrainStorageTestGrainExtendedKey')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:322:Azure.Data.Tables;Azure.Data.Tables.TableClient.UpdateEntityAsync(!!0,Azure.ETag,Azure.Data.Tables.TableUpdateMode,System.Threading.CancellationToken):Orleans.Persistence.AzureStorage.AzureTableDataManager`1.UpdateTableEntryAsync[](?)	request_block	Failed	1m 44.9412445s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Streaming.AQStreamingTests.AQ_11_ManySame_ManyProducerGrainsManyConsumerClients	6	127.0.0.1:10002PUT/devstoreaccount1/OrleansGrainState(PartitionKey='8a459f2d776242f595488cceae2cdebc_cHVic3VicmVuZGV6dm91cw%3D%3D_QXp1cmVRdWV1ZVByb3ZpZGVyL251bGwvYjViZDg4ZWM5OTg0NDAxOWI3NTZjMDRlZDNjNDE3NGI%3D',RowKey='Orleans.Streams.PubSubRendezvousGrain')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:322:Azure.Data.Tables;Azure.Data.Tables.TableClient.UpdateEntityAsync(!!0,Azure.ETag,Azure.Data.Tables.TableUpdateMode,System.Threading.CancellationToken):Orleans.Persistence.AzureStorage.AzureTableDataManager`1.UpdateTableEntryAsync[](?)	request_block	Failed	48.7986341s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Streaming.AQStreamingTests.AQ_06_ManyDifferent_ManyProducerGrainManyConsumerClients	6	127.0.0.1:10002PUT/devstoreaccount1/OrleansGrainState(PartitionKey='9466551e9fc34685bf1d2a17dda9668e_cHVic3VicmVuZGV6dm91cw%3D%3D_QXp1cmVRdWV1ZVByb3ZpZGVyL251bGwvZDE4ZjlhMGI4MWU0NDRkN2JmN2FjYmUzNTY4MjgyZWU%3D',RowKey='Orleans.Streams.PubSubRendezvousGrain')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:322:Azure.Data.Tables;Azure.Data.Tables.TableClient.UpdateEntityAsync(!!0,Azure.ETag,Azure.Data.Tables.TableUpdateMode,System.Threading.CancellationToken):Orleans.Persistence.AzureStorage.AzureTableDataManager`1.UpdateTableEntryAsync[](?)	request_block	Failed	45.033134s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.TimerTests.ReminderTests_AzureTable.Rem_Azure_Basic_Restart	0	127.0.0.1:10002PATCH/devstoreaccount1/OrleansReminders(PartitionKey='bf70b4c3e68b43ca88cd515770da340d_A3A93927',RowKey='cmVtaW5kZXJ0ZXN0Z3JhaW4y_NzhhZjJmZWFmYWUxNDJkNmI3MzM0NWFmZjc2ZmZmNWU%3D-DEFAULT_REMINDER')/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:201:Azure.Data.Tables;Azure.Data.Tables.TableClient.UpsertEntityAsync(!!0,Azure.Data.Tables.TableUpdateMode,System.Threading.CancellationToken):Orleans.Reminders.AzureStorage.AzureTableDataManager`1.UpsertTableEntryAsync[](?)	request_block	Failed	38.1734675s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Persistence.PersistenceGrainTests_AzureTableGrainStorage.Grain_AzureTableGrainStorage_Delete	1	127.0.0.1:10002POST/devstoreaccount1/OrleansGrainState/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:171:Azure.Data.Tables;Azure.Data.Tables.TableClient.AddEntityAsync(!!0,System.Threading.CancellationToken):Orleans.Persistence.AzureStorage.AzureTableDataManager`1.CreateTableEntryAsync[](?)	request_block	Failed	1m 41.4259264s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Persistence.PersistenceGrainTests_AzureTableGrainStorage.Grain_GuidKeyExtended_AzureTableGrainStorage_Read_Write	1	127.0.0.1:10002POST/devstoreaccount1/OrleansGrainState/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:171:Azure.Data.Tables;Azure.Data.Tables.TableClient.AddEntityAsync(!!0,System.Threading.CancellationToken):Orleans.Persistence.AzureStorage.AzureTableDataManager`1.CreateTableEntryAsync[](?)	request_block	Failed	1m 40.0508219s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Streaming.AQStreamingTests.AQ_06_ManyDifferent_ManyProducerGrainManyConsumerClients	5	127.0.0.1:10002POST/devstoreaccount1/OrleansGrainState/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:171:Azure.Data.Tables;Azure.Data.Tables.TableClient.AddEntityAsync(!!0,System.Threading.CancellationToken):Orleans.Persistence.AzureStorage.AzureTableDataManager`1.CreateTableEntryAsync[](?)	request_block	Failed	44.0640901s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Streaming.AQStreamingTests.AQ_11_ManySame_ManyProducerGrainsManyConsumerClients	5	127.0.0.1:10002POST/devstoreaccount1/OrleansGrainState/$format=application/json;odata=minimalmetadata	C:\Users\yinfang\orleans\src\Azure\Shared\Storage\AzureTableDataManager.cs:171:Azure.Data.Tables;Azure.Data.Tables.TableClient.AddEntityAsync(!!0,System.Threading.CancellationToken):Orleans.Persistence.AzureStorage.AzureTableDataManager`1.CreateTableEntryAsync[](?)	request_block	Failed	43.2046837s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Streaming.AQStreamingTests.AQ_11_ManySame_ManyProducerGrainsManyConsumerClients	4	127.0.0.1:10001DELETE/devstoreaccount1/testcluster-2023-01-10t03-49-40-422-0/messages/e3f6d882-c1ea-4218-a4da-472684df0938/popreceipt=0KK/z73y2ggBAAAA	C:\Users\yinfang\orleans\src\Azure\Orleans.Streaming.AzureStorage\Storage\AzureQueueDataManager.cs:308:Azure.Storage.Queues;Azure.Storage.Queues.QueueClient.DeleteMessageAsync(string,string,System.Threading.CancellationToken):Orleans.AzureUtils.AzureQueueDataManager.DeleteQueueMessage(?)	request_block	Failed	1m 42.6805352s	Assertion			
Tester.AzureUtils.Streaming.AQStreamingTests.AQ_06_ManyDifferent_ManyProducerGrainManyConsumerClients	2	127.0.0.1:10001POST/devstoreaccount1/testcluster-2023-01-10t04-43-14-964-1/messages	C:\Users\yinfang\orleans\src\Azure\Orleans.Streaming.AzureStorage\Storage\AzureQueueDataManager.cs:202:Azure.Storage.Queues;Azure.Storage.Queues.QueueClient.SendMessageAsync(string):Orleans.AzureUtils.AzureQueueDataManager.AddQueueMessage(?)	request_block	Failed	40.1579757s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Streaming.AQStreamingTests.AQ_11_ManySame_ManyProducerGrainsManyConsumerClients	2	127.0.0.1:10001POST/devstoreaccount1/testcluster-2023-01-10t03-47-26-966-1/messages	C:\Users\yinfang\orleans\src\Azure\Orleans.Streaming.AzureStorage\Storage\AzureQueueDataManager.cs:202:Azure.Storage.Queues;Azure.Storage.Queues.QueueClient.SendMessageAsync(string):Orleans.AzureUtils.AzureQueueDataManager.AddQueueMessage(?)	request_block	Failed	41.0640832s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Lease.AzureBlobLeaseProviderTests.ProviderCanReleaseLeases	2	127.0.0.1:10000PUT/devstoreaccount1/test-blob-container-name/testleasecategory-2eb14ceb-281c-40ce-b3aa-1f26648380d3.json/comp=lease	C:\Users\yinfang\orleans\src\Azure\Orleans.Streaming.AzureStorage\Providers\Lease\AzureBlobLeaseProvider.cs:64:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobLeaseClient.AcquireAsync(System.TimeSpan,Azure.RequestConditions,System.Threading.CancellationToken):Orleans.LeaseProviders.AzureBlobLeaseProvider.Acquire(?)	request_block	Failed	39.5921799s	Azure.RequestFailedException			
Tester.AzureUtils.Lease.AzureBlobLeaseProviderTests.ProviderCanReleaseLeases	1	127.0.0.1:10000PUT/devstoreaccount1/test-blob-container-name/testleasecategory-def03dd9-9d43-43a4-a407-d7fa700656dd.json	C:\Users\yinfang\orleans\src\Azure\Orleans.Streaming.AzureStorage\Providers\Lease\AzureBlobLeaseProvider.cs:62:Azure.Storage.Blobs;Azure.Storage.Blobs.BlobClient.UploadAsync(System.IO.Stream,Azure.Storage.Blobs.Models.BlobHttpHeaders,System.Collections.Generic.IDictionary`2[string,string],Azure.Storage.Blobs.Models.BlobRequestConditions,System.IProgress`1[long],System.Nullable`1[Azure.Storage.Blobs.Models.AccessTier],Azure.Storage.StorageTransferOptions,System.Threading.CancellationToken):Orleans.LeaseProviders.AzureBlobLeaseProvider.Acquire(?)	request_block	Failed	42.3245468s	Azure.RequestFailedException			
Tester.AzureUtils.Persistence.PersistenceGrainTests_AzureBlobStore_Json.Grain_AzureBlobStore_Delete_Json	3	127.0.0.1:10000GET/devstoreaccount1/grainstate/UnitTests.Grains.GrainStorageTestGrain-Z3JhaW5zdG9yYWdldGVzdA%3D%3D_MTQ2YzJiNzgwNWZiNDQ4OGEwZDdhMzI0NmZjNGQ1NjM%3D.json	C:\Users\yinfang\orleans\src\Azure\Orleans.Persistence.AzureStorage\Providers\Storage\AzureBlobStorage.cs:62:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadContentAsync():Orleans.Storage.AzureBlobGrainStorage.ReadStateAsync[](?)	request_block	Failed	2m 1.0985127s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Persistence.PersistenceGrainTests_AzureBlobStore_Json.Grain_AzureBlobStore_Delete_Json	1	127.0.0.1:10000PUT/devstoreaccount1/grainstate/UnitTests.Grains.GrainStorageTestGrain-Z3JhaW5zdG9yYWdldGVzdA%3D%3D_ZDQ4YTA0NWUwYWI3NDliOWE5ZTcxYmMwOTdkMTM1OTU%3D.json	C:\Users\yinfang\orleans\src\Azure\Orleans.Persistence.AzureStorage\Providers\Storage\AzureBlobStorage.cs:187:Azure.Storage.Blobs;Azure.Storage.Blobs.BlobClient.UploadAsync(System.BinaryData,Azure.Storage.Blobs.Models.BlobUploadOptions,System.Threading.CancellationToken):Orleans.Storage.AzureBlobGrainStorage.<>c__DisplayClass11_1`1.<WriteStateAndCreateContainerIfNotExists>b__0()	request_block	Failed	1m 57.66059s	Orleans.Serialization.CodecNotFoundException			
Tester.AzureUtils.Persistence.PersistenceGrainTests_AzureBlobStore_Json.Grain_AzureBlobStore_Delete_Json	0	127.0.0.1:10000DELETE/devstoreaccount1/grainstate/UnitTests.Grains.GrainStorageTestGrain-Z3JhaW5zdG9yYWdldGVzdA%3D%3D_ZTM1ZWUxMDlmNzMzNDdkODk1YzNhN2Q4NjhlN2Y0ZTQ%3D.json	C:\Users\yinfang\orleans\src\Azure\Orleans.Persistence.AzureStorage\Providers\Storage\AzureBlobStorage.cs:150:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobBaseClient.DeleteIfExistsAsync(Azure.Storage.Blobs.Models.DeleteSnapshotsOption,Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Orleans.Storage.AzureBlobGrainStorage.<>c__DisplayClass10_0`1.<ClearStateAsync>b__0()	request_block	Failed	2m 9.5521261s	Orleans.Serialization.CodecNotFoundException

Storage (Auzre Storage; ~120 min)

Run python driver.py -e storage

After the tests finish, there will be two folders~\rainmaker\results\storage-injection-round_XXXX.XX.XX.XX.XX.XX created with a bug_inspection.csv files inside.

One bug_inspection.csv contains 28 records indicating the 11 bugs founded, and the other file contains one record indicating another bug found by different injection policy, e.g.:

Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.UserMetadata_List_AlsoReturnsMetadata	4	127.0.0.1:10000GET/devstoreaccount1/itest/restype=containercomp=listdelimiter=/	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureContainerBrowser.cs:30:Azure.Storage.Blobs;Azure.Storage.Blobs.Models.GetBlobsByHierarchyAsyncCollection.GetNextPageAsync(string,System.Nullable`1[int],bool,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureContainerBrowser.ListFolderAsync(?)	timeout_blind	Failed	1m 35.6286816s	Azure.RequestFailedException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.Delete_create_and_delete_doesnt_exist	6	127.0.0.1:10000GET/devstoreaccount1/itest/restype=containercomp=listdelimiter=/	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureContainerBrowser.cs:30:Azure.Storage.Blobs;Azure.Storage.Blobs.Models.GetBlobsByHierarchyAsyncCollection.GetNextPageAsync(string,System.Nullable`1[int],bool,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureContainerBrowser.ListFolderAsync(?)	timeout_blind	Failed	1m 35.7069493s	Azure.RequestFailedException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.List_All_DoesntCrash	6	127.0.0.1:10000GET/devstoreaccount1/itest/restype=containercomp=listdelimiter=/	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureContainerBrowser.cs:30:Azure.Storage.Blobs;Azure.Storage.Blobs.Models.GetBlobsByHierarchyAsyncCollection.GetNextPageAsync(string,System.Nullable`1[int],bool,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureContainerBrowser.ListFolderAsync(?)	timeout_blind	Failed	1m 35.7225137s	Azure.RequestFailedException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.Exists_existing_blob_returns_true	3	127.0.0.1:10000GET/devstoreaccount1/itest/restype=containercomp=listdelimiter=/	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureContainerBrowser.cs:30:Azure.Storage.Blobs;Azure.Storage.Blobs.Models.GetBlobsByHierarchyAsyncCollection.GetNextPageAsync(string,System.Nullable`1[int],bool,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureContainerBrowser.ListFolderAsync(?)	timeout_blind	Failed	1m 35.519366s	Azure.RequestFailedException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.GetBlob_Root_valid_returns_some	5	127.0.0.1:10000GET/devstoreaccount1/itest/restype=containercomp=listdelimiter=/	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureContainerBrowser.cs:30:Azure.Storage.Blobs;Azure.Storage.Blobs.Models.GetBlobsByHierarchyAsyncCollection.GetNextPageAsync(string,System.Nullable`1[int],bool,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureContainerBrowser.ListFolderAsync(?)	timeout_blind	Failed	1m 35.7692982s	Azure.RequestFailedException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.GetBlob_for_one_file_succeeds	3	127.0.0.1:10000GET/devstoreaccount1/itest/restype=containercomp=listdelimiter=/	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureContainerBrowser.cs:30:Azure.Storage.Blobs;Azure.Storage.Blobs.Models.GetBlobsByHierarchyAsyncCollection.GetNextPageAsync(string,System.Nullable`1[int],bool,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureContainerBrowser.ListFolderAsync(?)	timeout_blind	Failed	1m 35.910065s	Azure.RequestFailedException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.GetBlob_for_one_file_succeeds	2	127.0.0.1:10000GET/devstoreaccount1/itest/restype=container	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:549:Azure.Storage.Blobs;Azure.Storage.Blobs.BlobContainerClient.GetPropertiesAsync(Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.GetPartsAsync(?)	timeout_blind	Failed	1m 35.5506541s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.UserMetadata_List_AlsoReturnsMetadata	3	127.0.0.1:10000GET/devstoreaccount1/itest/restype=container	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:549:Azure.Storage.Blobs;Azure.Storage.Blobs.BlobContainerClient.GetPropertiesAsync(Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.GetPartsAsync(?)	timeout_blind	Failed	1m 35.5323889s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.Open_copy_to_memory_stream_succeeds	3	127.0.0.1:10000GET/devstoreaccount1/itest/restype=container	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:549:Azure.Storage.Blobs;Azure.Storage.Blobs.BlobContainerClient.GetPropertiesAsync(Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.GetPartsAsync(?)	timeout_blind	Failed	1m 35.535095s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.List_All_DoesntCrash	5	127.0.0.1:10000GET/devstoreaccount1/itest/restype=container	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:549:Azure.Storage.Blobs;Azure.Storage.Blobs.BlobContainerClient.GetPropertiesAsync(Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.GetPartsAsync(?)	timeout_blind	Failed	1m 35.6910935s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.GetBlob_Root_valid_returns_some	4	127.0.0.1:10000GET/devstoreaccount1/itest/restype=container	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:549:Azure.Storage.Blobs;Azure.Storage.Blobs.BlobContainerClient.GetPropertiesAsync(Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.GetPartsAsync(?)	timeout_blind	Failed	1m 35.5196505s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.Exists_existing_blob_returns_true	2	127.0.0.1:10000GET/devstoreaccount1/itest/restype=container	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:549:Azure.Storage.Blobs;Azure.Storage.Blobs.BlobContainerClient.GetPropertiesAsync(Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.GetPartsAsync(?)	timeout_blind	Failed	1m 35.6084837s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.Delete_create_and_delete_doesnt_exist	5	127.0.0.1:10000GET/devstoreaccount1/itest/restype=container	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:549:Azure.Storage.Blobs;Azure.Storage.Blobs.BlobContainerClient.GetPropertiesAsync(Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.GetPartsAsync(?)	timeout_blind	Failed	1m 35.5192954s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.Exists_existing_blob_returns_true	1	127.0.0.1:10000HEAD/devstoreaccount1/itest/5c8017cd-73ef-4f3d-9494-02908a90e0cd	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:500:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobBaseClient.GetPropertiesAsync(Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.ExistsAsync(?)	timeout_blind	Failed	1m 35.5194569s	Azure.RequestFailedException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.UserMetadata_List_AlsoReturnsMetadata	2	127.0.0.1:10000HEAD/devstoreaccount1/itest/c960b59a-d866-492c-b909-93083ce54a6b	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:500:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobBaseClient.GetPropertiesAsync(Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.ExistsAsync(?)	timeout_blind	Failed	1m 35.5350607s	Azure.RequestFailedException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.Delete_create_and_delete_doesnt_exist	2	127.0.0.1:10000GET/devstoreaccount1/itest/restype=containercomp=listprefix=c1722582-7d49-4afc-9690-223181d448dc	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:478:Azure.Storage.Blobs;Azure.Storage.Blobs.Models.GetBlobsAsyncCollection.GetNextPageAsync(string,System.Nullable`1[int],bool,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.DeleteAsync(?)	timeout_blind	Failed	1m 35.6130161s	Azure.RequestFailedException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.List_All_DoesntCrash	1	127.0.0.1:10000GET/devstoreaccount1/comp=listinclude=metadata	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:413:Azure.Storage.Blobs;Azure.Storage.Blobs.Models.GetBlobContainersAsyncCollection.GetNextPageAsync(string,System.Nullable`1[int],bool,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.ListContainersAsync(?)	timeout_blind	Failed	1m 35.7380035s	Azure.RequestFailedException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.GetBlob_for_one_file_succeeds	1	127.0.0.1:10000HEAD/devstoreaccount1/itest/beb06823-cfd8-493c-8fac-0d18ea4fdce0	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:393:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobBaseClient.GetPropertiesAsync(Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.GetBlobAsync(?)	timeout_blind	Failed	1m 35.6284979s	Azure.RequestFailedException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.GetBlob_Root_valid_returns_some	0	127.0.0.1:10000GET/devstoreaccount1/itest/restype=container	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:384:Azure.Storage.Blobs;Azure.Storage.Blobs.BlobContainerClient.GetPropertiesAsync(Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.GetBlobAsync(?)	timeout_blind	Failed	1m 35.6911283s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.UserMetadata_List_AlsoReturnsMetadata	1	127.0.0.1:10000PUT/devstoreaccount1/itest/7e7835b3-7d7e-4654-baed-588508fb1eef/comp=metadata	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:369:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobBaseClient.SetMetadataAsync(System.Collections.Generic.IDictionary`2[string,string],Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.SetBlobAsync(?)	timeout_blind	Failed	1m 35.6131434s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.GetBlob_for_one_file_succeeds	0	127.0.0.1:10000PUT/devstoreaccount1/itest/93cb6b50-47c6-45e2-bd47-61822a65b769	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:147:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlockBlobClient.UploadAsync(System.IO.Stream,Azure.Storage.Blobs.Models.BlobHttpHeaders,System.Collections.Generic.IDictionary`2[string,string],Azure.Storage.Blobs.Models.BlobRequestConditions,System.Nullable`1[Azure.Storage.Blobs.Models.AccessTier],System.IProgress`1[long],System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.WriteAsync(?)	timeout_blind	Failed	1m 35.5351076s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.Open_copy_to_memory_stream_succeeds	2	127.0.0.1:10000PUT/devstoreaccount1/itest/c9a2b733-e3be-4fd2-82cf-ce79d8be1db2	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:147:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlockBlobClient.UploadAsync(System.IO.Stream,Azure.Storage.Blobs.Models.BlobHttpHeaders,System.Collections.Generic.IDictionary`2[string,string],Azure.Storage.Blobs.Models.BlobRequestConditions,System.Nullable`1[Azure.Storage.Blobs.Models.AccessTier],System.IProgress`1[long],System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.WriteAsync(?)	timeout_blind	Failed	1m 35.5663436s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.Exists_existing_blob_returns_true	0	127.0.0.1:10000PUT/devstoreaccount1/itest/fc2a83b3-3b09-4225-bafd-d288c4b3ec26	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:147:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlockBlobClient.UploadAsync(System.IO.Stream,Azure.Storage.Blobs.Models.BlobHttpHeaders,System.Collections.Generic.IDictionary`2[string,string],Azure.Storage.Blobs.Models.BlobRequestConditions,System.Nullable`1[Azure.Storage.Blobs.Models.AccessTier],System.IProgress`1[long],System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.WriteAsync(?)	timeout_blind	Failed	1m 35.6284526s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.UserMetadata_List_AlsoReturnsMetadata	0	127.0.0.1:10000PUT/devstoreaccount1/itest/c9fe0cbf-b026-4023-beae-56f1c42a8a66	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:147:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlockBlobClient.UploadAsync(System.IO.Stream,Azure.Storage.Blobs.Models.BlobHttpHeaders,System.Collections.Generic.IDictionary`2[string,string],Azure.Storage.Blobs.Models.BlobRequestConditions,System.Nullable`1[Azure.Storage.Blobs.Models.AccessTier],System.IProgress`1[long],System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.WriteAsync(?)	timeout_blind	Failed	1m 35.7226594s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.Delete_create_and_delete_doesnt_exist	0	127.0.0.1:10000PUT/devstoreaccount1/itest/9ff1cb7c-7b73-4f3d-b072-4d988dfdfb50	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:147:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlockBlobClient.UploadAsync(System.IO.Stream,Azure.Storage.Blobs.Models.BlobHttpHeaders,System.Collections.Generic.IDictionary`2[string,string],Azure.Storage.Blobs.Models.BlobRequestConditions,System.Nullable`1[Azure.Storage.Blobs.Models.AccessTier],System.IProgress`1[long],System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.WriteAsync(?)	timeout_blind	Failed	1m 35.5978554s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.Open_copy_to_memory_stream_succeeds	1	127.0.0.1:10000GET/devstoreaccount1/itest/16534f05-58ae-47cc-babd-a67ce64818ff	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:121:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadAsync(System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.OpenReadAsync(?)	timeout_blind	Failed	1m 35.6977699s	System.Xml.XmlException			
Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.Open_copy_to_memory_stream_succeeds	0	127.0.0.1:10000HEAD/devstoreaccount1/itest/eae0a717-b59f-4063-aa41-af5453e8894d	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:117:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlobBaseClient.GetPropertiesAsync(Azure.Storage.Blobs.Models.BlobRequestConditions,System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.OpenReadAsync(?)	timeout_blind	Failed	1m 37.2439637s	Azure.RequestFailedException						

Storage.Net.Tests.Integration.Blobs.AzureEmulatedBlobStorageTest.Exists_existing_blob_returns_true	0	127.0.0.1:10000PUT/devstoreaccount1/itest/c9f445d0-abea-4856-a005-fa874d479d28	C:\Users\yinfang\storage\src\Azure\Storage.Net.Microsoft.Azure.Storage.Blobs\AzureBlobStorage.cs:147:Azure.Storage.Blobs;Azure.Storage.Blobs.Specialized.BlockBlobClient.UploadAsync(System.IO.Stream,Azure.Storage.Blobs.Models.BlobHttpHeaders,System.Collections.Generic.IDictionary`2[string,string],Azure.Storage.Blobs.Models.BlobRequestConditions,System.Nullable`1[Azure.Storage.Blobs.Models.AccessTier],System.IProgress`1[long],System.Threading.CancellationToken):Storage.Net.Microsoft.Azure.Storage.Blobs.AzureBlobStorage.WriteAsync(?)	request_block	Failed	1m 37.2539017s	System.NotSupportedException

EF Core (~25 min)

  1. Close the Torch tool by running ProfOff.ps1 in ~\rainmaker\infra\torch-tool
  2. Start the CosmosDB emulator first (it has already been started in the VM). When encountering any weird problems, please always try to restart/reset the emulator and retry.
  3. Open the Windows system proxy with port specifying at 18081.
  4. Start Torch tool again via ProfOn.ps1 in ~\rainmaker\infra\torch-tool

Run python driver.py -e efcore

After the tests finish, there will be a folder ~\rainmaker\results\efcore-injection-round_XXXX.XX.XX.XX.XX.XX created with a bug_inspection.csv file inside.

The bug_inspection.csv contains the following seven records indicating the seven bugs founded, e.g.:

Microsoft.EntityFrameworkCore.Cosmos.EndToEndCosmosTest.Can_add_update_delete_detached_entity_end_to_end_async	7	127.0.0.1:8081GET/dbs/pUdlAA==/colls/pUdlAPhdsrA=/pkranges	C:\Users\yinfang\efcore\src\EFCore.Cosmos\Storage\Internal\CosmosClientWrapper.cs:899:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.FeedIteratorInlineCore.ReadNextAsync(System.Threading.CancellationToken):Microsoft.EntityFrameworkCore.Cosmos.Storage.Internal.CosmosClientWrapper.DocumentAsyncEnumerable.AsyncEnumerator.MoveNextAsync(?)	request_block	Failed	1m 16.3278278s	Microsoft.Azure.Cosmos.CosmosException			
Microsoft.EntityFrameworkCore.Cosmos.EndToEndCosmosTest.Can_add_update_delete_end_to_end_async	7	127.0.0.1:8081GET/dbs/u5x9AA==/colls/u5x9AOKTUF0=/pkranges	C:\Users\yinfang\efcore\src\EFCore.Cosmos\Storage\Internal\CosmosClientWrapper.cs:899:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.FeedIteratorInlineCore.ReadNextAsync(System.Threading.CancellationToken):Microsoft.EntityFrameworkCore.Cosmos.Storage.Internal.CosmosClientWrapper.DocumentAsyncEnumerable.AsyncEnumerator.MoveNextAsync(?)	request_block	Failed	52.0692867s	Microsoft.Azure.Cosmos.CosmosException			
Microsoft.EntityFrameworkCore.Cosmos.EndToEndCosmosTest.Can_add_update_delete_detached_entity_end_to_end_async	6	127.0.0.1:8081POST/dbs/CosmosEndToEndTest/colls/CustomerContext/docs	C:\Users\yinfang\efcore\src\EFCore.Cosmos\Storage\Internal\CosmosClientWrapper.cs:783:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.FeedIteratorInlineCore.ReadNextAsync(System.Threading.CancellationToken):Microsoft.EntityFrameworkCore.Cosmos.Storage.Internal.CosmosClientWrapper.DocumentEnumerable.Enumerator.MoveNext()	request_block	Failed	1m 15.5644816s	Microsoft.Azure.Cosmos.CosmosException		
Microsoft.EntityFrameworkCore.Cosmos.EndToEndCosmosTest.Can_add_update_delete_end_to_end_async	6	127.0.0.1:8081GET/dbs/CosmosEndToEndTest/colls/CustomerContext/docs/Customer%7C42	C:\Users\yinfang\efcore\src\EFCore.Cosmos\Storage\Internal\CosmosClientWrapper.cs:624:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.ContainerInlineCore.ReadItemStreamAsync(string,Microsoft.Azure.Cosmos.PartitionKey,Microsoft.Azure.Cosmos.ItemRequestOptions,System.Threading.CancellationToken):Microsoft.EntityFrameworkCore.Cosmos.Storage.Internal.CosmosClientWrapper.CreateSingleItemQueryAsync(DbContext _, ValueTuple`4 parameters, CancellationToken cancellationToken)	request_block	Failed	1m 4.7639387s	Microsoft.Azure.Cosmos.CosmosException		
Microsoft.EntityFrameworkCore.Cosmos.EndToEndCosmosTest.Can_add_update_delete_end_to_end_async	5	127.0.0.1:8081DELETE/dbs/CosmosEndToEndTest/colls/CustomerContext/docs/Customer%7C42	C:\Users\yinfang\efcore\src\EFCore.Cosmos\Storage\Internal\CosmosClientWrapper.cs:413:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.ContainerInlineCore.DeleteItemStreamAsync(string,Microsoft.Azure.Cosmos.PartitionKey,Microsoft.Azure.Cosmos.ItemRequestOptions,System.Threading.CancellationToken):Microsoft.EntityFrameworkCore.Cosmos.Storage.Internal.CosmosClientWrapper.DeleteItemOnceAsync(?)	request_block	Failed	1m 11.094677s	System.InvalidOperationException		
Microsoft.EntityFrameworkCore.Cosmos.EndToEndCosmosTest.Can_add_update_delete_end_to_end_async	4	127.0.0.1:8081PUT/dbs/CosmosEndToEndTest/colls/CustomerContext/docs/Customer%7C42	C:\Users\yinfang\efcore\src\EFCore.Cosmos\Storage\Internal\CosmosClientWrapper.cs:350:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.ContainerInlineCore.ReplaceItemStreamAsync(System.IO.Stream,string,Microsoft.Azure.Cosmos.PartitionKey,Microsoft.Azure.Cosmos.ItemRequestOptions,System.Threading.CancellationToken):Microsoft.EntityFrameworkCore.Cosmos.Storage.Internal.CosmosClientWrapper.ReplaceItemOnceAsync(?)	request_block	Failed	1m 4.4414136s	System.InvalidOperationException			
Microsoft.EntityFrameworkCore.Cosmos.EndToEndCosmosTest.Can_add_update_delete_end_to_end_async	3	127.0.0.1:8081GET/dbs/CosmosEndToEndTest/colls/CustomerContext	C:\Users\yinfang\efcore\src\EFCore.Cosmos\Storage\Internal\CosmosClientWrapper.cs:280:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.ContainerInlineCore.CreateItemStreamAsync(System.IO.Stream,Microsoft.Azure.Cosmos.PartitionKey,Microsoft.Azure.Cosmos.ItemRequestOptions,System.Threading.CancellationToken):Microsoft.EntityFrameworkCore.Cosmos.Storage.Internal.CosmosClientWrapper.CreateItemOnceAsync(?)	request_block	Failed	49.8019812s	System.InvalidOperationException

FHIR Server (~30 min)

  1. Close the Torch tool by running ProfOff.ps1 in ~\rainmaker\infra\torch-tool
  2. Start the CosmosDB emulator first (it has already been started in the VM). When encountering any weird problems, please always try to restart/reset the emulator and retry.
  3. Open the Windows system proxy with port specifying at 18081.
  4. Start Torch tool again via ProfOn.ps1 in ~\rainmaker\infra\torch-tool

Run python driver.py -e fhirserver

After the tests finish, there will be a folder ~\rainmaker\results\fhir-server-injection-round_XXXX.XX.XX.XX.XX.XX created with a bug_inspection.csv file inside.

The bug_inspection.csv contains the following 11 records indicating the 11 bugs founded, e.g.:

GivenExportJobDoesNotExist_WhenRequestingExportStatus_ThenServerShouldReturnNotFound	0	127.0.0.1:8081POST/dbs/health/colls/fhirR4/docs	C:\Users\Ze\fhir-server\src\Microsoft.Health.Fhir.CosmosDb\Features\Queries\CosmosQuery.cs:77:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.FeedIteratorInlineCore`1.ReadNextAsync(System.Threading.CancellationToken):Microsoft.Health.Fhir.CosmosDb.Features.Queries.CosmosQuery`1.ExecuteNextAsync[](?)	timeout_injection	Failed	1m 2.494346s    Assertion													
GivenExportJobDoesNotExist_WhenRequestingExportStatus_ThenServerShouldReturnNotFound	1	127.0.0.1:8081GET/dbs/health/colls/fhirR4	C:\Users\Ze\fhir-server\src\Microsoft.Health.Fhir.CosmosDb\Features\Storage\CollectionInitializer.cs:65:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.DatabaseInlineCore.CreateContainerIfNotExistsAsync(string,string,System.Nullable`1[int],Microsoft.Azure.Cosmos.RequestOptions,System.Threading.CancellationToken):Microsoft.Health.Fhir.CosmosDb.Features.Storage.CollectionInitializer.<>c__DisplayClass7_0.<InitializeCollection>b__2(?)	timeout_injection	Failed	41.8393246s	Assertion							
GivenExportJobDoesNotExist_WhenRequestingExportStatus_ThenServerShouldReturnNotFound	12	127.0.0.1:8081GET/dbs/health/colls/fhirR4/sprocs/replaceSingleResource_128E1A9FC2FB21F135F6CB64C8AAD80FC86890047FEEE05A04990B7D3E5C8EF2	C:\Users\Ze\fhir-server\src\Microsoft.Health.Fhir.CosmosDb\Features\Storage\StoredProcedures\StoredProcedureInstaller.cs:33:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.Scripts.ScriptsInlineCore.ReadStoredProcedureAsync(string,Microsoft.Azure.Cosmos.RequestOptions,System.Threading.CancellationToken):Microsoft.Health.Fhir.CosmosDb.Features.Storage.StoredProcedures.StoredProcedureInstaller.ExecuteAsync(?)	timeout_injection	Failed	54.3123858s	Assertion																						
GivenExportJobDoesNotExist_WhenRequestingExportStatus_ThenServerShouldReturnNotFound	2	127.0.0.1:8081GET/dbs/health/colls/fhirR4	C:\Users\Ze\fhir-server\src\Microsoft.Health.Fhir.CosmosDb\Features\Storage\CosmosClientExtensions.cs:20:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.ContainerInlineCore.ReadContainerAsync(Microsoft.Azure.Cosmos.ContainerRequestOptions,System.Threading.CancellationToken):Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosClientExtensions.TryGetContainerAsync(?)	timeout_injection	Failed	38.7684649s	Assertion														
GivenExportJobDoesNotExist_WhenRequestingExportStatus_ThenServerShouldReturnNotFound	3	127.0.0.1:8081GET/dbs/health/colls/fhirR4	C:\Users\Ze\fhir-server\src\Microsoft.Health.Fhir.CosmosDb\Features\Storage\CosmosClientReadWriteTestProvider.cs:28:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.ContainerInlineCore.UpsertItemAsync(!!0,System.Nullable`1[Microsoft.Azure.Cosmos.PartitionKey],Microsoft.Azure.Cosmos.ItemRequestOptions,System.Threading.CancellationToken):Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosClientReadWriteTestProvider.PerformTest(?)	timeout_injection	Failed	44.6689991s	Assertion															
GivenExportJobDoesNotExist_WhenRequestingExportStatus_ThenServerShouldReturnNotFound	4	127.0.0.1:8081GET/dbs/health/colls/fhirR4/docs/__healthcheck__	C:\Users\Ze\fhir-server\src\Microsoft.Health.Fhir.CosmosDb\Features\Storage\CosmosClientReadWriteTestProvider.cs:35:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.ContainerInlineCore.ReadItemAsync(string,Microsoft.Azure.Cosmos.PartitionKey,Microsoft.Azure.Cosmos.ItemRequestOptions,System.Threading.CancellationToken):Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosClientReadWriteTestProvider.PerformTest(?)	timeout_injection	Failed	44.7413621s	Assertion																
GivenExportJobDoesNotExist_WhenRequestingExportStatus_ThenServerShouldReturnNotFound	5	127.0.0.1:8081POST/dbs/health/colls/fhirR4/docs	C:\Users\Ze\fhir-server\src\Microsoft.Health.Fhir.CosmosDb\Features\Storage\CosmosDbDistributedLock.cs:113:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.ContainerInlineCore.CreateItemAsync(!!0,System.Nullable`1[Microsoft.Azure.Cosmos.PartitionKey],Microsoft.Azure.Cosmos.ItemRequestOptions,System.Threading.CancellationToken):Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosDbDistributedLock.TryAcquireLock(?)	timeout_injection	Failed	1m 8.4223522s	Assertion																
GivenExportJobDoesNotExist_WhenRequestingExportStatus_ThenServerShouldReturnNotFound	6	127.0.0.1:8081DELETE/dbs/health/colls/fhirR4/docs/LockDocument:UpgradeLock:2	C:\Users\Ze\fhir-server\src\Microsoft.Health.Fhir.CosmosDb\Features\Storage\CosmosDbDistributedLock.cs:161:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.ContainerInlineCore.DeleteItemAsync(string,Microsoft.Azure.Cosmos.PartitionKey,Microsoft.Azure.Cosmos.ItemRequestOptions,System.Threading.CancellationToken):Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosDbDistributedLock.ReleaseLock(?)	timeout_injection	Failed	1m 3.3462925s	Assertion									
GivenExportJobDoesNotExist_WhenRequestingExportStatus_ThenServerShouldReturnNotFound	7	127.0.0.1:8081POST/dbs/health/colls/fhirR4/docs	C:\Users\Ze\fhir-server\src\Microsoft.Health.Fhir.CosmosDb\Features\Storage\CosmosDbDistributedLock.cs:232:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.ContainerInlineCore.UpsertItemAsync(!!0,System.Nullable`1[Microsoft.Azure.Cosmos.PartitionKey],Microsoft.Azure.Cosmos.ItemRequestOptions,System.Threading.CancellationToken):Microsoft.Health.Fhir.CosmosDb.Features.Storage.CosmosDbDistributedLock.LockKeepAliveLoop(?)	timeout_injection	Failed	1m 22.8287113s	Assertion															
GivenExportJobDoesNotExist_WhenRequestingExportStatus_ThenServerShouldReturnNotFound	8	127.0.0.1:8081GET/dbs/health	C:\Users\Ze\fhir-server\src\Microsoft.Health.Fhir.CosmosDb\Features\Storage\FhirCosmosClientInitializer.cs:131:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.CosmosClient.CreateDatabaseIfNotExistsAsync(string,Microsoft.Azure.Cosmos.ThroughputProperties,Microsoft.Azure.Cosmos.RequestOptions,System.Threading.CancellationToken):Microsoft.Health.Fhir.CosmosDb.Features.Storage.FhirCosmosClientInitializer.<>c__DisplayClass8_0.<InitializeDataStore>b__0(?)	timeout_injection	Failed	39.0765396s	Assertion																
GivenExportJobDoesNotExist_WhenRequestingExportStatus_ThenServerShouldReturnNotFound	9	localhost:8081GET/	C:\Users\Ze\fhir-server\src\Microsoft.Health.Fhir.CosmosDb\Features\Storage\FhirCosmosClientInitializer.cs:84:Microsoft.Azure.Cosmos.Client;Microsoft.Azure.Cosmos.Fluent.CosmosClientBuilder.Build():Microsoft.Health.Fhir.CosmosDb.Features.Storage.FhirCosmosClientInitializer.CreateCosmosClient(CosmosDataStoreConfiguration configuration)	timeout_injection	Failed	37.9451614s	Assertion

Sleet (~15 min)

  1. Add the environment variables "AWS_ACCESS_KEY_ID" and "AWS_ACCESS_KEY_ID" with your AWS keys, and remember to use a new PowerShell to make the environment variables valid;
  2. Add the mockserver certificate to your machine;
  3. Open the Windows system proxy with port specifying at 18081.

Run python driver.py -e sleet

After the tests finish, there will be a folder ~\rainmaker\results\sleet-injection-round_XXXX.XX.XX.XX.XX.XX created with a bug_inspection.csv file inside.

The bug_inspection.csv contains the following two records indicating the two bugs founded, e.g.:

Sleet.AmazonS3.Tests.AmazonS3FileSystemTests.GivenAS3AccountVerifyBucketOperations	1	sleet-test-1f95f06f-7484-4f92-9465-712bcf541a6d.s3.us-east-2.amazonaws.comPUT/	C:\Users\Ze\Sleet\src\SleetLib\FileSystem\AmazonS3FileSystem.cs:122:AWSSDK.S3;Amazon.S3.AmazonS3Client.Amazon.Runtime.SharedInterfaces.ICoreAmazonS3.EnsureBucketExistsAsync(string):Sleet.AmazonS3FileSystem.CreateBucket(?)	timeout_injection	Failed	6.9402689s	Amazon.S3.AmazonS3Exception											
Sleet.AmazonS3.Tests.AmazonS3FileSystemTests.GivenAS3AccountVerifyBucketOperations	3	sleet-test-0808c58b-816c-4ffd-bbba-e7cfceb30d8c.s3.us-east-2.amazonaws.comDELETE/	C:\Users\Ze\Sleet\src\SleetLib\FileSystem\AmazonS3FileSystem.cs:196:AWSSDK.S3;Amazon.S3.AmazonS3Client.DeleteBucketAsync(string,System.Threading.CancellationToken):Sleet.AmazonS3FileSystem.DeleteBucket(?)	timeout_injection	Failed	8.4416305s	Amazon.S3.AmazonS3Exception
Why sometimes there are different numbers of alarm records in the bug_inspection.csv file?
  1. It is because multiple test cases have excercised the same SDK APIs, and we inject faults to the same place in different test rounds. So there are multiple alarm records indicating the same bug. For instance, the Orleans bug reproduction before has 30 records denoting 18 bugs.
  2. Some tests were running nondeterministicly in different environmental settings. They could be flaky tests.

Note that here we try to reproduce the bugs found by Rainmaker with selective tests. To run the fault injection in the full mode (which will take longer time), you can configure the value full_test_or_vanilla as True in the configuration file.

Test plan generation

To generate test plan when vanilla run finishes, use python .\test_planner.py -d <path_to_the_result_dir>.

We have provided the previously collected vanilla result files in the VM, so you can run the following commands to generate the test plans for each application accordingly:

python .\test_planner.py -d "Botbuilder-dotnet_2022.05.06.06.00.32"
python .\test_planner.py -d "DistributedLock_2022.05.13.05.29.54"
python .\test_planner.py -d "Orleans_2022.02.25.00.09.00"
python .\test_planner.py -d "Alpakka_2022.09.13.23.43.12"
python .\test_planner.py -d "ServiceBus.AttachmentPlugin_2022.06.01.15.24.27"
python .\test_planner.py -d "fhir-server_2022.08.14.03.09.23"
python .\test_planner.py -d "Insights_2022.06.12.18.16.23"
python .\test_planner.py -d "IronPigeon_2022.09.13.22.07.03"
python .\test_planner.py -d "sleet_2022.07.25.11.54.00"
python .\test_planner.py -d "storage_2022.06.01.13.24.28"
python .\test_planner.py -d "storage-aws_2022.07.18.13.57.02"

After the command finishes, you will see the test plan outcome (with the estimated running time and the number of test plans) in the PowerShell like this:

Objective 0 (covering all tests in test suite - randomly pick one call site for each test): 2.9411
Number of test plans: 155
Objective 1 (covering unique SDK API call sites): 0.3344
Number of test plans: 47
Objective 2 cover test and unique SDK call site in a pair-wise way: 2.9414
Number of test plans: 155
Objective 3 cover every combination of (test, unique SDK call sites): 18.8992
Number of test plans: 951
Objective 4 cover every combination of (test, SDK call sites): 0.2800
Number of test plans: 99494

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

24 stars

Watchers

2 watching

Forks

1 fork
Report repository

Releases

No releases published

Packages

No packages published

Contributors 3

  •  
  •  
  •  

Languages