Performance limitations

[timdbowden]

Hi there, what are the performance limits on this? We are hitting "Reached maximum number of concurrent runs per pipeline" running from Blob Storage to Azure SQL Server

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: b7ee5166-ff40-f4ed-ce07-5312f935e388
• Version Independent ID: 68f01b43-e3b2-003b-24cb-446a08aa0c1b
• Content: Create event-based triggers in Azure Data Factory
• Content Source: articles/data-factory/how-to-create-event-trigger.md
• Service: data-factory
• GitHub Login: @douglaslMS
• Microsoft Alias: douglasl

[douglaslMS]

@timdbowden Azure Data Factory limits are documented here: Data Factory limits. Maximum concurrent runs per pipeline is currently 100.

[timdbowden]

Does this mean that the event based trigger is not suite to instances where we are receiving 100+ blob per second?

…

On Mon, Oct 1, 2018 at 6:18 PM Douglas Laudenschlager < ***@***.***> wrote: @timdbowden <https://github.com/timdbowden> Azure Data Factory limits are documented here: Data Factory limits <https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#data-factory-limits>. Maximum concurrent runs per pipeline is currently 100. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15909 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFPB77ULt6VoikiPGG_VTwJ29A0GHkfLks5ugk5LgaJpZM4XCPIC> .

[douglaslMS]

Sent question to Data Factory team.

[douglaslMS]

#in-progress

[cesher]

The default Data Factory concurrent pipeline runs limit is set to 100. If you have an event trigger that creates events faster than your pipeline can process than you have a couple of options. One, and the method we recommend, is to change your trigger configuration so that it fires less frequently. Instead of having the trigger fire for each blob created and/or deleted you could instead use a control file that signals when a "batch" of files should be processed. If however, this is not feasible for your scenario and you need to process each file individually, than the only other alternative is to request a pipeline concurrency limit increase from the product team.

[douglaslMS]

#please-close