Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Chad Vidovcich (GitHub)
- The event processor error handler will now raise warning when an unhandled exception propagated from the event processing handler causing partition processing to fault and restart.
- Fixed an issue with event processor startup validation where an invalid consumer group was not properly detected.
-
EventProcessorClientandBlobCheckpointStorewill now detect when an ownership blob has been deleted externally while the processor is running and gracefully recover. -
Samples now each have a table of contents to help discover and navigate to the topics discussed for a scenario. (A community contribution, courtesy of chadvidovcich)
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Daniel Marbach (GitHub)
- The
BlobCheckpointStoreimplementation used internally by the processor has been made public and now conforms to theCheckpointStorecontract, allowing it to be used with custom processor implementations.
-
Based on a new series of profiling and testing in real-world application scenarios, the default values for processor load balancing are being updated to provide better performance and stability. The default load balancing interval was changed from 10 seconds to 30 seconds. The default ownership expiration interval was changed from 30 seconds to 2 minutes. The default load balancing strategy has been changed from balanced to greedy.
-
Added additional heuristics for the
EventProcessorClientload balancing cycle to help discover issues that can impact processor performance and stability; these validations will produce warnings should potential concerns be found. -
EventProcessorClientwill now log a verbose message indicating what event position was chosen to read from when initializing a partition. -
Removed allocations from Event Source logging by introducing
WriteEventoverloads to handle cases that would otherwise result in boxing toobject[]via params array. (A community contribution, courtesy of danielmarbach) -
Enhanced documentation to call attention to the need for the Azure Storage container used with the processor to exist, and highlight that it will not be implicitly created.
- The
BlobCheckpointStoreimplementation used internally by the processor has been made public and now conforms to theCheckpointStorecontract, allowing it to be used with custom processor implementations.
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Daniel Marbach (GitHub)
-
Removed allocations from Event Source logging by introducing
WriteEventoverloads to handle cases that would otherwise result in boxing toobject[]via params array. (A community contribution, courtesy of danielmarbach) -
Enhanced README documentation to call attention to the need for the Azure Storage container used with the processor to exist, and highlight that it will not be implicitly created.
- Added
FullyQualifiedNamespace,EventHubName, andConsumerGroupto the partition context associated with events dispatched for processing.
- Based on a new series of profiling and testing in real-world application scenarios, the default values for processor load balancing are being updated to provide better performance and stability. The default load balancing interval was changed from 10 seconds to 30 seconds. The default ownership expiration interval was changed from 30 seconds to 2 minutes. The default load balancing strategy has been changed from balanced to greedy.
-
Added additional heuristics for the
EventProcessorClientload balancing cycle to help discover issues that can impact processor performance and stability; these validations will produce warnings should potential concerns be found. -
EventProcessorClientwill now log a verbose message indicating what event position was chosen to read from when initializing a partition.
- Dependencies have been updated to resolve an error when creating
EventSourceinstances when used with Xamarin.
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Andrey Shihov (GitHub)
-
Fixed an issue with refreshing authorization where redundant requests were made to acquire AAD tokens that were due to expire. Refreshes will now coordinate to ensure a single AAD token acquisition.
-
Fixed an issue with authorization refresh where attempts may have been made to authorize a faulted link. Links that fail to open are no longer be considered valid for authorization.
- Documentation has been enhanced to provide additional context for client library types, notably detailing non-obvious validations applied to parameters and options members.
- Fixed an issue where partition processing would ignore cancellation when the processor was shutting down or partition ownership changed and continue dispatching events to the handler until the entire batch was complete. Cancellation will now be properly respected.
- Added the ability to adjust the connection idle timeout using the
EventHubConnectionOptionsavailable within the options for each client type.
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Daniel Marbach (GitHub)
-
When stopping, the
EventProcessorClientwill now attempt to force-close the connection to the Event Hubs service to abort in-process read operations blocked on their timeout. This should significantly help reduce the amount of time the processor takes to stop in many scenarios. (Based on a community prototype contribution, courtesy of danielmarbach) -
When the
EventProcessorClientdetects a partition being stolen outside of a load balancing cycle, it will immediately surrender ownership rather than waiting for a load balancing cycle to confirm the ownership change. This will help reduce event duplication from overlapping ownership of processors. -
The
ConnectionOptionsavailable when creating a processor now support registering a callback delegate for participating in the validation of SSL certificates when connections are established. This delegate may override the built-in validation and allow or deny certificates based on application-specific logic. -
The
ConnectionOptionsavailable when creating a processor now support setting a custom size for the send and receive buffers of the transport.
-
The
EventProcessorClientwill now properly respect another another consumer stealing ownership of a partition when the service forcibly terminates the active link in the background. Previously, the client did not observe the error directly and attempted to recover the faulted link which reasserted ownership and caused the partition to "bounce" between owners until a load balancing cycle completed. -
The
EventProcessorClientwill now be less aggressive when considering whether or not to steal a partition, doing so only when it will correct an imbalance and preferring the status quo when the overall distribution would not change. This will help reduce event duplication due to partitions moving between owners.
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Daniel Marbach (GitHub)
-
When stopping, the
EventProcessorClientwill now attempt to force-close the connection to the Event Hubs service to abort in-process read operations blocked on their timeout. This should significantly help reduce the amount of time the processor takes to stop in many scenarios. (Based on a community prototype contribution, courtesy of danielmarbach) -
When the
EventProcessorClientdetects a partition being stolen outside of a load balancing cycle, it will immediately surrender ownership rather than waiting for a load balancing cycle to confirm the ownership change. This will help reduce event duplication from overlapping ownership of processors. -
The
ConnectionOptionsavailable when creating a processor now support registering a callback delegate for participating in the validation of SSL certificates when connections are established. This delegate may override the built-in validation and allow or deny certificates based on application-specific logic. -
The
ConnectionOptionsavailable when creating a processor now support setting a custom size for the send and receive buffers of the transport.
-
The
EventProcessorClientwill now properly respect another another consumer stealing ownership of a partition when the service forcibly terminates the active link in the background. Previously, the client did not observe the error directly and attempted to recover the faulted link which reasserted ownership and caused the partition to "bounce" between owners until a load balancing cycle completed. -
The
EventProcessorClientwill now be less aggressive when considering whether or not to steal a partition, doing so only when it will correct an imbalance and preferring the status quo when the overall distribution would not change. This will help reduce event duplication due to partitions moving between owners.
-
The processor will now perform validation of core configuration and permissions at startup, in order to attempt to detect unrecoverable problems more deterministically. Validation is non-blocking and will not delay claiming of partitions. One important note is that validation should be considered point-in-time and best effort; it is not meant to replace monitoring of error handler activity.
-
Partition initialization has been moved to a background operation. This will allow partitions to be more efficiently managed and speed up ownership claims, especially when using the
LoadBalancingStrategy.Greedyconfiguration or when the processor is recovering from some error conditions.
-
Dependencies have been updated to resolve security warnings for CVE-2021-26701. (The Event Hubs client library does not make use of the vulnerable components, directly or indirectly)
-
The processor will no longer inappropriately determine that it should attempt to steal partitions from itself or when the load is balanced but there is an uneven ownership distribution. Previously, stealing was attempted but no candidates were found, leading to log spam but no interruption in processing.
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Daniel Marbach (GitHub)
-
The
EventProcessorClientnow supports shared key and shared access signature authentication using theAzureNamedKeyCredentialandAzureSasCredentialtypes in addition to the connection string. Use of the credential allows the shared key or SAS to be updated without the need to create a new processor. -
Multiple enhancements were made to the AMQP transport paths for reading events to reduce memory allocations and increase performance. (A community contribution, courtesy of danielmarbach)
- The AMQP library used for transport has been updated, fixing several issues including a potential unobserved
ObjectDisposedExceptionthat could cause the host process to crash. (see: release notes)
- Updating package bindings for
Azure.Messaging.EventHubsto synchronize on v5.4.0-beta.1.
-
Fixed an issue where long-lived credentials (more than 49 days) were overflowing refresh timer limits and being rejected.
-
Added detection and recovery for a race condition that occurred when the Event Hubs service closed a connection or link after the client had validated its state and was performing an operation; this will now be properly retried with a fresh connection/link.
-
Extended retry scenarios to include generic I/O exceptions, as they are typically transient network failures.
-
Extended retry scenarios to include authorization failures, as the Event Hubs services do not differentiate between authentication and authorization, callers cannot reason about whether an
Unauthorizedreturn from an operation indicates that the call will never succeed or may trigger a credential renewal that may allow success.
-
Additional options for tuning load balancing have been added to the
EventProcessorClientOptions. -
It is now possible to specify a custom endpoint to use for establishing the connection to the Event Hubs service in the
EventHubConnectionOptionsfor the processor. -
Interactions with Blob Storage have been tuned for better performance and more efficient resource use. This will also improve start-up time, especially when using the
Greedyload balancing strategy. -
Errors occurring in the Event Hubs service or active transport are now preserved in full and propagated as an inner exception; this will provide deeper context for diagnosing and troubleshooting exceptions.
-
Documentation used for auto-completion via Intellisense and other tools has been enhanced in many areas, addressing gaps and commonly asked questions.
-
Upgraded the
Microsoft.Azure.Amqplibrary to resolve crashes occurring in .NET 5. -
The calculation for authorization token expiration has been fixed, resulting in fewer token refreshes and network requests.
-
Additional options for tuning load balancing have been added to the
EventProcessorClientOptions. -
Documentation used for auto-completion via Intellisense and other tools has been enhanced in many areas, addressing gaps and commonly asked questions.
-
Upgraded the
Microsoft.Azure.Amqplibrary to resolve crashes occurring in .NET 5. -
The calculation for authorization token expiration has been fixed, resulting in fewer token refreshes and network requests.
- An issue with package publishing which blocked referencing and use has been fixed.
-
The
EventDatarepresentation has been extended with the ability to treat theBodyasBinaryData.BinaryDatasupports a variety of data transformations and allows the ability to provide serialization logic when sending or receiving events. Any type that derives fromObjectSerializer, such asJsonObjectSerializercan be used, with Schema Registry support available via theSchemaRegistryAvroObjectSerializer. -
EventDatahas been integrated with the new Schema Registry service, via use of theObjectSerializerwithBinaryData.
Note: Azure Schema Registry is a new hosted schema repository service provided by Azure Event Hubs, and may not yet be available in all regions or Azure clouds.
- Introduction of an option for the various event consumers allowing the prefetch cache to be filled based on a size-based heuristic rather than a count of events. This feature is considered a special case, helpful in scenarios where the size of events being read is not able to be known or predicted upfront and limiting resource use is valued over consistent and predictable performance.
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Daniel Marbach (GitHub)
-
The approach used for creation of checkpoints has been updated to interact with Azure Blob storage more efficiently. This will yield major performance improvements when soft delete was enabled and minor improvements otherwise.
-
The
EventProcessorClientwill now perform an eager validation of connection strings upon creation. Previously, validation was deferred until a partition was claimed which made debugging difficult. -
Fixed an issue where failure to create an AMQP link would lead to an AMQP session not being explicitly closed, causing connections to the Event Hubs service to remain open until a garbage collection pass was performed.
-
Load balancing will now detect when it has reached a balanced state more accurately; this will allow it to operate more efficiently when
LoadBalancingStrategy.Greedyis in use. -
The
EventProcessorClientnow supports a configurable strategy for load balancing, allowing control over whether it claims ownership of partitions in a balanced manner (default) or more aggressively. The strategy may be set in theEventProcessorClientOptionswhen creating the processor. More details about strategies can be found in the associated documentation. -
The
EventProcessorClientOptionsnow support setting aPrefetchCountandCacheEventCountfor performance tuning. More details about each can be found in the associated documentation. -
Connection strings for each of the clients now supports a
SharedAccessSignaturetoken, allowing a pre-generated SAS to be used for authorization. -
Load balancing now has better recognition for being in a recovery state and will aggressively reclaim partitions for which it is the recognized owner, regardless of whether the current instance made the ownership claim. Previously, those partitions were redistributed on a 1-by-1 basis as part of the standard cycle.
- Bug in TaskExtensions.EnsureCompleted method that causes it to unconditionally throw an exception in the environments with synchronization context
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Daniel Marbach (GitHub)
-
Load balancing will now detect when it has reached a balanced state more accurately; this will allow it to operate more efficiently when
LoadBalancingStrategy.Greedyis in use. -
The approach used for creation of checkpoints has been updated to interact with Azure Blob storage more efficiently. This will yield major performance improvements when soft delete was enabled and minor improvements otherwise.
-
Load balancing now has better recognition for being in a recovery state and will aggressively reclaim partitions for which it is the recognized owner, regardless of whether the current instance made the ownership claim. Previously, those partitions were redistributed on a 1-by-1 basis as part of the standard cycle.
-
The underlying AMQP library has been enhanced for more efficient resource usage, particularly when no events are available; this will result in a noticeable reduction in memory use. (A community contribution, courtesy of danielmarbach)
-
The
EventProcessorClientwill now perform an eager validation of connection strings upon creation. Previously, validation was deferred until a partition was claimed which made debugging difficult. -
Fixed an issue where failure to create an AMQP link would lead to an AMQP session not being explicitly closed, causing connections to the Event Hubs service to remain open until a garbage collection pass was performed.
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Daniel Marbach (GitHub)
-
The
EventProcessorClientnow supports a configurable strategy for load balancing, allowing control over whether it claims ownership of partitions in a balanced manner (default) or more aggressively. The strategy may be set in theEventProcessorClientOptionswhen creating the processor. More details about strategies can be found in the associated documentation. -
The
EventProcessorClientOptionsnow support setting aPrefetchCountandCacheEventCountfor performance tuning. More details about each can be found in the associated documentation.
-
Surfacing of exceptions has been fixed to consistently preserve the stack trace in cases where it was previously lost. (A community contribution, courtesy of (danielmarbach](https://github.com/danielmarbach))
-
A cleanup sweep was performed to tune small areas to be more efficient and perform fewer allocations.
- The set of features from v5.1.0-preview.1 are now generally available. This includes the
EventProcessor<TPartition>andPartitionReceivertypes which focus on advanced application scenarios which require greater low-level control.
- Minor enhancements to reduce allocations and improve efficiency
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- The
EventProcessorClienthas been enhanced to derive from the newEventProcessor<TPartition>primitive, bringing improvements to stability, resilience, and performance.
- Event batches are now protected against modification while publishing is actively taking place.
-
Validation for the Event Hubs fully qualified namespace has been improved, allowing for more deterministic failures when creating clients.
-
Exceptions surfaced will now properly remember their context in all scenarios; previously, some deferred cases unintentionally reset the context.
-
Logging for the Storage Manager for Azure Blobs now follows the common pattern for other Event Hubs types, as well as providing additional context and information.
-
The diagnostic scope for activities will now complete in a more deterministic manner. (A community contribution, courtesy of christothes)
-
Diagnostic activities have been extended with additional information about events being processed and with additional environmental context.
-
Parsing of connection strings is now more permissive for the
Endpointkey, allowing additional formats that result from common mistakes when building the string rather than copying the value from the portal. -
The partition manager has been renamed to
StorageManagerto better represent its purpose. (A community contribution, courtesy of marusyk)
- The tests for load balancing and the Event Processor client have been tuned to remove dependencies on Azure resources and run more efficiently. (A community contribution, courtesy of christothes)
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Christopher Scott (GitHub)
- A migration guide is now available for those moving from the 4.x version of the
Microsoft.Azure.EventHubslibraries to the 5.0.1 version under theAzure.Messaging.EventHubsnamespace.
- Namespaces have been reorganized to align types to their functional area, reducing the number of types in the root namespace and offering better context for where a type is used. Cross-functional types have been left in the root while specialized types were moved to the
Producer,Consumer, orProcessornamespaces.
-
Load balancing has been tuned for better performance and lower resource use. (A community contribution, courtesy of christothes)
-
Reduction of reliance on Azure resources for Event Processor tests. (A community contribution, courtesy of christothes)
-
Logging has been implemented for Event Processor operations interacting with storage. (A community contribution, courtesy of christothes)
-
Logging has been implemented for general Event Processing operations, including background execution.
-
A bug with resuming from a storage checkpoint was fixed, ensuring that processing resumes from the next available event rather than reprocessing the event from which the checkpoint was created.
-
The protected
On[EventName]members have been marked private to reduce the public surface and reduce confusion. They provided no benefit over providing a handler and the cognative cost was not justified.
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Christopher Scott (GitHub)
-
This version is a reorganization of the Event Hubs client library packages, moving the
EventProcessorClientinto its own package and evolving this version into an opinionated implementation on top of Azure Storage Blobs. This is intended to offer a more streamlined developer experience for the majority of scenarios and allow developers to more easily take advantage of the processor. -
A large portion of the public API surface, including members and parameters, have had adjustments to their naming in order to improve discoverability, provide better context to developers, and better conform to the Azure SDK Design Guidelines for .NET.
-
The API for the
EventProcessorClienthas been revised, adopting an event-driven model that aligns to many of the .NET base class library types and reduces complexity when constructing the client. -
The handlers for event processing will now process events on an individual basis, allowing developers to control whether or not they wish to handle events as they're available or batch events before processing.
-
The mechanism for creating checkpoints has been bundled with the event dispatched for processing; this allows a checkpoint to be created with a clear association with a given event, removing the need for developers to explicitly pass an event to a dedicated checkpoint manager.
-
The concept of a plug-in model for the durable storage used by the processor has been removed; this package has taken a strong dependency on Azure Storage Blobs and no longer requires developers to manipulate an abstraction on top of storage.
-
Checkpoints are now ensuring case-insensitivity for metadata to guard against inadvertent creation of multiple checkpoints for the same Event Hub / Consumer Group / Partition combination due to mixed casing.
-
Removed the concept of explicitly closing or disposing the processor; the lifespan of resources is now managed implicitly within the scope of starting and stopping the processor.
-
Synchronous members have been added for starting and stopping the event processor.