-
The
EventDatarepresentation has been extended with the ability to treat theBodyasBinaryData.BinaryDatasupports a variety of data transformations and allows the ability to provide serialization logic when sending or receiving events. Any type that derives fromObjectSerializer, such asJsonObjectSerializercan be used, with Schema Registry support available via theSchemaRegistryAvroObjectSerializer. -
EventDatahas been integrated with the new Schema Registry service, via use of theObjectSerializerwithBinaryData. -
When publishing events to Event Hubs, timeouts or other transient failures may introduce ambiguity into the understanding of whether a batch of events was received by the service. To assist in this scenario, the option to publish events idempotently across all retries of a publish operation has been added to the
EventHubProducerClient.
Note: The idempotent publishing feature is new to the Event Hubs service, and Azure Schema Registry is a new hosted schema repository service provided by Azure Event Hubs. Both offerings may not yet be available in all regions or Azure clouds.
- Introduction of an option for the various event consumers allowing the prefetch cache to be filled based on a size-based heuristic rather than a count of events. This feature is considered a special case, helpful in scenarios where the size of events being read is not able to be known or predicted upfront and limiting resource use is valued over consistent and predictable performance.
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Daniel Marbach (GitHub)
-
The underlying AMQP library has been enhanced for more efficient resource usage; this will result in a noticeable reduction in memory use in common consuming scenarios. (A community contribution, courtesy of danielmarbach)
-
All clients will now perform an eager validation of connection strings upon creation. Previously, validation was performed just before a service operation in some scenarios which made debugging difficult.
-
An additional level of resilience was added to some corner case scenarios where establishing an AMQP link failed with what may be a transient issue.
-
Fixed an issue where failure to create an AMQP link would lead to an AMQP session not being explicitly closed, causing connections to the Event Hubs service to remain open until a garbage collection pass was performed.
-
The
EventProcessor<TPartition>now supports a configurable strategy for load balancing, allowing control over whether it claims ownership of partitions in a balanced manner (default) or more aggressively. The strategy may be set in theEventProcessorOptionswhen creating the processor. More details about strategies can be found in the associated documentation. -
The
EventHubConsumerClientpipeline for reading events from a single partition was reworked to improve efficiency and make use of the new configuration options forPrefetchCountandCacheEventCount. -
The
ReadEventOptionsused with theEventHubConsumerClientnow support setting aPrefetchCountandCacheEventCountfor performance tuning. More details about each can be found in the associated documentation. -
Logging for the core send and receive operations against the Event Hubs service can now be correlated by an
OperationIdin the logs and detail the number of retries attempted for the operation. -
Connection strings for each of the clients now supports a
SharedAccessSignaturetoken, allowing a pre-generated SAS to be used for authorization.
- Bug in TaskExtensions.EnsureCompleted method that causes it to unconditionally throw an exception in the environments with synchronization context
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Daniel Marbach (GitHub)
-
The underlying AMQP library has been enhanced for more efficient resource usage; this will result in a noticeable reduction in memory use in common consuming scenarios. (A community contribution, courtesy of danielmarbach)
-
All clients will now perform an eager validation of connection strings upon creation. Previously, validation was performed just before a service operation in some scenarios which made debugging difficult.
-
Fixed an issue where failure to create an AMQP link would lead to an AMQP session not being explicitly closed, causing connections to the Event Hubs service to remain open until a garbage collection pass was performed.
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Daniel Marbach (GitHub)
-
The
ReadEventOptionsused with theEventHubConsumerClientnow support setting aPrefetchCountandCacheEventCountfor performance tuning. More details about each can be found in the associated documentation. -
The
EventHubConsumerClientpipeline for reading events from a single partition was reworked to improve efficiency and make use of the new configuration options forPrefetchCountandCacheEventCount.
-
The
EventProcessor<TPartition>now supports a configurable strategy for load balancing, allowing control over whether it claims ownership of partitions in a balanced manner (default) or more aggressively. The strategy may be set in theEventProcessorOptionswhen creating the processor. More details about strategies can be found in the associated documentation. -
The
EventProcessorClientOptionsnow support setting aPrefetchCountandCacheEventCountfor performance tuning. More details about each can be found in the associated documentation.
- Logging for the core send and receive operations against the Event Hubs service can now be correlated by an
OperationIdin the logs and detail the number of retries attempted for the operation.
-
Surfacing of exceptions has been fixed to consistently preserve the stack trace in cases where it was previously lost. (A community contribution, courtesy of danielmarbach)
-
An additional level of resilience was added to some corner case scenarios where establishing an AMQP link failed with what may be a transient issue.
-
A cleanup sweep was performed to tune small areas to be more efficient and perform fewer allocations.
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- The set of features from v5.1.0-preview.1 are now generally available. This includes the
EventProcessor<TPartition>andPartitionReceivertypes which focus on advanced application scenarios which require greater low-level control.
- A set of events may now be published without an explicit batch; a batched approach will be used when communicating with the Event Hubs service, with an implicit batch created on the sender's behalf.
-
The transport producers used for sending events to a specific partition are now managed by a pool with sliding expiration to enable more efficient resource use and cleanup. (A community contribution, courtesy of albertodenatale)
-
Timing operations have been refactored to make use of a more efficient approach with fewer allocations. (A community contribution, courtesy of danielmarbach)
-
Fixed a bug with EventDataBatch; it is now thread-safe.
-
Minor enhancements to reduce allocations and improve efficiency
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
-
A new primitive,
EventProcessor<TPartition>, has been implemented to serve as an extensibility point for creating a custom event processor instance. It offers built-in fault tolerance, load balancing, and structure while allowing tuning for low-level network configuration, processing of events in batches, and customization for the storage of checkpoints. More detail can be found in the design proposal. -
A new primitive,
PartitionProcessor, has been implemented to serve as a low-level means of reading batches of events from a single partition with greater control over network configuration. More detail can be found in the design proposal.
- Event batches are now protected against modification while publishing is actively taking place.
-
Exceptions surfaced will now properly remember their context in all scenarios; previously, some deferred cases unintentionally reset the context.
-
Validation for the Event Hubs fully qualified namespace has been improved, allowing for more deterministic failures when creating clients.
-
The diagnostic scope for activities will now complete in a more deterministic manner. (A community contribution, courtesy of christothes)
-
Diagnostic activities have been extended with additional information about events being processed and with additional environmental context.
-
Parsing of connection strings is now more permissive for the
Endpointkey, allowing additional formats that result from common mistakes when building the string rather than copying the value from the portal. -
LastEnqueuedEventPropertiescan now be compared for structural equality.
- For special cases, the live tests may be instructed to use existing Azure resources instead of dynamically creating dedicated resources for the run. (A community contribution, courtesy of albertodenatale)
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Alberto De Natale (GitHub)
-
A migration guide is now available for those moving from the 4.x version of the
Microsoft.Azure.EventHubslibraries to the 5.0.1 version under theAzure.Messaging.EventHubsnamespace. -
A bug was fixed that would intermittently cause a failure that caused retries to abort, potentially preventing recovery from transient failures.
-
Several minor performance and efficiency improvements have been implemented.
-
Namespaces have been reorganized to align types to their functional area, reducing the number of types in the root namespace and offering better context for where a type is used. Cross-functional types have been left in the root while specialized types were moved to the
Producer,Consumer, orProcessornamespaces. -
The hierarchy of custom exceptions has been flattened, with only the
EventHubsExceptionremaining. The well-known failure scenarios that had previously been represented as stand-alone types are now exposed by a newReasonproperty to allow for applying exception filtering and other logic where inspecting the text of an exception message wouldn't be ideal.
Thank you to our developer community members who helped to make the Event Hubs client libraries better with their contributions to this release:
- Alberto De Natale (GitHub)
-
A bug with the use of Azure.Identity credential scopes has been fixed; Azure identities should now allow for proper authorization with Event Hubs resources.
(A community contribution, courtesy of albertodenatale) -
A bug with the renewal of connection string-based credentials has been fixed; clients using a connection string will now properly refresh when necessary instead of experiencing an error.
-
Some performance tuning has been done, focusing on reducing allocations associated with frequently used types where possible.
-
The
EventHubsConsumerClientis no longer bound to a partition, an owner level, or tracking of last enqueued event properties; these attributes may be specified when requesting to read events, allowing more granular control over behavior without the need to create multiple clients. -
Events may now be read across all partitions of an Event Hub using the
ReadEventsmethod of the consumer client. This is intened for exploring Event Hubs and not recommended for production use; for production scenarios, please consider using theEventProcessorClientas a more robust alternative. -
Events read using the consumer client are now accompanied by a
PartitionContexthelping to identify the source partition for an event and allowing for partition-specific operations, such as reading the last enqueued event properties for that partition.
- The
EventHubsProducerClientclient is no longer bound to a partition for sending events to a specific partition. It will now accept a partition identifier as part of the options when creating an event batch. This option may not be used with a partition key in the same batch; only one or the other may be specified. This allows for more flexibility when publishing events without the need to create multiple clients.
-
Test and sample infrastructure has been updated to work with Azure.Identity credentials in addition to connection strings.
(A community contribution, courtesy of albertodenatale) -
A sample demonstrating the use of an Azure Active Identity principal with Event Hubs is now available.
(A community contribution, courtesy of albertodenatale) -
The
EventHubsSharedKeyCredentialhas been removed from this release for further design and improvements; it is intended to be reintroduced in the future.
-
A large portion of the public API surface, including members and parameters, have had adjustments to their naming in order to improve discoverability, provide better context to developers, and better conform to the Azure SDK Design Guidelines for .NET.
-
The
EventProcessorClienthas been moved into its own package and evolved into an opinionated implementation on top of Azure Storage Blobs. This is intended to offer a more streamlined developer experience for the majority of scenarios and allow developers to more easily take advantage of the processor. -
A collection of internal supporting types were moved into a new shared library to allow them to be shared as source between the Event Hubs client libraries. The tests assoociated with these types were also moved to the shared library for locality with the source being tested.
Thank you to our developer community members who helped to make the Azure SDKs better with their contributions to this release:
- Alberto De Natale (GitHub)
-
The early stages of a large refactoring of the Event Hub client type hierarchy is complete; Event Hub producers and consumers are now stand-alone clients that can be created independently, and which may or may not share a connection to the Event Hubs service at the discretion of the developer.
Please Note: These changes are part of a larger effort that is currently in process. The client types have not yet been refactored to remove affinity to a specific partition which leaves some operations feeling somewhat awkward with the new structure.
-
Client types using Azure identity for authorization should now be working properly; a bug was fixed to allow the proper authorization scopes to be requested.
(A community contribution, courtesy of albertodenatale) -
Service communication is now being performed by an internal communication stack; previously, it had been delegated to an internal copy of the legacy client library.
-
Cancellation tokens are now supported throughout the client hierarchy and across operations.
-
Telemetry is now emitting a
User-Agentmember, which is standard across other client libraries in the Azure SDK. -
With the new hierarchy allowing clients to be more self-contained, the configuration of retry policies and timeouts have been made more consistent and easier to reason about. Each may be configured as part of the options for a client and will retain either the value provided by developers or the default. There is no longer a notion of inherited values from a "parent" client.
-
The body of an
EventDatainstance is now available as a stream, for convenience. -
The information about the last event enqueued to an Event Hub partition is now presented on-demand as an immutable object, if the tracking option was enabled. This ensures that the properties are stable and consistent when being read, rather than being subject to in-place updates each time a new event is received.
-
Included the Event Hubs fully qualified namespace as part of the checkpoint information, ensuring that there is no conflict between Event Hubs instances in different regions using the same Event Hub and consumer group names.
-
Distributed tracing support has been added to the Event Processor.
-
Fixed date parsing for time zones ahead of UTC in the Event Hub Consumer when tracking of the last event was disabled.
-
Updated dependencies to take advantage of newer client libraries for identity management, blob storage, and .NET Core improvements.
-
Improved stability and performance with refactorings around hot paths and areas of technical debt.
-
The full set of system properties was reintroduced to the
EventDatafor events received from the service. In addition to the well-known properties for identifying an event's place in the partition (Offset,SequenceNumber, andEnqueuedTime), any additional metadata sent as part of an event will appear in theSystemPropertiesdictionary. -
The
EventHubConsumercan be configured to track information about the last event to be enqueued into its associated partition in order to allow for monitoring the backlog of events to be processed without the need to make explicit calls to request partition properties from theEventHubClient. -
Consuming events using
SubscribeToEventshas been refactored for better performance and lower resource use.
-
Creation of an
EventProcessorhas been refined with the goal of allowing simpler use for standard scenarios while maintaining flexibility for complex ones. TheEventProcessoris now able to manage partition processor instances for implementations with a default constructor. -
The
EventProcessorwill now coordinate with other activeEventProcessorinstances for a given Event Hub and consumer group in order to share partitions and ensure that processing work is balanced between them.EventProcessorinstances may be activated or deactivated independently and the remaining processors will redistribute partitions as needed. -
The interface for processing partitions has been changed to a base class, in order to allow developers to make the decision which of the available methods they would like to override, rather than being forced to implement each one.
-
The client library for Event Hubs is now emitting diagnostics information for distributed tracing. (see: proposal)
-
Some public types were scoped in a way that made them difficult to mock for library consumers. These have been re-scoped to
protected internalfor better testability.EventDataand metadata types were the significant instances.
-
Public members of type
DateTimewere converted toDateTimeOffsetto explicitly represent that they are UTC -
Members of type
DateTimeOffsethave had theUtcsuffix removed from their names, as theDateTimeOffsettype removes ambiguity about the time zone -
Members representing a partition offset that were incorrectly exposed as
stringhave been converted tointto provide a unified type. -
Renamed all instances of
EventHubPathtoEventHubNameto align with the usage context and unify on the chosen semantics across the client library for different languages. -
Ongoing minor tweaks to formatting, wording, and spelling of documentation and XML document comments.
-
Reintroduced the
EventDataBatch, allowing for publication of a batch of events with known size constraint. This is intended to ensure that publishers can build batches without the potential for an error when sending and to allow publishers with bandwidth concerns to control the size of each batch published. -
Enhanced the
EventHubProducerto allow creation of anEventDataBatchand to accept them for publication.
-
Added a means to subscribe to the events exposed by an
EventHubConsumerin the form of an asynchronous iterator. Using the iterator, consumers are able to use the familiarforeachpattern to enumerate events as they are available and process them. Optionally, consumers may specify a maximum time to wait for events to arrive which, when exceeded, will emit a null event if none were available, returning control to the loop and allowing cancellation or other processing needs. -
Introduced the initial concept of a new version of the
EventProcessor, intended as a neutral framework for processing events across all partitions for a given Event Hub and in the context of a specific Consumer Group. This early preview is intended to allow consumers to test the new design using a single instance that does not persist checkpoints to any durable store.
-
Created an exception hierarchy that incorporates the exceptions exposed by the legacy client library, covering the same set of scenarios with minor changes to naming to better clarify the context.
-
Ensured that custom Event Hubs exceptions are properly translated to the new versions, no longer exposing legacy exception types.
-
Removed the exposed retry policies in favor of a set of retry options based on the options found in
Azure.Corein order to keep the API familiar. -
An abstract
EventHubsRetryPolicyhas been created to serve as the contract for custom retry policy implementations. -
An option for fixed retry has been added to accompany the exponential retry that was in place previously. Operation timeouts have been moved from the associated client options and incorporated into the retry options and retry policies.
Version 5.0.0-preview.1 is a preview of our efforts in creating a client library that is developer-friendly, idiomatic to the .NET ecosystem, and as consistent across different languages and platforms as possible. The principles that guide our efforts can be found in the Azure SDK Design Guidelines for .NET.
For more information, please visit: https://aka.ms/azure-sdk-preview1-net.