Epic: Eventing Framework in .NET 9 #53219
Comments
captainsafia
added
the
Epic
dotnet-issue-labeler
bot
added
the
needs-area-label
captainsafia
removed
the
needs-area-label
mkArtakMSFT
added
the
area-minimal
captainsafia
removed
the
area-minimal
|
I'd like to chime in here a bit because I think a couple of points might be relevant to consider. Right now there exist a couple of message handler frameworks. Some that come to mind are:
And many more exist. Given that this is supposed to become a first party implementation of similar capabilities, is there going to be a call for usage scenarios so we can give feedback on which capabilities are used and how or is this going to be handled entirely internally? I expect that this implementation will at least in part leverage source generation similar to how https://github.com/martinothamar/Mediator and https://github.com/JasperFx/wolverine do it. Albeit the difference between those two is that Wolverine does runtime source generation vs. Mediator using Roslyn source generators. If it does use source generation I'd like to point out that special care should be taken looking at the integration with existing other source generators. Specifically because there still is no way to chain them together. It would be great if the capability to create MessageHandlers on the fly using Source generation would still allow them to be used from within whatever you're going to build. I'd love for the aspnetcore team to champion changes in Roslyn that lead to us being able to chain Source Generators 😉 . Given that the Razor generator already presents similar issues. Runtime code generation will likely not work that well together with AOT unless you do a two-pass system similar to how Wolverine handles it. All in all I think having a first party implementation as something people can fall back on or that other libraries can expand on is a good thing. I'd just like it to be as usable as possible in the end :) |
|
Isn´t this exactly what you already tried with WCF when it came out? What will you do differently this time around to prevent the mess that WCF was for, especially for one-way messaging? |
|
I would request and highly recommend that in building this you introduce a common abstraction layer that can be used by third party tools, much like the |
|
Why not support existing OSS tools like Wolverine or MediatR or smth else instead of building smth from scratch in-house? |
|
One question pops into my mind: How are schema registries covered? Is "Expose APIs for serializing and deserializing messages resolved from providers" meant to support schema registries out of the box, or is is meant to provide an interface to integrate a schema registry on your own? |
|
This is interesting. Some thoughts from experience with existing libraries in the space.
|
|
With outbox feature, please. |
One man library. The tools are designed to promote their creators. The design and functionality depend on the owner. When someone is dissatisfied, they create another very similar tool to fulfil their needs. Create APIs and Layers to standardise these tools seems much better to me. |
Imo, that's exactly against the spirit and nature of OSS. Tools are not designed to promote developers(while this might be a good side effect). Those tools are built to solve problems, and people collaborate together to solve them. Now, MS looks at the existing ecosystem selects popular areas, and develops tools that essentially replace existing OSS alternatives. So what happens next? Use of those tools will obviously drop, and eventually, they will disappear. Why maintain some OSS tool when you can just sit, wait, and hope MS will be kind enough to develop the thing that solves your problem? This is not OSS. |
|
Just as a point of clarification, MediatR is NOT comparable to this feature set, nor would plug in to a common event bus interface. MediatR is strictly in-process and in-memory. Not for durable async message queues. And also evidently MediatR exists solely to promote myself. I like this idea though. It should be easier to adopt event-driven architectures. It would be nice if it were like DI, logging, even EF Core where there is some default implementation but you can add your own 3rd party implementation. |
|
The risk is that, similar to what happened with DI, the "common interface" effectively defines the implementation and stifles innovation. There was a great thread many moons ago where the author of Simple Injector was trying to explain why the DI interface was hostile to the fundamental premise of Simple Injector. IIRC the conclusion was ~"this is a solved problem that doesn't need innovation" which was off base considering Simple Injector was a reasonably popular project that was taking a different tack that wasn't achievable with the "common interface". Not sure how that all finally resolved in the end. My desire would be to always be able to opt out and also to be able to rebuild whatever is given from smaller bits of the framework. |
FWIW, there's a company behind Wolverine (JasperFx), with similar commercial support offerings behind NServiceBus, MassTransit, and Rebus to name a few others. That argument doesn't really fly. Even Wolverine as the newest kid on the block has technical roots going back a decade. NServiceBus and MassTransit are ~15 years old at least. |
|
Worth noting if you find this exciting, that we already offer all the features described in Brighter (and more @voroninp we have your outbox and inbox already): https://github.com/BrighterCommand/Brighter So if you are looking to work with a mature messaging framework, battle-tested at scale, (that also supports .NET Framework) feel free to check us out and start using events today instead of waiting. We have been doing this for 10+ years, so our offering comes with a wealth of experience. |
|
But to the larger point. The .NET space is perhaps more richly served than any other ecosystem with messaging tools, both commercial, commercial support and solely OSS. Competition amongst those projects has created a strength in offerings that other languages don't have. When MS enters the space that diversity and competition will be crushed, leading to less innovation and choice. There is no argument that MS needed to enter this space, .NET developers were more richly served than any other ecosystem. Instead this is just the usual MS playbook - crush successful OSS - own all the things. When I spoke about a .NET Renaissance many years ago I was clear that MS had a duty of care as the 500lbs gorilla in the room not to do this. But whilst previous flattening of .NET OSS could be seen as ignorance or lack of care, they know well enough by now, that this can only be seen as malice. |
|
FWIW, I'm voting hard against any set of common abstractions for the entry point to messaging, and even more so to any kind of standardized interface signature for message handlers. That's a recipe for a mediocre developer experience that won't make .NET "the best platform for cloud native applications". This isn't a place for a "Conforming Container" kind of approach. |
|
Please, No. We are thankfully blessed with a lot of good libraries at various layers for event-ish type stuff:
Again, all of the above are extremely useful in the context of eventing or event driven systems. I don't see how one could make a sane abstraction however, since many of them are context dependent yet cannot be easily simplified. They mostly have enough different but unique semantics to where a common abstraction could easily be either of little utility or just dangerous. My apologies for any firmness here, I have concerns that another 'conforming container' across all of these paradigms would risk turning the story of event driven systems in .NET into a cautionary tale about good intentions stifling innovation and flexibility, potentially at the benefit of specific vendors despite the unique opportunities each of the above mentioned cases would handle. I hope the community does not decide to add a 'conforming container, but for events', especially if it happens to conform best to a brand new system without clear benefit compared to existing ones aside from 'being first party' or 'well you need it anyway because ASPNETCORE'. If we wanted to fix eventing at a better level, I would suggest:
Points One and Three, I think are fairly easy, it's just a matter of putting them where people can get to them. That middle point, It actually solves a lot of other problems in the .NET space... More Importantly, It helps the authors of all of the above things give better results to the community, as they have done for so long. If you -do- choose to go down the route of a common abstraction, I would hope that it is a public process and you involve the above mentioned parties as well as others. |
|
Just keeping an eye on this as a maintainer / co-creator of https://github.com/FoundatioFx/Foundatio. I just hope this doesn't follow the path of IDistributedCacheClient that I've seen used by very few. Please engage early and often with leaders who have been in this space a long time. |
That is definitely an interesting one. Brighter uses a reactor/proactive model (depending on whether you use async) and has a single threaded message pump to preserve ordering (which you scale out). This also means we don't run into problems with thread pool exhaustion, or blocking on semaphores trying to limit work in flight at high scale. We use a custom SerializationContext based on an article by Stephen Toub to have async callbacks use that thread. But is a little gnarly and a framework solution to that problem would have re-usable value. |
I love foundatio ❤️❤️❤️ |
|
Maybe one could compare the amount of money & effort that would be needed to develop a de novo fit-for-use eventing platform against the money & effort needed to create a set of template projects? For Event Driven Architectures, having the right set of templates may be 80% of the developer win. |
|
If this does move forward, please also add:
With some solutions, it can be hard to even figure out what the DLQ is named... |
|
Also, Kafka should be considered as it greatly impacts design. |
This proposal is about message queues. Kafka is not a queue, I don't think it should be considered. |
|
Hi, I'm one of the maintainers of https://github.com/asyncapi/saunter, a code-first documentation tool for AsyncAPI. We've been thinking about how to better get automatic API documentation, rather than having to manually add attributes to existing code. We are very keen to make sure that something like the This will enable us to build out our library to automatically produce an AsyncAPI doc. Something like this: builder.Services.AddEventQueues();
builder.Services.AddEventQueueExplorer(); // msft provided, could be automatically added as part of .AddEventQueues();
builder.Services.AddAsyncApiSchemaGeneration(); // asyncapi/saunter provided, depends on EventQueueExplorer |
|
Yes, please! |
What you are saying seems to be a very valid reason as to wee need a standard. MediatR is strictly in-process and in-memory but that's possibly just yet another implementation of the those common libraries. The transport, queue mechanism should be all abstract. |
|
Excuse, me~ |
It's most likely going to be vendor agnostic with some integration pattern with Azure. |
|
@captainsafia @davidfowl where is the best place to keep up with the work being done on this? |
|
The only issue that I have with this is that it's all very ASP.Net centric. Even though this isn't specific for building web apps. Is this just a messaging issue? These new features that are not really tied to a specific runtime (like ASP.Net), I'd like to see them spoken of in a broader context. In examples and also just in general discussions because console apps, desktop apps and mobile apps need to feel included as well. Also, I feel it would help developers feel like these new BCL features* are valid "tools" they can use to solve issues in their applications that are not just web apps. *Well, not BCL, but pretty low level. |
|
Any news on this? Which preview is this scheduled for? Thanks! |
|
What I am missing in the API draft is a way to return some result. This disdinguishes it clearly from other messaging libraries. Have you considered including at least an optional possibility in the API spec to return result values from handlers to the caller? Even if .Net does not really implement this feature, other third party libs could plug into the official API. I would be curious, what your thoughts are on this? |
|
There hasn't been any news on this since it's announcement (more or less); Is this one of those things where you keep it hidden until it's well underway, or has this been delayed until .Net 10+? |
|
good question, I'm also wondering why the sudden silence.
…On Sun, Jun 2, 2024 at 12:45 PM Kenneth Hoff ***@***.***> wrote:
There hasn't been any news on this since it's announcement (more or less);
Is this one of those things where you keep it hidden until it's well
underway, or has this been delayed until .Net 10+?
—
Reply to this email directly, view it on GitHub
<#53219 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABHD72Q6LAK2OFAA53WIDM3ZFLSUPAVCNFSM6AAAAABBR66QL2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBTG43TMNBQHE>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
I can't read everything on this thread, but having a programming model that is the same between ASP.NET Core and Azure Functions is much desired. At times, I have wanted to switch between the two and it's a bunch of unnecessary work. |
|
Having a standard eventing framework may be great enabler for better AsyncAPI (like OpenAPI spec, but for async APIs) support. For example so that in a schema first - code second scenario the generated code can be more abstract. |
|
Any updates? |
|
Hey everyone, we’ve got some updates on the .NET eventing framework and wanted to share where things stand. The "excitement" around the eventing framework is undeniable, and it’s clear that expectations are high. We’re taking the time needed to ensure we deliver something that will truly enhance the .NET ecosystem and work well with existing frameworks. We’re committed to the idea that the best designs emerge from collaboration, that’s why we plan on engaging in an open design process for the eventing framework. We’re inviting the community to join us from the very beginning, to make sure the framework meets developers’ needs and aligns with the design principles we set out. To get this right, we’ve decided to delay the release of the eventing framework to a subsequent version of .NET. It’s a significant decision, but it allows us to focus on other essential features for .NET 9. In the meantime, we’re collaborating with Azure Functions to integrate more closely with Aspire and ASP.NET Core, which we believe is important for laying the groundwork for the eventing framework. We understand that some of you may be curious about the timing of this update. Our team has been concentrating on other critical components of .NET 9, but we recognize the importance of keeping the community informed. Going forward, we’re committed to providing more frequent updates. We’re excited to see your contributions to the eventing framework! |
|
Thank you @davidfowl for taking the time to post this update. |
|
Fair enough, thank you @davidfowl <https://github.com/davidfowl> for
getting us posted. I'm seriously looking forward to seeing this eventing
framework in .net 9.x
…On Wed, Jun 26, 2024 at 5:11 AM Jean-Sebastien Carle < ***@***.***> wrote:
Thank you @davidfowl <https://github.com/davidfowl> for taking the time
to post this update.
—
Reply to this email directly, view it on GitHub
<#53219 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABHD72XYX2FP5NXVEG2AG5TZJIPM3AVCNFSM6AAAAABBR66QL2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJQGM4TQMBUGQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
In the meantime, if you have messaging needs, we have been fulfilling them for .NET developers for over 10 years, and are compatible with both .NET Framework and .NET |
|
Well that's a little disappointing. I've been looking for a good framework to use in serverless lambda functions on AWS... I don't need the full feature sets of a lot of these frameworks as there are various ways to do them with AWS services, but feel like I could use some of their functionality. I recently had to rip some out and have found other alternative framework owner's discussions about how they are refusing to add AoT support as they don't see the need and it is significant work. AoT on lambda is a best practice for cold-start performance. I had hope that at least a framework provided by the .NET team would actually work properly with the modern framework regardless of what runtime options you chose for your project. |
Just recently came across this thread. I have my own Service Bus that I created when NServiceBus went commercial. It has been around for quite some time. It would be great if Microsoft can provide something that is standardized. It will be quite tricky to get just right as there are many moving parts. I'm guessing that something like Dapr pub/sub may have influenced this move? As someone else mentioned, I think it is more messaging as opposed to eventing. That is something that I didn't quite like about the Dapr implementation, that it seems event-focused. For messaging, there are three message types: The intent behind each type is quite different. In fact, I've recently started seeing an overlap between document messages and over-the-wire DTOs... but that's another discussion. Anyhow, I was wondering how the "open design process" would work. Also, you mentioned that "We’re inviting the community to join us from the very beginning...". How would one be invited; unless it is a more general statement related to the open design process. It would be very interesting to get involved in something like this. I, for one, would love to no longer have to maintain a bunch of code for different queue providers that are abstracted away. Having a message-oriented framework available in |
|
I wanted to take this opportunity to commend & thank @davidfowl @captainsafia and the rest of the team on the direction they're taking with this Planning to "work well with existing frameworks" and "engaging in an open design process" is a BIG DEAL. We as a community should be just as vocal when we see Microsoft engaging positively and collaboratively with the ecosystem as we do when they misstep. If you believe this is a positive development, let them know! Comment/emoji on the issue, tweet, LinkedIn, whatever. Please and thank you 🙏🏼 |
|
Good decision @davidfowl |
|
One thing I want to call out as respectfully as possible (Since I am no longer at MS or MSR) is that some of the available "open source" libraries might use licenses that do not allow Microsoft to reuse them or make use of that code in the standard library or base class library, and so Microsoft is kind of forced to rewrite things from scratch here in order to make sure that all the IP ownership is legit/buttoned up in a way that keeps the lawyers and the accountants happy. For example: I'm in the process of writing a CQRS based system in .Net and I would love to be able to just use a BCL solution for the event processing, but looking around what I'm finding is that many of these libraries are missing features or have conflicting ideas about what SOLID and DRY mean, so after research due to my requirements, I'm going to have to write it from scratch because it has distributed system requirements that make choosing some of the available libraries and the patterns they use under the hood effectively the same as intentionally designing for an intentional single point of failure that would violate SLA/Uptime/compliance requirements and make them unusable. I'm looking forward to a functional Eventing framework, but I also feel that its CRITICAL to make sure that it can actually used for serious engineering leveraging generics and all the rest, and not just a few toy azure functions that may not do much, otherwise it will just end up like some of the past projects that others have already called out as ineffective or often unused due to the design constraints imposed on them. In the end, I really hope I get to delete a large amount of the code I need to write to support CQRS systems, and that it ends up being a BLC import and thus natively supported, but I also worry that what I end up writing turns out to be better than what the BCL will eventually provide, and I think its fair that I call this out as it seems to be a common fear by others. Please, take the time you need to do this right, to make it effective and general use, and to support distributed systems, generics, snapshots, and generic parameters for queries, in the best possible way. |
|
Curious as to what you think those failures by existing frameworks are? We would love to fix them. Brighter is used by a lot of FinTechs (amongst others) for high-volume and reliable transactions. So it would be interesting to see where you think we fail. |
BrennanConroy
added
area-minimal
and others
This issue captures issues related to the "eventing framework" work area in .NET 9. Issues and categorizations are subject to change as design and prototyping is underway.
This eventing framework will allow developers to write applications that support processing messages from various queue providers in their application.
Major categories of work include:
More details here:
#53219 (comment)
The text was updated successfully, but these errors were encountered: