New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatibility with netstandard/net core #6

Closed
mollyporph opened this Issue Nov 10, 2017 · 61 comments

Comments

Projects
None yet
10 participants
@mollyporph

mollyporph commented Nov 10, 2017

This is a collection of blockers and potential solutions for moving from WindowsAzure.ServiceBus (net45) to Microsoft.Azure.ServiceBus (1.0.0: net451, netstandard1.3. 2.0.0: net461, netstandard2.0)

With Microsoft.Azure.ServiceBus Microsoft has moved the management api to a new project: Microsoft.Azure.Management.Servicebus.
This new project is an attempt by Microsoft to align the servicebus sdk to their ARM-everywhere philosophy.

This introduces a couple of issues. The first and foremost is that all management commands are executed via ARM and thus requires Azure Active Directory credentials instead of the much simpler Service Bus connectionstring. Maintainers have received the criticism and have a plan to re-introduce management APIs (follow the issue here: Azure/azure-service-bus-dotnet#65)
tl;dr They will gradually re-implement it with a goal of having it all implemented by summer 2018.

Will update with more findings

@hulvei3

This comment has been minimized.

hulvei3 commented Feb 7, 2018

What is the current status on compatibility with netstandard? It's hard to sense if the limitation is still on place or not.

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Feb 7, 2018

As far as I know, Rebus.AzureServiceBus can still not run on .NET Core – not even on .NET Core 2, even though it will readily try, because it will fails due to WindowsAzure.ServiceBus trying to use System.Configuration, which is not available.

@hulvei3

This comment has been minimized.

hulvei3 commented Feb 7, 2018

Okay. This is a shame - we actually planned for migrating to Azure from MSSQL very soon. And we already have dependencies that targets NETStandard. If we would make a temp solution for us, by forking, how what is the culbrit of this problem? (if we have to include Azure AD creds, so be it)

@Meyce

This comment has been minimized.

Contributor

Meyce commented Feb 7, 2018

For whatever the host application is, can you have the host app use a .Net Framework version? I have several web APIs and Windows Services where the underlying project is .Net Core but I'm continuing to target .Net framework 4.7, which has worked out well thus far. This would work fine as long as you were running on Windows, and you wouldn't have to rework the ASB transport.

If you do try and rework the transport, you'll need to use an app registration in Azure AD as a service principal and I think you have to give it the owner role on your subscription. I don't think there are currently any roles targeted at ASB, but I may be wrong on that. The "updated" transport would need to bring in the ARM ASB lib as well to handle all the management operations that the new .Net Standard ASB lib can't do anymore.

There hasn't been much from Microsoft recently on the state of the .Net Standard ABS library. They have talked about adding the necessary features back in, but it'll likely be at least a few more months before this happens.

@hulvei3

This comment has been minimized.

hulvei3 commented Feb 7, 2018

Well, that would not work for us, as we use Rebus in our own slim wrapper lib. This lib is then shared among every components (being services, APIs etc) - some of them being targeted NET Core. So unless we accept that we would remove the NETStandard support in our wrapper lib, I don't see any way for us to consume this right now it seems.

Thanks for the response

@Meyce

This comment has been minimized.

Contributor

Meyce commented Feb 7, 2018

Okay.

If you do look into forking/updating the existing transport the bulk of the work will be in the two transport classes.

I've used the ASB ARM lib a bit. You'll need to get the ((client id/client secret) or cert), authority, app id into the transports so that you can get a bearer token for the Azure Management API. Once you have that the management operations are pretty straightforward. The bearer token will have an expiration, so it will need to have the ability to go out and get another access token in the future. This would have to be done with username/password token flow, so I don't think there will be any possibility of refresh tokens. The ADAL lib would help with the authentication. After that, all the send/receive/etc. methods in the transport will need to be updated, and there will be some differences there with the new library.

If you didn't want to bring in the ARM ASB lib, you may be able to do all of the managment operations via ASB HTTPs endpoint. I think it would work fine without having to use Azure AD credentials. You would need to parse apart the endpoint=sb://{domain/keyname/key} into something that can be used with an HttpClient which I've done with Regex in the past. I think all the management ops can be done over HTTPs, and that "should" allow a straight port to the ASB .Net Standard lib without alot of underlying changes. Then, once Microsoft updates the .Net Standard lib and puts the management features back in, usage of the HTTP endpoint could be updated to use the features in the lib.

@KennethJakobsen

This comment has been minimized.

KennethJakobsen commented Jun 6, 2018

You should be able to do it with this workaround (https://github.com/Mossharbor/AzureWorkArounds.ServiceBus) and the official .net standard package

@hulvei3

This comment has been minimized.

hulvei3 commented Jun 6, 2018

Thx for the efforts guys. But we actually moving to RabbitMQ instead as SQL itself doesn't scale for us anymore.

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Jun 6, 2018

@KennethJakobsen that's actually pretty interesting! Did you try it?

@KennethJakobsen

This comment has been minimized.

KennethJakobsen commented Jun 6, 2018

@mookid8000 No not yet, but actually MS is aiming at june to implement management again, due to popular demand. Unless i misunderstood something.
Azure/azure-service-bus-dotnet#65 (comment)

@KennethJakobsen

This comment has been minimized.

KennethJakobsen commented Jun 8, 2018

Seems like management is back on the menu boys! Azure/azure-service-bus-dotnet#481

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Jun 8, 2018

Wow – I hope they accept that PR soon 🤞

@frostebite

This comment has been minimized.

frostebite commented Jun 14, 2018

To clarify, without the above PR linked there.

Netcore (2.0) cannot work with rebus.azureservicebus?

I'm getting:
Rebus.Injection.ResolutionException : Could not resolve Rebus.AzureServiceBus.AzureServiceBusTransport with decorator depth 0 - registrations: Rebus.Injection.Injectionist+Handler ---- System.IO.FileNotFoundException : Could not load file or assembly 'Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified. at Rebus.Injection.Injectionist.ResolutionContext.Get[TService]()

using servicecollection:

var services = new ServiceCollection();
services.AutoRegisterHandlersFromAssemblyOf<handler>();
services.AddRebus(configure => configure
 .Transport(t => t.UseAzureServiceBus("...", "transportQueuething"));
var provider = services.BuildServiceProvider();
provider.UseRebus();
@frostebite

This comment has been minimized.

frostebite commented Jun 14, 2018

@KennethJakobsen

This comment has been minimized.

KennethJakobsen commented Jun 14, 2018

@frostebite are you making a new version of Rebus.AzureSeriviceBus or are you trying to make the existing implentation work?

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Jun 14, 2018

If I understand things correctly - in order to make Rebus.AzureServiceBus work on .NET Core 2, it would be necessary to port the code in Rebus.AzureServiceBus to make it work with Microsoft's new ASB driver, and then make it use the AzureWorkarounds library to carry out the management operations.

As far as I know, noone has taken a stab at doing this yet. If I weren't so bogged down with work at the moment, I would do it myself, but unfortunately this is around number 5 on my list of priorities right now.

@KennethJakobsen

This comment has been minimized.

KennethJakobsen commented Jun 14, 2018

@mookid8000 I have some vacation coming up in a couple of weeks i'll take a stab at it then if noone else beats me to the punch, i'm just too caught up in work to do it right now.

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Jun 14, 2018

Yeah well, me2 😄 let's see who gets to the vacation first then.

We'd probably better coordinate in here, so we don't end up coding the same things at the same time 😁

@frostebite

This comment has been minimized.

frostebite commented Jun 15, 2018

I was just trying to assert that my understanding was correct that the exception I was facing was due to it being netcore 2.0 and that it won't be able to work until the issues being discussed here are fixed.

Sounds like that is the case.

I'm super happy to assign one of my team members to work on this, but we've only just discovered rebus so it would be much better to be able to work with one of you or to be able to bother you with questions.

Any suggestions are welcome :).

Also with regard to the workaround, is it worth doing that or waiting for the PR to land that changes the behaviour in the original microsoft package?

@frostebite

This comment has been minimized.

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Jun 28, 2018

@frostebite let's hope they get that merged and release quickly 🤠

@Meyce

This comment has been minimized.

Contributor

Meyce commented Jun 28, 2018

I was able to get the Rebus.AzureServiceBus.Tests working with the branch of the netstandard ASB driver that contains the new management components. I cloned Microsoft's ASB repo and used the branch with the management components contained in Azure/azure-service-bus-dotnet#481, pulled in as a project reference of a fork of Rebus.AzureServiceBus.

It was just a quick and dirty try to update the transport to work with the new driver so it's a bit messy, but I didn't hit any major issues converting to the new driver, unless the problems I hit with net452 can't be resolved see 6. below.

A few things of not while converting it.

  1. Exceptions. The Transport uses exceptions types from the driver to determine transient errors. Some of these exception types no longer exist. Further investigation needed.
  2. The send and receive operations are split across different types. The old driver had send/receive operations on the entity client(queueClient/topicClient). Now the entity client only contains the Send operations. A MessageReceiver is used to receive messages. I put the mesage receiver into a concurrent dictionary same as the clients, but I don't think this is necessary as there would only ever be 1 for the input address.
  3. The Complete/Abandon/Defer methods are all on the MessageReceiver. they used to be on the BrokeredMessage, so it's necessary to get the message receiver to perform operations on the message.
  4. It looks like the message receiver can accept a delegate for handling incoming messages. It may be worthwhile to look into what the message receiver is doing here, as it may allow for simplification of the receive code.
  5. All the client/receiver methods are async. I didn't look into changing internal methods to async. I just used GetAwaiter().GetResult() where Rebus was already using the synchronous methods on the old driver.
  6. I had some issues targeting net452. I've commented it out for now and am targeting net461/netstandard2.0. Needs further research.
  7. Version conflicts with newtonsoft between Rebus using v9 and ASB driver using v10.
@KennethJakobsen

This comment has been minimized.

KennethJakobsen commented Jun 29, 2018

The Management Operations are now merged in.

@frostebite

This comment has been minimized.

frostebite commented Jul 2, 2018

@KennethJakobsen @mookid8000 with regards to the issue with netcore in rebus. Is it looking like a few weeks? Or a few months for that to be resolved, now that the ASB management operations are back?

Thanks

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Jul 3, 2018

My ambition is to take a serious look at these things within the next weeks. I am currently on vacation though, so I can guarantee that I will no do anything about it the next 7-8 days, but after that, there's a good chance that I will do it 😄

@Jeern

This comment has been minimized.

Jeern commented Jul 11, 2018

I need this too - so I am cheering for you too @mookid8000 :) Please do send updates on the timeline if possible, BTW. I lost access to the Rebus Slack channel has it been discontinued ?

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Jul 11, 2018

@Jeern

This comment has been minimized.

Jeern commented Jul 11, 2018

That's fair :) We have long periods where we do not develop on Rebus related code at all, and then long periods where we do. I hope we will be active again, and perhaps we will need the pro subscription. BTW. you accidentaly added your contact info to the post. You should probably delete it

@frostebite

This comment has been minimized.

frostebite commented Jul 18, 2018

@mookid8000 any news on this front?

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Jul 18, 2018

@frostebite I have time to take a stab at it tomorrow evening.

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Jul 20, 2018

@Meyce w00t! that changes everything!!

@runes83

This comment has been minimized.

runes83 commented Aug 7, 2018

Have any status on this?

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Aug 7, 2018

It's almost complete - you can see the work on the net-core-again branch.

Only thing missing seems to be fixing the test that verifies that the peek lock can be automatically renewed. Haven't had time to really sit down and figure it out, so if anyone feels like taking a look at it, it would be awesome.

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Aug 7, 2018

I just pushed Rebus.AzureServiceBus-6.0.0-b01 to NuGet.org – it supports .NET Standard 2.0 as the only platform, because that's what's supported by Microsoft's new driver...

....and then I updated our Fleet Manager DEV environment to use that, so we'll see how it goes when we get it punished some 😈

If anyone feels like trying it, I would be very happy!

@ctcoulter

This comment has been minimized.

ctcoulter commented Aug 7, 2018

thanks for taking the time, man! so much appreciated. will hopefully have a chance to tinker with it over the next few days

@Meyce

This comment has been minimized.

Contributor

Meyce commented Aug 14, 2018

The Microsoft.Azure.ServiceBus 3.1.0 is out of preview.

@Meyce

This comment has been minimized.

Contributor

Meyce commented Aug 14, 2018

@mookid8000 did you get the test for peek lock/auto renew fixed? I pull you latest down and there weren't any failing tests.

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Aug 14, 2018

@Meyce yes!

@runes83

This comment has been minimized.

runes83 commented Aug 19, 2018

To recap, can we now run Rebus with Azure servicebus on .net core (even linux) ?

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Aug 19, 2018

@mookid8000 mookid8000 self-assigned this Aug 19, 2018

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Aug 19, 2018

Just released Rebus.AzureServiceBus 6.0.0-b02 which now depends on Microsoft.Azure.ServiceBus 3.1.0

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Aug 23, 2018

so... anyone tried it? (besides me, who's currently running two out of three Fleet Manager environments on the new version...)

@Meyce

This comment has been minimized.

Contributor

Meyce commented Aug 23, 2018

Yes, I've done some testing with it, and I am using it in a service deployed as an Azure WebJob to a couple different environments and have had no issues.

@mpaul31

This comment has been minimized.

mpaul31 commented Aug 23, 2018

@Meyce how are you hosting your endpoint within the WebJob? Topper is not compatible with .net core so are you using the WebJobs SDK?

@Meyce

This comment has been minimized.

Contributor

Meyce commented Aug 23, 2018

@mpaul31, In netcoreapp2.1 there is a generic host builder. Its is a generic version of the WebHostBuilder from aspnet core. The generic builder gives some similar facilities as TopShelf service stop/start, etc. I copied the WebShutdownWatcher class from the Azure WebJobs SDK lib. It's the only thing I needed from the SDK. It just watches the file system for a file to show up on the file system that signals the job should stop/shutdown. I tied that classes cancellation token to the stop functionality of the generic host.

This is irrelevant for this thread, so if you need any additional description, I'll put together a repo to demonstrate all the parts.

@mpaul31

This comment has been minimized.

mpaul31 commented Aug 23, 2018

@Meyce thanks i would love to see a repo when you have time!

@Meyce

This comment has been minimized.

Contributor

Meyce commented Aug 23, 2018

@mpaul31 Check out the repo linked below.
https://github.com/Meyce/Core-Rebus-WebJob

@mpaul31

This comment has been minimized.

mpaul31 commented Aug 23, 2018

@Meyce thank you for taking the time to set this up!

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Aug 30, 2018

I'm about to lay the finishing touches on Rebus' new Azure Service Bus driver, but in my tests I found that it would receive messages at about half the rate of the old driver.

E.g. in Rebus' Azure Service Bus prefetch performance test I would receive ~100 msg/s with the old driver, but with the new driver it would lay around ~50 msg/s.

Anyone experienced anything similar?

@Meyce

This comment has been minimized.

Contributor

Meyce commented Aug 30, 2018

I typically don't have enough messages coming through where I would notice a drop off.

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Aug 31, 2018

I did a couple of experiements yesterday, and it just seems the new driver is generally slower.

Looks like it's possible to account for it with Rebus though, simply by increasing the allowed max parallelism like this:

.Options(o => {
	o.SetMaxParallelism(100);
})

Rebus.AzureServiceBus 6.0.0-rc1 is out now 🌮 🎉

@Meyce

This comment has been minimized.

Contributor

Meyce commented Sep 4, 2018

RC1 is referencing Rebus(5.0.0-b14). Will the driver release with the 5.0.0 version of Rebus?

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Sep 4, 2018

Will the driver release with the 5.0.0 version of Rebus?

That's the plan, yes.

Usually, it's pretty easy to upgrade Rebus dependencies, even though they're major increments.

There has been no wire-level breaking changes since before version 2, so you can upgrade one endpoint at a time.

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Sep 10, 2018

so.... I've been running the new Azure Service Bus transport on all Fleet Manager environments for a while now, and it seems to run as it should. I'll close this one for now 🙂

@mookid8000 mookid8000 closed this Sep 10, 2018

@Meyce

This comment has been minimized.

Contributor

Meyce commented Sep 10, 2018

Will there be a new non-rc version of Rebus.AzureServiceBus sometime soon?

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Sep 10, 2018

yeah sure 😄

@mookid8000

This comment has been minimized.

Member

mookid8000 commented Sep 10, 2018

I've pushed Rebus.AzureServiceBus 6.0.0 to NuGet.org now – it should become visible within a few minutes 🎉 🎋

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment