Management Operations

[sealightPT]

In .Net we had a NamespaceManager and we could do things like:

if (!NamespaceManager.QueueExists(_commandQueueName))
{
    NamespaceManager.CreateQueue(_commandQueueName);
}

How can we do it with this .Net Standard library?

[jtaubensee]

Hi @sealightPT - So far we have decided not to include management operations in this library. As a part of Azure Resource Manager, we now have multiple ways to dynamically provision/manage entities.

The functionality you are looking for is now included in the Microsoft.Azure.Management.ServiceBus library:

• GitHub repo - https://github.com/Azure/azure-sdk-for-net/tree/AutoRest/src/ResourceManagement/ServiceBus
• NuGet package - https://www.nuget.org/packages/Microsoft.Azure.Management.ServiceBus/
• Sample - https://github.com/Azure-Samples/service-bus-dotnet-management

Using Resource manager is the standard way to manage Azure Resources, and we have chosen as a team to adhere to that standard. We also think that the separation is a good idea, and we do not want to encourage patterns where management operations are invoked at every run-time operation.

For example checking if a queue exists before each send, will cripple your performance. As much as this sounds obvious, we do see it regularly.

[SeanFeldman]

I can see this question repeating itself in the future. Perhaps README.MD should contain a link/reference to the new management library/samples @jtaubensee ?

[jtaubensee]

Good point @SeanFeldman - I will add that.

[jtaubensee]

Closing based on #66

[barkhashah]

@SeanFeldman Can you please point us to the github link that points to the latest 1.0.0 management library released on 30th june. The above links are old and reference to pre-release versions.

thanks
barkha shah

[barkhashah]

@SeanFeldman https://github.com/Azure/azure-sdk-for-net/tree/AutoRest/src/ResourceManagement/ServiceBus
This link is pointing to old source code. Management library 1.0.0 is way different than this. When we use nuget package manager, we do get the latest release however, I need to see the source code.

[SeanFeldman]

You're looking at a specific branch. That team might be releasing from different branches. Raise an issue in appropriate repo to get an answer since I'm not a Microsoft-ie 🙂

…

On Aug 4, 2017 14:31, "barkhashah" ***@***.***> wrote: @SeanFeldman <https://github.com/seanfeldman> https://github.com/Azure/ azure-sdk-for-net/tree/AutoRest/src/ResourceManagement/ServiceBus This link is pointing to old source code. Management library 1.0.0 is way different than this. When we use nuget package manager, we do get the latest release however, I need to see the source code. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#65 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABP7tkUAco-kA63a8jPQVHry7PvSFZMrks5sU3-3gaJpZM4LeHvm> .

[xplicit]

Asked this question along time ago Azure-Samples/service-bus-dotnet-management#9 but didn't get an answer. In new management API I did not find how to check queue existing by using connection string.

What is the equivalent for this code snippet from WindowsAzure.ServiceBus in new management API?

var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);

namespaceManager.QueueExists(queueName);
namespaceManager.CreateQueue(new QueueDescription(queueName));

[SeanFeldman]

What is the equivalent for this code snippet from WindowsAzure.ServiceBus in new management API?

The good news - you can with Microsoft.Azure.Management.ServiceBus.Fluent.
The bad news - it's not a 3 liner anymore.

var azureCredentials = SdkContext.AzureCredentialsFactory.FromFile(path); // or different way
var serviceBusManager = ServiceBusManager.Authenticate(azureCredentials, subscriptionId);

// namespace manager
var @namespace = serviceBusManager.Namespaces.GetByResourceGroup(resourceGroupName, namespaceName);

// check if queue exists
var queue = @namespace.Queues.List().FirstOrDefault(x => x.Name == queueName);
if (queue == null)
{
   // create a queue
   var createdQueue = @namespace.Queues.Define(queueName)
                    .WithSizeInMB(1024)
                    .WithMessageMovedToDeadLetterQueueOnMaxDeliveryCount(10)
                    .WithMessageLockDurationInSeconds(30)
                    .Create();
}

[SeanFeldman]

I would like to re-open this issue.
Here is the reason why it should be addressed and discussed.

The old client provided management operations out of the box. For good or for bad, it was providing the ability to perform CRUD operations and report message counts. With the new client, customers will be facing a few challenges. First, there will be sheer conversion of the ASB code. Then, there will be the shift to the AAD based authentication and ARM based approach. And then, there will be wheel reinvention since every single customer will have to implement what NamespaceManager was already doing. For that implementation, there's an Azure ResourceManagement project, that is auto-generating API either based on AutoRest branch or Fluent branch.

Here's an example of a typical confusion and amount of work each customer will have to go through to get the NamespaceManager equivalent: finding number of messages. Or an example posted here that was answered by an MVP (and not the repo maintainers) that leads to a documentation page that is far from a good example. The separate management library is rough edge and requires plumbing code to be written and tested by each customer for now obvious reason. Not to mention that there clearly a gap between ASB and Management teams:

• ServiceBus management client v1.0 feedback
• Trying to get answers for the missing API
• Reporting non-implemented properties

While there's no doubt that eventually management operations and run-time operations should not be mixed and the right step in that direction is to split the library into two, there's got to be a better story for users.

Update 2017-09-27
An option to bring to the table - a new Microsoft backed OSS to provide NamespaceManager like functionality.

[SeanFeldman]

/cc @Azure/azure-service-bus-write

[xplicit]

@SeanFeldman I still don't understand how can I get resourceGroupName, subsriptionId and other settings when I have only this connection string?

Endpoint=sb://domain.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<key is here>

Can you point me to the code which does that job? Seems that old windows version of Azure Service Bus is closed source (or maybe I could not find its source) so I was not able to look how NamespaceManager implemented and reuse that code to keep compatibility between .NET and .NET Core library.

[SeanFeldman]

@xplicit ironically, precisely the reason I'm asking to re-open this issue and re-evaluate what's been done so far.

I still don't understand how can I get resourceGroupName, subsriptionId and other settings when I have only this connection string?

You can't. The management library doesn't connect using connection string. Instead, you need to go through Azure Active Directory. See Authentication with Azure Management Libraries doco that explains what is needed and how to get it using Azure CLI (locally or using the portal).

[sbosell]

I can understand the need for connecting via the resource manager to create service bus accounts but not to create queues inside of a service bus. It is a pretty common operation for most systems which I think most if not all of the test code that has been published does. Can you please add back the capability to manage queues without having to use the Azure management libraries?

[mythz]

Have to agree, requiring a separate library or use a UI to manually create queue's is absurd for an MQ library. I'm surprised anyone puts up with this limitation, It makes everything so much more difficult and you're likely losing Customers at the door when they hit this first hurdle.

Checking/Creating RDBMS tables also shouldn't be done at runtime either, but App's should still be able to create them on Startup if they don't exist. It would be unacceptable if an RDBMS handicapped ORM's from not being able to create them within the library, likewise for MQ Brokers. I don't know any other data store or MQ with this limitation.

[niemyjski]

Yeah, it is pretty absurd but I understand why it was done initially. However, making it so hard to do anything (like requiring AAD based authentication) is pretty unacceptable. Having to use AAD just to consume a service bus in a deployed app.... This is a security and deployment nightmare. Why can't I just use my existing connection string?

[shahbarkha]

It would be terrific if we can address this issue where one does not have to use the AAD to just create the entities in the Service Bus. Connection string should be enough for the user to deal with creating entities like it used to be in the old library. I have submitted PR for Foundatio community that has new implementation of azure service bus and it looks really ugly having to deal with management library in the service bus implementation.
@SeanFeldman I would really love it if you could review my PR and give any comments whenever you have a chance :)
FoundatioFx/Foundatio.AzureServiceBus#1

[SeanFeldman]

tagging @ChristianWolf42

[SeanFeldman]

An option to bring to the table - a new Microsoft backed OSS to provide NamespaceManager like functionality.

[janpieterz]

This sounds very odd, is the plan to move all Azure management operations to separate libraries backed by Azure AD? Like, are we going to need Azure AD credentials and resource manager/subscription knowledge to create tables in a hosted SQL database for example? Create tables, containers in storage? 😕

It sounds way better to use non-management and management connection strings (like we've been doing since forever with read/write/create permissions in SQL) than to jump through all these extra hoops to get the right stuff and need knowledge of subscription and resource groups into applications and tests to run and be deployed to new/changed underlying infra.

Please provide the NamespaceManager like capabilities. Either through something open sourced or simply included in the default library, with no need for extra configuration items (just connection strings) and no need for knowledge for apps where they're going to run. This approach will considerably increase configuration and subscription management with, for us at least, no return.

[nemakam]

@0xced
Yes. Aiming at June.

[mscrivo]

@SeanFeldman and @nemakam We've been having a ton of problems with SB lately, lots of throttling, random exceptions. We've written a ton of code to handle all the failure conditions and we're finally able to tread water at least. Anyhow, all that aside, we've done what you guys have asked and switched from using the GET API call to get queue counts to using the Metrics API instead for counts and the numbers we're see are way off. For example, for one of our queues, GET is reporting 11m active messages, whereas the metrics API is reporting 800k. Any idea what's going on there?

Again, we use these numbers to automatically scale up/down our various worker tasks and need these counts to be somewhat accurate. We used to poll the GET on each queue every 3-5 mins, and now we're querying the metrics every 5.

[SeanFeldman]

Any idea what's going on there?

Metrics do not report accurate numbers at the moment AFAIK.
@ShubhaVijayasarathy might help and shed some light on the recent developments.

[awr]

Huh. Then why the recommendation to use it over regular counts?

Does it work on some skus and not others? Seems like the numbers on Premium are more accurate than on Standard. Any timelines on when this will work correctly? Otherwise we'll have to switch back to the old approach once again...

[nemakam]

@mscrivo
800k to 11m. That's way off. This might need some investigation to be done. Could you raise a support ticket online (through Azure Portal) and our Support executives can find out about the discrepancy you are seeing.

[mscrivo]

@nemakam already done.

[RogerDawson]

@mscrivo , I hear you on the service bus issues, we have our own retry logic on top of the service bus retry logic to try and cope with the SB issues but they still get through. The quality of SB standard has nose dived over the past year. We've been pushed by MS to SB premium in some cases to reduce the exceptions and throttling but that's not got great performance/scalability and the costs are astronomical in comparison to standard. It's easily our biggest maintenance headache and support generator on Azure. Sorry that this is slightly off-topic.

[mscrivo]

@RogerDawson thanks for that. It's reassuring that we're not the only ones.

[SeanFeldman]

Removed Backlog milestone as it's a WIP.

[SeanFeldman]

@nemakam should this be 3.1.0 or 4.0.0?

[nemakam]

There is no breaking change. 3.1.0 seems valid according to SemVer.

[SeanFeldman]

🎉 😍 🥂
On behalf of the entire ASB .NET customer community - THANK YOU!!!