New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Management Operations #65

Closed
sealightPT opened this Issue Jan 9, 2017 · 93 comments

Comments

Projects
None yet
@sealightPT
Copy link

sealightPT commented Jan 9, 2017

In .Net we had a NamespaceManager and we could do things like:

if (!NamespaceManager.QueueExists(_commandQueueName))
{
    NamespaceManager.CreateQueue(_commandQueueName);
}

How can we do it with this .Net Standard library?

@jtaubensee

This comment has been minimized.

Copy link
Contributor

jtaubensee commented Jan 9, 2017

Hi @sealightPT - So far we have decided not to include management operations in this library. As a part of Azure Resource Manager, we now have multiple ways to dynamically provision/manage entities.

The functionality you are looking for is now included in the Microsoft.Azure.Management.ServiceBus library:

Using Resource manager is the standard way to manage Azure Resources, and we have chosen as a team to adhere to that standard. We also think that the separation is a good idea, and we do not want to encourage patterns where management operations are invoked at every run-time operation.

For example checking if a queue exists before each send, will cripple your performance. As much as this sounds obvious, we do see it regularly.

@jtaubensee jtaubensee changed the title How to check if an entity already exists and create it if not (queue, topic, subscription) Management Operations Jan 9, 2017

@jtaubensee jtaubensee self-assigned this Jan 9, 2017

@SeanFeldman

This comment has been minimized.

Copy link
Collaborator

SeanFeldman commented Jan 9, 2017

I can see this question repeating itself in the future. Perhaps README.MD should contain a link/reference to the new management library/samples @jtaubensee ?

@jtaubensee

This comment has been minimized.

Copy link
Contributor

jtaubensee commented Jan 9, 2017

Good point @SeanFeldman - I will add that.

@jtaubensee jtaubensee referenced this issue Jan 9, 2017

Merged

ReadMe update #66

@jtaubensee

This comment has been minimized.

Copy link
Contributor

jtaubensee commented Jan 9, 2017

Closing based on #66

@barkhashah

This comment has been minimized.

Copy link

barkhashah commented Aug 4, 2017

@SeanFeldman Can you please point us to the github link that points to the latest 1.0.0 management library released on 30th june. The above links are old and reference to pre-release versions.

thanks
barkha shah

@barkhashah

This comment has been minimized.

Copy link

barkhashah commented Aug 4, 2017

@SeanFeldman https://github.com/Azure/azure-sdk-for-net/tree/AutoRest/src/ResourceManagement/ServiceBus
This link is pointing to old source code. Management library 1.0.0 is way different than this. When we use nuget package manager, we do get the latest release however, I need to see the source code.

@SeanFeldman

This comment has been minimized.

Copy link
Collaborator

SeanFeldman commented Aug 4, 2017

@xplicit

This comment has been minimized.

Copy link

xplicit commented Aug 26, 2017

Asked this question along time ago Azure-Samples/service-bus-dotnet-management#9 but didn't get an answer. In new management API I did not find how to check queue existing by using connection string.

What is the equivalent for this code snippet from WindowsAzure.ServiceBus in new management API?

var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);

namespaceManager.QueueExists(queueName);
namespaceManager.CreateQueue(new QueueDescription(queueName));
@SeanFeldman

This comment has been minimized.

Copy link
Collaborator

SeanFeldman commented Aug 30, 2017

What is the equivalent for this code snippet from WindowsAzure.ServiceBus in new management API?

The good news - you can with Microsoft.Azure.Management.ServiceBus.Fluent.
The bad news - it's not a 3 liner anymore.

var azureCredentials = SdkContext.AzureCredentialsFactory.FromFile(path); // or different way
var serviceBusManager = ServiceBusManager.Authenticate(azureCredentials, subscriptionId);

// namespace manager
var @namespace = serviceBusManager.Namespaces.GetByResourceGroup(resourceGroupName, namespaceName);

// check if queue exists
var queue = @namespace.Queues.List().FirstOrDefault(x => x.Name == queueName);
if (queue == null)
{
   // create a queue
   var createdQueue = @namespace.Queues.Define(queueName)
                    .WithSizeInMB(1024)
                    .WithMessageMovedToDeadLetterQueueOnMaxDeliveryCount(10)
                    .WithMessageLockDurationInSeconds(30)
                    .Create();
}
@SeanFeldman

This comment has been minimized.

Copy link
Collaborator

SeanFeldman commented Sep 10, 2017

I would like to re-open this issue.
Here is the reason why it should be addressed and discussed.

The old client provided management operations out of the box. For good or for bad, it was providing the ability to perform CRUD operations and report message counts. With the new client, customers will be facing a few challenges. First, there will be sheer conversion of the ASB code. Then, there will be the shift to the AAD based authentication and ARM based approach. And then, there will be wheel reinvention since every single customer will have to implement what NamespaceManager was already doing. For that implementation, there's an Azure ResourceManagement project, that is auto-generating API either based on AutoRest branch or Fluent branch.

Here's an example of a typical confusion and amount of work each customer will have to go through to get the NamespaceManager equivalent: finding number of messages. Or an example posted here that was answered by an MVP (and not the repo maintainers) that leads to a documentation page that is far from a good example. The separate management library is rough edge and requires plumbing code to be written and tested by each customer for now obvious reason. Not to mention that there clearly a gap between ASB and Management teams:

While there's no doubt that eventually management operations and run-time operations should not be mixed and the right step in that direction is to split the library into two, there's got to be a better story for users.

Update 2017-09-27
An option to bring to the table - a new Microsoft backed OSS to provide NamespaceManager like functionality.

@SeanFeldman

This comment has been minimized.

Copy link
Collaborator

SeanFeldman commented Sep 10, 2017

/cc @Azure/azure-service-bus-write

@xplicit

This comment has been minimized.

Copy link

xplicit commented Sep 10, 2017

@SeanFeldman I still don't understand how can I get resourceGroupName, subsriptionId and other settings when I have only this connection string?

Endpoint=sb://domain.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<key is here>

Can you point me to the code which does that job? Seems that old windows version of Azure Service Bus is closed source (or maybe I could not find its source) so I was not able to look how NamespaceManager implemented and reuse that code to keep compatibility between .NET and .NET Core library.

@SeanFeldman

This comment has been minimized.

Copy link
Collaborator

SeanFeldman commented Sep 11, 2017

@xplicit ironically, precisely the reason I'm asking to re-open this issue and re-evaluate what's been done so far.

I still don't understand how can I get resourceGroupName, subsriptionId and other settings when I have only this connection string?

You can't. The management library doesn't connect using connection string. Instead, you need to go through Azure Active Directory. See Authentication with Azure Management Libraries doco that explains what is needed and how to get it using Azure CLI (locally or using the portal).

@sbosell

This comment has been minimized.

Copy link

sbosell commented Sep 12, 2017

I can understand the need for connecting via the resource manager to create service bus accounts but not to create queues inside of a service bus. It is a pretty common operation for most systems which I think most if not all of the test code that has been published does. Can you please add back the capability to manage queues without having to use the Azure management libraries?

@mythz

This comment has been minimized.

Copy link

mythz commented Sep 12, 2017

Have to agree, requiring a separate library or use a UI to manually create queue's is absurd for an MQ library. I'm surprised anyone puts up with this limitation, It makes everything so much more difficult and you're likely losing Customers at the door when they hit this first hurdle.

Checking/Creating RDBMS tables also shouldn't be done at runtime either, but App's should still be able to create them on Startup if they don't exist. It would be unacceptable if an RDBMS handicapped ORM's from not being able to create them within the library, likewise for MQ Brokers. I don't know any other data store or MQ with this limitation.

@niemyjski

This comment has been minimized.

Copy link

niemyjski commented Sep 12, 2017

Yeah, it is pretty absurd but I understand why it was done initially. However, making it so hard to do anything (like requiring AAD based authentication) is pretty unacceptable. Having to use AAD just to consume a service bus in a deployed app.... This is a security and deployment nightmare. Why can't I just use my existing connection string?

@shahbarkha

This comment has been minimized.

Copy link

shahbarkha commented Sep 12, 2017

It would be terrific if we can address this issue where one does not have to use the AAD to just create the entities in the Service Bus. Connection string should be enough for the user to deal with creating entities like it used to be in the old library. I have submitted PR for Foundatio community that has new implementation of azure service bus and it looks really ugly having to deal with management library in the service bus implementation.
@SeanFeldman I would really love it if you could review my PR and give any comments whenever you have a chance :)
FoundatioFx/Foundatio.AzureServiceBus#1

@SeanFeldman

This comment has been minimized.

Copy link
Collaborator

SeanFeldman commented Sep 15, 2017

@SeanFeldman

This comment has been minimized.

Copy link
Collaborator

SeanFeldman commented Sep 27, 2017

An option to bring to the table - a new Microsoft backed OSS to provide NamespaceManager like functionality.

@janpieterz

This comment has been minimized.

Copy link

janpieterz commented Sep 27, 2017

This sounds very odd, is the plan to move all Azure management operations to separate libraries backed by Azure AD? Like, are we going to need Azure AD credentials and resource manager/subscription knowledge to create tables in a hosted SQL database for example? Create tables, containers in storage? 😕

It sounds way better to use non-management and management connection strings (like we've been doing since forever with read/write/create permissions in SQL) than to jump through all these extra hoops to get the right stuff and need knowledge of subscription and resource groups into applications and tests to run and be deployed to new/changed underlying infra.

Please provide the NamespaceManager like capabilities. Either through something open sourced or simply included in the default library, with no need for extra configuration items (just connection strings) and no need for knowledge for apps where they're going to run. This approach will considerably increase configuration and subscription management with, for us at least, no return.

@0xced

This comment has been minimized.

Copy link

0xced commented May 28, 2018

Well, it turns out that implementing GetQueues and GetTopics was fairly easy. A pull request is waiting for you: Mossharbor/AzureWorkArounds.ServiceBus#6 😉

0xced added a commit to 0xced/SbManager that referenced this issue May 29, 2018

[WIP] Switch to Microsoft.Azure.ServiceBus
The [`Microsoft.Azure.ServiceBus`](https://www.nuget.org/packages/Microsoft.Azure.ServiceBus/) package conforms to .NET Standard and thus works fine on Linux and macOS (using Mono). The WindowsAzure.ServiceBus package can't run on Mono since it depends kernel32.dll.

Required incidental changes:

* Upgrade target framework from v4.6 to v4.7.1
* Swith to async code
* Upgrade `Newtonsoft.Json` from version 9 to version 10
* Delete IsBodyConsumed and State for which I have found no equivalent
* Take a dependency on `Mossharbor.AzureWorkArounds.ServiceBus` since management is not (yet) supported with a service bus connection string, see Azure/azure-service-bus-dotnet#65
* Use a local version of `Mossharbor.AzureWorkArounds.ServiceBus` since `GetQueues()` and `GetTopics()` are not implemented in the published package Mossharbor/AzureWorkArounds.ServiceBus#6

0xced added a commit to 0xced/SbManager that referenced this issue May 29, 2018

[WIP] Switch to Microsoft.Azure.ServiceBus
The [`Microsoft.Azure.ServiceBus`](https://www.nuget.org/packages/Microsoft.Azure.ServiceBus/) package conforms to .NET Standard and thus works fine on Linux and macOS (using Mono). The WindowsAzure.ServiceBus package can't run on Mono since it depends on kernel32.dll.

Required incidental changes:

* Upgrade target framework from v4.6 to v4.7.1
* Swith to async code
* Upgrade `Newtonsoft.Json` from version 9 to version 10
* Delete IsBodyConsumed and State for which I have found no equivalent
* Take a dependency on `Mossharbor.AzureWorkArounds.ServiceBus` since management is not (yet) supported with a service bus connection string, see Azure/azure-service-bus-dotnet#65
* Use a local version of `Mossharbor.AzureWorkArounds.ServiceBus` since `GetQueues()` and `GetTopics()` are not implemented in the published package Mossharbor/AzureWorkArounds.ServiceBus#6
@nemakam

This comment has been minimized.

Copy link
Member

nemakam commented May 30, 2018

@0xced
Yes. Aiming at June.

@mscrivo

This comment has been minimized.

Copy link
Contributor

mscrivo commented Jun 1, 2018

@SeanFeldman and @nemakam We've been having a ton of problems with SB lately, lots of throttling, random exceptions. We've written a ton of code to handle all the failure conditions and we're finally able to tread water at least. Anyhow, all that aside, we've done what you guys have asked and switched from using the GET API call to get queue counts to using the Metrics API instead for counts and the numbers we're see are way off. For example, for one of our queues, GET is reporting 11m active messages, whereas the metrics API is reporting 800k. Any idea what's going on there?

Again, we use these numbers to automatically scale up/down our various worker tasks and need these counts to be somewhat accurate. We used to poll the GET on each queue every 3-5 mins, and now we're querying the metrics every 5.

@SeanFeldman

This comment has been minimized.

Copy link
Collaborator

SeanFeldman commented Jun 1, 2018

Any idea what's going on there?

Metrics do not report accurate numbers at the moment AFAIK.
@ShubhaVijayasarathy might help and shed some light on the recent developments.

@awr

This comment has been minimized.

Copy link

awr commented Jun 1, 2018

Huh. Then why the recommendation to use it over regular counts?

Does it work on some skus and not others? Seems like the numbers on Premium are more accurate than on Standard. Any timelines on when this will work correctly? Otherwise we'll have to switch back to the old approach once again...

@nemakam

This comment has been minimized.

Copy link
Member

nemakam commented Jun 1, 2018

@mscrivo
800k to 11m. That's way off. This might need some investigation to be done. Could you raise a support ticket online (through Azure Portal) and our Support executives can find out about the discrepancy you are seeing.

@mscrivo

This comment has been minimized.

Copy link
Contributor

mscrivo commented Jun 1, 2018

@nemakam already done.

0xced added a commit to 0xced/SbManager that referenced this issue Jun 3, 2018

[WIP] Switch to Microsoft.Azure.ServiceBus
The [`Microsoft.Azure.ServiceBus`](https://www.nuget.org/packages/Microsoft.Azure.ServiceBus/) package conforms to .NET Standard and thus works fine on Linux and macOS (using Mono). The WindowsAzure.ServiceBus package can't run on Mono since it depends on kernel32.dll.

Required incidental changes:

* Upgrade target framework from v4.6 to v4.7.1
* Swith to async code
* Upgrade `Newtonsoft.Json` from version 9 to version 10
* Delete IsBodyConsumed and State for which I have found no equivalent
* Take a dependency on `Mossharbor.AzureWorkArounds.ServiceBus` since management is not (yet) supported with a service bus connection string, see Azure/azure-service-bus-dotnet#65
* Use a local version of `Mossharbor.AzureWorkArounds.ServiceBus` since `GetQueues()` and `GetTopics()` are not implemented in the published package Mossharbor/AzureWorkArounds.ServiceBus#6

0xced added a commit to 0xced/SbManager that referenced this issue Jun 3, 2018

[WIP] Switch to Microsoft.Azure.ServiceBus
The [`Microsoft.Azure.ServiceBus`](https://www.nuget.org/packages/Microsoft.Azure.ServiceBus/) package conforms to .NET Standard and thus works fine on Linux and macOS (using Mono). The WindowsAzure.ServiceBus package can't run on Mono since it depends on kernel32.dll.

Required incidental changes:

* Upgrade target framework from v4.6 to v4.7.1
* Swith to async code
* Upgrade `Newtonsoft.Json` from version 9 to version 10
* Delete IsBodyConsumed and State for which I have found no equivalent
* Take a dependency on `Mossharbor.AzureWorkArounds.ServiceBus` since management is not (yet) supported with a service bus connection string, see Azure/azure-service-bus-dotnet#65
* Use a local version of `Mossharbor.AzureWorkArounds.ServiceBus` since `GetQueues()` and `GetTopics()` are not implemented in the published package Mossharbor/AzureWorkArounds.ServiceBus#6
@RogerDawson

This comment has been minimized.

Copy link

RogerDawson commented Jun 4, 2018

@mscrivo , I hear you on the service bus issues, we have our own retry logic on top of the service bus retry logic to try and cope with the SB issues but they still get through. The quality of SB standard has nose dived over the past year. We've been pushed by MS to SB premium in some cases to reduce the exceptions and throttling but that's not got great performance/scalability and the costs are astronomical in comparison to standard. It's easily our biggest maintenance headache and support generator on Azure. Sorry that this is slightly off-topic.

@mscrivo

This comment has been minimized.

Copy link
Contributor

mscrivo commented Jun 4, 2018

@RogerDawson thanks for that. It's reassuring that we're not the only ones.

@SeanFeldman SeanFeldman referenced this issue Jun 6, 2018

Closed

Management? #479

0xced added a commit to 0xced/SbManager that referenced this issue Jun 6, 2018

[WIP] Switch to Microsoft.Azure.ServiceBus
The [`Microsoft.Azure.ServiceBus`](https://www.nuget.org/packages/Microsoft.Azure.ServiceBus/) package conforms to .NET Standard and thus works fine on Linux and macOS (using Mono). The WindowsAzure.ServiceBus package can't run on Mono since it depends on kernel32.dll.

Required incidental changes:

* Upgrade target framework from v4.6 to v4.7.1
* Swith to async code
* Upgrade `Newtonsoft.Json` from version 9 to version 10
* Delete IsBodyConsumed and State for which I have found no equivalent
* Take a dependency on `Mossharbor.AzureWorkArounds.ServiceBus` since management is not (yet) supported with a service bus connection string, see Azure/azure-service-bus-dotnet#65
* Use a local version of `Mossharbor.AzureWorkArounds.ServiceBus` since `GetQueues()` and `GetTopics()` are not implemented in the published package Mossharbor/AzureWorkArounds.ServiceBus#6

0xced added a commit to 0xced/SbManager that referenced this issue Jun 7, 2018

[WIP] Switch to Microsoft.Azure.ServiceBus
The [`Microsoft.Azure.ServiceBus`](https://www.nuget.org/packages/Microsoft.Azure.ServiceBus/) package conforms to .NET Standard and thus works fine on Linux and macOS (using Mono). The WindowsAzure.ServiceBus package can't run on Mono since it depends on kernel32.dll.

Required incidental changes:

* Upgrade target framework from v4.6 to v4.7.1
* Swith to async code
* Upgrade `Newtonsoft.Json` from version 9 to version 10
* Delete IsBodyConsumed and State for which I have found no equivalent
* Take a dependency on `Mossharbor.AzureWorkArounds.ServiceBus` since management is not (yet) supported with a service bus connection string, see Azure/azure-service-bus-dotnet#65
* Use a local version of `Mossharbor.AzureWorkArounds.ServiceBus` since `GetQueues()` and `GetTopics()` are not implemented in the published package Mossharbor/AzureWorkArounds.ServiceBus#6

@SeanFeldman SeanFeldman removed this from the Backlog milestone Jun 27, 2018

@SeanFeldman

This comment has been minimized.

Copy link
Collaborator

SeanFeldman commented Jun 27, 2018

Removed Backlog milestone as it's a WIP.

@SeanFeldman

This comment has been minimized.

Copy link
Collaborator

SeanFeldman commented Jun 27, 2018

@nemakam should this be 3.1.0 or 4.0.0?

@nemakam

This comment has been minimized.

Copy link
Member

nemakam commented Jun 27, 2018

There is no breaking change. 3.1.0 seems valid according to SemVer.

@nemakam nemakam added this to the 3.1.0 milestone Jun 27, 2018

@nemakam nemakam closed this in #481 Jun 28, 2018

nemakam added a commit that referenced this issue Jun 28, 2018

Management Operations - CRUD of entities (#481)
Introduces Serivce Bus Management operations - CRUD of entities
Fixes #65 

The API is exposed through `ManagementClient` which uses HTTP underneath to perform these operations. 
Summary of exposed operations:
1. `GetQueue` returns the `QueueDescription` which contains static information about the queue. 
1. `QueueDescription` is required to Create or Update the queue.
1. `ManagementClient.GetQueueRuntimeInfo` returns `QueueRuntimeInfo` which contains runtime information of the queue like the message count and the size.
1. `GetQueues` will return list of `QueueDescription` and will not contain runtime information.

Majority of the operations are similar to the ones exposed in older client (`Microsoft.ServiceBus` -> `NamespaceManager`), with following differences:
1. `GetQueue` now returns only static information as part of `QueueDescription` and runtime  information is part of `QueueRuntimeInfo`.
1. Default value of `DuplicateDetection` feature is 1 minute.
@SeanFeldman

This comment has been minimized.

Copy link
Collaborator

SeanFeldman commented Jun 29, 2018

🎉 😍 🥂
On behalf of the entire ASB .NET customer community - THANK YOU!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment