Driver for Azure Resource Graph

[rcobb-scwx]

Hi mstic team,

I wrote a custom driver/data environment that leverages the Azure Python SDK to hit the Azure Resource Graph. This lets us query for resources using the power of the YAML templates. I know about azure_data, but I think the Resource Graph has access to a much wider body of security relevant information.

Is this something that folks would be interested in if I opened a PR? If so, is there anyone in particular I could work with to get that over the line?

If this isn't the right channel to ask, feel free to nuke this issue.

All the best,
Ryan

[ianhelle]

Ryan,
That looks awesome. I would love to have this in msticpy.
I can definitely help. @petebryan also.

[rcobb-scwx]

Fantastic. I have asked our legal to work through the CLA paperwork. I will update as soon as I can.

[petebryan]

That's awesome @rcobb-scwx - I actually had a backlog item in my queue to create this exact thing! Its great to see someone has already done it.

[rcobb-scwx]

I'm glad to hear it. I have a few questions that I'll lob over before the CLA paperwork is finished, so we can make a little progress at least over the next few business days.

[rcobb-scwx]

Good news! I think the CLA hurdles have been cleared. The ball is back in my court. I will get a PR opened up this week.

[rcobb-scwx]

Hello folks,

My PR is just about ready to roll. I have a few quick questions:

• Where/how would you prefer me to add the dependencies for azure-mgmt-resourcegraph? I figured both conda-reqs-pip.txt and requirements.txt but was not sure.
• For my driver to work, I had to explicitly add self._loaded = True in the class init. I don't see this in other classes. Is this alright/normal?
• I noticed that there is no convention around logging. It looks like most messages are printed (except for warnings.warn and one of the Azure auth modules uses logging.basicConfig. If I wanted to include more verbose logging, is there a preferred way?
• This isn't really a msticpy question, but more of a general one that I haven't figured out yet. It looks like the Resource Graph has some limitations on pagination. When I looked at how the azcli implemented pagination, it appears that pagination might be done automagically somewhere in the HTTP pipeline. But I have seen other implementations explicitly handle pagination. I don't have a large enough tenant to really test pagination, so written current this would simply behave as the azcli would -- whatever that implies. Any ideas on the best practice here?
• The test cases for drivers are pretty skimpy. I cargo-culted the tests for MDATP/SecurityGraph because I think the ResourceGraph is most similar to those drivers. Do I need anything else in regards to testing?

And lastly here are some TODO items for future work:

• Add support for resource change history queries (new preview APIs)
• Add support for selecting specific subscriptions from the list of subs

I think this will make more sense once the PR is submitted. I just wanted to get these out to make the process as easy as possible for you folks.

Thanks!
Ryan

[rcobb-scwx]

Good morning everyone. I just opened up my PR -- PTAL.

[ianhelle]

Hey @rcobb-scwx.
We had to switch this to another PR to deal with some issues in our build pipeline. This has just been reviewed and should be merged shortly.
Just to answer (belatedly):

• The requirements stuff should be sorted out - we have most azure libraries part of the "azure" extra (since some are big and time-consuming to install if you're not using them)
• The self._loaded thing is only really relevant for providers that have a distinct loading/initialization stage (i.e. something more than just importing) - what you did is what we do for most providers
• logging - yeah - I'm regretting not building this in from the start. Adding it is on our to-do list - just not a big priority vs. the number of people we have to do the work.
• not sure about the pagination question
• I think @petebryan has added more test cases with mocks - in fact Pete has done all the work on this. I'm just reply :-)
  TODO items
• the change history queries sound cool - esp if you are trying to investigate unauthorized mods to something
• for the subscription selection - if you want this to be interactive, we've created a bunch of widgets (e.g. SelectSubset) that would let you interactively prompt for subscription selection and then use the values in subsequent queries.

Thanks again for the great work on this - awesome to have this capability.

[rcobb-scwx]

The pleasure is all mine. I learned a lot doing it and am very happy to contribute back to such a useful project. I look forward to what comes next. Happy hunting!

[ianhelle]

This has now been merged via #167 into our main branch. Should be published in a few days!
Thx again!