Publishing of pySigma-datadog-backend

[apiazza-dd]

Hello Sigma Maintainers!

My name is Andrea Piazza and I’m a software engineer on Datadog’s Cloud SIEM team. We’re looking forward to offering support for Sigma Rules in our Cloud SIEM product and are wondering if you can please publish the pysigma-backend-datadog per this part of the ReadMe?

I wanted to bring attention to a couple of areas of our repo that are currently broken due to the Datadog package not being supported by Sigma yet, and I’m happy to help resolve these issues in any way I can:

• Our workflows and import statements are currently broken on the main branch because datadog is not currently supported in hte Sigma Library. We’ve created a pysigma-datadog-backend-local branch with a workaround hack for local testing and the tests we’ve written pass on this branch.
• Our workflows are currently failing with the following error:
  /home/runner/work/pysigma-backend-datadog/pysigma-backend-datadog/sigma does not contain any element
  I’m expecting this error to be resolved once the Datadog backend is supported. If not, I’d be more than happy to help troubleshoot any error that’s manifesting afterwards.
• I noticed that Thomas Patzke is listed as a maintainer on the other pySigma-backend- repositories and I’m wondering if we should also add you under the “maintainer” section as well?

On a personal note, I appreciate all of the effort you’ve put in to make contributing to this project so seamless. I really enjoyed working with the pySigma Library and am excited that our customers will be able to increase their coverage with Sigma rules!

I’m looking forward to your feedback, learning how to improve the pysigma-datadog-repo and how we can get this supported on pyPi. Please let me know how I can help with the next steps.

Thank you,

Andrea Piazza

[thomaspatzke]

Hi! Nice to see that Datadog now also gets supported by pySigma and Sigma CLI! To publish it and get the backend exposed to users of the CLI, a PR to the plugin directory is required. Feel free to provide a pull request, but I can also make this at the weekend.

Generally, the best would be to have a stable version released on PyPI. This makes it more accessible to developers who want to integrate the backend in custom tooling and is also cleaner than referring to a source repository or branches.

[apiazza-dd]

Hi Thomas! Thanks for the quick turnaround! I've submitted a PR for Datadog with a testing status to the plugin directory. Please let us know if there's anything else you need from us!

[thomaspatzke]

It's merged 😉

[apiazza-dd]

Thank you!

[apiazza-dd]

@thomaspatzke

Hi Thomas,

I'm trying to use the sigma-cli to convert a datadog rule using the following command:
sigma convert -t datadog_backend -p datadog_pipeline ../../andrea.piazza/sigma/sigma/rules/cloud/aws

which outputs queries with an extra @ sign.

Example:

@@evt.source:rds.amazonaws.com AND @@responseElements.pendingModifiedValues.masterUserPassword:* AND @@evt.name:ModifyDBInstance

I've made the fix in the repo here , but when I use the Sigma CLI, I'm not seeing the change picked up. The tests currently have the correct expected output, so I'm curious how the additional @ sign is being added.

I'm hoping the tool will output something like this:

@evt.source:rds.amazonaws.com AND @responseElements.pendingModifiedValues.masterUserPassword:* AND @evt.name:ModifyDBInstance

Any help is greatly appreciated!

Thanks!

Andréa

[thomaspatzke]

Hi Andréa!

I've tried your backend and identified the issue. The import error is caused by the dependency to the sigma package, which is not related to "our" Sigma 😉 This causes that the sigma namespace package clashes with the package from the unrelated package. It also broke the CLI when the Datadog backend was installed.

There were some further minor issues, one of them was the problem that the code of DatadogFieldMappingTransformation was completely commented out in a way that it caused a syntax error.

The double @ was caused because the pipeline was applied twice: as backend pipeline and again as pipeline passed on the command line. I've created another fix that disables requires_pipeline in the backend. This is not required because the backend_processing_pipeline is applied automatically with this backend.

Another small fix: I've replaced your custom UnsupportedSyntax exception with the pySigma SigmaFeatureNotSupportedByBackendError. This is the exception of choice for this purpose and has the advantage that CLI can skip rules not supported by the backend if requesed by the user.

I provided a pull request that fixes all issues and removes the warnings. Please check if this matches your expectations. While testing please also take care that you work with a clean environment, especially because your former working environment might still contain the unrelated sigma package. This can be ensured by first deleting the Poetry-managed virtual Python environment with poetry env remove python (last one might vary depending on your system) and reinstalling it with poetry install.

Regarding your first question: backend autodiscovery only works if the backend and pipeline is located within the sigma.backends and sigma.pipelines package, therefore it should be contained here to work smoothly with the pySigma plugin autodiscovery that is also utilized by the CLI. I think it doesn't matters anymore because the issue is fixed now.

And the second question: it's up to you. We can transfer it to SigmaHQ, you can also maintain it inside your DataDog organization. There are no real advantages or disadvantages from this decision. More important is the commitment of vendors to the pySigma backend because as usual in open source the Sigma project is also short in resources for development and maintenance. Furthermore, the PyPI release would have some advantages once you consider the backend as stable.

[apiazza-dd]

Hi Thomas!!

Thank you so much for all of your help, I can't begin to tell you how much we appreciate it!

I'm still noticing the rules being output with two @ signs when I use the following command with the sigma-cli. I'm wondering if this was also the case when you tested? The tests are succeeding with the single @ sign but the pipeline still applies them twice:

sigma convert -t datadog_backend -p datadog_pipeline ../../andrea.piazza/sigma/sigma/rules/cloud/aws -f siem_rule

@@evt.source:rds.amazonaws.com AND @@responseElements.pendingModifiedValues.masterUserPassword:* AND @@evt.name:ModifyDBInstance

I've tried reinstalling the Datadog backend in a completely new virtual environment.

In terms of the repo, can we transfer ownership to Sigma in case members of the open source community want to contribute? Datadog absolutely commits to the maintenance of our backend and are happy to help in anyway we can. This is such a cool project and we appreciate all the work you've put into it.

Thanks!

Andréa

[thomaspatzke]

Hi Andréa! The @ appears twice because of the -p datadog_pipeline, which I omitted while my tests.

Sure we can transfer it to SigmaHQ. A repository owner you have to initiate it in the danger zone of the repository settings.

[apiazza-dd]

Thank you so much, Thomas! I can't tell you how much we appreciate all of your help and work on the pySigma project.

We'll transfer the ownership to Sigma tomorrow and please let us know how we can help you maintain the Datadog backend!

[apiazza-dd]

Hi Thomas!

We're trying to transfer the repo to https://github.com/SigmaHQ and we need to have repo creation permission in the receiving org.
Do you know how we can do that?

Thanks!
Andréa

[thomaspatzke]

I forked the repository and gave you administrative access to the fork, this should have the same effect as transferring the original repository, except that the source repository still exists.

[apiazza-dd]

Thank you so much, Thomas! We really appreciate all of your help!