[Question] Need assistance in generating rules for Nginx logs

[Loginsoft-Research]

Hi Elastic Team,

We are trying to write new rules based on our research. We have been successful in writing rules from the Kibana SIEM dashboard, however on exporting the rules we get them in ndjson format. However, on writing a rule with the same query using detection_rules python module, it errors out.

Example: Rule for a vulnerability on Nginx

• On shipping the logs with filebeat nginx module, there are fields which are not in ECS.
• How should I go about writing rules for them using detection_rules module, as it errors out on using non-ECS fields?

Command: rlwrap python3 -m detection_rules create-rule ../nginx_rules/"testing".toml
Error log:

query (required): log.file.path: "/var/log/nginx/error.log" AND message: "checking"                                                                      
ecs_version (multi, comma separated): 1.5.0                                                                                                              
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/cyber/research/detection-rules/detection_rules/__main__.py", line 28, in <module>
    main()
  File "/home/cyber/research/detection-rules/detection_rules/__main__.py", line 25, in main
    root(prog_name="detection_rules")
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)

• Is there a possibility to export the rules in TOML format from Kibana instead of ndjson?
• How can we import rules in TOML format into Kibana?

[rw-access]

Hi @Loginsoft-Research do you have the full output? It looks like this only captures part of the traceback, but there should be more information as well as the exception message. If you're using a local fork, you can modify blob/main/etc/non-ecs-schema.json to add your exceptions when you aren't using ECS fields.

Also, I belive our minimum supported version of Python is 3.7. You might get lucky with 3.6, but we haven't tested that. https://github.com/elastic/detection-rules#getting-started

[brokensound77]

Hello 👋, thanks for the questions. Hopefully this helps:

---

On shipping the logs with filebeat nginx module, there are fields which are not in ECS.
How should I go about writing rules for them using detection_rules module, as it errors out on using non-ECS fields?

In addition to what @rw-access said, if the non-ECS fields are beats fields, then as long as you define the event.dataset and event.module (they need to be logged too) as expected by your defined index, it will pass validation. We do this as a means to ensure tighter rule logic.

Is there a possibility to export the rules in TOML format from Kibana instead of ndjson?

At the moment, the rules can only be exported in ndjson format. I just opened #83 to better support importing or creating rules which were exported from kibana, so you could do:
python -m detection_rules create-rule -c export.ndjson --required-only rules/my_custom_rule.toml

This would build the rule from your exported rule and prompt you for any additional needed information

How can we import rules in TOML format into Kibana?

Support for this was just added as of #50 and can be done with python -m detection_rules kibana-upload <list> <of> <files> [options]

As far as your error - as @rw-access mentioned, with the rest of the error information, we might be able to determine the issue and whether it will be solved by any of the mentioned PR's

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This has been closed due to inactivity. If you feel this is an error, please re-open and include a justifying comment.