Custom local.rules not showing up in kibana NIDS page

[dstults]

Previously, when performing the exact same experiment on securityonion-16.04.4.1iso install, all custom rules showed up by default on the NIDS page. As for the most recent update (16.04.6.3), there is no clear way to make them show there, you have to click into .

To replicate the issue, here are the steps:

Fresh install of Security Onion 16.04.6.3 ISO to hardware:
Two NICs, one facing management network, one monitoring mirrored port for test network
Setup for Production Mode, pretty much all defaults, suricata
create alert rules for /etc/nsm/local.rules and run rule-update
Log into scapy/msf on kalibox, send a few suspicious packets
Log into Kibana on SO, click on NIDS, and I see that they've all been registered:

But when I look for details on them, all I see are default rules (46/285 alerts):

The only way to see the custom rules is by going down on click on this:

Then all the custom rules are visible, but that was quite buried:

Current workaround:

The best workaround I've found is to navigate to the "Indicator" page (which doesn't have a default link on Home either, you have to navigate to the page above and then remove the IP filter):

Which then lets me see all the alerts for all the things at once (previously, this was visible on the NIDS page, however it is not anymore):

[dougburks]

Hi @DranKof ,

Since your local rules are showing up starting with GID:SID (example: [1:1000008:1]), it sounds like something might be going wrong during rule-update. Are you able to share the full output of that command?

[dstults]

Sure thing, here's the set of rules I was using before and am using now on both versions, as well as the output from running rule-update again just now.

local.rules

alert tcp any any -> any 23 (msg:"p23 telnet dst"; sid:1000001; rev:1;)
alert tcp any 23 -> any any (msg:"p23 telnet src"; sid:1000002; rev:1;)
alert tcp any any -> any 22 (msg:"p22 ssh dst"; sid:1000003; rev:1;)
alert tcp any 22 -> any any (msg:"p22 ssh src"; sid:1000004; rev:1;)
alert tcp any any -> any 4444 (msg:"p4444 msf dst"; sid:1000005; rev:1;)
alert tcp any 4444 -> any any (msg:"p4444 msf src"; sid:1000006; rev:1;)
alert tcp any any -> any 31337 (msg:"p31337 elite dst"; sid:1000007; rev:1;)
alert tcp any 31337 -> any any (msg:"p31337 elite src"; sid:1000008; rev:1;)
alert tcp any any -> any 666 (msg:"p666 hell dst"; sid:1000009; rev:1;)
alert tcp any 666 -> any any (msg:"p666 hell src"; sid:1000010; rev:1;)
alert tcp any any -> any 9999 (msg:"p9999 vulnserv dst"; sid:1000011; rev:1;)
alert tcp any 9999 -> any any (msg:"p9999 vulnserv src"; sid:1000012; rev:1;)
alert tcp 172.16.1.166 any -> any any (msg:"xpbox rev_tcp uplink"; sid:1001001; rev:1;)
alert tcp any any -> any any (msg:"get"; content:"get"; sid:1001002; rev:1;)
alert tcp any any -> any any (msg:"exe"; content:"exe"; sid:1001003; rev:1;)
alert tcp any any -> any any (msg:"shell"; content:"dir"; sid:1001004; rev:1;)
alert tcp any any -> any any (msg:"idiot overflow"; content:"AAAAAAAA"; sid:1001005; rev:1;)
alert tcp any any -> 172.16.1.166 any (msg:"possible xpbox buffer overflow"; dsize:>256; sid:1001006; rev:1;)

rule-update

student@onion-10:/etc/nsm/rules$ sudo rule-update
Wed Jan 29 22:08:23 UTC 2020
Backing up current local_rules.xml file.
Cleaning up local_rules.xml backup files older than 30 days.
Backing up current downloaded.rules file before it gets overwritten.
Cleaning up downloaded.rules backup files older than 30 days.
Backing up current local.rules file before it gets overwritten.
Cleaning up local.rules backup files older than 30 days.
LOCAL_NIDS_RULE_TUNING is enabled.
This will cause PulledPork to use the existing rules in /opt/emergingthreats/
instead of downloading new rules from the Internet.
If you want PulledPork to download new rules from the Internet,
set the following in /etc/nsm/securityonion.conf:
LOCAL_NIDS_RULE_TUNING=no
ENGINE=suricata, so we'll execute PulledPork with -T -S suricata-4.1.5.
Running PulledPork.
    https://github.com/shirkdog/pulledpork
      _____ ____
     `----,\    )
      `--==\\  /    PulledPork v0.7.3 - Making signature updates great again!
       `--==\\/
     .-~~~~-.Y|\\_  Copyright (C) 2009-2017 JJ Cummings, Michael Shirk
  @_/        /  66\_  and the PulledPork Team!
    |    \   \   _(")
     \   /-| ||'--'  Rules give me wings!
      \_\  \_\\
Prepping rules from emerging.rules.tar.gz for work....
        Done!
Reading rules...
Reading rules...
Modifying Sids....
        Done!
Processing /etc/nsm/pulledpork/enablesid.conf....
        Modified 0 rules
        Skipped 0 rules (already disabled)
        Done
Processing /etc/nsm/pulledpork/dropsid.conf....
        Modified 0 rules
        Skipped 0 rules (already disabled)
        Done
Processing /etc/nsm/pulledpork/disablesid.conf....
        Modified 63 rules
        Skipped 2 rules (already disabled)
        Done
Setting Flowbit State....
        Enabled 184 flowbits
        Enabled 1 flowbits
        Done
Writing /etc/nsm/rules/downloaded.rules....
        Done
Generating sid-msg.map....
        Done
Writing v1 /etc/nsm/rules/sid-msg.map....
        Done
Writing /var/log/nsm/sid_changes.log....
        Done
Rule Stats...
        New:-------0
        Deleted:---243
        Enabled Rules:----19764
        Dropped Rules:----0
        Disabled Rules:---7394
        Total Rules:------27158
No IP Blacklist Changes
Done
Please review /var/log/nsm/sid_changes.log for additional details
Fly Piggy Fly!
Restarting Barnyard2.
Restarting: onion-10-enp3s0
  * stopping: barnyard2 (spooler, unified2 format)                                                                                                                                                                       [  OK  ]
  * starting: barnyard2 (spooler, unified2 format)                                                                                                                                                                       [  OK  ]
Restarting IDS Engine.
Restarting: onion-10-enp3s0
  * stopping: suricata (alert data)                                                                                                                                                                                      [  OK  ]
  * starting: suricata (alert data)                                                                                                                                                                                      [  OK  ]

I'm not 100% sure about this, but I am pretty sure I also recall something else being different that may be a clue: I think I noticed that before when I set alerts, they would show up with the RT in red on the interface on the left in SGUIL (just like the default rule alerts), but in the latest version, when viewing the custom local.rules alerts in SGUIL (which are showing up), they actually show up with RT in yellow like the 'log' entries usually had. (Sorry, I'm not able to see the desktop and my previous screenshots from here to confirm.)

[dougburks]

Have you tried adding classtype to your rules?

Example:

alert tcp any any -> any 23 (msg:"p23 telnet dst"; classtype:attempted-user; sid:1000001; rev:1;)

[dstults]

It appears that adding classtypes makes them appear in the NIDS page. I guess that's easy enough, but it was nice when it showed them without. As for the SGUIL anomaly, after changing some "alert"s to "log"s, their colors didn't change either way, I think that's something different and/or I was mistaken and I just need to find out what the colors actually mean.

For your information, here are the updated results.
Local.rules:

alert tcp any any -> any 23 (msg:"p23 telnet dst"; priority:1; classtype:attempted-user; sid:1000001; rev:3;)
alert tcp any 23 -> any any (msg:"p23 telnet src"; priority:1; classtype:attempted-user; sid:1000002; rev:3;)
log tcp any any -> any 22 (msg:"p22 ssh dst"; priority:1; classtype:attempted-user; sid:1000003; rev:3;)
log tcp any 22 -> any any (msg:"p22 ssh src"; priority:1; classtype:attempted-user; sid:1000004; rev:3;)
alert tcp any any -> any 4444 (msg:"p4444 msf dst"; priority:1; classtype:misc-attack; sid:1000005; rev:3;)
alert tcp any 4444 -> any any (msg:"p4444 msf src"; priority:1; classtype:misc-attack; sid:1000006; rev:3;)
alert tcp any any -> any 31337 (msg:"p31337 elite dst"; priority:1; sid:1000007; rev:2;)
alert tcp any 31337 -> any any (msg:"p31337 elite src"; priority:1; sid:1000008; rev:2;)
alert tcp any any -> any 666 (msg:"p666 hell dst"; priority:1; sid:1000009; rev:2;)
alert tcp any 666 -> any any (msg:"p666 hell src"; priority:1; sid:1000010; rev:2;)
alert tcp any any -> any 9999 (msg:"p9999 vulnserv dst"; classtype:trojan-activity; priority:2; sid:1000011; rev:3;)
alert tcp any 9999 -> any any (msg:"p9999 vulnserv src"; classtype:trojan-activity; priority:8; sid:1000012; rev:3;)
alert tcp 172.16.1.166 any -> any any (msg:"xp rev_tcp uplink"; priority:1; classtype:trojan-activity; sid:1001001; rev:5;)
alert tcp any any -> any any (msg:"get"; content:"get"; priority:3; classtype:web-application-attack; sid:1001002; rev:3;)
alert tcp any any -> any any (msg:"exe"; content:"exe"; priority:9; classtype:web-application-attack; sid:1001003; rev:3;)
alert tcp any any -> any any (msg:"shell"; content:"dir"; priority:1; sid:1001004; rev:2;)
alert tcp any any -> any any (msg:"idiot overflow"; content:"AAAAAAAA"; priority:5; classtype:misc-attack; sid:1001005; rev:3;)
alert tcp any any -> 172.16.1.166 any (msg:"xp buffer overflow"; dsize:>128; priority:9; classtype:misc-attack; sid:1001006; rev:6;)

Rule-update seems to run normally (and everything seems to be working now).

NIDS: 2494 of 3009 alerts are showing on main page summary now. It would appear that those custom rules showing are those with classtypes:

Clicking further into the Dashboard / Indicator page confirms that the ones that didn't have class-type added were the ones counted in the earlier total (3009) but that weren't appearing in the summary on the NIDS page:

As for our class, most of us preferred the older version of SO when classifications were not needed to make alerts appear in the alert summary of the NIDS page--if that was changed because it goes against best practices, that's undestandable.

=============================================

Unrelated but since it came up, just in case you were curious or wanted to verify that the SGUIL behavior is acting as expected too, here are the screenshots from before adding classtypes and afterwards.

Before:

After:

[dougburks]

It sounds like the short answer is to make sure that your rules include classtype as that is considered best practice as evidenced by all the major rulesets.

Just out of curiosity, when you ran Setup, did you run standard Setup or did you explicitly run sosetup-minimal? Assuming you ran standard Setup, then you should be running the traditional Logstash config where the NIDS alert parser hasn't changed since 3/15/2018, meaning that Logstash should parse NIDS alerts identically from 16.04.4.1 to 16.04.6.3:
https://github.com/Security-Onion-Solutions/securityonion-elastic/commits/master/configfiles/1033_preprocess_snort.conf

[dougburks]

We did some further testing of both 16.04.4.1 and 16.04.6.3 and verified that the Logstash parsing of NIDS alerts is identical. There is a slight difference in Kibana visualizations that was allowing 16.04.4.1 to partially show NIDS alerts even when they weren't parsed properly. We also noticed that in 16.04.6.3, you can actually see the unparsed NIDS alerts on the NIDS dashboard. They won't show up in the visualizations at the top of the page but they should show up in the log panel at the bottom of the page.

Again, the best answer for both 16.04.4.1 and 16.04.6.3 is to make sure that your rules have classtype so that the alerts are parsed properly and visualizations display as expected.

If you have further questions or problems, please use the mailing list:
https://securityonion.readthedocs.io/en/latest/mailing-lists.html

Thanks!

[dstults]

Thank you, I tried getting a second machine running this morning but a static shock seems to have broken the hard drive I was using and I wasn't able to finish. Thank you for helping check.

In answer to your question earlier, we didn't know there was a so-setup-minimal -- it was the desktop install icon, then the desktop setup icon, then desktop setup icon again using the settings mentioned earlier. Yeah, I don't think there was a change in the parsing as the alerts were updating, were triggering, and were showing up in the total as well as everywhere else, just not on the NIDS summary. But adding the categories is simple enough.

We are looking up how to configure the Kibana pages (not in /etc/kibana apparently) so we can see specifically which configs were changed that affected whether categorized alerts were shown. If we have any more questions we will contact the mailing lists.

[bdesforges]

Can I suggest that we add a "classtype:misc-attack" to the example rule provided in the "Addition Local Rules" page of the documentation?
I was trying to make it work and ended up on this thread.
Thanks!

[dougburks]

I've updated https://docs.securityonion.net/en/16.04/local-rules.html.

If you have further questions or problems, please use the mailing list:
https://securityonion.readthedocs.io/en/latest/mailing-lists.html

Thanks!