idstools-rulecat error with current etpro ruleset.

[banjoey]

Here is the end of the run. Rules download properly, but the merged rule file is never updated. If I run with no arguments it grabs the open source rules and does not appear to error out.

Enabled 480 rules for flowbit dependencies.
Recording file /etc/suricata/rules/merged.rules with hash '94bbee8c808b0d5052f8917e51b0749a'.
Traceback (most recent call last):
File "/usr/local/bin/idstools-rulecat", line 12, in
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/idstools/scripts/rulecat.py", line 752, in main
write_merged(args.merged, rulemap)
File "/usr/local/lib/python2.7/dist-packages/idstools/scripts/rulecat.py", line 415, in write_merged
idstools.rule.parse_fileobj(open(filename)))
File "/usr/local/lib/python2.7/dist-packages/idstools/scripts/rulecat.py", line 454, in build_rule_map
if rule.id not in rulemap:
File "/usr/local/lib/python2.7/dist-packages/idstools/rule.py", line 158, in id
return (int(self.gid), int(self.sid))
TypeError: int() argument must be a string or a number, not 'NoneType'

Here is my config:

--etpro=
--suricata=/usr/local/bin/suricata
--merged=/etc/suricata/rules/merged.rules
--sid-msg-map-2=/etc/suricata/rules/gen-msg2.map
--sid-msg-map=/etc/suricata/rules/gen-msg.map
--enable=/etc/suricata/rules_enable.conf
--disable=/etc/suricata/rules_disable.conf
--modify=/etc/suricata/rules_modify.conf
--threshold-in=/etc/suricata/threshold.in
--threshold-out=/etc/suricata/threshold.config
--post-hook=sudo kill -USR2 $(cat /var/run/suricata.pid)
--temp=/etc/suricata/rules/raw/
-v

I've verified that all the paths listed above exist. Help is appreciated. Not sure if this is a problem with my version or a problem with the way I've configured it. Used pip to install, idstools (0.5.2)

[jasonish]

Well thats no good. I can't replicate here using ET-Pro, but the traceback gives me a good idea. Can you tell me what version of Suricata you are using?

Thanks,
Jason

[banjoey]

I'm using a branch of 3.0. dev-detect-grouping-v178 is the branch name.

[jasonish]

Do you enable/modify/disable files have any content in them? Specifically the modify one I guess.

There appears to be a rule loading that doesn't have an SID. I can't replicate this with ET-Open or ET-Pro. I do need to fixup the handling for a rule where no SID is present, which I'll do real soon, and log it. Could help with this.

[banjoey]

I have nothing in the modify file, but I do have content in enable and disable. Let me clear out my rulesets and try again. Maybe something just got corrupted.

[banjoey]

ok, I killed the .gz file and also removed my rules folder and regenerated from scratch. That seems to have taken care of it. I don't know if it was the .gz file or something in the rules folder that was the culprit. Sorry, I should've moved instead of rm'd...

[jasonish]

Ok. I'm going to add some better logging around this area anyways. Thanks.