[Bug] Suricata rule lookup crash

[L1ghtn1ng]

While trying to look at the rules on 17.1.4 64-bit I get the follwoing crash and this is even after a reboot

configd.py: [765dd116-17aa-4f24-87ea-61c7dcb83e14] Script action failed with Command '/usr/local/opnsense/scripts/suricata/queryInstalledRules.py /limit "10" /offset "0" /filter "" /sort_by "sid"' returned non-zero exit status 1 at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 477, in execute stdout=output_stream, stderr=error_stream) File "/usr/local/lib/python2.7/subprocess.py", line 541, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '/usr/local/opnsense/scripts/suricata/queryInstalledRules.py /limit "10" /offset "0" /filter "" /sort_by "sid"' returned non-zero exit status 1

[AdSchellevis]

@L1ghtn1ng can you execute the following on a console and post the output?

/usr/local/opnsense/scripts/suricata/queryInstalledRules.py /limit "10" /offset "0" /filter "" /sort_by "sid"

[L1ghtn1ng]

I get the following Traceback (most recent call last):   File "/usr/local/opnsense/scripts/suricata/queryInstalledRules.py", line 46, in <module>     rc.create()   File "/usr/local/opnsense/scripts/suricata/lib/rulecache.py", line 193, in create     'fieldvalues': ':' + (',:'.join(self._rule_fields))}, rules) sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

…

On Sun, 2017-04-02 at 01:59 -0700, Ad Schellevis wrote: @L1ghtn1ng can you execute the following on a console and post the output? /usr/local/opnsense/scripts/suricata/queryInstalledRules.py /limit "10" /offset "0" /filter "" /sort_by "sid" — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[AdSchellevis]

@L1ghtn1ng can you try 5f17abb ?

it looks there's some unicode text in your rules, which result in parsing issues in sqlite, because I don't have the same files, I can't properly test it here.

[L1ghtn1ng]

@AdSchellevis That did the trick, they now load. I am just using the rules that come in OPNsense

[L1ghtn1ng]

Can we get this pulled in for 17.1.5?

[AdSchellevis]

@L1ghtn1ng ok, thanks for confirming. Let's ask @fichtner if he can pull this one in.

[fichtner]

I think I have another case, if this is confirmed there too it's going into 17.1.5 for sure.

As for the trigger of this issue, I don't see anything in particular, maybe a Framework change in the ports? Or did ET Open rules maybe start embedding UTF-8? The timing for 17.1.4 is off: it was released on Wednesday, but reports for this problem are not older than 24 hours...

[L1ghtn1ng]

Probably the latter. As that would be the only variable here that would make sense, as you have said you cannot see anything in ports. Sent from phone

…

________________________________ From: Franco Fichtner <notifications@github.com> Sent: Monday, April 3, 2017 4:47:37 AM To: opnsense/core Cc: J.Townsend; Mention Subject: Re: [opnsense/core] [Bug] Suricata rule lookup crash (#1516) I think I have another case, if this is confirmed there too it's going into 17.1.5 for sure. As for the trigger of this issue, I don't see anything in particular, maybe a Framework change in the ports? Or did ET Open rules maybe start embedding UTF-8? The timing for 17.1.4 is off: it was released on Wednesday, but reports for this problem are not older than 24 hours... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1516 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ADddQr1RTd1R-gXHBEPn7kX-5_Npe_OAks5rsGvZgaJpZM4MwtTN>.

[fichtner]

Confirmed. Backport+Close.