[BUG] spamd_language_detector_init: cannot compile stop words od Debian 10

[predraggavrilovic]

Prerequisites

• [ X] Put an X between the brackets on this line if you have done all of the following:
  • Read about bug reporting in general: https://rspamd.com/doc/faq.html#how-to-report-bugs-found-in-rspamd
  • Enabled relevant debugging logs: https://rspamd.com/doc/faq.html#how-to-debug-some-module-in-rspamd
  • Checked the FAQs about Core files in case of fatal crash: https://rspamd.com/doc/faq.html#how-to-figure-out-why-rspamd-process-crashed
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=%E2%9C%93&q=is%3Aissue+user%3Arspamd
  • Checked that there is not already an experimental package or master branch

After fresh install or upgrade to rspamd "2.0-1~buster" there are warnings in rspamd.log:

2019-10-28 15:31:39 #2307(main) ; cfg; rspamd_language_detector_init: cannot compile stop words for 0 language group: pattern is not enclosed with : m\x{ed}
2019-10-28 15:31:39 #2307(main) ; cfg; rspamd_language_detector_init: cannot compile stop words for 1 language group: regexp parsing error: 'character value in \x{} or \o{} is too large' at position 6
2019-10-28 15:31:39 #2307(main) ; cfg; rspamd_language_detector_init: cannot compile stop words for 2 language group: regexp parsing error: 'character value in \x{} or \o{} is too large' at position 6
2019-10-28 15:31:39 #2307(main) ; cfg; rspamd_language_detector_init: cannot compile stop words for 3 language group: regexp parsing error: 'character value in \x{} or \o{} is too large' at position 6

Also rspamdadm configtest gives:

CPU doesn't have SSSE3 instructions set required for hyperscan, disable it
cannot compile stop words for 0 language group: pattern is not enclosed with : m\x{ed}
cannot compile stop words for 1 language group: regexp parsing error: 'character value in \x{} or \o{} is too large' at position 6
cannot compile stop words for 2 language group: regexp parsing error: 'character value in \x{} or \o{} is too large' at position 6
cannot compile stop words for 3 language group: regexp parsing error: 'character value in \x{} or \o{} is too large' at position 6
syntax OK

Steps to Reproduce

1. Install rspamd 2.0-1~buster on Debian 10
2. check logs or run rspamadm configtest

Versions

Additional Information

rspamd 2.0-1~buster on Debian 10.1

[predraggavrilovic]

Still present in rspamd 2.1-1~buster

[vstakhov]

I cannot reproduce it. Are you using pcre1 or pcre2? Anyway, non-hyperscan version of Rspamd is not something in top of my priorities.

[pmontepagano]

@vstakhov is the error about not being able to compile stop words related to hyperscan being disabled? I thought those were separate issues.

[vstakhov]

CPU doesn't have SSSE3 instructions set required for hyperscan, disable it - so it is caused by PCRE.

[pmontepagano]

Thanks! Changing VM's CPU to enable SSSE3 instructions and re-enabling hyperscan solved it for me.

[predraggavrilovic]

Vsevold, thank you for your time.
I also have enabled SSSE3 on my virtual machines and errors are gone.
As for pcre libs:
rspamd binary packages from rspamd site are linked against libpcre3 which is in debian consiedered old librararies, and packages are to be linked against libpcre2 which is newer despite number in its name.
For example, debian repository has rspamd 1.8 and debian buster backports has 1.9.4, both of which are old but not ancient, and both are linked against libpcre2-8-0 which is condiered newer.
So enabling hyperscan solves it, but maybe different pcre lib could provide functional failback

Best regards,

[vstakhov]

I do not support Debian packages, please stop using them...

[vstakhov]

I wish I could convince Debian maintainers to stop providing Rspamd in Debian repos. Probably, I should change a license for that, but I'm really tired of that crap...

[predraggavrilovic]

Just to be clear, I am not using debian packages, I am using binary packages from http://rspamd.com/apt-stable/
I was just noting that debian uses newer pcre lib for their packages, than upsteam rspamd does, and that for debian, on libpcre3 page there is information "New packages should use the newer pcre2 packages, and existing packages should migrate to pcre2."

Packages from http://rspamd.com/apt-stable are depending od libpcre3 not on libpcre2

Best regards,

[vstakhov]

Ah, I see, sorry. Yes, all packages should be built with pcre2 - I will try to fix it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Ryushin]

Can this be reopened? I'm setting up rspamd on a dual socket Opteron 6168 12 core system with 128GB of RAM. The Opteron 6168 does not have SSSE3. I'm using the Rspamd provided Debian packages.

/etc/init.d/rspamd restart [....] Restarting rapid spam filtering system: rspamd2020-06-12 10:32:20 #0(main) cfg; rspamd_config_post_load: CPU doesn't have SSSE3 instructions set required for hyperscan, disable it 2020-06-12 10:32:20 #0(main) <9senjh>; cfg; rspamd_language_detector_init: cannot compile stop words for 0 language group: regexp parsing error: 'character code point value in \x{} or \o{} is too large' at position 7; pattern: a\x{10d}koli 2020-06-12 10:32:20 #0(main) <9senjh>; cfg; rspamd_language_detector_init: cannot compile stop words for 1 language group: regexp parsing error: 'character code point value in \x{} or \o{} is too large' at position 6; pattern: \x{441}\x{430}\x{43c} 2020-06-12 10:32:20 #0(main) <9senjh>; cfg; rspamd_language_detector_init: cannot compile stop words for 2 language group: regexp parsing error: 'character code point value in \x{} or \o{} is too large' at position 6; pattern: \x{906}\x{92b}\x{942} 2020-06-12 10:32:20 #0(main) <9senjh>; cfg; rspamd_language_detector_init: cannot compile stop words for 3 language group: regexp parsing error: 'character code point value in \x{} or \o{} is too large' at position 6; pattern: \x{647}\x{646}\x{627}\x{644}\x{643}

[vstakhov]

That is likely fixed in the master. Using of such a CPU has a very bad impact on Rspamd performance JFYI.