'rspamc learn_spam' fails with 'HTTP error : 500, Unknown statistics error' #2678

yasuhirokimura · 2018-12-12T14:54:02Z

Classification (Please choose one option):

Reproducibility (Please choose one option):

Rspamd version:

1.8.3

Operation system, CPU:

FreeBSD 11.2-RELEASE amd64

Description (Please provide a descriptive summary of the issue):

About 2 weeks ago I found rspamc learn_spam fails as following.

yasu@eastasia[2041]% rspamc learn_spam ~/tmp/spam01.txt ~
Results for file: /home/yasu/tmp/spam01.txt (0.024 seconds)
HTTP error: 500, Unknown statistics error

I can't say exactly when but in the past it certainly worked well.

I removed all files under ${DBDIR} and start spam/ham learning from
scratch. At first 'rspamc learn_spam' completed successfully. But
after a week or so same error happened again.

rspamc learn_ham always succeeds.

Relevant logs (see details here):

When executing rspamc learn_spam following messages are written to
log file.

2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_controller_check_password: allow unauthorized connection from a trusted IP 127.0.0.1
2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_message_parse: loaded message; id: <20181127235149.DF964AE5EA9@mail.prospector.kz>; queue-id: <undef>; size: 4210; checksum: <21b55fd227635acf41c417c420722c42>                          
2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_mime_text_part_utf8_convert: converted from ISO_8859-1:1987 to UTF-8 inlen: 811, outlen: 811
2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_mime_part_detect_language: detected part language: en       
2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing
2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_task_process: learn error: Unknown statistics error         
2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_controller_learn_fin_task: cannot learn <20181127235149.DF964AE5EA9@mail.prospector.kz>: Unknown statistics error

Expected results:

rspamc learn_spam completes successfully.

Actual results:

rspamc learn_spam fails as following.

yasu@eastasia[2041]% rspamc learn_spam ~/tmp/spam01.txt ~
Results for file: /home/yasu/tmp/spam01.txt (0.024 seconds)
HTTP error: 500, Unknown statistics error

Configuration (e.g. `rspamadm configdump module`):

I don't make any customization about spam learning.

The text was updated successfully, but these errors were encountered:

vstakhov · 2018-12-12T14:58:54Z

IDK: enable bayes debugging at least.

yasuhirokimura · 2019-01-15T02:28:20Z

Last month I made clean install of FreeBSD 12.0-RELEASE on my mail server and also made clean install of Rspamd 1.8.3 with debug_modules = ["bayes"]; in ${CONFDIR}/local.d/logging.inc. Then same problem happened last week. There is part of rspamd.log in following URL.

https://www.utahime.org/Rspamd/rspamd.20190112042927.log

If you need learned mail then I can provide it. And since the problem is reproducible now I can provide other cases if necessary. Does this help you investigating this issue?

vstakhov · 2019-03-11T11:53:28Z

Are you using sqlite backend?

yasuhirokimura · 2019-03-12T05:24:40Z

Yes, I think so.

yasu@eastasia[4693]% rspamadm configdump classifier.bayes.backend
*** Section classifier.bayes.backend ***
"sqlite3"
*** End of section classifier.bayes.backend ***
yasu@eastasia[4694]%

stale · 2019-05-11T06:15:28Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

sem-hub · 2019-05-28T11:02:28Z

The same problem on FreeBSD with the log:
2019-05-28 13:57:17 #42580(controller) <9c769f>; csession; rspamd_task_process: learn error: Unknown statistics error, found on stage learn; classifier: (null)
2019-05-28 13:57:17 #42580(controller) <9c769f>; csession; rspamd_controller_learn_fin_task: cannot learn CA+1tZ+6G7fx1eJFe8V9DDTJVH5f1MfyAwz2pKQDY5Tus=JHxOw@mail.gmail.com: Unknown statistics error, found on stage learn; classifier: (null)

The same backend: sqlite3

Twilek-de · 2019-07-04T18:06:25Z

I am seeing this problem now as well (I don´t know how long it has existed already. I had a redis backend. I have switched to sqlite and now it seems to work again.

bong0 · 2019-10-27T14:33:09Z

Since a few weeks I'm having that issue as well, the log seems not so say more than the client output does. (the unknown statistics error)

My backed is redis, default install on Debian using the rspamd.com repo:

apt show rspamd
Package: rspamd
Version: 2.0-1~jessie
Maintainer: Vsevolod Stakhov <vsevolod@highsecure.ru>
[...]
APT-Sources: http://rspamd.com/apt-stable/ jessie/main amd64 Packages

Redis is not showing me errors in /var/log/redis.log;

Debug log:

[more ... "token" messages]
2019-10-27 15:28:06 #9075(controller) <966b1d>; bayes; bayes_learn_spam: token 10245623571907066200 <#rcv:45.155.37.0:esmtp:#F:Werner Klemm>: window: 2, total_count: 0, spam_count: 0, ham_count: 0
2019-10-27 15:28:06 #9075(controller) <966b1d>; bayes; bayes_learn_spam: token 539890791086949597 <#rcv:45.155.37.0:esmtp:#h:C-T:text/html; charset="UTF-8">: window: 3, total_count: 0, spam_count: 0, ham_count: 0
2019-10-27 15:28:06 #9075(controller) <966b1d>; bayes; bayes_learn_spam: token 1730648915295258020 <#rcv:45.155.37.0:esmtp:#hh:54e5275>: window: 4, total_count: 0, spam_count: 0, ham_count: 0
2019-10-27 15:28:06 #9075(controller) <966b1d>; csession; rspamd_task_process: learn error: Unknown statistics error, found when storing data on backend; classifier: (null)
2019-10-27 15:28:06 #9075(controller) <966b1d>; csession; rspamd_controller_learn_fin_task: cannot learn <d1fa3087ee3c3bcdb33ae481b9bb5dca@towpartsnow.com>: Unknown statistics error, found when storing data on backend; classifier: (null)

The error occurs no matter whether using rspamc or the web interface paste

nahuelange · 2019-11-04T09:13:47Z

I have the same error

snowirbis · 2020-03-29T02:24:36Z

Same here. FreeBSD 11.3 amd64 rspamd 2.2

arnowelzel · 2020-04-10T11:15:47Z

Same here, Ubuntu 18.04 amd64, rspamd 2.5, but with redis as backend. I get this error about 2 or 3 times per hour:

learn error: Unknown statistics error, found when storing data on backend; classifier: bayes

And this is the result of rspamadm configdump classifier.bayes.backend:

2020-04-10 13:13:38 #0(main) lua_ip_from_string: cannot parse ip:
*** Section classifier.bayes.backend ***
"redis"
*** End of section classifier.bayes.backend ***

Edit: the above result was caused by configuration error which I fixed. Now it's correct:

*** Section classifier.bayes.backend ***
"redis"
*** End of section classifier.bayes.backend ***

But I still get learn error: Unknown statistics error, found when storing data on backend; classifier: bayes.

norbusan · 2020-04-22T19:36:26Z

Just to chime in, same here, too, Debian buster, rspamd 2.5, redis.

2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_controller_check_password: allow unauthorized connection from a trusted IP 127.0.0.1
2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_message_parse: loaded message; id: <20200417154538.4771ca228a26@slickdeals.net>; queue-id: <undef>; size: 32282; checksum: <5b9b237748ee9acde82ce4219da439af>
2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_mime_part_detect_language: detected part language: en
2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_mime_part_detect_language: detected part language: en
2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing
2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_task_process: learn error: Unknown statistics error, found when storing data on backend; classifier: (null)
2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_controller_learn_fin_task: cannot learn <20200417154538.4771ca228a26@slickdeals.net>: Unknown statistics error, found when storing data on backend; classifier: (null)

bong0 · 2020-05-15T17:48:45Z

I finally resolved the issue in my case, though with the now unsupported version 2.2 on debian jessie (8.11). I want to share it nevertheless, it took the following:

I removed /etc/rspamd/local.d/statistic.conf, which previously contained

classifier "bayes" {
    servers = "127.0.0.1:6379";
}

I introduced /etc/rspamd/local.d/classifier-bayes.conf with the contents:

servers = "127.0.0.1";
backend = "redis";

Restart rspamd.

After uploading ham/spam via rspamc or the webinterface, I started seeing heavy output in redis-cli monitor :)

R3belWithoutAPause · 2020-05-31T00:07:13Z

This fixed my problem running Rspamd 2.5 on Slackware. Thank you!

* Add servers directive explicitly to classifer-bayes.conf * Fixes issue rspamd/rspamd#2678

v2b1n · 2021-04-06T05:15:30Z

I finally resolved the issue in my case, though with the now unsupported version 2.2 on debian jessie (8.11). I want to share it nevertheless, it took the following:
1. I removed `/etc/rspamd/local.d/statistic.conf`, which previously contained
classifier "bayes" {
    servers = "127.0.0.1:6379";
}
1. I introduced  `/etc/rspamd/local.d/classifier-bayes.conf` with the contents:
servers = "127.0.0.1";
backend = "redis";
1. Restart rspamd.
After uploading ham/spam via rspamc or the webinterface, I started seeing heavy output in redis-cli monitor :)

Thanks, fixed rspamd 2.7 @ debian stretch

i hadn't the local.d/statistics.conf, but

installing redis-server and
creating the /etc/rspamd/local.d/classifier-bayes.conf with the ^^ mentioned content fixed that issue for me.

detrout · 2021-09-07T23:20:58Z

I too had this problem on a system just upgraded to Debian Bullseye using the Debian rspamd 2.7-1, rpamd had previously been working with buster.

When I was having the 500 error with rspamc, I was also seeing messages that rspamd couldn't init the redis cache on startup.

2021-09-07 16:10:46 #18887(normal) <4zbh5i>; cfg; rspamd_redis_init: cannot init redis backend for BAYES_SPAM
2021-09-07 16:10:46 #18887(normal) <4zbh5i>; cfg; rspamd_stat_cache_redis_init: cannot init redis cache for BAYES_SPAM
2021-09-07 16:10:46 #18887(normal) <4zbh5i>; cfg; rspamd_stat_init: error adding cache redis for symbol BAYES_SPAM
2021-09-07 16:10:46 #18887(normal) <4zbh5i>; cfg; rspamd_stat_init: cannot init backend redis for statfile BAYES_SPAM
2021-09-07 16:10:46 #18887(normal) <4zbh5i>; cfg; rspamd_redis_init: cannot init redis backend for BAYES_HAM
2021-09-07 16:10:46 #18887(normal) <4zbh5i>; cfg; rspamd_stat_cache_redis_init: cannot init redis cache for BAYES_HAM
2021-09-07 16:10:46 #18887(normal) <4zbh5i>; cfg; rspamd_stat_init: error adding cache redis for symbol BAYES_HAM
2021-09-07 16:10:46 #18887(normal) <4zbh5i>; cfg; rspamd_stat_init: cannot init backend redis for statfile BAYES_HAM
2021-09-07 16:10:46 #18886(controller) <gptip9>; rrd; rspamd_rrd_open: rrd file opened: /var/lib/rspamd/rspamd.rrd
2021-09-07 16:10:46 #18884(main) <88e267>; main; main: listening for control commands on /var/lib/rspamd/rspamd.sock

Since I noticed /etc/rspamd/statistic.conf already included
backend = "redis";

I decided to create /etc/rspamd/local.d/classifier-bayes.conf with just setting the address.

servers="127.0.0.1";

After that rspamc worked and stopped giving me the 500 error, and the errors about not being able to create the redis backends went away

vstakhov added this to To do in Kanban via automation Mar 11, 2019

stale bot added the wontfix label May 11, 2019

stale bot closed this as completed May 18, 2019

Kanban automation moved this from To do to Done May 18, 2019

essandess added a commit to essandess/macports-ports that referenced this issue Jul 1, 2020

mail-server: Bugfix in rspamd classifer-bayes default configuration

5a5fde7

* Add servers directive explicitly to classifer-bayes.conf * Fixes issue rspamd/rspamd#2678

essandess mentioned this issue Jul 1, 2020

mail-server: Bugfix in rspamd classifer-bayes default configuration macports/macports-ports#7591

Merged

11 tasks

mf2k pushed a commit to macports/macports-ports that referenced this issue Jul 4, 2020

mail-server: Bugfix in rspamd classifer-bayes default configuration

bc2ddc5

* Add servers directive explicitly to classifer-bayes.conf * Fixes issue rspamd/rspamd#2678

phifogg mentioned this issue Sep 19, 2020

[BUG] unknown statistics error, found when storing data on backend; classifier: bayes #3339

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

'rspamc learn_spam' fails with 'HTTP error : 500, Unknown statistics error' #2678

'rspamc learn_spam' fails with 'HTTP error : 500, Unknown statistics error' #2678

yasuhirokimura commented Dec 12, 2018

vstakhov commented Dec 12, 2018

yasuhirokimura commented Jan 15, 2019

vstakhov commented Mar 11, 2019

yasuhirokimura commented Mar 12, 2019

stale bot commented May 11, 2019

sem-hub commented May 28, 2019

Twilek-de commented Jul 4, 2019

bong0 commented Oct 27, 2019 •

edited

nahuelange commented Nov 4, 2019

snowirbis commented Mar 29, 2020

arnowelzel commented Apr 10, 2020 •

edited

norbusan commented Apr 22, 2020 •

edited

bong0 commented May 15, 2020 •

edited

R3belWithoutAPause commented May 31, 2020

v2b1n commented Apr 6, 2021 •

edited

detrout commented Sep 7, 2021 •

edited

'rspamc learn_spam' fails with 'HTTP error : 500, Unknown statistics error' #2678

'rspamc learn_spam' fails with 'HTTP error : 500, Unknown statistics error' #2678

Comments

yasuhirokimura commented Dec 12, 2018

Classification (Please choose one option):

Reproducibility (Please choose one option):

Rspamd version:

Operation system, CPU:

Description (Please provide a descriptive summary of the issue):

Relevant logs (see details here):

Expected results:

Actual results:

Configuration (e.g. rspamadm configdump module):

vstakhov commented Dec 12, 2018

yasuhirokimura commented Jan 15, 2019

vstakhov commented Mar 11, 2019

yasuhirokimura commented Mar 12, 2019

stale bot commented May 11, 2019

sem-hub commented May 28, 2019

Twilek-de commented Jul 4, 2019

bong0 commented Oct 27, 2019 • edited

nahuelange commented Nov 4, 2019

snowirbis commented Mar 29, 2020

arnowelzel commented Apr 10, 2020 • edited

norbusan commented Apr 22, 2020 • edited

bong0 commented May 15, 2020 • edited

R3belWithoutAPause commented May 31, 2020

v2b1n commented Apr 6, 2021 • edited

detrout commented Sep 7, 2021 • edited

Configuration (e.g. `rspamadm configdump module`):

bong0 commented Oct 27, 2019 •

edited

arnowelzel commented Apr 10, 2020 •

edited

norbusan commented Apr 22, 2020 •

edited

bong0 commented May 15, 2020 •

edited

v2b1n commented Apr 6, 2021 •

edited

detrout commented Sep 7, 2021 •

edited