-
-
Notifications
You must be signed in to change notification settings - Fork 376
/
ChangeLog
1497 lines (1441 loc) · 62.9 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1.2.0:
* New dynamic updates plugin
* Regular expressions map support
* Faster radix trie algorithm
* Faster siphash for AVX2 supporing CPUs (used in fuzzy hashes)
* PCRE2 support
* Allow quoted and slashed keys in map
* Add proper support of DNS resolvers balancing (#552)
* Rework includes and configuration system for better local changes support
* New keypairs framework for signing and encryption
* Added support for dynamic modules and workers
* Allow to dump configuration with help comments
* Rework once_received module
- Fix priority for `good_hosts`
- If a good host has been found do not add once_received symbols
- Fix priorities for strict once_received
- Add ability to whitelist IP addresses
* Implement support of signed maps for HTTP and file maps
* Add command to sync fuzzy storage (#533)
* Rework system of symbols and actions registration
It is possible now to use priorities when adding symbols to metrics and
override scores for symbols with lower priority with the scores with
high priority.
* Add auth support and db selection for redis stats
* Improve composite rules application
* Add ignore_received option
* Fix critical issue with inconsistent resorting
* Fix `all` in spf redirects
* Add punycoded versions for IDN domains (#554)
* Improve sorting order for symbols cache
* Add lockless logging for processes management
* Allow to specify flags for metric symbols
* Load images height and width from style attribute (#538)
* Override DNS requests limits for SPF and DKIM
* Fix resetting symbols to their default values in WebUI
* Improve configuration agility for redis stats
* Allow to set db and password for redis in stat_convert
* Import the latest libucl
* LUA API changes:
- Add rspamd_version function to LUA API
- Add lua_cryptobox module
- Add lua_map module
- Add task:set_metric_action lua API method
- Fix race condition in lua_tcp module
- Fix a lot of issues in lua_redis module
- Rework and abstract lua maps API
- Add util.strlen_utf8 lua function
- Add lua functions for caseless comparison
- Allow optional symbols registration
- Add config:add_map table form method, add regexp maps
- Add task:has_urls method
- Add task:has_flag method
- Add html tags methods to lua_html
- Add task:get_dns_req
* Plugins changes:
- Add support for WLBLEval SA plugin
- Use caseless comparison in SA and DMARC plugins
- Allow SA plugin to set scores for rspamd symbols
- Add regexp maps support to multimap
- Allow filenames match in multimap
- Add more filters for the existing map types
- Fix html images rules to reduce FP rates
* New rules:
- LONG_SUBJ - too long subject
- MIME_BAD_ATTACHMENT - bad attachement type
- RDNS_NONE - no reverse DNS record for sender's IP
- Fix MISSING_MIMEOLE rule for modern OE
* Many other bugfixes, memory leaks plugs thanks to:
- Coverity scan
- New gcc-6 warnings
- valgrind manual iterations
* Documentation improvements:
- FAQ list: https://rspamd.com/doc/faq.html
- Reworked quick start guide
- Added documentation for all active modules
* Other changes:
- Dropped Ubuntu Vivid support
- Added Ubuntu Xenial support
- Rework build system for rspamd and rmilter
1.1.4:
* Print traceback on lua errors in lua config
* Fix leaks in lua error paths
* Improve 'R_EMPTY_IMAGE' rule
* Fix metas memoization in SA plugin
* Properly set `flag` in fuzzy replies
* Fix arguments order
* Fix issue with out-of-boundary reading
* Fix issues found by coverity
* Same result checking error found by coverity
* Fix varargs processing (found by coverity)
* Fix error in printing hex
* Reduce weights for some hfilter patterns
* Add aliases for task:get_from_ip:
- task:get_addr
- task:get_from_addr
- task:get_ip
* Rework once_received module
- Fix priority for `good_hosts`
- If a good host has been found do not add once_received symbols
- Fix priorities for strict once_received
- Add ability to whitelist IP addresses
* Fix `MISSING_MIMEOLE` rule for modern OE
* Treat meta tags as embedded tags (#501)
1.1.3:
* Fix DSN rules when SMTP from is unavailable
* Fix statconvert routine to avoid lua module usage
* Set a sane quark for configtest to avoid NULL to be printed in logs
* Support c11 if available
* Fix parsing of ip:port strings
* Add more diagnostic for lua subr errors
* Fix task:set_from_ip lua method
* Add basic routines for digital signatures
* Add tool for digital signatures
* Add plain open file API method for atomic open
* Fix parsing nested braces inside logger vars
* Pre filters now actually skip processing
* Add pre-filter mode for multimap
* Switch to apache 2 license
1.1.2:
* Fix stat_cache closing
* Add checkpoints to sqlite3 learn cache
* Do not recompile lua generated headers all the time
* Increase number of messages learned
* Fix issues with dual stack and hfilter
* Disable MID checks for hfilter by default
* Fix cache definitions in multiple classifier and no type
* Don't crash if learn cache failed to initialize
* Fix googlegroups support in maillist plugin
* Rework flags LUA API:
- Allow to check for a specific flag
- Add `learn_spam`, `learn_ham` and `broken_headers` flags
- Unify internal functions
* Add `BROKEN_HEADERS` rule
* Add support for forged confirmation headers (by @AdUser)
* Allow `any`, `mime` and `smtp` for get_from/get_recipients
* Add mime types checking plugin
* Add rule to detect spammers attempts to cheat mime parsing
* Rework parsing of IP addresses in configuration (better IPv6 support)
* Add `util.parse_mail_address` function to LUA API
* Add lua sqlite3 module
* Implement synchronous redis call
* Ratelimit: avoid possible indexing of nil value (Fixes #498) (by @fatalbanana)
* Add stat_convert command to convert stats tokens from sqlite3 to redis
* Implement redis advanced lua api with pipelining
* Fix memory leak on redis stat (#500)
* Fix user/language learn count in sqlite statistics (#496) (by @fatalbanana)
* Fix build with custom pcre
* Fix fuzzy relearning (#498)
* Improve planning of asynchronous tasks
* Show slow rules in log
* Add warning for slow regexps
* Add base32 decode/encode routines to lua util
* Allow converting of learn cache from sqlite to redis
* Add methods to check if a messages has from/rcpts
* Improve and fix multimap plugin:
- Restore 'header' maps
- Add filters for headers
- Add 'email:addr', 'email:user', 'email:domain' and 'email:name' filters
- Add generic regexp filters
* Disable reload command in rc scripts
* Improve runtime CPU dispatcher for libcryptobox
* Add preliminary support of digital signatures via ed25519
* Add detection for RDRAND support
* Print configuration of crypto on start
* A in SPF presumes AAAA lookup as well
1.1.1:
* Fix duplicated XBL symbol
* Reduce log severity for ratelimit missing servers
* Fix XBL composite to avoid duplicate symbols
* Reduce weight of URL_ONLY rule due to FP rate
* Disable fuzzy hashes from the metadata for now
* Fix processing of empty messages (#486)
* Always treat DNS timeouts as temporary fail for SPF
* Fix issue with SPF double IP stack (#483)
* Use X-Forwarded-For when checking secure_ip (#488)
* Fix hash calculation for sqlite stats
* Fix memory corruption on punycode
* Fix strings allocation in punycode
* Fix error message (#491)
1.1.0:
* Incompatible change: sqlite3 and per_user behaviour:
Now both redis and sqlite3 follows the common principles for per-user
statistics:
1) If per-user statistics is enabled check per-user tokens ONLY
2) If per-user statistics is not enabled then check common tokens ONLY
If you need old behaviour, then you'd need to use separate classifier
for per-user statistics.
* Implement redis statistics backend and cache
* Implement autolearning for statistics
* Reworked statistics architecture from scratch
* Add hyperscan (https://github.com/01org/hyperscan) engine for regular
expressions:
- add lazy loader for hyperscan databases
- rework regexp cache to have joint pcre/hyperscan scanning
- implement hyperscan pre-filter support
- add compilation guards for bad expressions
- implement `rspamadm control recompile` command
- implement hyperscan cache monitoring
- slides: <https://highsecure.ru/rspamd-hyperscan.pdf>
* Implement flexible task logging
* Rework fuzzy worker:
- it is now possible to run multiple fuzzy workers;
- implement lazy writing as sqlite3 is bad at concurrent writing;
- add retries for simple sql commands in fuzzy backend;
- use fine-grained transactions for fuzzy;
- implement new multi-pubkeys mode;
- allow encrypted only storages;
- rework statistics for fuzzy;
- add `rspamadm control fuzzystat` command for extended statistics;
- implement human readable output for the previous command;
- add condition script for learning fuzzy storage;
* Various fixes to SPF:
- fix `redirect` records;
- fix domains when parsing mx/ptr/a records in includes/redirects;
- fix issues with multiple addresses in SPF records;
- ignore SPF results in case of DNS failure;
- adjust TTL of records when resolving subelements of SPF records;
- always select `v=spf1` line if it is available
- do not cache records with DNS failure in subrequests;
- ignore records with temporary fails during subrequests resolving;
- fix `RDNS_RC_NOREC` support;
* Add clang plugin for static analysis:
- implement static checks for `rspamd_printf` format strings;
* Add 'allow_raw_input' option for non-mime messages
* Recognize types using libmagic
* Fix parsing of IPv6 received headers
* Add new interface of communication between workers in rspamd
* Add support for named socketpairs
* Don't write URLs by default as it is too verbose
* Set status for HTTP replies
* Try load `rspamd.conf.override`
* Implement words decaying for text parts to limit many checks
* Improve support of SA rules and plugins:
- add check_for_shifted_date and check_for_missing_to_header eval rules;
- add 'check_relays_unparseable' support;
- add `check_for_mime('mime_attachement')` function;
- use new re_cache interface for all SA rules;
- add support for `Mail::SpamAssassin::Plugin::MIMEHeader`;
- add support of 'special' SA headers to `exists` function;
- fix issue when SA metas contain other metas;
- fix freemail rules;
* Many fixes to the URL parser
* Match any newline character in regexps
* Fix resolving of upstreams and detection of poor IPv6 configurations
* Parse upstreams selection algorithm from the configuration line
* Add `reresolve` command to the control interface
* Generate fuzzy hashes from task metadata (URLs and headers)
* Add method to check if IP is local and `local_addrs` option
* Implement forced timeout for delayed filters
* Disable fast path of pcre-jit as it seems to be broken
* Bayes fixes:
- new normalizer function;
- really use weights of tokens from the OSB algorithm;
- restore multiple classifiers support;
* Rules changes:
- add `R_SUSPICIOUS_URL` rule that detects obfuscated URL's;
- improve empty image rule;
- rework `FORGED_RECIPIENTS` rule;
- reduce weight of `SUSPICIOUS_RECIPS`;
- fix `*_NORESOLVE_MX` symbols in hfilter;
- add `SUBJ_ALL_CAPS` rule with support of UTF8
- add spamhaus SBL to uribl
- fix `SUSPICIOUS_RECIPS` and `SORTED_RECIPS` rules
- remove `R_TO_SEEMS_AUTO` as it generates a lot of FP;
- add new Message-ID regexp for Thunderbird (by @moisseev);
* Plugins changes:
- allow ratelimit plugin to set symbol instead of pre-result
- support IP DNS black lists for URIBL (e.g spamhaus SBL);
- drop deprecated SURBL bits (by @fatalbanana)
- rename `JP_SURBL_MULTI` to `ABUSE_SURBL` (by @fatalbanana)
- add `SURBL_BLOCKED` (by @fatalbanana)
- add `CR_SURBL`
- SURBL: allow fallthrough to default symbol (by @fatalbanana)
- Settings: fix IP match (by @fatalbanana)
- SURBL: add missing symbols to metric (by @fatalbanana)
- allow processing images urls for SURBL
- unconditionally disable SPF for authenticated users and local networks
* Rework ratelimit plugin
- switch to `rates` instead of old and stupid strings to setup;
- check if a bucket is zero and disable the corresponding limits'
- turn off all buckets by default;
- check either `rcpt` or `user` buckets, not all together'
- document new `rates` and `symbol` options;
- inform user about what buckets are used in the configuration;
* Add neural network **experimental** plugin
* Add a sample script to learn neural network from rspamd logs
* Add documentation strings support to rspamd:
- add strings for the main configuration options;
- document workers options;
- add internal plugin options;
- create `rspamadm confighelp` routine;
- implement human readable output for the previous command;
- add subtree search support;
- add keyword search support;
* Documentation improvements, tutorials section, statistics description
* Many other minor and major bugfixes not noted here
1.0.11:
* Fix spf redirects
* Fix domains when parsing mx/ptr/a records in includes/redirects
* Fix unfolded base64 encoding
* Fix GError use-after-free
* Do not rewrite the original url when using redirector
* Fix parsing of fragment in urls
* Fix processing of HTML tags
* Improve empty image rule
* Avoid long double type
* Fix tokens weights in OSB algorithm
* Improve debugging for bayes
1.0.10:
* Fix settings application (#416)
* Fix another issue with fixed strings
* Fix hash function invocation
* Use the proper string for make_dns_request in lua_http
* Fix scan time output
* Update webui:
- fix labels for greylisting
- fix dimension of scan time
1.0.9:
* Emergency fix in keyed blake2 to fix fuzzy hashes and encrypted password
* Support passwords longer than 64 symbols
1.0.8:
* Add function to traverse AST atoms
* Allow dependencies on rspamd symbols for SA metas
* Fix memory corruption when timeout is removed in fuzzy check
* Fix encrypted fuzzy add processing
* Avoid use-after-free in controller session destructor
* Use session pool instead of task pool in fuzzy check
* Fix assembly in i386 mode (#413, #412)
1.0.7:
* Plugged memory leaks in internet address object & html parser
* Fixed static build
* Fixed multiple sigchld processing
* Fixed deletion of signal events after event processing loop
* Fixed build on ARM (#404 - reported by @Gottox)
* Fixed setting the default mask for SPF.
* Fixed sanitisation of HTTP query values
* Fixed parsing of the last header in encrypted HTTP messages
* Additions and fixes for test suite & benchmarks
* Added openssl aes-256-gcm support to libcryptobox & HTTP server
* Implemented support for starting multiple HTTP servers
* Implemented batch accept in HTTP server
* Added module to get data from HTTP headers (#285 - reported by @msimerson)
* Added `rspamadm control` command
* Added ability to sort counters output.
* Added ability to specify custom headers for rspamc client
* Fix architecture detection
* Converted history storage to the UCL format
* Allow flexible number of rows in history
* Fix action badges in WebUI
* Add universal cryptobox hash API
* Migrated to the optimized blake2b implementation adopted from Andrew Moon
* Allow explicit loading of specific modules
* Always load settings module
* Allow to add symbols from settings
* Fix double free in the controller fuzzy learn command
* Avoid endless loop when cannot open sqlite db
* Updated libucl
1.0.6:
* Fix build on i386
* Update CentOS7 service file patch (by @fatalbanana)
* Fix path to rspamadm in Debian init script (by @fatalbanana)
* Fix broken '_SC_GETPW_R_SIZE_MAX' on FreeBSD
* Fix portability issues
* Use cryptobox chacha for libottery
* Better support of 32 bit builds
* Fix header name tokens setup
* Fix levenstein distance method for words
* Add workaround for old libevent (#400)
* Fix microseconds in termination timer
* Fix some more issues with fixed strings
* Explicitly test CPU instructions even after CPUID call
* Do not check out of boundary memory
* Do not output broken emails
* Fix unknown symbols registration
* Handle SIGILL using longjmp
* Block signals when exiting event loop
* Fix incorrect allocation size
* Slightly optimize alignment
* Restore rspamd -t for compatibility
* Add more sanity checks for emails
1.0.5:
* Add rspamd control interface:
- support `stat` command to get runtime stats of rspamd workers
- support `reload` command to reload runtime elements (e.g. sqlite3 databases)
* Rework curve25519 library for modular design:
- add Sandy2x implementation by Tung Chou
- fix CPU detection for variables loading assembly
- add testing for curve25519 ECDH
* New fixed strings library
* Add `R_SUSPICIOUS_IMAGES` rule
* Enable mmap in sqlite3
* Use new strings in the HTTP code
* Improve google perftools invocation
* Improve performance profiling in http test
* Reorganize includes to reduce namespace pollution
* Allow specific sections printing in configdump command
* Rework workers signals handlers to be chained if needed
* Update socketpair utility function
* Add control_path option for rspamd control protocol
* Fix ownership when listening on UNIX sockets
* Rework signals processing in main
* Remove extra tools from rspamd (they live in rspamadm now)
* Remove global rspamd_main
* Add global timeout for the overall task processing (8 seconds by default)
* Sanitize NULL values for fuzzy backend
* Store NM between encrypt/decrypt
* Add textpart:get_words_count method
* Fix generic DNS request in lua
* Tune hfilter weights
* Add support of IPv6 in hfilter
* Fix parsing of HTTP headers with IP addresses
* Sync with the recent libucl
* Various minor bugfixes
1.0.4:
* Add configdump routine to rspamadm
* Implement retransmits for fuzzy_check plugin
* Fix events processing for learning anf checking fuzzy hashes
* Avoid dependency on unneeded and uncompatible glib include
* Add `historyreset` command to the controller
* Fix loading of tokenizer config from dump (#389)
* Add sorting hints for the history
* Allow custom lua scripts for users/languages extraction (#388)
* Do not add FORGED_RECIPIENTS when 'To' is missing (#387)
* Do not add R_UNDISC_RCPT when 'To' is missing (#387)
* Add encryption to fuzzy check plugin
* Add encryption for fuzzy storage
* Add new epoch for encrypted fuzzy request
* Add encryption for `rspamd.com` storage
* Remove gmime processing for LDA mode as it is deadly broken
* Add routine to find end of headers position in mime messages
* Fix LDA headers folding
* Init libraries in rspamc client as well to avoid locale issues
* Avoid collision with locally installed includes
* Allocate and free memory with the same allocator in rspamadm (#385)
* Preserve expired fuzzy hashes counter
* Improvements in webui:
- Add favicon.ico
- Rework history table
- Fix sorting for the history
- Migrate to bootstrap 3 and jquery 2
- Fix css bugs
- Add glyphicons
- Add reset history
- Improve history buttons
- Redraw graph to avoid display issues
- Webui is now MIT licensed to match licensing policy of rspamd
1.0.3:
* Fix piechart clean slice (#380)
* Fix controller crashes when GString is reallocated (#381)
* Correctly set locale before start
* Set C locale for numeric values
* Add rspamadm routine:
- add `pw` command to manage passwords
- add `help` command for displaying help
- add `configtest` command to check configuration files
- add `keypair` command for generating encryption keys
- add `fuzzy_merge` routine to merge fuzzy sqlite databases
- add a simple manual page for rspamadm
* Allow metric registration for composite expressions
* Add strict mode for configtest
* Add logger counters
* Save and show learned messages count (#383)
* Add `no_stat` flag
* Add `task:set_flag` and `task:get_flags` (#382)
* Enable foreign keys in sqlite3
* Remove orphaned shingles from fuzzy storage
* Optimize synchronization steps for fuzzy storage
* Allow delayed conditions registration
* Add lua API for conditions registering
1.0.2:
* Fix critical bug in webui that prevents password from being sent
* Rework webui view:
- Switch to d3.js for graphs
- Improve piechart look
- Rework colors for piechart
- Fix layout for symbols
- Fix refresh button
* Add descriptions for whitelist maps
* Fix build on arm (#379)
* Fix issue with the last element in the radix trie
* Add more tests for radix trie algorithm
* Allow to extract URLs from query strings of other URLs (#361)
* Initialize rrd fields before writing to file
* Fix double free if no password has been specified
1.0.1:
* Add writing to rrd from the controller
* Fixed lots of bugs in rrd code
* Adopt new DNS API in hfilter plugin (by @AlexeySa)
* Allow only one controller process to manage rrd file
* Set event base for fuzzy calls
* Improve fuzzy IO errors logging
* Add rra extraction function to rrd library
* Add graph handler to the controller
* Cache correct passwords to avoid too high CPU usage when working with webui
* Controller sockets are owned by router do not export them to task
* Optimize logging by skipping hash table search if it's empty
* Fix loading issue whith broken statfiles
* Print assertions from glib to rspamd logger
* Load legacy `lua/rspamd.local.lua`
* Update webui with some fixes to learning and scanning
1.0.0:
* Rework symbols processing:
- Improve sorting logic for symbols
- Organize processing into multiple stages
- Added asynchronous watchers for symbols
- Added ability to organize dependencies between symbols
* Fixed URL redirector:
- Use optimized POE loop
- Organize dependencies
- Fix startup
* New sqlite3 backend:
- Allow to have per-languages and per-user statistics
- Allow sqlite3 to be used as statistics backend
* Store tokenizer configuration within statfiles
* Improve bayes statistics:
- Use headers and images metainformation in bayes
- Suggest using of pre-processed tokens for statistics
- Fix tokens normalization for OSB algorithm
* Rewrite url parsing:
- Fix numerous issues with url extraction and normalization
- Fix mailto urls
* Fix settings plugin to allow custom actions scores
* Improve rbl plugin
* Allow capturing patterns in rspamd lua regexp library
* Add GTUBE support
* Fix spamc legacy support
* Add DKIM support to RBL module
* Fix issues with multiple DKIM signatures
* Fix issue if rspamd cannot create statfiles (#331)
* Rework parts and task structure:
- Now text_parts, parts and received are arrays
- Pre-allocate arrays with some reasonable defaults
- Use arrays instead of lists in plugins and checks
- Remove unused fields from task structure
- Rework mime_foreach callback function
- Remove deprecated scan_milliseconds field
* Add ip_score plugin support (not enabled by default):
- Can check for asn/country and network using DNS lookups
- Can store and load reputation from redis server
* Improve PARTS_DIFFER rule to count merely different words
* New HTML parser:
- Parses HTML parts using a set of state machines
- Extracts useful data and exports it to lua functions:
+ Styles
+ Images
+ URLs
+ Colors
+ Structure elements
- Added HTML rules for some checks
* New version of LUA DNS API
* Table versions of many functions in LUA API
* Improve rspamc client:
- Print execution time
- Allow executing of external commands and passing output to them
- Allow mime output mode when rspamc alters message according to rspamd
checks and send it to an external command or stdout
* Allow scanning of local files using HTTP requests
* Rework configuration system:
- Rules are now moved from the $CONFDIR to $RULESDIR to avoid ambiguity
- All modules configurations are now split in $CONFDIR/modules.d/* to
simplify upgrades
- Move hfilter to plugins
- Allow plugins and rules to define default scores to simplify metrics
setup
- Include overrides for all modules to honor local/automatic parameters
- Tune scores for many modules
* Rework and enable DMARC plugin
* Add whitelist plugin for SPF/DKIM/DMARC based whitelisting
* Add some common domains to whitelists shipped with rspamd
* Rework logging:
- Now each log entry supports module name and a `tag`. Tag is used to
identify unique objects (such as tasks) when checking log files
- It is possible to turn on debugging for the specific modules
- Systemd logging is fixed
* Improve spamassassin plugin.
- Now headers are matched more like SA
- Improve support of Message-ID
- Add support of ToCc header type
- Fix :addr and :name in headers regexps
* Resurrect rrd support code
* Save controller stats between restarts
* Fixed tonns of bugs
* Added tonns of minor improvements and features
* Added more unit tests
* Create functional tests framework
* Added documentation for missing modules
* Added rpm/deb repositories and scripts
* Updated WebUI and libucl externals
0.9.10:
* Do not dereference null pointer on learning.
* Fix some extreme cases in BAYES.
* Add a workaround to avoid bad HTML messages breaking.
* Build with -O2 flags by default.
* Add constraints to limit DNS requests count per task.
* Add workaround for SURBL DNS flood.
* Set error if rspamd cannot learn anything.
0.9.9:
* Don't use RWL_SPAMHAUS_WL (unknown result) for whitelisting (by @fatalbanana)
* Import updated public suffix list (by @fatalbanana)
* Remove debug message
* Fix settings (by @fatalbanana)
* Remove duplicated symbol registration
* Use WAL for fuzzy storage
* RBL fixes (by @fatalbanana):
- silence errors;
- yield unknown results from RBLs;
- fix scoring for DNSWL;
- fix use of RBL name as symbol;
- ignore RBL names that would not be yielded;
* Support captures in regular expressions
* Add captures support to lua_regexp
* Support dist on FreeBSD and Darwin
* Add RCVD_IN_DNSWL_NONE as whitelisting exclusion (by @fatalbanana)
* Multiple fixes to URL detection:
- support port definition;
- fix query and path recognition;
- fix parsing of multiple slashes in URL;
- fix parsing query just after port;
- fix path field in `url:to_table` method;
- improve support of IP based URLs.
* Set ignore_whitelists = true for RECEIVED_SPAMHAUS_XBL (by @fatalbanana)
* Add GTUBE support
* Ignore User header in SA mode
0.9.8:
* Fix critical bug in bayes classifier (#305)
* Fix critical bug in RBL module (by @fatalbanana)
* Fix and rework settings plugin.
* Fix get_all_opts for a case of non-iterable options.
* Use tld for redirector's matching.
0.9.7:
* Add whitelist_exception setting to RBL module (by @fatalbanana)
* Don't use RWL_MAILSPIKE_POSSIBLE or DNSWL_BLOCKED for whitelisting (by
@fatalbanana)
* Fix extreme cases in bayes classifier.
* Fix parsing of urls with '?' at the end of hostname.
* Update interface.
* Fix number of issues with webui interaction.
* Fix saving maps.
* Allow user@ and @domain matches in multimap.
* Fix issues with bounces From processing.
* Fix abs/fabs misuse.
* Fix builds on suse and arch linux distributions.
0.9.6:
* Fix memory leak if mime cannot be parsed.
* Fix dkim cache expiration.
* Fix issues with redirector HTTP response.
* Fix abnormal connection closing with certains messages with a high score
(issue #296)
* Fix redirector installation.
* Use specific POE loop for some systems.
* Fix number of issues in URL redirector.
* Fix selecting URLs for sending to redirector.
0.9.5:
* Avoid double free when extending HTTP message.
* Fix double free if multiple classifiers are defined.
* Fix misprint in spamassassin plugin.
* Fix cpuid invocation on i386.
* Fix ownership issues for zero-copy decode.
* Allow __len metamethod on rspamd{text}.
* Add base64 decoding lua utility.
* Fix build on FreeBSD
* Skip spaces at the beginning of mime messages.
* DBL_ABUSE_REDIR should not have significant weight.
* Allow to split by lua_regexp rspamd{text} objects.
* Allow to specify custom stop pattern for lua_tcp.
0.9.4:
* Fix critical bugs in tokenization algorithm
* Write unit tests for tokenization
* Add documentation for lua_tcp
* Switch off legacy tokenization by default.
* Fix critical bugs in words normalization
* Add lua bindings to tokenizer.
* Implement storing of HTTP headers inside task
* Add lua API to accerss HTTP headers data
* Implemented base64 encoding suitable for MIME
* Use caseless hash and equal functions for HTTP request headers.
* Improve debian architectures support (by @dottedmag)
0.9.3:
* Revert incorrect regexp change that broke the default rules
* Fix lua_tcp module
0.9.2:
* Fix error on spawning unique workers.
* Add preliminary version of generic LUA TCP requests API.
* Use lua 5.1 if luajit is not available (Arm64, PowerPC, s390x etc)
* Fix fuzzy mime strings with only type.
* Improve thunderbird sanity checks.
* Fix critical bug on matching regular expressions.
* Make hiredis optional dependency.
* Fix multiple bugs in daemon reloading
0.9.1:
* Restore utf8 validation for regular expressions to avoid crashes
* Fix symbols displaying in the interface
* Add symbol groups to the interface
* Fix maps ID parsing in the controller
* Add multimap and regexp modules documentation
* Backport fixes from libucl
* Fix debian package (by @dottedmag)
* Rework XXH32 invocations
0.9.0:
* Add support of the fast and secure protocol level encryption:
- curve25519 is used for key exchange;
- chacha20/poly1305 cryptobox construction for bulk encryption;
- zero latency overhead;
- encrypting and balancing HTTP proxy worker
* Rework expressions and create new expressions library:
- aggressive optimizations based on the abstract syntax tree;
- abstract expressions support (regular expressions, functions, lua modules
composites and so on)
- New comparision and '+' operators support
- New greedy algorithm to minimize execution time of expressions and
all symbols
- Dynamic expressions benchmark and reoptimizations
* Many improvements to the LUA API:
- reworked logger module allowing to do pretty print of the most of lua
types (including tables and userdata classes)
- reworked lua redis and lua HTTP to support more features
- added opaque type for passing large text chunks without copying
- new regexp module with many auxiliary functions (e.g. `re:split`)
* LuaJIT is now the default requirement for rspamd allowing to speed up lua
execution by a large margin (however, plain lua is still supported)
* New plugins:
- spamassassin rules plugin that allows to load and re-use the most of
SA rules natively
- DMARC plugin that evaluates SPF and DKIM policies to the domain policies
- many old plugins has been reworked to implement new features and improve
stability
* New aho-corasic trie implementation from @mischasan that allows to load and
use hundreds of thousands of patterns with no influence on load
* Support of PCRE JIT and PCRE JIT fast path modes that significantly improves
the performance of regular expressions if supported by PCRE
* New URLs parser and extractor:
- removed legacy code that was useless for url finding
- reworked algorithms of URL parsing for more precise and accurate results
- added top-level-domains tree from http://publicsuffix.org
- improved emails parsing
- removed many phishing false positives due to TLD tree check
* New statistics infrastructure:
- created a separate layer of statistic library
- improved OSB-Bayes by re-weighting tokens according to the original
academic paper and `crm114` implementation, which reduced false positives
rate significantly
- created learn cache to avoid double learning of statistics and providing
an efficient way to re-learn class for a message
- created abstract layers for different statistics backends
- implemented new tokenization algorithms with fast or secure (siphash)
hashes to generate statistics features
* Reworked utf8 tokenization that previously corrupted all UTF8 words (minor
incompatibility with old fuzzy hashes with utf-8 symbols)
* SPF module has been completely rewritten to support complex cases of
`include` and `redirect` within SPF records
* DKIM module now supports multiple signatures
* Controller passwords can now be stored encrypted by `PBKDF2-HMAC` in the
configuration file
* Many hand-written HTTP clients has been replaced with the common rspamd
http module
* New test framework:
- import lua `telescope` test framework
- add unit tests for many rspamd modules and routines
- create a unit test for each possible bug found
- use luajit ffi for testing C code
- added preliminary support of functional testing by creating tasks from lua
* Randomize hash seed to avoid certain hash tables vulnerabilities
* Documentation improvements:
- added documentation for the vast majority of rspamd modules
- added documentation for rspamd protocol
- added documentation for the most of rspamd LUA extensions
* Fixed tonns of bugs and memory leaks
* Added tonns of minor features
0.8.3:
* Various critical fixes in distribution (by @dottedmag and @fatalbanana)
* Fixed bugs in url detector to parse certain patterns
* Add default host and helo for a client
* Some sanity checks for tokenizer and classifier
* Reiterate on systemd support
* Fix missing symbol registration
* Add support of spamc compatible output
* Filter double-dots in rbl.lua validate_dns (by @fatalbanana)
* Update ucl submodule due to critical bugfix
0.8.2:
* Create fuzzy db if it does not exist
* Fix: Centos init script: configtest() (by @AlexeySa)
* Enable one_shot for RECEIVED_SPAMHAUS_XBL - Fixes #102 (by @fatalbanana)
* Update Exim patch (by @fatalbanana)
* Fix processing of unix sockets.
* Allow applying settings to authenticated users (by @fatalbanana)
* Make settings priorities work as documented (by @fatalbanana)
* Fix race condition in symbols planner
* Add DNSWL_BLOCKED symbol (by @fatalbanana)
* Make Exim pass usernames to rspamd (by @fatalbanana)
* Update RBL module (by @fatalbanana):
- fix indentation;
- collapse loops;
- avoid calling for un-needed information;
- allow disabling RBLs for authenticated users
* once_received.lua: Fix indentation & add exclusion for authenticated users (by @fatalbanana)
* hfilter.lua: Add exclusion for authenticated users (by @AlexeySa)
* Updates to hfilter rules (by @AlexeySa)
* Set empty <> user or addr for msgs without FROM (by @eneq123)
* Fix: attempt to index field '?' (a nil value) (by @eneq123)
* Fix: if not exist Date-header (by @AlexeySa)
* Add task:get_content() method.
* rbl.lua: Ignore private IP space (by @fatalbanana)
* Allow to check radix maps from lua by rspamd{ip}
* Make local exclusions configurable per-RBL (by @fatalbanana)
* Add rspamd_config:radix_from_config() (by @fatalbanana)
* Support emails dnsbl in rbl (by @fatalbanana)
* Complete rework of url extraction logic
* Allow customizations for unix sockets. (fixes #182)
* Set lua path according to rspamd settings.
* Import lua-functional for plugins stuff.
* Completely rewrite multimap plugin in functional style.
* Fix FORGED_MUA_THUNDERBIRD_MSGID (fixes #186)
* Check IPv6 addresses at dnswl.org and Spamhaus whitelist (by @fatalbanana)
* Add lowercase utility for utf8 strings.
* Various fixes to build system
* Updated debian configuration infrastructure (by @dottedmag)
0.8.1:
* Add sqlite and perl as dependencies for RPM/Debian packages (by @fatalbanana)
* Remove whitelist.lua from RPM file list (by @fatalbanana)
* Make Exim pass hostnames to rspamd (by @fatalbanana)
* Fix building on Fedora (by @fatalbanana)
* Add toggle for disabling installation of systemd units on Linux (by @fatalbanana)
* Fix double format rounding that caused output corruption (reported by @fatalbanana)
* Revert broken change for destructors ordering that led to memory corruption
* Do not reset symbols case of settings if parsed from lua (reported by @andrejzverev)
* Fix build on SunOS (by @wiedi)
* Fix multiple crashes on broken DKIM DNS records
* Fix critical issue with composites weights removing
* Fix memory corruption in composites processing code
* Ignore non-SPF TXT records when parsing SPF includes
0.8.0:
* New fuzzy check logic:
- use shingles algorithm for fuzzy matching
- use blake2 instead of md5 for larger output space
- combine fuzzy and strict matching
- allow to organize private storages by means of keys
- preserve compatibility with previous versions
* New fuzzy storage:
- use sqlite instead of own memory based hash tables
- rework commands interface
- add conversion from the old format
- add fuzzy match by shignles
- support old rspamd versions
* Add lemmatizing for words used in fuzzy hashes that allows to improve match
quality by using of the first forms of all words
* Rework language detection
* Fix several critical bugs, memory leaks and deadlocks:
- memory leak in HTML nodes parsing
- deadlock in logger code
- deadlock in signals processing
- crashes in fuzzy_storage
- crashes in tokenizers if the input was empty
* Import new libucl with several bugfixes and improvements
* Support listening on ipv6 addresses only
* Fix macro expansion in SPF module
* Several bugfixes in DKIM module
* Add load headers support for mime parts to the lua API
* Add documentation for:
- workers in general
- fuzzy_storage worker
- fuzzy_check plugin
- mimepart and textpart lua API modules
0.7.6:
* Apply boundary fix for dkim simple canonization
* Fix ping command
* Return nil if header was not found in lua_task
* Fix hang in upstreams revive logic
* Decode entitles when normalizing HTML parts
* Fix logic of finding URLs in HTML parts
* Do not include \0 into length of text when performing conversion to utf8
* Fix raw vs parsed reperesentations
Raw parts are now:
- decoded b64/qp, but *NOT* converted to utf-8
Processed parts are now:
- converted to UTF-8
- normalized if needed (e.g. HTML tags are stripped)
* Rework DKIM canonization to line based
* Fix fuzzy hashes addding
* Use more specific hash function for fuzzy
* Fix leaking of iconv descriptors
* Fix PTR resolving in lua resolver
* Rework spf module.
- Copy data to memory pool as cached record might be destroyed causing
freed memory being passed to the protocol output (use after free)
- Allow SPF_NEUTRAL policy to be handled separately
- Add R_SPF_NEUTRAL to the default config
* Rework `register_symbols` function
* Allow to disable components of hfilter
0.7.5:
* Fix owner when creating folder /run/rspamd (by @sfirmery)
* Fix IP validity checks
* Decode URLs obtained from HTML tags
* Fix crash with unweighted upstreams
* Stop processing headers in parts
* Set sockaddr.sa_family properly when connectig to upstreams
* Fix reload issues in surbl and fuzzy_check (reported by @citrin)
* Fix timeouts in redirector
* Improve lua errors reporting
* Fix lua closures processing in libucl
* Rework calling of lua functions from regexp module
* Choose raw regexp for raw headers
* Rework conversion to utf since glib one is broken
* Ignore SGML style tags in html
* Fix old bug with non-capturing https urls
* Fix memory corruption on fuzzy reload (reported by @citrin)
* Fix percents display in rspamc
* Fix buffer update for DKIM
* Do not validate utf for raw headers
0.7.4:
* Fix build under *BSD
* Detect HAN unicode script
* Implement language detection heuristic for text parts
* Fix time output in history
* Improve piechart coloring
* Fix \r\n conversion in DKIM module (reported by @citrin)
* Try to detect systems with no IPv6 support
* Fix multiple/single values in use settings (reported by @citrin)
* Rework IP addresses in upstreams:
- Select ipv4/unix addresses if they exist and use ipv6 for ipv6 only
upstreams (since the support of ipv6 is poor in many OSes and
environments)
- Free IP list on upstream destruction
- Add test cases for addresses selection
- Allow adding of free form IP addresses to upstreams
* Fix endiannes in lua_radix search (reported by @citrin)
* Soft shutdown should also set wanna_die flag (reported by @citrin)
* Stop use-after-free in event loop termination
* Fix processing of very short messages in DKIM (reported by @citrin)
* Detect systems without shared mutexes
* Fix issues with PTR and MX elements in SPF parser (reported by @citrin)
0.7.3:
* New upstreams code:
- simplify upstreams API;
- unify strings parsing in upstreams definition;
- add configuration options for the upstreams;
- for failed upstreams re-resolve their addresses;
- use all resolved addresses for an upstream (round-robin);
- implement stable hashing and use it by default for upstreams;
- add unit test for upstreams module.
* Rework signals processing in all rspamd workers:
- signals are now processed in the event loop;
- implement the most common signal handlers for all workers;
- add callbacks for workers specific signal handlers
* Fix critical issue with fuzzy storage:
Fuzzy stroage could not save any hashes on termination due to bugged
signals handling
* Fix roll history IP storage
* Rework ipv4/ipv6 handling in parsing addresses:
- turn off support of IPV6_V6ONLY socket option;
- create ipv6 socket prior to ipv4 one to handle systems with v6/v4
sockets enabled (Linux)
* Remove CBL as it's wholly included in Spamhaus XBL (by @fatalbanana)
* Remove nszones.com fake RBL (by @citrin)
* Fix upstreams interaction for fuzzy_check
* Verify spf PTR records (reported by @citrin)
* Fix spf MX records parsing
* Add compatibility for old libevent (by @yellowman)
* Sync bugfixes from libucl
0.7.2:
* Convert all maps to the compressed radix trie
* Allow IPv6 addresses in IP maps
* Remove dynamic items support from symbols cache
* Allow hex encoded output of strings