-
Notifications
You must be signed in to change notification settings - Fork 27
Expand file tree
/
Copy pathCHANGES
More file actions
1658 lines (931 loc) · 52.9 KB
/
CHANGES
File metadata and controls
1658 lines (931 loc) · 52.9 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2.2.0 | 2021-12-20 19:32:33 -0800
* Merge branch 'topic/christian/fix-python-unhashables' (Christian Kreibich, Corelight)
* topic/christian/fix-python-unhashables:
Add testcases for new Python SafeSubscriber class
Provide a SafeSubscriber class in Python to accommodate non-hashable types
Add immutable values to Python data type testsuite
Python cleanup: remove unneeded semicolons and trailing whitespace
* Add testcases for new Python SafeSubscriber class (Christian Kreibich, Corelight)
* Provide a SafeSubscriber class in Python to accommodate non-hashable types (Christian Kreibich, Corelight)
Broker's data model supports nested complex types, such as a set with table
members. When rendering such structures into Python, this could trigger
TypeError exceptions due to unhashable types.
The SafeSubscriber class and the corresponding Endpoint.make_safe_subscriber()
method ensure that immutable (hashable) Python types get used and that the
returned values are read-only. For sets the frozenset type achieves this, for
mapping types it's types.MappingProxyType. The API now also supports the latter
two types as inputs.
* Add immutable values to Python data type testsuite (Christian Kreibich, Corelight)
* Python cleanup: remove unneeded semicolons and trailing whitespace (Christian Kreibich, Corelight)
2.2.0-dev.17 | 2021-11-05 13:38:17 -0700
* Request at least Sphinx 2.0 to avoid a dependency problem in RTD (Christian Kreibich, Corelight)
* Introduce .readthedocs.yaml to avoid docs directory (Christian Kreibich, Corelight)
2.2.0-dev.14 | 2021-11-05 12:47:15 -0700
* Add macOS Monterey and drop Catalina in CI (Christian Kreibich, Corelight)
* Add Fedora 35 in CI (Christian Kreibich, Corelight)
2.2.0-dev.10 | 2021-08-26 14:30:39 -0700
* CI support refresh (Christian Kreibich, Corelight)
- Add Debian 11 (Bullseye)
- Drop Ubuntu 16.04
2.2.0-dev.8 | 2021-08-03 11:55:22 -0700
* Fix publisher deadlock when passing large batches (Dominik Charousset, Corelight)
2.2.0-dev.6 | 2021-07-28 19:08:17 -0700
* Fix output of build-type CFLAGS during configure (Tim Wojtulewicz, Corelight)
* Update HMAC key for benchmarking service (Tim Wojtulewicz, Corelight)
2.2.0-dev.3 | 2021-07-16 19:02:24 -0700
* Raise CAF dependency to 0.18.5 and update embedded version (Tim Wojtulewicz, Corelight)
2.1.0 | 2021-07-08 09:49:23 -0700
* Release 2.1.0.
2.1.0-dev.90 | 2021-07-08 09:49:16 -0700
* Raise CAF dependency to 0.18.4 and update embedded version (Tim Wojtulewicz, Corelight)
* Bring back process metrics (Dominik Charousset, Corelight)
2.1.0-dev.87 | 2021-07-02 10:02:22 -0700
* Fix is_prefix logic (Dominik Charousset, Corelight)
* Remove global topic variables (Dominik Charousset, Corelight)
Fix a potential memory corruption during static initialization by
avoiding global variables with non-trivial constructors.
2.1.0-dev.83 | 2021-06-28 08:44:56 -0700
* Fix heap-use-after-free during shutdown (Dominik Charousset, Corelight)
2.1.0-dev.81 | 2021-06-21 15:56:14 -0700
* CI support refresh to bring in line with Zeek (Christian Kreibich, Corelight)
* Update port used by benchmark host (Tim Wojtulewicz, Corelight)
2.1.0-dev.78 | 2021-06-07 09:22:35 -0700
* Remove noexcept from functions that may throw
Functions that access a `variant` of a `metric_view` may throw
`broker::bad_variant_access`. (Dominik Charousset, Corelight)
* Fix uninitialized field warnings (Dominik Charousset, Corelight)
2.1.0-dev.75 | 2021-06-01 10:18:00 -0700
* Make telemetry exporter configurable at runtime (Dominik Charousset, Corelight)
* Raise CAF dependency to 0.18.3 (Dominik Charousset, Corelight)
* Update CAF submodule (Dominik Charousset, Corelight)
* Enable Broker to collect cluster metrics (Dominik Charousset, Corelight)
* Add remaining meta data to scrape output (Dominik Charousset, Corelight)
* Add support for histograms to the scraper (Dominik Charousset, Corelight)
* Implement new publishing mode for local metrics (Dominik Charousset, Corelight)
2.1.0-dev.67 | 2021-05-24 15:22:34 -0700
* Change how actor_system_config::config_file_path is set (Jon Siwek, Corelight)
Mainly avoid conflict with an upcoming change
2.1.0-dev.66 | 2021-05-24 15:01:34 -0700
* Update embedded CAF to 0.18.3 (Jon Siwek, Corelight)
And migrate a now-deprecated actor_system_config::parse() usage.
2.1.0-dev.65 | 2021-04-19 16:14:38 -0700
* Fix broker-pipe crash with static CAF builds (Dominik Charousset, Corelight)
2.1.0-dev.63 | 2021-04-19 16:01:51 -0700
* Remove redundant code in "integration" test (Dominik Charousset, Corelight)
2.1.0-dev.61 | 2021-03-27 14:15:37 -0700
* Bump libbroker soversion to 4 (Jon Siwek, Corelight)
* Update bundled CAF to 0.18.2 (Jon Siwek, Corelight)
2.1.0-dev.58 | 2021-03-17 13:45:33 -0700
* Prevent use of LeakSanitizer on FreeBSD (Jon Siwek, Corelight)
2.1.0-dev.56 | 2021-03-08 12:27:52 -0800
* Improve error handling for data store backend creation failures (Jon Siwek, Corelight)
* Change broker::filesystem::detail functions that rely on
std::filesystem to use non-throwing calls
* Add error messages to some broker::filesystem::detail functions:
mkdirs(), remove(), and remove_all()
* Use sqlite3_errmsg() to emit reason for sqlite3_open() failures
* Change sqlite backend initialization failures to be non-fatal
Related to https://github.com/zeek/zeek/issues/1426
2.1.0-dev.54 | 2021-03-02 19:59:06 -0700
* Add lgtm configuration (Tim Wojtulewicz, Corelight)
2.1.0-dev.52 | 2021-02-26 17:57:58 -0800
* Fix to allow hashing of Python binding's broker.Count type (Jon Siwek, Corelight)
2.1.0-dev.50 | 2021-02-22 15:45:38 -0800
* Enable metrics exposition via BROKER_METRICS_PORT (Dominik Charousset, Corelight)
2.1.0-dev.48 | 2021-02-16 13:58:19 -0800
* Increase CPUs used for macOS CI builds (Jon Siwek, Corelight)
2.1.0-dev.47 | 2021-02-16 12:30:36 -0800
* broker-benchmark: Fix heap-buffer overflow when receiving stats (Dominik Charousset, Corelight)
* Fix metrics and TTL of locally published data (Dominik Charousset, Corelight)
* Implement more lightweight origin credit tracking (Dominik Charousset, Corelight)
The item-based tracking of message origins keeps all context for input
messages. However, benchmarking reveled that this nice-to-have property
comes at an expensive price.
The new approach for tracking how many inputs for each origin are
currently in the system leverages the reference count of Broker messages
that we need in any case rather than introducing additional state.
* Disable background deserialization (Dominik Charousset, Corelight)
Zeek configures only a few threads to the CAF thread pool. This may
introduce some heavy delays for the deserialization that takes place in
the background. Disabling this feature altogether makes sure that
messages will not end up in some queues even though it puts more
pressure on the multiplexer's thread.
2.1.0-dev.42 | 2021-02-11 12:10:05 -0800
* Disable syntax highlighting for some overview documentation (Jon Siwek, Corelight)
* Add Python bindings example docs for representing Zeek records (Jon Siwek, Corelight)
* Fix typo in data model documentation (Jon Siwek, Corelight)
2.1.0-dev.39 | 2021-01-25 21:17:05 -0800
* Add missing comments to example Windows Dockerfile (Jon Siwek, Corelight)
* Add Travis CI config for Windows OS (Jon Siwek, Corelight)
2.1.0-dev.36 | 2021-01-25 12:27:58 -0800
* Update embedded CAF to 0.18 (Jon Siwek, Corelight)
* Add CI support for running cluster benchmarks (Tim Wojtulewicz, Corelight)
2.1.0-dev.33 | 2021-01-21 16:15:32 -0800
* Fix deprecation warnings with latest CAF (Dominik Charousset, Corelight)
* Clean up remaining caf::hash::fnv TODO (Dominik Charousset, Corelight)
CAF 0.18 introduced an FNV hashing visitor that we can use instead of
rolling custom hash algorithms and hash combinators.
* Replace unsafe (to|from)_blob functions (Dominik Charousset, Corelight)
The utility functions `to_blob` and `from_blob` in
`broker/detail/blob.hh` silently ignored any error during serialization
and deserialization. Since the functions were only used by the SQLite
backend, the new implementations with proper error propagation (and
cleaned up interface) moved to `sqlite_backend.cc`.
2.1.0-dev.27 | 2021-01-19 21:07:18 -0800
* Update Cirrus CI naming for MacOS images (Christian Kreibich, Corelight)
The docs on https://cirrus-ci.org/guide/macOS/ changed from
"osx_instance" to "macos_instance", so let's reflect that.
2.1.0-dev.25 | 2021-01-12 20:19:56 -0800
* GH-165: Fix possible deadlock by implementing new origin-based credit strategy (Dominik Charousset, Corelight)
2.1.0-dev.16 | 2021-01-11 11:05:53 -0800
* Remove deprecated broker::memory/broker::sqlite enums (Jon Siwek, Corelight)
* Remove unused and broken rocksdb backend (Dominik Charousset, Corelight)
2.1.0-dev.12 | 2021-01-04 15:40:56 -0800
* broker-cluster-benchmark: Fix type mismatch and potential cleanup issue (Dominik Charousset, Corelight)
2.1.0-dev.10 | 2020-12-23 10:13:37 -0800
* Call `ctest` instead of `make test` in CI builds (Dominik Charousset, Corelight)
Setting `CTEST_OUTPUT_ON_FAILURE` and then calling `make test` seems to
lose the flag at some point. Calling `ctest` directly and passing
`--output-on-failure` makes sure that we actually get the test output in
case of an error.
2.1.0-dev.8 | 2020-12-21 10:53:48 -0800
* Add macOS Big Sur to CI (Jon Siwek, Corelight)
2.1.0-dev.6 | 2020-12-17 16:04:25 -0800
* Remove explicit use of config mode for finding CAF (Dominik Charousset, Corelight)
2.1.0-dev.4 | 2020-12-17 10:36:12 -0700
* Fix CAF configuration parameter names (Dominik Charousset, Corelight)
2.1.0-dev.2 | 2020-12-16 15:04:30 -0800
* Make sure status updates are visible before data (Dominik Charousset, Corelight)
2.1.0-dev | 2020-12-14 20:16:02 -0800
* Update NEWS for start of v2.1.0 development (Jon Siwek, Corelight)
1.5.0-dev.63 | 2020-12-14 10:38:14 -0800
* Update CAF submodule to fix handling of --libdir (Jon Siwek, Corelight)
1.5.0-dev.62 | 2020-12-12 18:28:25 -0800
* Change `configure --prefix` to not set CMAKE_INSTALL_LIBDIR (Jon Siwek, Corelight)
In places where that libdir option is used and not explicitly set
before, it will already correctly prepend CMAKE_INSTALL_PREFIX. Having
the configure script use the hardcoded 'lib' also prevented the
automatic use of 'lib64' on platforms that prefer it.
1.5.0-dev.61 | 2020-12-10 14:07:51 -0800
* Add Jenkinsfile for re-enabling Windows builds (Dominik Charousset, Corelight)
* Add Python context manager support (Dominik Charousset, Corelight)
For proper resource management/cleanup, the Python API now requires
using Endpoint, Subscriber, StatusSubscriber, and Store objects within a
`with` statement or alternatively doing an explicit call to the
``reset()`` method of subscriber/store objects before the associated
Endpoint's ``shutdown()`` method.
Co-authored-by: Jon Siwek <jsiwek@corelight.com>
* Disable Windows builds via Travis CI (Dominik Charousset, Corelight)
Remove the YAML file for Travis CI due to issues with getting Visual
Studio 2019 with support for `std::filesystem` working on Travis. This
is supposed to be a temporary measure until either finding a fix for the
setup or re-enabling Windows build through some other means.
* Fix handling of initial subscription set (Dominik Charousset, Corelight)
* Add missing to_string overloads (Dominik Charousset, Corelight)
* Make CMake scaffold more robust against re-runs (Dominik Charousset, Corelight)
* Clean up deprecation warnings for CAF 0.18 (Dominik Charousset, Corelight)
* Fix CAF paths for bundled CAF version (Dominik Charousset, Corelight)
* Migrate to CAF 0.18, update CAF submodule (Dominik Charousset, Corelight)
1.5.0-dev.43 | 2020-12-09 17:27:31 -0800
* Update SQLite to 3.34.0 (Jon Siwek, Corelight)
1.5.0-dev.42 | 2020-12-04 13:05:41 -0800
* Add FreeBSD 11.4 and 12.2 to CI (Jon Siwek, Corelight)
1.5.0-dev.40 | 2020-12-04 11:32:19 -0800
* Suppress interactive tzdata setup on Ubuntu (Dominik Charousset)
* Call update on Debian-based distros before install (Dominik Charousset)
* Update CI build matrix (Dominik Charousset)
* Update and streamline Docker files (Dominik Charousset)
1.5.0-dev.34 | 2020-12-02 11:05:55 -0800
* Update minimum required CMake to 3.5 (Jon Siwek, Corelight)
1.5.0-dev.32 | 2020-12-02 10:57:29 -0800
* Upgrade pybind11 to v2.6.1 and fix compatibility issues (Jon Siwek, Corelight)
1.5.0-dev.30 | 2020-11-26 18:12:38 +0000
* Update macOS CI to use system default Python 3 (Jon Siwek, Corelight)
* Remove Python 2 compatibility logic (Jon Siwek, Corelight)
* Update CMake logic to enforce Python >= 3.5 for Python Bindings (Jon Siwek, Corelight)
* Update Python Bindings docs for new Python >= 3.5 requirements (Jon Siwek, Corelight)
* Update cmake submodules with changes related to Python 2 EOL (Jon Siwek, Corelight)
1.5.0-dev.22 | 2020-10-21 15:37:57 -0700
* Move CI from Fedora 30 (EOL) to Fedora 32 (Jon Siwek, Corelight)
1.5.0-dev.20 | 2020-10-21 15:14:55 -0700
* Update Windows CI config (Jon Siwek, Corelight)
* Disable Cirrus Windows task: an updated Docker image on Cirrus-side
has unknown issues and trying to create one for ourselves also ended
with similar, unsuccessful results
* Provide an example Dockerfile that uses Visual Studio 2019, which may
help as a starting point for manual testing/development purposes
* Enable Travis for testing just Windows
1.5.0-dev.16 | 2020-08-21 11:41:06 -0700
* Respect --config-file= CLI parameters (Dominik Charousset, Corelight)
1.5.0-dev.14 | 2020-07-30 16:08:56 -0700
* Fix build with CAF 0.18 (Dominik Charousset)
1.5.0-dev.12 | 2020-07-27 18:17:13 -0700
* Use 127.0.0.1 instead of ::1 for cluster configs (Dominik Charousset)
* Call abort() on errors during cluster benchmarks (Dominik Charousset)
Orderly shutdowns seem to fail for some categories of errors. Calling
`abort` ensures that the program at least always terminates, even though
this means skipping cleanup of system resources such as open ports.
* Make cluster-benchmark tool modular, add new mode (Dominik Charousset)
Rather than using flags to change the behavior, the configuration of the
tool now asks for a mode. This makes the tool easier to extend and use.
Previously the behavior of mutualy exclusive flags (for example, setting
both `--generate-config` and `--dump-stats`) was not well defined.
Furthermore, this commits adds another mode: `shrink-generator-file`.
This enables users to reduce the size of recordings retrospectively.
* Add factory function for creating generator file headers (Dominik Charousset)
* Add new read_raw method to expose generator file input bytes (Dominik Charousset)
* Make broker-cluster-benchmark config file output independent of argv order (Dominik Charousset)
* Add integration test for broker-cluster-benchmark (Dominik Charousset)
* Support reading the cluster benchmark config from STDIN (Dominik Charousset)
1.5.0-dev | 2020-07-27 10:45:17 -0700
* Update NEWS for v1.5.0-dev placeholder (Jon Siwek, Corelight)
1.4.0 | 2020-07-27 10:28:25 -0700
* Release 1.4.0
1.4.0-dev.171 | 2020-07-24 15:28:49 -0700
* Update embedded CAF 0.17.6 (Jon Siwek, Corelight)
1.4.0-dev.170 | 2020-07-24 13:53:33 -0700
* Use std::move in some broker::store methods (Jon Siwek, Corelight)
1.4.0-dev.169 | 2020-07-24 11:28:34 -0700
* Silence self-assign-overloaded warning on Clang (Dominik Charousset)
1.4.0-dev.165 | 2020-07-16 18:30:03 -0700
* Update CAF to release/0.17 branch (Jon Siwek, Corelight)
1.4.0-dev.164 | 2020-07-14 14:56:23 -0700
* Properly initialize stack buffers in the flare (Dominik Charousset)
1.4.0-dev.162 | 2020-07-13 09:19:40 -0700
* GH-129: improve logging for stale store expiry reminders (Jon Siwek, Corelight)
1.4.0-dev.160 | 2020-07-10 16:51:18 -0700
* broker-cluster-benchmark: Avoid extra peerings on startup (Dominik Charousset)
* broker-cluster-benchmark: Fix loop detection and tree printing (Dominik Charousset)
1.4.0-dev.157 | 2020-07-08 08:33:03 -0700
* Change default console logging verbosity to "error" (Jon Siwek, Corelight)
1.4.0-dev.155 | 2020-07-07 12:40:20 -0700
* Improve error messages emitted from SQLite backend (Jon Siwek, Corelight)
* Fix typos in docs (Jon Siwek, Corelight)
* Add Python SQLite-backed data store examples (Jon Siwek, Corelight)
* Fix syntax highlighting for Python examples (Jon Siwek, Corelight)
* Fix docs to include snippets from correct header path (Jon Siwek, Corelight)
1.4.0-dev.150 | 2020-06-30 10:58:03 -0700
* Include store IDs in the topic (Dominik Charousset)
As discussed in #121, adding the store ID to the topic enables better
control over the amount of data a subscriber receives by selecting only
stores of interest.
1.4.0-dev.148 | 2020-06-23 16:10:07 -0700
* Use the right variable when opening peers.txt for recording data (Tim Wojtulewicz, Corelight)
1.4.0-dev.146 | 2020-06-12 18:01:03 -0700
* Differentiate between data store remove and expire events (Dominik Charousset)
* Remove the multimap for data store clones (Dominik Charousset)
1.4.0-dev.141 | 2020-06-02 10:53:09 -0700
* Change CI script to compile from build/Makefile (Jon Siwek, Corelight)
1.4.0-dev.140 | 2020-06-01 12:16:17 -0700
* Add store ID to insert, update and erase events (Dominik Charousset)
1.4.0-dev.138 | 2020-05-29 17:33:07 -0700
* Add benchmark targets to installed files (Tim Wojtulewicz, Corelight)
1.4.0-dev.136 | 2020-05-28 14:08:26 -0700
* Move recording logic to mixin (Dominik Charousset)
* Move data store management to mixin (Dominik Charousset)
* Move connection and notification logic to mixins (Dominik Charousset)
1.4.0-dev.131 | 2020-05-19 10:48:28 -0700
* Fix definition of broker::infinite (Dominik Charousset)
* Bump required CAF version to >= 0.17.5 (Dominik Charousset)
* Update CAF submodule (Dominik Charousset)
* Port Broker to CAF 0.18 (Dominik Charousset)
With the new forward compatibility macros in CAF 0.17.5, we can support
both CAF releases with a few select #ifdef blocks until switching fully
to CAF 0.18.
* Switch to _v suffix notation for atoms (Dominik Charousset)
* Use CAF's new type ID API (Dominik Charousset)
1.4.0-dev.121 | 2020-05-15 10:37:27 -0700
* Use BROKER_WINDOWS in conditionals for MinGW (Noah Treuhaft)
1.4.0-dev.118 | 2020-05-13 11:12:51 -0700
* Add CMake variable to disable doc examples (Tim Wojtulewicz, Corelight)
1.4.0-dev.116 | 2020-04-30 18:29:35 -0700
* Use inheritance for composing the core manager (Dominik Charousset)
* Refactor streaming scaffold (Dominik Charousset)
- Bundle streaming logic in new alm::stream_transport class
- Inherit from `caf::stream_manager` directly
- Remove the obsolete `stream_distribution_tree`
* Templatize the core policy (Dominik Charousset)
* Generalize node_message (Dominik Charousset, Corelight)
* Fix subscription management with filter_extend (Dominik Charousset, Corelight)
1.4.0-dev.105 | 2020-04-30 16:37:21 -0700
* Allow users to subscribe to data store "update events" (Dominik Charousset)
By subscribing to the new (reserved) topic topics::store_events, users can
now get insert, update, and erase events whenever values in the store are
added, changed, or removed::
auto x = subscriber.get();
if (auto insert = store_event::insert::make(x)) {
// ...
}
// ... same for store_event::update and store_event::erase ...
1.4.0-dev.85 | 2020-04-29 15:17:53 -0700
* Add new status codes for node discovery (Dominik Charousset)
The new status codes are currently unused, but become relevant when
switching to the ALM-based communication backend.
1.4.0-dev.81 | 2020-04-15 15:10:37 -0700
* GH-109: fix unusable subscriber.poll() method in Python bindings (Jon Siwek, Corelight)
1.4.0-dev.79 | 2020-04-15 15:08:12 -0700
* GH-106: Fix Python tests to work with Python 3.8 (Jon Siwek, Corelight)
* Change topic strings to not automatically alter slashes (Jon Siwek, Corelight)
Automatically removing trailing or consecutive slashes in topic strings
prevents uniquely targeting topic names to a node.
For example, publishing to "node-10/" should not match a subscription
for "node-1/", but that's what happens with prefix-based matching if the
trailing slash is automatically removed.
1.4.0-dev.77 | 2020-04-09 22:57:15 -0700
* Fix concurrency setup for the Cirrus/MSVC build (Dominik Charousset)
1.4.0-dev.75 | 2020-03-30 13:21:09 -0700
* Make RocksDB usage opt-in and add --enable-rocksdb configure flag
(Jon Siwek, Corelight)
1.4.0-dev.73 | 2020-03-27 11:11:51 -0700
* Add missing binding for broker_options member (Dominik Charousset)
* Update CAF submodule to fix MSVC/Windows build (Dominik Charousset)
1.4.0-dev.69 | 2020-03-26 12:00:05 -0700
* Update CAF submodule to fix GCC 7.2 build (Dominik Charousset)
1.4.0-dev.67 | 2020-03-23 10:22:10 -0700
* Increase template backtrace limit (Dominik Charousset)
1.4.0-dev.65 | 2020-03-13 22:21:28 -0700
* Add Windows task to Cirrus CI config (Jon Siwek, Corelight)
This also moves the CAF submodule from aux/caf/ to caf/ since
Windows does not allow naming a directory "aux".
1.4.0-dev.63 | 2020-03-13 13:38:40 -0700
* Add branch whitelist for CI notifications (Jon Siwek, Corelight)
1.4.0-dev.62 | 2020-03-09 19:21:47 -0700
* Add Cirrus CI config and remove Travis CI config (Jon Siwek, Corelight)
The CMake configuration also now prefers choosing a Python3 installation
over Python2 (since that's end-of-life) for building the Broker Python
bindings.
Some timeout values in unit tests were also fiddled to work better in
the CI environment.
1.4.0-dev.60 | 2020-03-09 12:54:05 -0700
* Fix a race condition in data store flare operations (Jon Siwek, Corelight)
The race (as seen via Zeek usage) goes like:
Thread A: enqueue item, get suspended
Thread B: sees mailbox has items
Thread B: dequeue item
Thread B: extinguish flare
Thread A: resume, fire flare
That ordering can leave the flare in an active state without any actual
items remaining in the mailbox.
This patch adds a mutex/lock such that extinguishing of the flare cannot
be interleaved between the enqueue and firing of the flare.
This likely relates to https://github.com/zeek/zeek/issues/838,
https://github.com/zeek/zeek/issues/716, as well as this thread
http://mailman.icsi.berkeley.edu/pipermail/zeek/2020-February/015062.html
1.4.0-dev.58 | 2020-03-06 11:53:58 -0800
* Send statuses and errors via regular topics (Dominik Charousset)
With this set of changes, status and error messages no longer bypass the
regular topic-based distribution. There are two new reserved topics:
topics::statuses and topics::errors. Messages to these two topics are never
forwarded to peers.
When emitting status or error messages, they get converted to data, which
is a vector. The new conversions between status <-> data and error <->
data actually accounts for the most code changes. However, this means
scripts can now easily access these messages as well. Subscribing to the
reserved topics grants easy access to statuses and errors now, allowing to
multiplex them with regular events.
The status_subscriber was rewritten to dispatch to a regular subscriber
internally.
* Implement conversion between error and data (Dominik Charousset)
* Implement can_convert_to for endpoint_info (Dominik Charousset)
* Add declarative inspection of data via contains<> (Dominik Charousset)
* Make ec convertible to and from string (Dominik Charousset)
* Add status_view type for operating on raw data (Dominik Charousset)
* Implement conversion between data and status (Dominik Charousset)
* Add from-string conversion to sc enum (Dominik Charousset)
1.4.0-dev.38 | 2020-03-06 10:39:42 -0800
* Fix cross compilation build (Dominik Charousset)
1.4.0-dev.35 | 2020-03-02 11:40:53 -0800
* Add missing unistd.h include (Jon Siwek, Corelight)
1.4.0-dev.34 | 2020-03-02 10:56:59 -0800
* Include CAF directly as submodule (Dominik Charousset)
1.4.0-dev.32 | 2020-03-02 10:30:03 -0800
* Check in CMake whether the <filesystem> code links (Dominik Charousset)
* Add instructions for MSVC builds (Dominik Charousset)
* Improve error logging in the SQLite backend (Dominik Charousset)
* Port the flare to Windows/MSVC (Dominik Charousset)
The `pipe` system call only exists on POSIX. On Windows, we need to
create a pair of connected sockets. Also, file descriptors and sockets
are different things on Windows.
Fortunately, CAF already offers a set of abstractions we can use to
adjust for the platform differences. We still have to make sure to use
recv/send on Windows instead of read/write as well as calling WSAStartup
and WSACleanup.
* Port recording utilities to Windows (Dominik Charousset)
* Port filesystem wrapper to Windows/MSVC (Dominik Charousset)
* Reimplement address with caf::ip_address for MSVC (Dominik Charousset)
The implementation of `broker::address` used many UNIX-specific
functions as well as functions marked as deprecated/unsafe on MSVC (e.g.
`sprintf`). Basing the implementation on `caf::ip_address` instead of
rolling a custom implementation avoids dealing with most issues in the
first place.
* Add MSVC to the CMake build harness (Dominik Charousset)
1.4.0-dev.18 | 2020-02-28 13:22:09 -0800
* Fix handling of OS-specific threading flags (Dominik Charousset)
1.4.0-dev.15 | 2020-02-25 17:56:28 -0800
* Support building Broker as CMake subdirectory (Dominik Charousset)
When building Broker as a subdirectory, we disable all compiler checks
and flag management and leave it to the parent project to provide valid
settings. In order to achieve "perfect forwarding" of all settings, we
include CAF as subdirectory rather than bundling it as an external
project.
1.4.0-dev.4 | 2020-02-22 11:54:14 -0800
* Fix various "maybe uninitialized" warnings (Jon Siwek, Corelight)
* Switch to using [[deprecated]] attribute (Jon Siwek, Corelight)
1.4.0-dev | 2020-02-08 11:47:27 -0800
* Update NEWS/VERSION for v1.4.0-dev (Jon Siwek, Corelight)
1.3.0 | 2020-02-08 11:02:01 -0800
* Release 1.3.0.
1.2.0-161 | 2020-02-08 11:01:43 -0800
* Update RTD version labeling for LTS (Jon Siwek, Corelight)
1.2.0-160 | 2020-02-08 10:41:13 -0800
* Fix RTD version number/label template (Jon Siwek, Corelight)
1.2.0-159 | 2020-02-08 10:38:58 -0800
* Fix RTD version number/label template (Jon Siwek, Corelight)
1.2.0-158 | 2020-02-08 10:34:43 -0800
* Display version number for more RTD version labels (Jon Siwek, Corelight)
1.2.0-157 | 2020-02-08 09:39:45 -0800
* Update embedded CAF to 0.17.4 (Jon Siwek, Corelight)
1.2.0-156 | 2020-02-07 21:05:22 -0800
* Update NEWS (Jon Siwek, Corelight)
1.2.0-155 | 2020-02-05 21:46:17 -0800
* Update CAF to current 0.17.4 release branch (Jon Siwek, Corelight)
1.2.0-154 | 2020-02-04 11:33:03 +0000
* GH-85: Improve Python bindings Event validity checks. (Jon Siwek,
Corelight)
This adds a valid() method to broker.zeek.Event objects that may be
called after creating them to check if the data it parsed was valid
event data.
Also extended the name() and args() methods of broker.zeek.Event to
first check for validity to avoid segfaulting. If the validity check
fails, a ValueError exception is now thrown.
1.2.0-151 | 2020-01-24 12:04:26 -0800
* Fix accidential error/log output (Dominik Charousset, Corelight)
* Extend log output format on the console (Dominik Charousset, Corelight)
1.2.0-147 | 2020-01-23 13:33:04 +0000
* Redesign Broker's configuration setup. (Dominik Charousset, Corelight)
The environment variables BROKER_DEBUG_VERBOSE and
BROKER_DEBUG_LEVEL overlapped and provided fuzzy semantics for a
user. This commit replaces the previous variables with
BROKER_CONSOLE_VERBOSITY and BROKER_FILE_VERBOSITY:
- set `BROKER_CONSOLE_VERBOSITY=info` for seeing high-level events such
as new connections, incoming peerings, etc.
- set `BROKER_CONSOLE_VERBOSITY=debug` for tapping into exchanged
messages and other low-level events (slow)
When debugging a distributed Broker setup, users can instead use
BROKER_FILE_VERBOSITY to record a test run and then collect the files
from all endpoints.
In all cases, Broker now omits CAF output. Advanced features are only
accessible from the (new) `broker.conf`. Here, experienced users can
remove the component files, tweak format strings, and so on.
The initialization process now also fixes ordering of the various
configuration parameters:
- `broker.conf` overrides hard-coded defaults
- environment variables override `broker.conf` entires
- command line arguments override environment variables
Finally, Broker now throws exceptions on invalid configuration entries
rather than silently dropping them. Also, the `endpoint` checks whether
the user asked for a help text and stops immediately instead of carrying
on normally.
1.2.0-141 | 2020-01-07 20:45:15 -0800
* Preserve optimization setting for sanitizer build (Jon Siwek, Corelight)
1.2.0-140 | 2020-01-03 10:45:38 -0700
* Add LeakSanitizer instrumentation macros
And disable checking LSAN leaks of sqlite3_open which seems to be a
false positive. (Jon Siwek, Corelight)
* Improve convenience options for enabling sanitizers
* Removes --enable-asan configure flag
* Removes ENABLE_ADDRESS_SANITIZER CMake variable
* Adds --sanitizers configure flag
* Adds BROKER_SANITIZERS CMake variable
* Sets C++ compiler flags according to the new options and also passes
them to the embedded CAF project
These new options would typically be used with the Debug build type and
also automatically sets -O1 unless NO_OPTIMIZATIONS environment variable
exists.
Related to https://github.com/zeek/broker/pull/77 (Jon Siwek, Corelight)
1.2.0-137 | 2019-12-02 16:29:47 -0800
* Add infinite-send-rate feature to broker-benchmark (Dominik Charousset, Corelight)
1.2.0-135 | 2019-11-24 10:13:37 -0800
* Use binary serializers, prepare for CAF changes (Dominik Charousset, Corelight)
- The stream serializers perform poorly and probably disappear in the
next CAF release. Consequently, always using the binary serializers
improves performance and also makes sure Broker is "future proof".
- Legacy support for serialize() functions probably fades away and
mixing `inspect` and `serialize` functions in Broker only has historic
reasons anyway. Sicking to one abstraction (`inspect`) is overdue.
- The binary_serializer might use any buffer type in future releases.
Instead of hardcoding `vector<char>`, we now ask the serializer for
its container type.
* Avoid using soon-to-be-removed message::take (Dominik Charousset, Corelight)
1.2.0-131 | 2019-11-15 17:49:37 -0800
* Add workaround to enable parallel Unix make for external project_caf (Simon Hardy-Francis, Corelight)
* Fix build for multi-config CMake generators (e.g. Xcode) (Jon Siwek, Corelight)
1.2.0-126 | 2019-11-11 13:47:17 -0800
* Make broker-cluster-benchmark log output configurable (Dominik Charousset, Corelight)
Also makes "quiet" log level the default.
1.2.0-123 | 2019-11-11 13:36:46 -0800
* Update embedded CAF to 0.17.3 (Jon Siwek, Corelight)
1.2.0-120 | 2019-10-30 13:22:30 -0700
* Set CMAKE_INSTALL_LIBDIR in CAF if set in broker/zeek (Johanna Amann, Corelight)
The INSTALL_LIB_DIR CMake variable was also removed in favor of
CMAKE_INSTALL_LIBDIR.
1.2.0-117 | 2019-10-28 18:06:57 -0700
* Update README to reflect CMake 3.0+ requirement (Jon Siwek, Corelight)
1.2.0-115 | 2019-10-28 16:53:41 -0700
* Require CMake 3 (Dominik Charousset, Corelight)
* Fix position of CMake project() (Dominik Charousset, Corelight)
The call to `project` must come after `cmake_minimum_required` in CMake
in order to get the correct policy settings.
* Fix conditions where broker-cluster-benchmark may hang (Dominik Charousset, Corelight)
1.2.0-108 | 2019-10-28 16:08:14 -0700
* Use Ubuntu 18.04 (Bionic) for Travis CI (Jon Siwek, Corelight)
Mostly as a simple way to get a C++17-capable compiler in the default
build environment.
* Require C++17 (Jon Siwek, Corelight)
1.2.0-102 | 2019-10-23 09:16:09 -0700
* Add -DNO_AUTO_LIBCPP when building CAF. (Robin Sommer, Corelight)
When setting the C++ compiler to clang, CAF by default switches to
compiling with -stdlib=libc++. As Broker doesn't do that, the two end
up linking against different libaries. Setting NO_AUTO_LIBCPP lets CAF
use the same library settings as Broker.
1.2.0-100 | 2019-10-21 17:35:16 -0700
* Add broker-cluster-benchmark tool (Dominik Charousset, Corelight)
Allows benchmarking arbitrary Broker networks based on previously
recorded data. e.g. can record all Broker messages in a Zeek cluster
and then replay them later in a simulated network for benchmarking
purposes. See tests/benchmark/README.md for details.
* Add new is_file utility function (Dominik Charousset, Corelight)
* Remove obsolete benchmark (Dominik Charousset, Corelight)
The old broker-stream-benchmark is very low level and no longer provides
any value with broker-cluster-benchmark in the mix.
* Give recognizable names to custom actor types (Dominik Charousset, Corelight)
* Streamline announcement of types (Dominik Charousset, Corelight)
1.2.0-60 | 2019-10-21 09:33:35 -0700
* Update embedded CAF to 0.17.2 (Jon Siwek, Corelight)
1.2.0-59 | 2019-10-21 11:41:35 +0200
* Fix check for cmake commands on POSIX shells (Dominik Charousset, Corelight)
* Prefer cmake3 command, add --cmake=PATH option (Dominik Charousset, Corelight)
1.2.0-56 | 2019-10-16 10:01:19 -0700
* Update to pybind11 v2.4.3 (Jon Siwek, Corelight)
1.2.0-54 | 2019-10-02 10:44:57 -0700
* Extend visibility of imported CAF library targets (Dominik Charousset, Corelight)
1.2.0-49 | 2019-09-30 10:09:24 -0700
* Let CAF parse the CLI of broker-benchmark instead of using getopt (Dominik Charousset)
* Remove YCM file (Dominik Charousset)