/
NEWS
5604 lines (4253 loc) · 242 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
This document summarizes the most important changes in the current Zeek
release. For an exhaustive list of changes, see the ``CHANGES`` file
(note that submodules, such as Broker, come with their own ``CHANGES``.)
Zeek 6.0.3
==========
This release fixes the following security issues:
- A specially-crafted series of packets containing nested MIME entities can
cause Zeek to spend large amounts of time parsing the entities. Due to the
possibility of receiving these packets from remote hosts, this is a DoS
risk.The fix included adds a new option (MIME::max_depth) to the MIME parser
that limits the depth the parser will attempt to follow the entity nesting. If
the limit is reached an exceeded_mime_max_depth weird is generated.
This release fixes the following bugs:
- CMake correctly passes along third-party package information when building
plugins. This ensures that, for example, the same paths to OpenSSL used in a
Zeek build are provided to a plugin build.
- Fix a problem with the HTTP analyzer where a signature regex ending in '$'
used to match against 'http-request-body' or 'http-reply-bdoy' will never
succeed. Thank you to GitHub user xb-anssi for this fix.
- The DNS analyzer now understands the Ed25519 and Ed448 signature algorithms.
- The SMB::State$recent_files field was not correctly expiring entries, leading
to unbounded state growth. This is fixed to correctly follow the &read_expire
condition on the field. Thank you to Slack user ya-sato for reporting this.
- The &create_expire attribute is now kept valid after clearing a table. After
switching the known scripts away from broker stores, the &create_expire value
of the local tables/sets of the known scripts wasn't in effect due to
Cluster::node_up() and Cluster::node_down() re-assigning these without keeping
the &create_expire attribute intact. This broke the "log hosts every 24h"
behavior.
- Zeek builds using the --binary-package argument and including Spicy will now
include all necessary Spicy symbols.
Zeek 6.0.2
==========
This release fixes the following security issues:
- A specially-crafted SSL packet could cause Zeek to leak memory and potentially
crash. Due to the possibility of receiving these packets from remote hosts,
this is a DoS risk. The fix included adds additional memory cleanup to the
x509 file analyzer.
- A specially-crafted series of FTP packets could cause Zeek to log entries for
requests that have already been completed, using resources unnecessarily and
potentially causing Zeek to lose other traffic. Due to the possibility of
receiving these packets from remote hosts, this is a DoS risk. The fix
included changes the way that we track the pending FTP commands, avoiding
possibly reusing the same value for subsequent commands.
- A specially-crafted series of SSL packets could cause Zeek to output a very
large number of unnecessary alerts for the same record. Due to the possibility
of receiving these packets from remote hosts, this is a DoS risk. The fix
included adds a new option SSL::max_alerts_per_record that caps the number of
alerts that can be generated for an SSL record. For TLS 1.3 this is capped at
1 as defined in RFC 8446. For non-TLS 1.3 it is a configurable value. A
SSL_excessive_alerts_in_record weird will be raised if the cap is exceeded.
- A specially-crafted series of SSL packets could cause Zeek to generate very
long ssl_history fields in the ssl.log, potentially using a large amount of
memory due to unbounded state growth. Due to the possibility of receiving
these packets from remote hosts, this is a DoS risk. The fix included adds a
new option SSL::max_ssl_history_length that caps this to 100 characters by
default. A SSL_max_ssl_history_length_reached weird will be raised if the cap
is exceeded.
- A specially-crafted IEEE802.11 packet could cause Zeek to overflow memory and
potentially crash. Due to the possibility of receiving these packets from
remote hosts, this is a DoS risk. The fix included adds additional bounds
checking to the IEEE802.11 packet analyzer.
This release fixes the following bugs:
- Fixed Spicy type names from causing collisions with existing Zeek types.
- On some systems with low values for the maximum number of file descriptors, it
was possible to run into crashes when doing DNS lookups if all of the file
descriptors were used. This is now avoided with better checking for the number
of available file descriptors before trying a lookup. Thank you to Zeek Slack
user h-mikami for reporting this issue.
- Tables backed by a Broker backend now correctly support deletion if they have
complex index types. Zeek previously reported an error when trying to delete
elements from these tables.
- A significant performance issue with Zeek's supervisor code was fixed,
revolving around the re-initialization of the Event Manager object used to
track events. Thank you to Jan Grashoefer for reporting this issue.
- The MaxMind DB code now cleans up after itself, resolving a memory leak with
the loaded database files.
- The ZeekJS submodule was updated to version 0.9.6, bringing fixes for
zeek.invoke and zeek.event crashes, garbage collection, and an issue where
Zeek may stop executing events from ZeekJS.
Zeek 6.0.1
==========
This release fixes the following security issues:
- File extraction limits were not correctly enforced for files containing large
amounts of missing bytes. Crafting files with large amounts of missing bytes
in them could cause Zeek to spend a long time processing data, allocate a lot
of main memory, and write a lot of data to disk. Due to the possibility of
receiving these packets from remote hosts, this is a DoS risk. The fix
included makes Zeek correctly enforce file size limits. We also added a new
option (default_limit_includes_missing) which allows to customize the behavior
when encountering large amounts of missed bytes in file. This issue was
discovered by Luca Cigarini.
- Sessions are sometimes not cleaned up completely within Zeek during shutdown,
potentially causing a crash when using the -B dpd flag for debug logging. This
is low priority because it only happens at shutdown and only when using that
flag. The fix included is to reorder shutdown to cleanup all the sessions
prior to tearing down the analyzers.
- A specially-crafted HTTP packet can cause Zeek’s filename extraction code to
take a long time to process the data. Due to the possibility of receiving
these packets from remote hosts, this is a DoS risk. The fix included adjusts
the regular expression used in the extract_filename_from_content_disposition()
script function to more-correctly match the data.
- A specially-crafted series of FTP packets made up of a CWD request followed by
a large amount of ERPT requests may cause Zeek to spend a long time logging
the commands. Due to the possibility of receiving these packets from remote
hosts, this is a DoS risk. The fix included is to prevent logging of pending
commands for FTP packets.
- A specially-crafted VLAN packet can cause Zeek to overflow memory and
potentially crash. Due to the possibility of receiving these packets from
remote hosts, this is a DoS risk. The fix included is to add some additional
length checking to the VLAN analyzer.
This release fixes the following bugs:
- Fixed a base64 decoding issue with the authorization field of HTTP request
headers that was sometimes causing Zeek to output error messages. Thank you
to GitHub user @progmboy for reporting and providing a fix for this issue.
- Ensure that Zeek builds use the internal version of Spicy instead of external
installations, unless specifically configured for that mode.
- Support was added for ``switch`` fields when exporting Spicy types to Zeek.
- A number of fixes were added to protect against potential unbounded state
growth with the SMB and DCE-RPC analyzers. SMB close requests will properly
tear down an related DCE-RPC analyzers. A new ``SMB::max_dce_rpc_analyzers``
script variable was added that allows finer control of how many DCE-RPC
analyzers are allowed to be created per SMB connection. Thanks to Zeek Slack
user Takomi Sugawara for reporting this issue.
- Fixed a regression in the UDP and TCP analyzers that was causing more data
than necessary to be forwarded to the next analyzer in the chain. Thanks to
Zeek Slack user Hiep Long Tan for reporting this issue.
- A connection's value is now updated in-place when its directionality is
flipped due to Zeek's heuristics (for example, SYN/SYN-ACK reversal or
protocol specific approaches). Previously, a connection's value was discarded
when flipped, including any values set in a ``new_connection()`` handler. A
new ``connection_flipped()`` event is added to allow updating custom state in
script-land.
- Fixed undefined symbols being reported from Spicy when building some of the
binary packages for Zeek.
- Loading ``policy/frameworks/notice/community-id.zeek`` now also automatically
community ID logging. In the past, loading the script had no effect unless
``policy/protocols/conn/community-id-logging.zeek`` was loaded before. This
was fairly unusual and hard to debug behavior.
- Spicy no longer registers an extra port for every port registered in a
plugin's .evt file.
- Timeouts in DNS resolution no longer cause uncontrolled memory growth.
- Fix check to skip DNS hostname lookups for notices that are not delivered via
email in ``policy/frameworks/notice/extend-email/hostnames``. Due to that
policy script being loaded in the Zeek's default ``local.zeek``, this
previously caused unneeded DNS lookups for every generated notice instead of
just those delivered via email.
Zeek 6.0.0
==========
Breaking Changes
----------------
- Zeek now treats private address space (i.e., non-routable IP address ranges)
as local by default, matching the intuition of many users that e.g. a
192.168/16 IP address should show up as local in the logs. To do this, Zeek
automatically adds ``Site::private_address_space`` to ``Site::local_nets`` at
startup. Subsequent runtime updates to ``Site::private_address_space``
propagate to ``Site::local_nets``, while updates to the latter don't affect
the former.
You're free to define ``Site::local_nets`` as before and do not need to update
your configurations. If you added standard private address space to
``Site::local_nets`` in the past, you no longer need to do so. This also
applies to zeekctl's ``networks.cfg`` file.
The new global Boolean ``Site::private_address_space_is_local``, true by
default, controls the behavior. A redef to false brings back Zeek's prior
behavior of considering private address space an unrelated concept, which will
come in handy for example when working with tests that compare results against
log baselines that have not yet been updated.
- Telemetry centralization and Prometheus exposition is not enabled by default
anymore. Previously, the manager node would open port 9911/tcp by default and
import all metrics from other nodes. For large clusters, the current implementation
introduces significant processing overhead on the manager even if the Prometheus
functionality is not used. While inconvenient, disable this functionality
(assumed to be used by few as of now) by default to preserve resources.
The script to enable centralization and the Prometheus endpoint is now
located in the ``policy/`` folder. Re-enable the old functionality with:
@load frameworks/telemetry/prometheus
You may experiment with increasing ``Broker::metrics_export_interval``
(default 1s) to reduce the extra overhead and communication at the expense
of stale metrics.
- Custom source tarballs require a ``repo-info.json`` file.
Note, should you be using official Zeek release tarballs only, or build
Zeek solely from git checkouts, this does not affect you.
However, if you're building your own Zeek source tarballs, it is now required
that a ``repo-info.json`` file exists at the top-level. The ``dist`` target was
extended to add this file and official Zeek release source tarballs will
contain it going forward.
The following command can be used to produce ``repo-info.json``:
python3 ./ci/collect-repo-info.py --only-git > ../path/to/tarballdir/repo-info.json
This is required to support the new ``-V`` / ``--build-info`` option that
provides information about git submodules and included plugins used during
the build. The ``ci/collect-repo-info.py`` tool runs at ``./configure`` time
and either collects the required information from a git clone (when git is
installed), or otherwise uses the content of a file named ``repo-info.json``.
If you see opportunities to extend ``repo-info.json`` with further information,
please get in touch.
- Plugin authors should raise the minimum required CMake version to 3.15 to
ensure compatibility with new CMake scaffolding included in this
release. Older versions will trigger a warning at configuration time and,
depending on the functionality included in the plugin, may trigger subsequent
errors during configuration or build.
- Zeek container images are not pushed to the zeekurity organization anymore.
Please switch to using the ``zeek/zeek`` image on DockerHub, or the images
published to ``public.ecr.aws/zeek/zeek``.
- The IRC_Data analyzer declaration has been moved to protocols/irc/IRC.h.
- The error message returned when using ``bro_init``, ``bro_done``, and
``bro_script_loaded`` events is now removed. removed. Usage of these events
has returned that error during script parsing for a few years, and time has
come to finally remove it.
New Functionality
-----------------
- Zeek now features experimental JavaScript support:
/* hello.js */
zeek.on('zeek_init', () => {
console.log('Hello, Zeek!');
});
$ zeek ./hello.js
Hello, Zeek!
When building Zeek on a system that features a recent (16.13+) version of the
libnode package with development headers, Zeek automatically includes the
externally-maintained ZeekJS plugin (https://github.com/corelight/zeekjs) as a
builtin plugin. This allows Zeek to load and execute JavaScript code located
in ``.js`` or ``.cjs`` files. When no such files are passed to Zeek, the
JavaScript engine and Node.js environment aren't initialized and there is no
runtime impact.
The Linux distributions Fedora 37 & 38, Ubuntu 22.10, and the upcoming Debian
12 release provide suitable packages. On other platforms, Node.js can be built
from source with the ``--shared`` option.
To disable this functionality, pass ``--disable-javascript`` to configure.
- Zeek now comes with Spicy support built in, meaning it can now
leverage any analyzers written in Spicy out of the box. While the
interface layer connecting Zeek and Spicy used to be implemented
through an external Zeek plugin, that code has now moved into the
Zeek code base itself. We also added infrastructure to Zeek that
enables its built-in standard analyzers to use Spicy instead of
Binpac. As initial (simple) examples, Zeek's Syslog and Finger
analyzers are now implemented in Spicy. While their legacy versions
remain available as fallbacks for now in case Spicy gets explicitly
disabled at build time, their use is deprecated and their code won't
be maintained any further. (Some of these Spicy updates were part of
Zeek 5.2 already, but hadn't been included in its NEWS section.)
- Zeek events now hold network timestamps. For scheduled events, the timestamp
represents the network time for which the event was scheduled for, otherwise
it is the network time at event creation. A new bif ``current_event_time()``
allows to retrieve the current event's network timestamp within the script-layer.
When Zeek sends events via Broker to other nodes in a cluster, an event's network
timestamp is attached to the Broker messages. On a receiving Zeek node executing a
handler for a remote event, ``current_event_time()`` returns the network time of
the sending node at the time the event was created.
The Broker level implementation allows to exchange arbitrary event metadata, but
Zeek's script and C++ APIs currently only expose network timestamp functionality.
- A new bif ``from_json()`` can be used to parse JSON strings into records.
type A: record { a: addr; };
local p = from_json({\"a\": \"192.168.0.1\"}", A);
if ( p$valid )
print (p$v as A)
Implicit conversion from JSON to Zeek types is implemented for bool, int, count,
real, interval (number as seconds) and time (number as unix timestamp), port
(strings in "80/tcp" notation), patterns, addr, subnet, enum, sets, vectors
and records similar to the rules of the input framework. Optional or default
record fields are allowed to be missing or null in the input.
- Zeek now provides native "Community ID" support with a new bif called
``community_id_v1()``. Two policy scripts ``protocols/conn/community-id-logging``
and ``frameworks/notice/community-id`` extend the respective logs with a
``community_id`` field the same way as the external zeek-community-id plugin
provides. A main difference to the external ``hash_conn()`` bif is that the
``community_id_v1()`` takes a ``conn_id`` record instead of a ``connection``.
Loading the new policy scripts and using the external zeek-community-id
plugin at the same time is unsupported.
- ZeekControl is now multi-logger aware. When multiple logger nodes are configured
in ZeekControl's node.cfg, by default the log archival logic adds a logger's name
as suffix to the rotated file name:
stats.11:18:57-11:19:00-logger-1.log.gz
stats.11:18:57-11:19:00-logger-2.log.gz
Previously, in a multi-logger setup, individual logger processes would overwrite
each other's log files during rotation, causing data loss.
For setups with a single logger, there's no change in behavior. The naming
of the final logs can be customized by providing an alternative
``make-archive-name`` script and using the new ``ZEEK_ARG_LOG_SUFFIX``
environment variable.
- A supervisor controlled Zeek cluster is now multi-logger aware. This avoids
loggers overwriting each other's log files within a single log-queue directory.
By default, a logger's name is appended to the rotated logs by zeek-archiver.
- Introduce a new command-line option ``-V`` / ``--build-info``. It produces
verbose output in JSON format about the repository state and any included
plugins.
- The X.509 certificate parser now exposes the signature type that is given inside
the signed portion of the certificate.
- The SSL parser now parses the CertificateRequest handshake message. There is a new
``ssl_certificate_request`` event and a new ``parse_distinguished_name`` function.
We also added the ``protocols/ssl/certificate-request-info`` policy script, that
adds some additional information to ``ssl.log``.
- Add logging metrics for streams (``zeek-log-stream-writes``) and writers
(``zeek-log-writer-writes-total``).
- Add networking metrics via the telemetry framework. These are enabled
when the ``misc/stats`` script is loaded.
zeek-net-dropped-packets
zeek-net-link-packets
zeek-net-received-bytes
zeek-net-packet-lag-seconds
zeek-net-received-packets-total
Except for lag, metrics originate from the ``get_net_stats()`` bif and are
updated through the ``Telemetry::sync()`` hook every 15 seconds by default.
- The DNS analyzer now parses RFC 2535's AD ("authentic data") and CD ("checking
disabled") flags from DNS requests and responses, making them available in
the ``dns_msg`` record provided by many of the ``dns_*`` events. The existing
``Z`` field remains unchanged and continues to subsume the two flags, for
backward compatibility.
- The supervisor framework can now start worker nodes that read from a trace file.
- Zeek can be prevented from updating ``network_time()`` to the current time
by setting ``allow_network_time_forward=F``. Together with ``set_network_time()``
or a custom plugin, this allows control of ``network_time()`` without Zeek
interfering.
- The setting ``Pcap::non_fd_timeout`` can be used to configure the timeout
used by non-selectable packet sources in the idle case (default 20usec).
This value has previously been hard-coded, but increasing it can significantly
reduce idle CPU usage in low packet rate deployments.
- Zeek now supports a new ``@pragma`` directive. It currently allows suppressing
deprecation warnings in Zeek scripts by opening with
``@pragma push ignore-deprecations`` and closing with ``@pragma pop``.
This particularly helps in situations where use of the Zeek base scripts, for
example to populate a deprecated field for API compatibility, would otherwise
trigger deprecation warnings.
- The ``Reporter`` class was extended by a ``Deprecation()`` method to use
for logging deprecations rather than using ad-hoc ``Warning()`` calls.
- The network statistics record type features a new ``pkts_filtered`` field for
reporting the number of packets that the interface filtered before hand-off to
Zeek. Packet source implementations are free to fill this field as
feasible. The default pcap packet source does not provide this information
because its availability depends on the libpcap version.
- Packet statistics (packets received, packets dropped, bytes received, packets
seen on link, and packets filtered) are now reported to the Telemetry
framework, under the ``zeek_net`` prefix.
- Zeek's cluster framework provides the new ``get_node_count(node_type: NodeType)``
function to obtain the number of nodes for a given node type as defined in the
cluster layout. Furthermore, ``broadcast_topics`` was added as a collection of
broker topics that can be used to reach all nodes in a cluster.
- The new ``Cluster::Experimental`` namespace has been introduced to Zeek's cluster
framework to provide experimental features. Based on practical experiences and the
adoption of an experimental feature, it may become a regular feature or be removed
in future releases. Experimental features are loaded via:
``@load policy/frameworks/cluster/experimental``
- Zeek's cluster framework provides two new experimental events:
- ``cluster_started``: This event will be broadcasted from the manager once all
cluster-level connections have been established based on the given cluster layout.
If any node restarts (including the manager itself), the event will neither be
rebroadcasted nor raised locally for the restarted node.
- ``node_fully_connected``: This event will be sent to the manager and raised
locally once a cluster node has successfully conducted cluster-level handshakes
for all its outgoing connections to other cluster nodes based on the given cluster
layout.
Note: There is no tracking of cluster node connectivity. Thus, there is no guarantee
that all peerings still exist at the time of these events being raised.
- The IEEE 802.11 packet analyzer gains the ability to parse encapsulated A-MSDU
packets, instead of just dropping them. It also gains the ability to properly
recognize CCMP-encrypted packets. These encrypted packets are currently
dropped to Zeek's inability to do anything with them.
- Add packet analzyers for LLC, SNAP, and Novell 802.3, called from the Ethernet
and VLAN analyzers by default.
- Environment variables for the execution of log rotation postprocessors can
be set via ``Log::default_rotation_postprocessor_cmd_env``.
- The ``record_field`` record was extended by ``optional`` and ``record_fields()``
can now be used to determine the optionality of record fields.
- The ``ip4_hdr`` record was extended by ``DF``, ``MF``, ``offset`` and ``sum``
to aid packet-level analysis use-cases.
- Zeek now supports parsing the recently standardized DTLS 1.3. Besides the protocol
messages being correctly parsed and raising the typical SSL/TLS events, the biggest
visible change is the newly added ``ssl_extension_connection_id`` event.
- The NTP analyzer now recognizes when client and server mode messages disagree
with the notion of "originator" and "responder" and flips the connection. This
can happen in packet loss or packet re-ordering scenarios. Such connections will
have a ``^`` added to their history.
- New bifs for ``ceil()`` and ``log2()`` have been added.
- Seeds for deterministic processing can now also be set through a new environment
variable called ``ZEEK_SEED_VALUES``. The format is expected to contain 21
positive numbers separated by spaces.
Changed Functionality
---------------------
- The base distribution of the Zeek container images has been upgraded to
Debian 12 "bookworm" and JavaScript support was enabled.
- When ``get_file_handle()`` is invoked for an analyzer that did not register
an appropriate callback function, log a warning and return a generic handle
value based on the analyzer and connection information.
- The ``&on_change`` attribute of set and tables is propagated through ``copy()``.
- Revert back to old method of preallocating ``PortVal`` objects for all valid
port numbers, as it was implemented prior to the Windows port. Not
preallocating these objects saves a minor amount of memory for short runs of
Zeek, but comes at a performance cost for having to allocate the objects every
time a new port is seen plus do map lookups for each port. This memory savings
is mostly lost for long runs of Zeek, since all of the ports will likely end
up allocated in time.
If the version from the Windows port is desired, a new configure option
``--disable-port-prealloc`` will disable the preallocation and enable the map
lookup version.
- The main-loop has been changed to process all ready IO sources with a
zero timeout in the same loop iteration. Previously, two zero-timeout
sources would require two main-loop iterations. Further, when the main-loop
is polling IO sources with file descriptors, zero timeout IO sources are
added to the list of sources to be processed as well.
The intervals to decide when Zeek checks FD-based IO sources for readiness
have been made configurable through ``io_poll_interval_default`` and
``io_poll_interval_live`` for ease of testing, development and debugging
of the main-loop.
- Zeek does not arbitrarily update ``network_time()`` to current time anymore.
When a packet source is providing a constant stream of packets, packets
drive network time. Previously, Zeek updated network time to current
time in various situations, disregarding timestamps of network packets.
Zeek will now update ``network_time()`` only when a packet source has been
inactive/idle for an interval of ``packet_source_inactivity_timeout``
(default 100msec). When a worker process suddenly observes no packets, timer
expiration may initially be delayed by ``packet_source_inactivity_timeout``.
- Calling ``suspend_processing()`` when reading traces does not update network
time to the current time anymore. Instead, Zeek keeps ``network_time()``
according to the trace file. This causes scheduled events to not fire once
``suspend_processing()`` is called, which seems more reasonable than
arbitrarily setting ``network_time()`` to current time. Processing can still
be continued from broker events or input readers.
- Previously, Zeek would process and dispatch events for the very first packet
in a trace file in order to initialize time, even if ``suspend_processing()``
was called in a ``zeek_init()`` handler. This has been changed such that the
first packet will only be processed once ``continue_processing()`` has been
invoked again. Some background around the previous behavior can be found
in GH-938. Given that the ``network_time_init()`` event explicitly signals
initialization of network time, this behavior seems more reasonable.
- If an event is scheduled with a 0.0sec timeout from a ``zeek_init()`` handler
that also invokes ``suspend_processing()``, the scheduled event will fire
immediately with ``network_time()`` still yielding ``0.0``. Previously,
``network_time()`` was set to the current time. The new behavior provides
more deterministic operation and aligns with timers stopping during a
``suspend_processing()``.
- Broker no longer initializes network time to current time when processing
input. Particularly in combination with pcap processing this was not desirable
behavior.
- The IO loop's poll interval is now correctly reduced from 100 to 10 for
live packet sources. This should lower CPU usage for deployments with
non-selectable packet sources.
- Zeek's CMake scaffolding has received an overhaul for modernizing the build
system and to make it easier to maintain going forward. Plugins can now use a
declarative interface for adding all sources, BIFs, etc. in one block instead
of using the previous begin/end functions. While the old plugin functions
still exist for backward compatibility, the underlying codebase requires newer
CMake features. Plugin authors should raise their minimum required CMake
version to 3.15, to match Zeek's.
- The IRC data analyzer does not extract DCC acknowledgements to files anymore.
Instead, ``irc_dcc_send_ack`` is raised with the bytes acknowledged by the
recipient.
- The IRC base script now use ``file_sniff()`` instead of ``file_new()`` for
DCC file transfers to capture ``fuid`` and inferred MIME type in irc.log.
- The ``ignore_checksums`` script variable now reflects the correct value
when using the ``-C`` command-line flag.
- Support for ARUBA GRE tunnels now covers all of the known protocol type values
for those tunnels.
- The vlan field reported by the AF_PACKET packet source is now properly
masked to exclude PCP and DEI bits. Previously, these bits were included
and could cause invalid vlan values > 4095 to be reported.
- Libpcap based packet source now avoids the 32bit wraparound of link and
dropped packet counters as reported by users.
- The `ssl_history` field in ssl.log indicates that the letter `j` is reserved
for hello retry requests. However, this logging was never fully implemented;
instead, hello retry requests were logged like as a server hello (with the letter
`s`). This oversight was fixed, and hello retry requests are now correctly logged.
- When per-connection SMB parser state (read offsets, tree ids, ...) exceeds
``SMB::max_pending_messages`` (default 1000), Zeek discards such per-connection
state and raises a new ``smb2_discarded_messages_state()`` event. This event is
used to reset script-layer SMB state. This change provides protection against
unbounded state growth due to partial or one-sided SMB connections.
Setting ``SMB::max_pending_messages`` to 0 can be used to switch back to the
previous behavior of not discarding state. Setting ``SMB::enable_state_clear``
to ``F`` skips the script-layer state clearing logic.
- Fix ``disable_analyzer()`` builtin function crashing when attempting to
disable connection's root analyzers.
- Zeek script vectors now support negative indices.
local v = vector(1, 2, 3);
print v[-1]; # prints 3
- Function parameters are rendered by Zeekygen as ``:param x`` rather than just
``:x:``. This allows to group parameters Zeek's documentation.
Removed Functionality
---------------------
- Mixing vector and scalar operands for binary expressions, like addition,
multiplication, etc., is now an error.
- Using deprecated ``when`` semantics without capturing variables is now an error.
- Referencing local variables in a more outer scope than where they were declared
is now an error
Deprecated Functionality
------------------------
- The cluster framework's ``worker_count`` has been deprecated in favor of the
new function ``get_active_node_count(node_type: NodeType)`` that can be used
to obtain the number of nodes of a given type the calling node is currently
connected to.
Zeek 5.2.0
==========
Breaking Changes
----------------
- Zeekctl now assigns network ports to workers starting at port 27760. This
fixes an issue where workers were starting up with ports within Linux's
ephemeral port range, and were potentially failing to startup due the ports
already being in use. This change may require changes in firewall/routing
configurations between hosts in a Zeek cluster. This should not affect
clusters running on FreeBSD, as that OS uses a different range for ephemeral
ports.
- Zeekctl support for the AF_PACKET plugin specific options (af_packet_*) has
been integrated into zeekctl directly. Upgrading to Zeek 5.2 with a builtin
AF_PACKET packet source (default on Linux) requires an upgrade of zeekctl
to the version bundled with Zeek to continue using these options.
- The blank identifier ``_`` cannot be used in expressions and options anymore.
Outside of obfuscation exercises, this should have little real-world impact.
- A new ``mysql_eof`` event has been introduced and the ``mysql_ok`` event
is not raised in its place or artificially anymore. The base scripts were
adapted accordingly. Users of ``mysql_ok()`` likely need to switch to
``mysql_eof()``.
- Zeek will now exit at startup if an external plugin (e.g. from a package) is
discovered to have the same name as a built-in plugin. See below for the
change regarding the AF_PACKET plugin now being built-in for an example of
this potentially being triggered.
- DNS query type strings were updated to match the current standardized list of
strings. This changes the string reported for a small subset of query types:
30: Changed from "EID" to "NXT"
31: Changed from "NIMLOC" to "EID"
32: Changed from "NB" to "NIMLOC"
- The ``--with-caf`` option for the ``configure`` script was removed. Broker now
requires specific versions of CAF per Zeek release, and passing an
externally-built version of CAF often lead to build failures.
New Functionality
-----------------
- Experimental support added for building and running Zeek on Microsoft Windows
environments. This is considered experimental due to the fact that our
standard testing setup (btest) doesn't run properly on Windows. This will be
fixed in the future. In the meantime we have done some basic testing against
builds done with Visual Studio 2019. Information on how to build on Windows is
available in the Zeek documentation. Note also that Spicy is currently
unsupported and will be fixed in the future.
The feature as checked into the repository is not considered production-ready.
There are many bugs to squash and features to improve, and we will be steadily
fixing things over the next few months.
The Zeek team wants to give a huge thank you to the team at Microsoft for all
of their effort in completing this port.
- Zeek container images are now being published to zeek/zeek and zeek/zeek-dev
rather than zeekurity/zeek and zeekurity/zeek-dev on Docker Hub (and continue
to be published to public.ecr.aws) Further, container images for amd64 and
arm64 platforms are now available. Main driver for the latter was to allow
usage of the official container images on Apple's M1 systems.
- Zeekctl support for using ``af_packet`` as ``lb_method`` has been added.
- New ``analyzer_confirmation_info`` and ``analyzer_violation_info`` events with
accompanying record types ``AnalyzerConfirmationInfo`` and
``AnalyzerViolationInfo`` have been added. These supersede
``analyzer_confirmation`` and ``analyzer_violation``, which have been
deprecated.
- Added helpers to determine protocol, packet or file analyzer based on
``AllAnalyzers::Tag`` values named ``is_protocol_analyzer()``,
``is_packet_analyzer()`` and ``is_file_analyzer()``.
- File analyzers can now raise analyzer violations to the script-layer via the
new ``AnalyzerViolation()`` method.
- Packet and file analyzers can now be disabled and enabled at runtime using the
``Analyzer::enable_analyzer()`` and ``Analyzer::disable_analyzer()``
wrappers. While initially for protocol analyzers only, these have been
extended to work for packet and file analyzers. This now allows to leverage
``Analyzer::disabled_analyzers`` for these kinds of analyzers.
- The blank identifier ``_`` can now be used to ignore loop variables of
different types without type clash errors. This allows to do the following
within the same scope:
local vec = vector("a", "b", "c");
for ( _, v in vec )
print v;
for ( i, _ in vec )
print v;
Iterating over only the values of a table can be done by ignoring the full
index with a single blank identifier. Due to the internal structure of Zeek
tables, this can result in a performance improvement.
local tab = table(["a", 1, T] = "a1T", ["b", 2, F] = "b2f");
for ( _, v in tab )
print v;
It's also possible ignore individual indices of different types with the blank
identifier ``_`` as follows:
for ( [_, i, _], v in tab )
print i, v;
As noted under breaking changes, the blank identifier ``_`` cannot be
referenced in expression anymore.
- It is now possible to put trailing commas within table, vector, set and record
construction. For example, the following code is now valid, which can make for
more uniform style and smaller diffs.
local vec = vector(
"1",
"2",
);
local tab: table[string] of count = [
["a"] = 1,
["b"] = 2,
];
Function calls and record constructors can have a trailing comma after the
last argument.
Analyzer::schedule_analyzer(
chan$orig_h,
chan$resp_h,
chan$resp_p,
Analyzer::ANALYZER_FTP_DATA,
5mins,
);
- Re-introduce event groups. Allow the ``&group`` attribute on event and hook
handlers for annotating them with one or more event groups. These groups can
be disabled and enable during runtime. Disabling an event group implies
disabling all event and hook handlers that are part of it.
The main difference to a previous implementation in (very) old Zeek versions
is its granularity: It is now possible to toggle individual event handlers
(event handler bodies). The original implementation worked at the level of
events, disabling or enabling all event handlers for a given event at once.
Additionally, support for hooks was added as these are structurally similar to
events.
The BIFs ``disable_event_group()`` and ``enable_event_group()`` are
re-instantiated and allow controlling event groups based on the group
attribute.
Additionally, event and hook handlers are implicitly placed into event module
groups based on the module they are implemented in. All events implemented in
a given module can be toggled with ``disable_module_events()`` and
``enable_module_events()``.
- Extend the ``Logging::Stream`` record with an ``event_groups`` field and
toggle these during ``Log::disable_stream`` and ``Log::enable_stream``
invocations. This allows for explicit/manual opt-in performance optimizations
by turning off event handlers at runtime that are only needed for log
generation.
- On Linux, the AF_PACKET packet source plugin
(https://github.com/zeek/zeek-af_packet-plugin) is included as a builtin
plugin by default. To select this packet source, prefix the interface name
with ``af_packet``.
zeek -i af_packet::eth0
- Usage of ``break`` and ``next`` statements is now validated. It was previously
possible to place these outside of ``for``, ``while`` or ``switch`` statements
without any error indication.
- Add two BIFs ``get_identifier_declaring_script()`` and
``get_record_field_declaring_script()`` to query the declaring scripts for
identifiers and record fields from Zeek scripts.
- Extend the SSH analyzer to produce new events (``ssh2_ecc_init``,
``ssh2_gh_gex_init``, ``ssh2_gss_init``, ssh2_rsa_secret``) to detect when SSH
client and server roles are reversed.
- Analyzers found in the new ``Analyzer::requested_analyzers`` set will be
enabled at ``zeek_init()`` time. The set can be populated via
:zeek:see:`redef`. This change only has an effect in settings where
``Analyzer::disable_all`` is changed to ``T``. By default, all analyzers
continue to be enabled.
- A new ``analyzer.log`` was added to log all analyzer violations and optionally
analyzer confirmations. This log can be useful during development of new
analyzers as well as for collecting operational data in production
environments.
- Expose configurability of for SQLite's synchronous and journal_mode PRAGMAs
for SQLite backed Broker data stores. Setting these to synchronous=normal
and journal_mode=wal can significantly improve throughput at the cost of
some durability in the presence of power loss or OS crash. In the context
of Zeek, this is likely more than acceptable.
Additionally, add integrity_check and failure_mode options to support
detecting and deleting corrupted SQLite database at store initialization.
- A new ``join_string_set`` BIF was added, replacing the existing script-level
version from utils/strings.zeek.
- A new ``&ordered`` attribute for tables and sets was added. This attribute
causes iteration over a table/set to return elements in the order of their
insertion.
- A new ``-D`` argument was added to the ``configure`` script to allow passing
parameters directly to the underlying CMake call.
- Added parsing for the challenge and response fields to the NTLM analyzer.
- A new ``FTP::max_command_length`` value was added to script-land, defaulting
to 100. This value is used by the FTP analyzer to limit the size of commands
accepted by the analyzer. A ``FTP_max_command_length_exceeded`` weird is
raised for any violations of that length.
- The MySQL analyzer has been extended to detect when client and server negotiate
to use a SSL encrypted session. This allows analysis of the subsequent SSL
handshake. The service field for encrypted MySQL connections in the conn.log
will have entries for both, mysql and ssl.
Changed Functionality
---------------------
- Violations for packet analyzers that have sessions attached with them will be
raised once only. Further, analyzer confirmations are not raised after a
violation.
- The parameter given to ``enum_names()`` can now be a string naming the enum
type, rather than the type itself.
- The ``type_name`` of enum types produced by ``record_fields()`` now includes
the actual type name rather than just ``"enum"``.
- Passing non-string ``sep`` and ``def`` arguments to ``cat_sep()`` isn't a
fatal error anymore. More descriptive error messages are produced, too.
- The number of analyzer violation events that can be raised by protocol
analyzer instances is now capped by the const
``max_analyzer_violation_events``.
- The number of analyzer violation events that can be raised by protocol and
file analyzer instances is now capped by the const
``max_analyzer_violation_events``. Its default is 1000 and the main purpose
is to prevent analyzers from scheduling too many ``analyzer_violation_info``
events before the DPD ``max_violations`` script-level logic has a chance to
run and disable the problematic analyzer.
- The TCP analyzer now continues processing payload for some
connections missing initial packets where it would previously have
stopped. This fixes a few cases where we already had the logic to
continue in place, but we still ended up considering them partial.
- Count underflows via ``--c`` or subtract from statements (``c = c - 1``) are
now consistently warned about. Previously, underflows through ``--c`` were
treated as runtime errors, while "subtract from" underflows were silently
accepted. The following (surprising behavior) now causes a warning, too:
$ zeek -e 'print 1 - 2'
expression warning in <command line>, line 1: count underflow (1 - 2)
18446744073709551615
- The MQTT scripts registering the analyzer and DPD signatures have been moved
from the policy folder to base and are loaded by default.
- Notices created for files transferred over multiple connections will now be
associated with one of the connections rather than none.
- The MySQL analyzer has been switched to parse in little endian. This avoids
analyzer violations due to out of bound errors for length encoded strings.
- Non-fatal errors when setting up BPF filtering will no longer cause Zeek to
exit, but instead will log the error in reporter.log and continue processing.
- The languages reported for the ``keyboard_layout`` field in rdp.log were
updated to match the current standardized set of languages. Unknown layout
values now attempt to fallback to a "parent" layout if one is available.
- In the cluster management framework, the controller now supports Broker's
WebSocket data transport for communication with clients. It listens on TCP
port 2149 for this purpose. zeek-client now likewise uses the WebSocket
transport, removing its runtime dependency on the Broker library and enabling
standalone installation. The client still bundles with Zeek by default but is
now also available on PyPI and installable via ``pip install zeek-client``.
The documentation provides additional details.
Deprecated Functionality
------------------------
- The global ``disabling_analyzer()`` hook has been deprecated and replaced
with ``Analyzer::disabling_analyzer()`` that has the same semantics.
- The ``analyzer_confirmation`` and ``analyzer_violation`` events have been
deprecated in favor of the more generic ``analyzer_confirmation_info`` and
``analyzer_violation_info`` events.
- The const values for toggling individual tunnel packet analyzers have been
deprecated in favor of using ``Analyzer::disable_analyzer()`` directly. This
affects:
Tunnel::enable_ip
Tunnel::enable_ayiya
Tunnel::enable_teredo
Tunnel::enable_gtpv1
Tunnel::enable_gre
Setting these explicitly to F can be achieved by leveraging
``Analyzers::disabled_analyzers``, for example:
redef Analyzer::disabled_analyzers += { PacketAnalyzer::ANALYZER_GRE };
- The ``zeek::merge_type_list()`` function has been deprecated. Please consider
the partially compatible and saner ``zeek::maximal_type()`` instead. See
GH-2604 for context.
- The pre-authentication data field (pa_data) available in certain Kerberos
events now exposes the (encrypted) PA-ENC-TIMESTAMP field (padata-type=2).
- The ``SupressWeirds()`` method in the ContentLine analyzer was deprecated in
favor of the correctly-spelled ``SuppressWeirds()`` method.
- The `bro` symlink has finally been removed.
Zeek 5.1.0
==========
Breaking Changes
----------------
- The ``Packet::{l2,l3}_checksummed`` variables were reworked to correctly match
the network layers that they apply to. A new ``Packet::l4_checksummed``
variable was added to cover the transport layer. See this GitHub issue for
more detail: https://github.com/zeek/zeek/issues/2183.
- The STREAM mode of the ASCII reader now behaves like `tail -F`: when file is
removed/replaced, it will start tracking the new file. See
https://github.com/zeek/zeek/pull/2097 for more detail
- The Dictionary and PDict classes are now C++ templates. This may cause
plugin/package builds to fail due to needing to modify uses of them to match.
- By default, ``files.log`` does not have the fields ``tx_hosts``, ``rx_hosts``
and ``conn_uids`` anymore. These have been replaced with the more commonly
used ``uid`` and ``id`` fields. They can be re-instantiated by loading the
following policy script through ``local.zeek``:
@load frameworks/files/deprecated-txhosts-rxhosts-connuids
Note, however, that this script will be removed with Zeek 6.1. Consumers
of ``files.log`` should convert to using the singular ``uid`` and ``id``
fields instead.
- The ``files.log`` is now unrolled consistently. That is, when Zeek associates