Zeekify intel-limiter.

J-Gras · J-Gras · commit 259b15a052ef · 2022-02-21T18:57:02.000+01:00
diff --git a/LICENSE b/LICENSE
@@ -1,4 +1,4 @@
-Copyright (c) 2019 by Jan Grashoefer
+Copyright (c) 2019-2022 by Jan Grashoefer
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:
diff --git a/README.md b/README.md
@@ -4,11 +4,11 @@ This package provides limiting mechanisms for Zeek's intelligence framework. Thi
 
 ## Installation
 
-The scripts are available as package for the [Bro/Zeek Package Manager](https://github.com/zeek/package-manager) and can be installed using the following command: `bro-pkg install intel-limiter`
+The scripts are available as package for the [Zeek Package Manager](https://github.com/zeek/package-manager) and can be installed using the following command: `zkg install intel-limiter`
 
 ## General Usage
 
-To enable per item thresholds and heavy hitter suppression make sure the package is loaded: `bro-pkg load intel-limiter`
+To enable per item thresholds and heavy hitter suppression make sure the package is loaded: `zkg load intel-limiter`
 
 ### Matching Thresholds
 
@@ -20,22 +20,22 @@ With the package loaded, intel items might specify a new meta data field `meta.m
 ```
 Once a match is triggered, the internal match counter will be reset and the next match is triggered after the threshold is reached again. The threshold counters are managed per item. If two items that are obtained from different sources share the *same* indicator, thresholds do not interfere. For example, if thresholds of two and four are specified for a single indicator, every fourth time the indicator is seen a hit containing all meta data will be generated, while every second reporting will cause a match containing only the meta data of the item that defines the threshold of two.
 
-**Note:** As Bro 2.6 does not pass modified hook parameters along the chain, hits will be reported in the correct frequency but contain the meta data of all items.
+**Note:** Since version 2.6, Zeek does not pass modified hook parameters along the chain. Thus, hits will be reported in the correct frequency but contain the meta data of all items.
 
 ### Heavy Hitter Suppression
 
 To enable heavy hitter suppression, a heavy hitter interval and threshold have to be specified. The option `heavy_hitter_interval` defines the time window in which at most `heavy_hitter_threshold` hits might occur. If more hits are observed in that time window, the indicator is removed. By default the threshold is set to zero which disables suppression. When removing an heavy hitter, a reporter warning is generated to inform the user about the removal.
 
 ## Background
 
-The script `item-threshold.bro` implements per item thresholds and is loaded by default. Per item thresholds are realized using two additional meta data fields:
+The script `item-threshold.zeek` implements per item thresholds and is loaded by default. Per item thresholds are realized using two additional meta data fields:
 * `matching_threshold` defines the number of reports required to trigger a match
 * `recent_matches` implements a counter for recent matches
 
 To prevent matching of an item before the specified threshold is reached, the script handles the `extend_match` hook of the intelligence framework and checks the number of recent hits.
 For further details on the intel framework see the corresponding [blog post](https://blog.zeek.org/2016/12/the-intelligence-framework-update.html).
 
-The script `heaver-hitter.bro` implements heavy hitter suppression simply by tracking hits per indicator in a table. Note that item removal and reporting a warning is a very simple strategy. In more complex environments one would want to report heavy hitters using a dedicated feedback channel, e.g. implemented with broker.
+The script `heaver-hitter.zeek` implements heavy hitter suppression simply by tracking hits per indicator in a table. Note that item removal and reporting a warning is a very simple strategy. In more complex environments one would want to report heavy hitters using a dedicated feedback channel, e.g. implemented with broker.
 
 ## Acknowledgments
 
diff --git a/scripts/__load__.zeek b/scripts/__load__.zeek
diff --git a/scripts/heavy-hitter.zeek b/scripts/heavy-hitter.zeek
diff --git a/scripts/item-threshold.zeek b/scripts/item-threshold.zeek
diff --git a/testing/Baseline/scripts.test-heavy-hitter/output b/testing/Baseline/scripts.test-heavy-hitter/output
@@ -6,12 +6,12 @@
 #open	2019-08-31-19-35-18
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	seen.indicator	seen.indicator_type	seen.where	seen.node	matched	sources	fuid	file_mime_type	file_desc
 #types	time	string	addr	port	addr	port	string	enum	enum	string	set[enum]	set[string]	string	string	string
-1.000011	-	-	-	-	-	1.0.0.0	Intel::ADDR	SOMEWHERE	bro	Intel::ADDR	source1	-	-	-
-1.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	bro	Intel::ADDR	source1	-	-	-
-2.000011	-	-	-	-	-	1.0.0.0	Intel::ADDR	SOMEWHERE	bro	Intel::ADDR	source1	-	-	-
-2.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	bro	Intel::ADDR	source1	-	-	-
-3.000011	-	-	-	-	-	1.0.0.0	Intel::ADDR	SOMEWHERE	bro	Intel::ADDR	source1	-	-	-
-6.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	bro	Intel::ADDR	source1	-	-	-
+1.000011	-	-	-	-	-	1.0.0.0	Intel::ADDR	SOMEWHERE	zeek	Intel::ADDR	source1	-	-	-
+1.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	zeek	Intel::ADDR	source1	-	-	-
+2.000011	-	-	-	-	-	1.0.0.0	Intel::ADDR	SOMEWHERE	zeek	Intel::ADDR	source1	-	-	-
+2.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	zeek	Intel::ADDR	source1	-	-	-
+3.000011	-	-	-	-	-	1.0.0.0	Intel::ADDR	SOMEWHERE	zeek	Intel::ADDR	source1	-	-	-
+6.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	zeek	Intel::ADDR	source1	-	-	-
 #close	2019-08-31-19-35-18
 >> Run 0:
 >> Run 1:
diff --git a/testing/Baseline/scripts.test-item-threshold/output b/testing/Baseline/scripts.test-item-threshold/output
@@ -6,9 +6,9 @@
 #open	2019-08-27-18-33-06
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	seen.indicator	seen.indicator_type	seen.where	seen.node	matched	sources	fuid	file_mime_type	file_desc
 #types	time	string	addr	port	addr	port	string	enum	enum	string	set[enum]	set[string]	string	string	string
-3.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	bro	Intel::ADDR	source1	-	-	-
-6.000011	-	-	-	-	-	3.0.0.0	Intel::ADDR	SOMEWHERE	bro	Intel::ADDR	source1	-	-	-
-7.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	bro	Intel::ADDR	source1	-	-	-
+3.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	zeek	Intel::ADDR	source1	-	-	-
+6.000011	-	-	-	-	-	3.0.0.0	Intel::ADDR	SOMEWHERE	zeek	Intel::ADDR	source1	-	-	-
+7.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	zeek	Intel::ADDR	source1	-	-	-
 #close	2019-08-27-18-33-06
 >> Run 0:
 >> Run 1:
diff --git a/testing/Baseline/scripts.test-multiple-thresholds/output b/testing/Baseline/scripts.test-multiple-thresholds/output
@@ -6,9 +6,9 @@
 #open	2019-08-31-15-15-42
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	seen.indicator	seen.indicator_type	seen.where	seen.node	matched	sources	fuid	file_mime_type	file_desc
 #types	time	string	addr	port	addr	port	string	enum	enum	string	set[enum]	set[string]	string	string	string
-2.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	bro	Intel::ADDR	source1,source2	-	-	-
-5.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	bro	Intel::ADDR	source1,source2	-	-	-
-7.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	bro	Intel::ADDR	source1,source2	-	-	-
+2.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	zeek	Intel::ADDR	source2,source1	-	-	-
+5.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	zeek	Intel::ADDR	source2,source1	-	-	-
+7.000011	-	-	-	-	-	2.0.0.0	Intel::ADDR	SOMEWHERE	zeek	Intel::ADDR	source2,source1	-	-	-
 #close	2019-08-31-15-15-42
 >> Run 0:
 >> Run 1:
diff --git a/testing/btest.cfg b/testing/btest.cfg
@@ -6,8 +6,8 @@ IgnoreDirs  = .tmp
 IgnoreFiles = *.tmp *.swp
 
 [environment]
-BRO_SEED_FILE=%(testbase)s/random.seed
-BROPATH=`bro-config --bropath`:%(testbase)s/../scripts
+ZEEK_SEED_FILE=%(testbase)s/random.seed
+ZEEKPATH=`zeek-config --zeekpath`:%(testbase)s/../scripts
 TZ=UTC
 LC_ALL=C
 TRACES=%(testbase)s/traces
diff --git a/testing/scripts/test-heavy-hitter-cluster.zeek b/testing/scripts/test-heavy-hitter-cluster.zeek
@@ -1,11 +1,11 @@
 # @TEST-SERIALIZE: comm
 #
-# @TEST-EXEC: btest-bg-run manager-1 BROPATH=$BROPATH:.. CLUSTER_NODE=manager-1 bro %INPUT
-# @TEST-EXEC: btest-bg-run worker-1  BROPATH=$BROPATH:.. CLUSTER_NODE=worker-1 bro -r $TRACES/ticks.pcap --pseudo-realtime %INPUT
+# @TEST-EXEC: btest-bg-run manager-1 ZEEKPATH=$ZEEKPATH:.. CLUSTER_NODE=manager-1 zeek %INPUT
+# @TEST-EXEC: btest-bg-run worker-1  ZEEKPATH=$ZEEKPATH:.. CLUSTER_NODE=worker-1 zeek -r $TRACES/ticks.pcap --pseudo-realtime %INPUT
 # @TEST-EXEC: btest-bg-wait -k 13
 # @TEST-EXEC: btest-diff manager-1/intel.log
 
-# @TEST-START-FILE cluster-layout.bro
+# @TEST-START-FILE cluster-layout.zeek
 redef Cluster::nodes = {
 	["manager-1"] = [$node_type=Cluster::MANAGER, $ip=127.0.0.1, $p=37757/tcp],
 	["worker-1"]  = [$node_type=Cluster::WORKER,  $ip=127.0.0.1, $p=37760/tcp, $manager="manager-1"],
@@ -39,7 +39,7 @@ event Cluster::node_up(name: string, id: string)
 # Worker
 
 @if ( Cluster::local_node_type() == Cluster::WORKER )
-event bro_init()
+event zeek_init()
 	{
 	suspend_processing();
 	}
diff --git a/testing/scripts/test-heavy-hitter.zeek b/testing/scripts/test-heavy-hitter.zeek
@@ -1,20 +1,19 @@
-# @TEST-EXEC: bro -r $TRACES/ticks.pcap heavy-hitter %INPUT
+# @TEST-EXEC: zeek -r $TRACES/ticks.pcap heavy-hitter %INPUT
 # @TEST-EXEC: cat intel.log > output
 # @TEST-EXEC: cat .stdout >> output
 # @TEST-EXEC: btest-diff output
 
-# @TEST-START-FILE intel.dat
-#fields	indicator	indicator_type	meta.source	meta.desc
-1.0.0.0	Intel::ADDR	source1	this host is bad
-2.0.0.0	Intel::ADDR	source1	this host is bad
-# @TEST-END-FILE
-
-redef Intel::read_files += { "intel.dat" };
 redef enum Intel::Where += { SOMEWHERE };
 redef Intel::heavy_hitter_interval = 4sec;
 redef Intel::heavy_hitter_threshold = 2;
 redef table_expire_interval = 2sec;
 
+event zeek_init()
+	{
+	Intel::insert([$indicator="1.0.0.0", $indicator_type=Intel::ADDR, $meta=[$source="source1"]]);
+	Intel::insert([$indicator="2.0.0.0", $indicator_type=Intel::ADDR, $meta=[$source="source1"]]);
+	}
+
 global runs = 0;
 
 event connection_established(c: connection)
diff --git a/testing/scripts/test-item-threshold.zeek b/testing/scripts/test-item-threshold.zeek
@@ -1,4 +1,4 @@
-# @TEST-EXEC: bro -r $TRACES/ticks.pcap item-threshold %INPUT
+# @TEST-EXEC: zeek -r $TRACES/ticks.pcap item-threshold %INPUT
 # @TEST-EXEC: cat intel.log > output
 # @TEST-EXEC: cat .stdout >> output
 # @TEST-EXEC: btest-diff output
@@ -17,6 +17,23 @@ redef Intel::read_files += { "intel_plain.dat", "intel_threshold.dat" };
 redef enum Intel::Where += { SOMEWHERE };
 redef Intel::default_matching_threshold = 3;
 
+# Wait for intel data to be loaded
+global intel_data = 0;
+
+event zeek_init()
+	{
+	suspend_processing();
+	}
+
+hook Intel::filter_item(item: Intel::Item)
+	{
+	++intel_data;
+	if ( intel_data == 2 )
+		continue_processing();
+	}
+
+# Test scenario
+
 global runs = 0;
 
 event connection_established(c: connection)
diff --git a/testing/scripts/test-multiple-thresholds.zeek b/testing/scripts/test-multiple-thresholds.zeek
@@ -1,18 +1,19 @@
-# @TEST-EXEC: bro -r $TRACES/ticks.pcap item-threshold %INPUT
+# @TEST-EXEC: zeek -r $TRACES/ticks.pcap item-threshold %INPUT
 # @TEST-EXEC: cat intel.log > output
 # @TEST-EXEC: cat .stdout >> output
 # @TEST-EXEC: btest-diff output
 
-# @TEST-START-FILE intel_threshold.dat
-#fields	indicator	indicator_type	meta.source	meta.desc	meta.matching_threshold
-2.0.0.0	Intel::ADDR	source1	this host is bad	2
-2.0.0.0	Intel::ADDR	source2	this host is bad	4
-# @TEST-END-FILE
-
-redef Intel::read_files += { "intel_threshold.dat" };
 redef enum Intel::Where += { SOMEWHERE };
 redef Intel::default_matching_threshold = 3;
 
+event zeek_init()
+	{
+	Intel::insert([$indicator="2.0.0.0", $indicator_type=Intel::ADDR, $meta=[
+		$source="source1", $matching_threshold=2]]);
+	Intel::insert([$indicator="2.0.0.0", $indicator_type=Intel::ADDR, $meta=[
+		$source="source2", $matching_threshold=4]]);
+	}
+
 global runs = 0;
 
 event connection_established(c: connection)
diff --git a/zkg.meta b/zkg.meta
@@ -4,3 +4,5 @@ credits = Jan Grashoefer <jan.grashoefer@gmail.com>
 tags = intel, limits, threshold
 script_dir = scripts
 test_command = cd testing && btest -d
+depends =
+  zeek >=3.0

Original file line number	Diff line number	Diff line change
`@@ -1,4 +1,4 @@`
`1`		`-Copyright (c) 2019 by Jan Grashoefer`
	`1`	`+Copyright (c) 2019-2022 by Jan Grashoefer`
`2`	`2`
`3`	`3`	`Redistribution and use in source and binary forms, with or without`
`4`	`4`	`modification, are permitted provided that the following conditions are met:`