Expose all netflow input options through netflow module (elastic#14628)

The exposes all of the netflow input configuration options through the netflow/log fileset. Fixes elastic#14617

Filebeat's module code was changed to allow for manifests to declare variables without default values. This means that module manifests can be written without duplicating the default values of the input types that they wrap (e.g. don't duplicate the default socket timeout value). It also changes the template evaluation to be more strict in that referencing a variable that does not exist will now cause an error instead of evaluating to "<no value>".

The zeek/dhcp module referred to a variable that was not declared in its manifest. This was fixed.

(cherry picked from commit e08c6ec)

CHANGELOG.next.asciidoc

@@ -404,6 +404,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add more filesets to Zeek module. {pull}14150[14150]
 - Add attack_pattern_kql field to MISP threat indicators. {pull}14470[14470]
 - Add fileset to the Zeek module for the intel.log. {pull}14404[14404]
+- Add more configuration options to the Netflow module. {pull}14628{14628}
 
 *Heartbeat*
 - Add non-privileged icmp on linux and darwin(mac). {pull}13795[13795] {issue}11498[11498]

filebeat/docs/inputs/input-common-udp-options.asciidoc

@@ -18,6 +18,12 @@ The host and UDP port to listen on for event streams.
 
 [float]
 [id="{beatname_lc}-input-{type}-udp-read-buffer"]
-=== `read_buffer`
+==== `read_buffer`
 
 The size of the read buffer on the UDP socket.
+
+[float]
+[id="{beatname_lc}-input-{type}-udp-timeout"]
+==== `timeout`
+
+The read and write timeout for socket operations.

filebeat/docs/modules/netflow.asciidoc

@@ -51,6 +51,10 @@ traffic from network devices.
 `var.max_message_size`:: The maximum size of the message received over UDP.
 The default is `10KiB`.
 
+`var.read_buffer`:: The size of the read buffer on the UDP socket.
+
+`var.timeout`:: The read and write timeout for socket operations.
+
 `var.expiration_timeout`:: The time before an idle session or unused template is
 expired. Only applicable to v9 and IPFIX protocols. A value of zero disables
 expiration.
@@ -59,6 +63,11 @@ expiration.
 processing. Use this setting to avoid packet-loss when dealing with occasional
 bursts of traffic.
 
+`var.custom_definitions`:: A list of paths to field definitions YAML files.
+These allow to update the NetFlow/IPFIX fields with vendor extensions and to
+override existing fields. See <<filebeat-input-netflow,netflow input>> for
+details.
+
 :has-dashboards!:
 
 :fileset_ex!:

filebeat/fileset/fileset.go

@@ -173,10 +173,9 @@ func (fs *Fileset) evaluateVars(beatVersion string) (map[string]interface{}, err
 			return nil, fmt.Errorf("Variable doesn't have a string 'name' key")
 		}
 
-		value, exists := vals["default"]
-		if !exists {
-			return nil, fmt.Errorf("Variable %s doesn't have a 'default' key", name)
-		}
+		// Variables are not required to have a default. Templates should
+		// handle null default values as necessary.
+		value := vals["default"]
 
 		// evaluate OS specific vars
 		osVals, exists := vals["os"].(map[string]interface{})
@@ -268,7 +267,7 @@ func resolveVariable(vars map[string]interface{}, value interface{}) (interface{
 // the delimiters are set to `{<` and `>}` instead of `{{` and `}}`. These are easier to use
 // in pipeline definitions.
 func applyTemplate(vars map[string]interface{}, templateString string, specialDelims bool) (string, error) {
-	tpl := template.New("text")
+	tpl := template.New("text").Option("missingkey=error")
 	if specialDelims {
 		tpl = tpl.Delims("{<", ">}")
 	}

x-pack/filebeat/module/netflow/_meta/docs.asciidoc

@@ -46,6 +46,10 @@ traffic from network devices.
 `var.max_message_size`:: The maximum size of the message received over UDP.
 The default is `10KiB`.
 
+`var.read_buffer`:: The size of the read buffer on the UDP socket.
+
+`var.timeout`:: The read and write timeout for socket operations.
+
 `var.expiration_timeout`:: The time before an idle session or unused template is
 expired. Only applicable to v9 and IPFIX protocols. A value of zero disables
 expiration.
@@ -54,6 +58,11 @@ expiration.
 processing. Use this setting to avoid packet-loss when dealing with occasional
 bursts of traffic.
 
+`var.custom_definitions`:: A list of paths to field definitions YAML files.
+These allow to update the NetFlow/IPFIX fields with vendor extensions and to
+override existing fields. See <<filebeat-input-netflow,netflow input>> for
+details.
+
 :has-dashboards!:
 
 :fileset_ex!:

x-pack/filebeat/module/netflow/log/config/netflow.yml

@@ -4,3 +4,18 @@ host: '{{.netflow_host}}:{{.netflow_port}}'
 max_message_size: '{{.max_message_size}}'
 expiration_timeout: '{{.expiration_timeout}}'
 queue_size: {{.queue_size}}
+
+{{if .timeout}}
+timeout: '{{.timeout}}'
+{{end}}
+
+{{if .read_buffer}}
+read_buffer: '{{.read_buffer}}'
+{{end}}
+
+{{ if .custom_definitions}}
+custom_definitions:
+{{range .custom_definitions}}
+- '{{ . }}'
+{{end}}
+{{end}}

x-pack/filebeat/module/netflow/log/manifest.yml

@@ -11,6 +11,9 @@ var:
     default: 30m
   - name: queue_size
     default: 8192
+  - name: read_buffer
+  - name: timeout
+  - name: custom_definitions
 
 ingest_pipeline: ingest/pipeline.yml
 input: config/netflow.yml

x-pack/filebeat/module/zeek/dhcp/manifest.yml

@@ -10,6 +10,8 @@ var:
       - /usr/local/var/logs/current/dhcp.log
   - name: tags
     default: [zeek.dhcp]
+  - name: community_id
+    default: true
 
 ingest_pipeline: ingest/pipeline.json
 input: config/dhcp.yml