[Filebeat] module for palo_alto (pan-os) logs

[adriansr]

This is a module for Palo Alto Networks PAN-OS logs received via Syslog.

• Compatibility
• ECS Mappings for TRAFFIC
• ECS Mappings for THREAT
• Categorization of events
  • event.category
  • event.action
  • event.outcome
• Pending tasks

Compatibility

It has been tested with logs for PAN-OS version 7.1 to 9.0. However, it is expected to work with earlier versions as the log format is compatible.

ECS Mappings for TRAFFIC

These are the PAN-OS to ECS field mappings as well as those fields still not in ECS that are added under the palo_alto prefix:

PAN-OS Field	ECS Field	Non-standard field
Receive Time	event.created
Serial Number	observer.serial_number
Type	event.category
Subtype	event.action
Generated Time	@timestamp
Source IP	client.ip source.ip
Destination IP	server.ip destination.ip
NAT Source IP		palo_alto.source.nat.ip
NAT Destination IP		palo_alto.destination.nat.ip
Rule Name		palo_alto.ruleset
Source User	client.user.name source.user.name
Destination User	server.user.name destination.user.name
Application	network.application
Source Zone		palo_alto.source.zone
Destination Zone		palo_alto.destination.zone
Ingress Interface		palo_alto.source.interface
Egress Interface		palo_alto.destination.interface
Session ID		palo_alto.flow_id
Source Port	client.port source.port
Destination Port	destination.port server.port
NAT Source Port		palo_alto.source.nat.port
NAT Destination Port		palo_alto.destination.nat.port
Flags	abels
Protocol	network.transport
Action	event.outcome
Bytes	network.bytes
Bytes Sent	client.bytes destination.bytes
Bytes Received	server.bytes source.bytes
Packets	network.packets
Start Time	event.start
Elapsed Time	event.duration
Category		palo_alto.url.category
Sequence Number		palo_alto.sequence_number
Packets Sent	server.packets destination.packets
Packets Received	client.packets source.packets
Device Name	observer.hostname

ECS Mappings for THREAT

PAN-OS Field	ECS Field	Non-standard field
Receive Time	event.created
Serial Number	observer.serial_number
Type	event.category
Subtype	event.action
Generated Time	@timestamp
Source IP	client.ip source.ip
Destination IP	server.ip destination.ip
NAT Source IP		palo_alto.source.nat.ip
NAT Destination IP		palo_alto.destination.nat.ip
Rule Name		palo_alto.ruleset
Source User	client.user.name source.user.name
Destination User	server.user.name destination.user.name
Application	network.application
Source Zone		palo_alto.source.zone
Destination Zone		palo_alto.destination.zone
Ingress Interface		palo_alto.source.interface
Egress Interface		palo_alto.destination.interface
Session ID		palo_alto.flow_id
Source Port	client.port source.port
Destination Port	destination.port server.port
NAT Source Port		palo_alto.source.nat.port
NAT Destination Port		palo_alto.destination.nat.port
Flags	labels
Protocol	network.transport
Action	event.outcome
Miscellaneous	url.original	palo_alto.threat_file_or_url
Threat ID		palo_alto.threat_id
Category		palo_alto.url.category
Severity	log.level
Direction	network.direction
Source Location	source.geo.country_iso_code
Destination Location	destination.geo.country_iso_code
PCAP_id		palo_alto.network.pcap_id
Filedigest		palo_alto.file.hash
User Agent	user_agent.original
File Type	file.type
X-Forwarded-For	network.forwarded_ip
Referer	http.request.referer
Sender	source.user.email
Subject		palo_alto.subject
Recipient	destination.user.email
Device Name	observer.hostname

Categorization of events

This module uses the following ECS fields for categorization of events:

event.category

Log type	event.category
TRAFFIC	network_traffic
THREAT	security_threat

event.action

event.category	event.action
network_traffic	flow_started
	flow_terminated
	flow_dropped
	flow_denied
security_threat	data_match
	file_match
	flood_detected
	packet_attack
	scan_detected
	spyware_detected
	url_filtering
	virus_detected
	exploit_detected
	wildfire_veredict
	wildfire_virus_detected

event.outcome

event.category	event.outcome	description
network_traffic	allow	session was allowed by policy
	deny	session was denied by policy
	drop	session was dropped silently
	drop ICMP	session was silently dropped with an ICMP unreachable message to the host or application
	reset both	session was terminated and a TCP reset is sent to both the sides of the connection
	reset client	session was terminated and a TCP reset is sent to the client
	reset server	session was terminated and a TCP reset is sent to the server
security_threat	alert	threat or URL detected but not blocked
	allow	flood detection alert (?)
	deny	flood detection mechanism activated and deny traffic based on configuration
	drop	threat detected and associated session was dropped
	reset-client	threat detected and a TCP RST is sent to the client
	reset-server	threat detected and a TCP RST is sent to the server
	reset-both	threat detected and a TCP RST is sent to both the client and the server
	block-url	URL request was blocked because it matched a URL category that was set to be blocked
	block-ip	threat detected and client IP is blocked
	random-drop	flood detected and packet was randomly dropped
	sinkhole	DNS sinkhole activated
	syncookie-sent	syncookie alert
	block-continue	(URL subtype only) a HTTP request is blocked and redirected to a Continue page with a button for confirmation to proceed
	continue	(URL subtype only) response to a block-continue URL continue page indicating a block-continue request was allowed to proceed
	block-override	(URL subtype only) a HTTP request is blocked and redirected to an Admin override page that requires a pass code from the firewall administrator to continue
	override-lockout	(URL subtype only) too many failed admin override pass code attempts from the source IP. IP is now blocked from the block-override redirect page
	override	(URL subtype only) response to a block-override page where a correct pass code is provided and the request is allowed
	block	(Wildfire only) file was blocked by the firewall and uploaded to Wildfire

Pending tasks

• rebase after New processor extract_array #11761#11761 is merged
• dashboards

Closes #9199

[jsoriano]

Great work here, and good thing we can define pipelines in YAML now 😉 Added some minor comments, feel free to ignore them.

[jsoriano]

I think this should be beta.

[jsoriano]

Suggested change

	func TestExtractArrayProcessor_String(t *testing.T) {
	func TestExtractArrayProcessorString(t *testing.T) {

[jsoriano]

Suggested change

	func TestExtractArrayProcessor_Run(t *testing.T) {
	func TestExtractArrayProcessorRun(t *testing.T) {

[jsoriano]

Remove till a dashboard is available?

[jsoriano]

We should probably add a copy_fields processor 🙂

[andrewkroh]

This is looking good to me in terms of the ECS fields that it's populating. Left a few minor suggestions.

[andrewkroh]

Maybe we should call this pan-os instead of the generic log. I think Palo Alto might have other log producing products than PAN-OS.

[adriansr]

Yes, makes sense.

However, I chose pan_os instead of pan-os, for a few reasons:

• It makes sense to keep the custom fields in a subgroup, from palo_alto.* to palo_alto.pan-os.*. In this case, the pan-os component in a field name causes a lot of trouble with painless, it complicates if conditions because you can't do ctx?.palo_alto?.pan-os as it is interpreted as a subtraction.
• It looks kind of ugly to mix underscore and hyphen as in palo_alto.pan-os
• Other filesets / submodules use underscore
• ECS suggests to use underscore.

On the other side, pan-os is the well-known name. WDYT?

[andrewkroh]

SGTM. The benefits of pan_os outweigh the benefits of pan-os.

[andrewkroh]

The updates here LGTM.