Understanding log ingestion, parsing, and custom XML pipeline workflow in Security Onion

[nayadmohamed]

Version

2.4.211

Installation Method

Other (please provide detail below)

Description

other (please provide detail below)

Installation Type

Standalone

Location

other (please provide detail below)

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

128GB

Storage for /

1TB

Storage for /nsm

1TB

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Hello,
I am currently trying to better understand how logs are ingested and processed internally in Security Onion standalone deployments.
I am doing an internship and I am still learning the architecture, so I may misunderstand some concepts.
My current goal is to ingest custom XML logs into Security Onion, parse them correctly, and use them later for detections and dashboards.
However, I am struggling to understand the full ingestion pipeline and where parsing actually happens.

For example:

• For SSH logs, I can follow the logs up to /var/log/secure
• But after that, I get lost regarding:
  • where parsing happens
  • where ECS mapping occurs
  • how logs travel through the stack
  • where filters/pipelines are applied

I tried exploring:

• Logstash pipelines
• SOC Pipelines section in the UI
• Elastic integrations
• Sigma detections

But I still do not clearly understand the recommended workflow.
My questions are:

1. What is the normal log flow in Security Onion standalone ?
   For example:

   • File → Filebeat/Elastic Agent → Logstash? → Elasticsearch ?
   • Where exactly does parsing occur ?

2. If I have custom XML logs:

   • Where should I place the files so Security Onion reads them ?
   • What component should I use to ingest them?
     • Elastic Agent ?
     • Filebeat ?
     • Logstash ?
     • SOC Pipelines ?

3. Where are:

   • inputs
   • filters/parsing
   • outputs

typically configured in Security Onion ?

4. What is the recommended approach for parsing XML logs and mapping them into ECS fields?

5. Is there documentation or an example pipeline for custom log parsing in Security Onion?

I think part of my confusion comes from not understanding which layer is responsible for:

• collection
• parsing
• normalization
• detection

I would really appreciate any guidance on the recommended architecture/workflow before I go further.

Thank you very much.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

https://docs.securityonion.net/en/3/main/ingest/ shows the ingest pipeline for logs based on deployment. That is a good starting point to understand how the logs are processed and indexed.