Skip to content

Rootless-Ghost/LogNorm

BranchesTags

Repository files navigation

LogNorm

Log Source Normalizer — Sysmon / WEL / Wazuh / syslog / CEF → ECS-lite schema

Part of the Nebula Forge detection engineering platform. LogNorm is the normalization gateway for the pipeline: its ECS-lite JSON output is the shared data currency accepted by DriftWatch, ClusterIQ, AtomicLoop, and HuntForge.

version port python framework part%20of license

What it does

LogNorm ingests raw log files from five source types and normalizes every record to a consistent ECS-lite JSON schema. Normalized events are stored in SQLite and can be exported as JSON or CSV. The Flask web UI provides an upload → normalize → view → export workflow; the CLI handles file-based automation.

Pipeline Position

Nebula Forge pipeline — LogNorm highlighted

purple-loop: AtomicLoop → LogNorm → ClusterIQ → HuntForge → DriftWatch → repeat

Screenshots

Dashbaord

LogNorm Dashboard

Upload

LogNorm Upload

Source adapters

Key Source Input format Coverage
sysmon Microsoft Sysmon XML (wevtutil export) EventIDs 1–29 (process, network, file, registry, DNS, WMI, pipes)
wel Windows Event Log CSV (Get-WinEvent / EndpointTriage Security.csv) 4624/4625 logon, 4688 process, 4698 tasks, 4720/4726 accounts, 4740 lockout, 7045 service
wazuh Wazuh alerts NDJSON / JSON array / single object All rule-based alerts; Windows data.win.*, syscheck, audit blocks
syslog Linux auth.log / syslog Text (RFC 3164 / RFC 5424 / journald) SSH, sudo, PAM, cron, useradd, su, systemd
cef CEF / Generic JSON `CEF:0 ...

Quick start

# Install
pip install flask pyyaml

# Copy config
cp config.example.yaml config.yaml

# Run web app (default port 5006)
python app.py

# Or specify port
python app.py --port 5006

Open http://127.0.0.1:5006

CLI

# Normalize Sysmon XML to JSON (stdout)
python cli.py --input sysmon.xml --source sysmon --pretty

# Normalize WEL CSV to output file
python cli.py --input security.csv --source wel --output normalized.json

# Normalize Wazuh alerts to CSV
python cli.py --input alerts.json --source wazuh --output out.csv --format csv

# Normalize auth.log (strip original_log to reduce size)
python cli.py --input /var/log/auth.log --source syslog --no-original-log

# Read from stdin
cat sysmon.xml | python cli.py --source sysmon --stdin

# All options
python cli.py --help

API endpoints

Method Path Description
GET /api/health Health check — {"status":"ok","tool":"lognorm","version":"1.0.0"}
GET /api/sources List supported source types and descriptions
POST /api/normalize Normalize single record (JSON body)
POST /api/normalize/batch Normalize file (multipart) or record list (JSON)
GET /api/records List stored events (pagination + filters)
GET /api/record/<id> Fetch single stored event by UUID
GET /api/sessions List normalization sessions
GET /api/export Export as JSON or CSV (`?format=json
DELETE /api/records Clear all stored records

POST /api/normalize

POST /api/normalize
{"source_type": "sysmon", "raw": "<Event>...</Event>"}

→ {"success": true, "event": {<ECS-lite>}, "session_id": "uuid"}

POST /api/normalize/batch

// JSON body
POST /api/normalize/batch
{"source_type": "wazuh", "raw": "...full file content..."}
{"source_type": "syslog", "records": ["Apr 3 ...", "Apr 3 ..."]}

// File upload (multipart/form-data)
POST /api/normalize/batch
  file=<binary>  source_type=sysmon

→ {"success": true, "events": [{<ECS-lite>}, ...], "failed": 0,
   "total": 45, "session_id": "uuid", "filename": "sysmon.xml"}

ECS-lite schema reference

Schema version: 1.0 — Required fields are event.id, event.created, event.source_type, log.source_type. All other fields are optional and omitted when empty.

Top-level

Field Type Required Description
schema_version string yes Always "1.0"
tags array yes Free-form tags (e.g. T1059.001, atomic-test)

event

Field Type Required Description
event.id string yes UUID v4 — unique identifier for this normalized event
event.created ISO8601 yes Timestamp the event was created (UTC)
event.source_type string yes Adapter: sysmon | wel | wazuh | syslog | cef
event.category array no ECS category: process | network | file | registry | authentication | iam | driver
event.type array no ECS type: start | end | creation | deletion | access | change | connection | protocol
event.action string no Human-readable action from the source log
event.outcome string no success | failure | unknown
event.severity integer no Normalized 0–100 severity
event.original_event_id string no EventID from source (e.g. 1, 4624)

host

Field Type Required Description
host.name string no Hostname of the originating endpoint
host.hostname string no Same as host.name
host.ip array no IP addresses associated with the host
host.os.type string no windows | linux | macos
host.os.name string no Full OS name

process

Field Type Required Description
process.pid integer no Process ID
process.ppid integer no Parent process ID
process.name string no Image name (e.g. powershell.exe)
process.executable string no Full path to executable
process.command_line string no Full command line including arguments
process.hash.md5 string no MD5 of process image (lowercase hex)
process.hash.sha256 string no SHA-256 of process image (lowercase hex)
process.parent.pid integer no Parent PID
process.parent.name string no Parent image name
process.parent.executable string no Full path to parent executable
process.parent.command_line string no Parent command line

network

Field Type Required Description
network.direction string no ingress | egress | internal
network.transport string no tcp | udp | icmp
network.destination.ip string no Destination IP
network.destination.port integer no Destination port
network.destination.domain string no Destination hostname / domain
network.source.ip string no Source IP
network.source.port integer no Source port

file

Field Type Required Description
file.path string no Full file path
file.name string no Filename without directory
file.extension string no Extension without leading dot
file.hash.md5 string no MD5 (lowercase hex)
file.hash.sha256 string no SHA-256 (lowercase hex)
file.size integer no Size in bytes

registry

Field Type Required Description
registry.path string no Full registry path
registry.key string no Registry key name
registry.value.name string no Value name
registry.value.type string no Value type (REG_SZ, REG_DWORD, …)
registry.value.data string no Value data

user

Field Type Required Description
user.name string no Account username
user.domain string no Domain or workgroup
user.id string no SID or UID

log

Field Type Required Description
log.source_type string yes Adapter used
log.source_tool string no Tool that generated the original log
log.original_event_id string no Original event identifier
log.original_log object no Raw source record preserved verbatim

Nebula-Forge integration

nebula-dashboard config.yaml — add to the tools: block:

lognorm:
  label:       "LogNorm"
  url:         "http://127.0.0.1:5006"
  health_path: "/api/health"
  description: "Log source normalizer — Sysmon / WEL / Wazuh / syslog / CEF → ECS-lite"
  category:    "Normalize"

detection-pipeline / purple-loop config.yaml:

lognorm_url: "http://127.0.0.1:5006"

Architecture

LogNorm/
├── app.py                  Flask web app + all API endpoints
├── cli.py                  Standalone CLI
├── config.example.yaml     Configuration template
├── adapters/
│   ├── base.py             BaseAdapter — parse() + parse_file()
│   ├── sysmon.py           Sysmon XML (EventIDs 1–29)
│   ├── wel.py              Windows Event Log CSV
│   ├── wazuh.py            Wazuh JSON / NDJSON
│   ├── syslog.py           Linux auth.log / syslog / journald
│   └── cef.py              CEF + generic JSON
├── core/
│   ├── engine.py           NormalizationEngine — dispatch + fallback
│   ├── models.py           make_ecs_event() factory + helpers
│   ├── schema.py           Field reference + source descriptions
│   └── storage.py          SQLite — sessions + events tables
├── static/css/style.css    Nebula Forge dark theme
├── static/js/main.js       Upload, normalize, render, pagination
└── templates/
    ├── base.html           Sidebar layout
    ├── index.html          Upload + normalize UI
    ├── records.html        Browse stored events
    └── schema.html         ECS-lite field reference

Fallback pattern (consistent with detection-pipeline / ir-chain): if the SQLite write fails, the normalized events are saved to ./output/fallback_<source>_<timestamp>_<session>.json so no data is lost.

Home lab

Validated against:

  • Wazuh 4.14.4 at <YOUR_WAZUH_IP> — Wazuh adapter targets this alert format
  • Windows Agent — Sysmon (SwiftOnSecurity config) + EndpointTriage Security.csv
  • Linux Agent — auth.log / journald output
  • Splunk on Linux Agent — WEL CSV exports

License

This project is licensed under the MIT License — see the LICENSE file for details.

Built by Rootless-Ghost

Part of the Nebula Forge security tools suite.

About

Log Source Normalizer — Sysmon / WEL / Wazuh / syslog / CEF → ECS-lite schema | Part of Nebula Forge

Topics

python flask log-analysis ecs sysmon siem normalization wazuh blue-team detection-engineering

Resources

Readme

License

MIT license

Security policy

Security policy
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages