Webflow IDS: Real-Time Intrusion Detection System

This application was built to add an additional layer of protection and provide a deeper understanding of traffic for Webflow websites.

1. Log Collection with Caddy

Caddy Server: The Caddy server acts as a reverse proxy for your domain. It has been configured to log all HTTP traffic to a JSON file (access.json). This file contains detailed information about incoming requests, including:

• Timestamps
• IP addresses
• Request methods
• URIs
• User agents
• Status codes

2. Log Parsing with Fluent Bit

Fluent Bit: A log processor that monitors the access.json file generated by the Caddy server. It uses a Lua script (process_payload.lua) to parse and reformat log entries.

• Lua Script: Extracts relevant fields (e.g., remote_ip, method, uri, user_agent, status) from the log records. It handles potential nested data structures within the log entries, ensuring the data is correctly formatted into JSON objects that the IDS expects.
• Data Forwarding: Fluent Bit forwards these parsed and reformatted log entries to a Go-based Intrusion Detection System (IDS) running on localhost:3000.

3. Intrusion Detection System (IDS) in Go

HTTP Server: The Go application acts as an HTTP server that listens for incoming log data from Fluent Bit.

• Log Handling:

  • logHandler: This endpoint accepts an array of JSON objects representing the log entries. It parses each entry to extract fields like:
    • Timestamp
    • Remote IP
    • Request method
    • URI
    • User agent
    • Status

• Intrusion and Anomaly Detection: For each log entry, the system performs:

  • Signature-based Detection: Checks if the log matches any known attack patterns loaded from a signatures.json file.
  • Anomaly Detection: Tracks IP address activity to identify suspicious behavior, such as excessive requests in a short time frame.

• Data Storage: Maintains a list of detected anomalies and intrusions, along with general traffic statistics.

• Dashboard and Reporting:

  • statsHandler: This endpoint returns the aggregated statistics (e.g., total requests, detected intrusions, anomalies) for display on a dashboard.
  • Home Page: Displays the dashboard using an index.html template.

4. System Overview

Flow:

1. The Caddy server logs HTTP traffic.
2. Fluent Bit monitors and reads these logs, using a Lua script to extract and format relevant data.
3. Fluent Bit sends the parsed logs to the Go IDS application.
4. The Go IDS processes each log entry for potential intrusions or anomalies and maintains traffic statistics.
5. The IDS provides a dashboard for viewing these statistics, allowing for real-time monitoring of traffic and potential security threats.

Main Goal

This entire setup provides a real-time intrusion detection system for monitoring web traffic, extracting key data points, identifying potential security threats, and displaying the results on a web-based dashboard.