Pattern Extractor

A Python tool to define, extract, and structure data from semi-structured text files and logs using user-defined patterns.

Features

• Define extraction patterns using regular expressions or simple keyword rules within a configuration file.
• Process multiple input files or live streams based on configured patterns.
• Export extracted data into various structured formats like CSV, JSON, or YAML.
• Support for grouping and naming captured data fields from patterns.
• Command-line interface for easy execution and integration into scripts.

Installation

To install the pattern-extractor, you can clone the repository and install it:

git clone https://github.com/your-username/pattern-extractor.git
cd pattern-extractor
pip install .

Alternatively, if it's published on PyPI:

pip install pattern-extractor

Usage

The pattern-extractor tool takes a configuration file defining your patterns and one or more input files (or standard input) to process. It then outputs the extracted data in a specified format.

1. Define Your Patterns (patterns.yaml)

Create a YAML or JSON file (e.g., patterns.yaml) to specify your extraction rules. Each pattern requires a name, a regex or keywords definition, and an output_format.

Here's an example patterns.yaml:

patterns:
  - name: "ServerError"
    regex: "ERROR (?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<level>\w+): (?P<message>.+) in file (?P<file>.+) at line (?P<line>\d+)"
    output_format: "json"
  - name: "AccessLogEntry"
    regex: '(?P<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - - \[(?P<datetime>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) (?P<protocol>\S+)" (?P<status>\d+) (?P<size>\d+)'
    output_format: "csv"
  - name: "InfoMessage"
    keywords:
      - "INFO"
      - "Application started"
    output_format: "json"

2. Prepare Your Input Data (sample.log)

Create a file with the semi-structured text you want to extract data from. For example, sample.log:

INFO 2023-10-27 10:00:01: Application started.
ERROR 2023-10-27 10:05:30 ERROR: Failed to connect to DB in file database.py at line 123
192.168.1.1 - - [27/Oct/2023:10:06:01 +0000] "GET /index.html HTTP/1.1" 200 1234
WARN 2023-10-27 10:07:15: Low disk space warning.
ERROR 2023-10-27 10:10:00 ERROR: Null pointer exception in file worker.py at line 45

3. Run the Extractor

Execute the pattern-extractor command-line tool, specifying your configuration file and input files. You can also redirect standard input.

pattern-extractor --config patterns.yaml --input sample.log --output extracted_data.json

Command-line Options:

• --config <file>: Path to the pattern configuration file (YAML or JSON).
• --input <file> [<file> ...]: One or more input files to process. If not provided, reads from stdin.
• --output <file>: Path to the output file. If not provided, prints to stdout.
• --format <csv|json|yaml>: Override the output format specified in patterns (or set a global default). This applies if a pattern doesn't specify its own output_format. Default is JSON.
• --verbose: Enable verbose output for debugging.

4. View Extracted Data (extracted_data.json)

After running the command, extracted_data.json will contain structured data extracted according to your patterns.

[
  {
    "pattern_name": "InfoMessage",
    "original_line": "INFO 2023-10-27 10:00:01: Application started.",
    "extracted_data": {}
  },
  {
    "pattern_name": "ServerError",
    "original_line": "ERROR 2023-10-27 10:05:30 ERROR: Failed to connect to DB in file database.py at line 123",
    "extracted_data": {
      "timestamp": "2023-10-27 10:05:30",
      "level": "ERROR",
      "message": "Failed to connect to DB",
      "file": "database.py",
      "line": "123"
    }
  },
  {
    "pattern_name": "AccessLogEntry",
    "original_line": "192.168.1.1 - - [27/Oct/2023:10:06:01 +0000] \"GET /index.html HTTP/1.1\" 200 1234",
    "extracted_data": {
      "ip": "192.168.1.1",
      "datetime": "27/Oct/2023:10:06:01 +0000",
      "method": "GET",
      "path": "/index.html",
      "protocol": "HTTP/1.1",
      "status": "200",
      "size": "1234"
    }
  },
  {
    "pattern_name": "ServerError",
    "original_line": "ERROR 2023-10-27 10:10:00 ERROR: Null pointer exception in file worker.py at line 45",
    "extracted_data": {
      "timestamp": "2023-10-27 10:10:00",
      "level": "ERROR",
      "message": "Null pointer exception",
      "file": "worker.py",
      "line": "45"
    }
  }
]

For patterns configured with output_format: "csv", the output for those specific extractions would be formatted as CSV rows.