Skip to content
Hayabusa: Simple and Fast Full-Text Search Engine for Massive System Log Data
CSS HTML Python TeX JavaScript Makefile
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
presentation/was add was research group presentaionw Dec 20, 2017
config.ini add ini file Apr 27, 2017
logrotate-crontab fix bug and add store cron May 27, 2017


Hayabusa: A Simple and Fast Full-Text Search Engine for Massive System Log Data


  • Pure python implement
  • Parallel SQLite processing engine
  • SQLite3 FTS(Full Text Search)
  • Core-scale architecture


  • Design of the directory structure

    • By specifying a search range of time in ”the directory path + yyyy + mm + dd + hh + min.db”, the search program can select the search time systematically.
  • StoreEngine

    • sample code
    import os.path import sqlite3
    db_file = ’test.db’ log_file = ’1m.log’
    if not os.path.exists(db_file):
        conn = sqlite3.connect(db_file) conn.execute("CREATE VIRTUAL TABLE SYSLOG USING FTS3(LOGS)");
    conn = sqlite3.connect(db_file)
    with open(log_file) as fh:
        lines = [[line] for line in fh] 
        conn.executemany(’INSERT INTO SYSLOG VALUES ( ? )’, lines) 
  • SearchEngine

    • sample command
    $ python -h
    usage: [-h] [--time TIME] [--match MATCH] [-c] [-s] [-v]
    optional arguments:
      -h, --help     show this help message and exit
      --time TIME    time explain regexp(YYYY/MM/DD/HH/MIN). eg: 2017/04/27/10/*
      --match MATCH  matching keyword. eg: noc or 'noc Login'
      -e             exact match
      -c             count
      -s             sum
      -v             verbose
     $ python --time 2017/05/11/13/* --match 'keyword' -c 
  • Architecture image Hayabusa Architecture

Search condition

  • case-insensitive

    • no distinguish uppercase or lowercase
  • Exact match

    -e --match ''
  • AND

    --match 'Hello World'
  • OR

    --match 'Hello OR World'
  • NOT

    --match 'Hello World -Wide'

    --match '"Hello World"'
    --match '\"\"' <- IP address case(same as -e flag)
    --match '\"\" src sent' <- PHRASE + AND search
  • asterisk(*)

    --match 'H* World'
  • HAT

    --match '^Hello World'

Development environment

  • CentOS 7.3
  • Python 3.5.1(use anaconda packages)
  • SQLite3(version 3.9.2)

Dependency softwares

  • Python 3
  • SQLite3
  • GNU Parallel


Compare with Apache Spark

  • Hayabusa and Spark time comparison

  • Comarison of distributes Spark environment and the stand-alone Hayabusa

You can’t perform that action at this time.