Skip to content

Quick Start

Jump to bottom
sat edited this page Jul 15, 2026 · 1 revision

Quick-Start

This page walks through the smallest realistic end-to-end logdag analysis, using the sample SSH log data bundled in the tutorial/ directory of the logdag repository.

The tutorial uses a 2,000-line SSH log excerpt from loghub (SSH_2k.log). It requires amulog to be installed. See Installation if you have not installed logdag yet.

For background on what each step does in the full pipeline, see Overview and Pipeline. For deeper detail on configuration options see Configuration, and for DAG generation details see Generating DAGs.

Overview of the steps

1. Build an amulog DB   (amulog db-make)
2. Write a logdag config file
3. Build the time-series feature DB   (python -m logdag.source make-evdb-log-all)
4. Estimate causal DAGs   (python -m logdag make-dag)
5. Show results   (python -m logdag show-list / show-subgraphs)

Step 0 — Prerequisites

You need:

  • logdag installed (including amulog and pcalg). See Installation.
  • The tutorial/ directory from the logdag repository (contains SSH_2k.log, ssh_amulog.conf, ssh_logdag.conf, and a log2seq parser script ssh_parser.py).

Change into the tutorial/ directory before running the commands below.

Step 1 — Build the amulog database

logdag reads log events from an amulog database. Build it from the sample SSH log file with:

amulog db-make -c ssh_amulog.conf

After this command finishes, verify the database with:

amulog show-db-info -c ssh_amulog.conf

Expected output:

[DB status]
Registered log lines : 2000
Term : 2022-12-10 06:55:46+09:00 - 2022-12-10 11:04:45+09:00
Log templates : 23
Log template groups : 23
Hosts : 1

This tells you there are 2,000 log lines from one host (LabSZ), classified into 23 log-template groups. These template groups become the nodes in the causal DAG.

Step 2 — The logdag config file

The tutorial ships with a ready-to-use config file ssh_logdag.conf. Here is what the key sections look like:

[general]
# Date range to match the sample data
evdb_whole_term = 2022-12-10 00:00:00, 2022-12-11 00:00:00
evdb_unit_term  = 24h
evdb_unit_diff  = 24h
evdb_binsize    = 1m
log_source      = amulog
evdb            = sql          # SQLite3 feature DB (no extra install needed)

[database_amulog]
source_conf = ssh_amulog.conf  # path to the amulog config
event_gid   = ltid             # use log-template ID as event group ID

[database_sql]
database         = sqlite3
sqlite3_filename = ssh_logdag.db

[dag]
# Same date range as [general]
whole_term      = 2022-12-10 00:00:00, 2022-12-11 00:00:00
source          = log
area            = all
unit_term       = 24h
unit_diff       = 24h
ci_func         = gsq          # G-squared test for binary/discrete data
cause_algorithm = pc           # PC algorithm (default)
output_dir      = ssh_result
output_dag_format = json

The [database_amulog] section points to the amulog config (source_conf). That is how logdag locates the log event database built in Step 1.

For your own data you would adjust the evdb_whole_term and whole_term date ranges to match your log data, and update source_conf to point to your amulog config file. See Configuration for all options and Configuration Options for the full reference.

Step 3 — Build the time-series feature database

logdag first converts the raw log events into a time-series feature database (evdb). This is done with the logdag.source sub-tool:

python -m logdag.source make-evdb-log-all -c ssh_logdag.conf

This command reads occurrence-time sequences from the amulog DB, applies filters (periodicity removal, linear-trend removal as configured in [filter]), and stores the resulting per-(host × log-template) time series in ssh_logdag.db (SQLite3).

Depending on the size of the log dataset this may take from a few seconds to several minutes.

Step 4 — Estimate causal DAGs

Once the feature database is ready, run causal discovery:

python -m logdag make-dag -c ssh_logdag.conf

logdag splits the analysis term (whole_term) into job windows based on unit_term / unit_diff, runs the PC algorithm on each window, and writes the resulting LogDAG objects to the ssh_result/ directory.

The make-dag subcommand supports parallel execution with -p N (N processes). For large datasets, -p 4 or higher reduces wall-clock time:

python -m logdag make-dag -c ssh_logdag.conf -p 4

Step 5 — Show results

List all DAGs

python -m logdag show-list -c ssh_logdag.conf

Expected output:

all_20221210 23 14

Each line shows: DAG name (area + date), number of event nodes, number of causal edges found. Here there is one DAG (all_20221210, the single day in the sample) with 23 nodes and 14 edges.

Show causal subgraphs (text)

python -m logdag show-subgraphs -c ssh_logdag.conf all_20221210

Expected output (excerpt):

Subgraph 0 (4 nodes)
8@LabSZ:8 -> 0@LabSZ:0
8@LabSZ:8 <-> 9@LabSZ:9
6@LabSZ:6 -> 0@LabSZ:0
...

Edge notation: <eid>@<host>:<ltid> -> <eid>@<host>:<ltid>. -> is a directed causal edge; <-> is a bidirected (undirected) edge where the orientation could not be determined.

Add --detail to see representative log messages for each node:

python -m logdag show-subgraphs --detail -c ssh_logdag.conf all_20221210

Plot the DAG

python -m logdag plot-dag -c ssh_logdag.conf -o output.png all_20221210

Add -f no_isolated to omit nodes without any edges:

python -m logdag plot-dag -c ssh_logdag.conf -o output.pdf -f no_isolated all_20221210

The graph is written to the specified file. Supported formats depend on matplotlib (png, pdf, svg, etc.).

Available subcommands at a glance

The full list of logdag subcommands:

$ python -m logdag -h
usage: __main__.py SUBCOMMAND [options and arguments] ...

subcommands:
  dump-events: Output event node definition in readable format
  dump-input: Output causal analysis input in pandas csv format
  make-args: Initialize arguments for pc algorithm
  make-dag: Generate causal DAGs
  make-dag-stdin: make-dag interface for pipeline processing
  plot-dag: Generate causal DAG view
  plot-node-ts: Generate node time-series view
  show-args: Show arguments recorded in argument file
  show-deafult-config: Show configuration defaults
  show-edge: Show edges related to given conditions
  show-edge-list: Show all edges in a DAG
  show-full-config: Show virtual configuration considering defaults
  show-group-stats: Show stats classified by amulog tags
  show-list: Show abstracted results of DAG generation
  show-netsize: Show distribution of connected subgraphs in DAGs
  show-netsize-list: Show connected subgraphs in every DAG
  show-node-list: Show node definitions of the input
  show-node-ts: Show time-series of specified nodes in csv format
  show-stats: Show sum of nodes and edges
  show-stats-by-threshold: Show sum of edges by thresholds
  show-subgraphs: Show edges in each connected subgraphs
  update-event-label: Overwrite labels for log events

try "__main__.py SUBCOMMAND -h" to refer detailed subcommand usage

And for the feature-DB building tool:

$ python -m logdag.source -h
usage: __main__.py SUBCOMMAND [options and arguments] ...

subcommands:
  drop-features: Drop feature data (except original data) in feature DB
  make-evdb-log-all: Load log data from amulog and store features
  make-evdb-snmp: Load telemetry data from rrd and store features
  make-evdb-snmp-all: Load telemetry data from rrd and store features
  make-evdb-snmp-org: Load telemetry data from rrd and store
  make-evdb-snmp-tests: Store 1 feature from a specified source
  show-snmp-stats: Show event counts in telemetry features

See CLI Commands for a full reference of all subcommands and their options across all four CLI tools (logdag, logdag.source, logdag.eval, logdag.visual).

Using your own log data

For log data other than the tutorial sample, the basic workflow is the same. You need to:

  1. Build an amulog database for your logs (see the amulog documentation).
  2. Write a logdag config file:
    • Set [database_amulog] source_conf to your amulog config.
    • Set the date ranges in [general] evdb_whole_term and [dag] whole_term to span your log data.
    • Adjust evdb_unit_term, unit_term, and unit_diff to control the analysis window size (e.g., 24h for day-by-day).
    • Choose a feature-DB backend in [general] evdb (sql for SQLite3, the default; influx for InfluxDB v1; influx3 for InfluxDB 3 Core). See Data Sources for backend details.
  3. Run Steps 3–5 above with your config file.

A useful starting point is to copy logdag/data/config.conf.default (shipped with the package) and adapt it. You can see all effective settings with:

python -m logdag show-full-config -c your_config.conf

Where to go next

  • Overview and Pipeline — the full three-phase pipeline and what each step does internally.
  • Generating DAGs — job splitting, area configuration, parallelism, and output layout.
  • Configuration — config-file model, key sections, and the relation to the amulog config.
  • Configuration Options — full option reference derived from config.conf.default.

Clone this wiki locally