Skip to content

Overview and Pipeline

Jump to bottom
sat edited this page Jul 15, 2026 · 1 revision

Overview and Pipeline

This page gives the end-to-end big picture of logdag: what it does, where it sits in its ecosystem, and how data flows from raw log events to a causal DAG and its downstream consumers.

For deeper, task-specific detail, follow the cross-links to Generating DAGs, Causal Discovery Methods, Data Sources, and Architecture Overview.

What logdag does

logdag infers a causal DAG (Directed Acyclic Graph) among time-series events extracted from system logs (syslog and similar). The goal is to estimate cause-effect relationships between log-event types — for example, which event tends to trigger which other event across hosts — for tasks such as syslog analysis and network fault root-cause estimation. It is a batch-oriented CLI tool aimed at research use.

The work happens in two conceptual stages:

  1. Time-series event extraction. Structured logs from amulog (occurrence-time sequences keyed by log-template group/ID) — and optionally SNMP/RRD numeric metrics — are turned into per-(host x event) time-series features. Optional filters can remove periodicity or linear trend.
  2. Causal DAG inference. The extracted time series are discretized into bins, assembled into a multivariate dataset, and fed to a causal-discovery algorithm (PC / LiNGAM) to estimate a DAG over the events. Prior knowledge (e.g. network topology) can constrain the search space.

The inferred DAG is then available for display and statistics, visualization and DAG-to-DAG comparison, and matching against trouble tickets for evaluation.

Ecosystem position

logdag does not stand alone; it is one component in the cpflat log-analysis toolchain.

        +------------------+        +--------------------------------+
        |     amulog       |        |             logdag             |
        | (log templating, | -----> | (causal DAG inference +        |
        |  log event DB)   |  logs  |  built-in display/eval/visual) |
        +------------------+        +--------------------------------+
              INPUT LAYER             output: NetworkX DiGraph
  • Input layer: amulog. logdag depends strongly on amulog (amulog>=0.5.0). amulog provides the log database (occurrence times per log template), and logdag also delegates configuration management, CLI dispatch, log-DB access, and anonymization to amulog's common infrastructure. logdag reads the amulog config path through its own [database_amulog] section.
  • Output: NetworkX DiGraph. The estimated causal graph is represented as a NetworkX DiGraph, wrapped in logdag's LogDAG container (graph plus the event-definition map), and persisted as pickle or JSON.
  • Visualization. logdag has a built-in visual subpackage (logdag.visual) for DAG comparison, scoring, and drawing of the resulting graphs — see Viewing and Filtering DAGs.
  • Causal-discovery libraries. logdag is method-agnostic — the algorithm is chosen by config ([dag] cause_algorithm), and each method brings its own library: PC uses pcalg (with gsq for the binary G-squared independence test), and LiNGAM uses the lingam library. Neither is privileged. See Causal Discovery Methods.

The three-phase pipeline

logdag's processing splits into three phases: (1) build the feature database, (2) generate the DAG, and (3) post-process (display / evaluate / visualize).

  PHASE 1: FEATURE DB (evdb)            PHASE 2: DAG GENERATION         PHASE 3: POST-PROCESSING
  CLI: logdag.source                    CLI: logdag (make-dag)          CLI: logdag / logdag.eval / logdag.visual

  +-------------------+
  | amulog log DB     |--+
  | (src_amulog)      |  |   +---------------------+
  +-------------------+  +-->| evgen_log /         |     +-----------------+
  +-------------------+      | evgen_snmp          |---->| feature DB      |
  | RRD files (SNMP)  |----->| (+ filter_log /     |     | (evdb backend): |--+
  | (src_rrd)         |      |  evpost)            |     | sql / influx /  |  |
  +-------------------+      +---------------------+     | influx3         |  |
                                                        +-----------------+  |
                                                                             v
                                          +-------------------------------------------------+
                                          | log2event.makeinput()                           |
                                          |   -> pandas DataFrame (rows = time bins,         |
                                          |      cols = event IDs) + EventDefinitionMap      |
                                          +-------------------------------------------------+
        use_evdb=false                              |
        (amulog direct read,  ......................+   (optional) merge_sync_event()
         bypasses evdb)                             v
                                          +---------------------------+
                                          | pknowledge                |
                                          | init_prior_knowledge()    |
                                          +---------------------------+
                                                      v
                                          +---------------------------+      +------------------------+
                                          | makedag.estimate_dag()    |----->| method adapters:       |
                                          | (dispatch on              |      |  pc_input  (PC/pcalg)  |
                                          |  cause_algorithm)         |      |  lingam_input          |
                                          +---------------------------+      +------------------------+
                                                      v
                                          +---------------------------+
                                          | showdag.LogDAG            |
                                          | (NetworkX DiGraph +       |
                                          |  EventDefinitionMap)      |
                                          | saved as pickle | json    |
                                          +---------------------------+
                                                      |
                  +-----------------------------------+-----------------------------------+
                  v                                   v                                   v
       +--------------------+            +------------------------+          +------------------------+
       | display & stats    |            | evaluation             |          | visualization &        |
       | showdag /          |            | logdag.eval            |          | comparison             |
       | showdag_filter     |            | (match_edge, trouble)  |          | logdag.visual          |
       | (show-* / plot-*)  |            |                        |          | (scoring / draw)       |
       +--------------------+            +------------------------+          +------------------------+

Phase 1 — Feature DB (evdb) construction

Run via the logdag.source CLI. See Data Sources for the full treatment.

  1. Source. src_amulog.AmulogLoader reads occurrence-time sequences from the amulog DB; SNMP numeric metrics come from RRD files via src_rrd.

  2. Event definition & aggregation granularity. Each event (one time series, which becomes one node of the DAG) is defined along two orthogonal aggregation axes, both set in [database_amulog]:

    • log-template axis — event_gid: ltgid (amulog template group, the default) or ltid (the raw per-template id). ltgid gives fewer, coarser event types.
    • host axis — host_tier: empty = keep the original host (legacy); a tier name aggregates events by the amulog host-group at that tier (e.g. midplane / rack). Coarser granularity on either axis means fewer, larger events — the main knob for keeping the causal-discovery problem tractable.

  3. Feature extraction & filtering. evgen_log.LogEventLoader / evgen_snmp.SNMPEventLoader turn those occurrences into time-series features, optionally applying filter_log.LogFilter (periodicity / linear-trend removal) or evpost (difference / anomaly score).

  4. Feature storage (evdb backend). The selected evdb backend stores the features, chosen by [general] evdb and dispatched in source/evgen_common._init_evdb:

    • sql — SQLite3 / MySQL (source/sqlts.py)
    • influx — InfluxDB v1 (source/influx.py)
    • influx3 — InfluxDB 3 Core (source/influx3.py)

    The influx3 backend is present and wired in the current code, but it is newer and less battle-tested than the sql and influx (v1) backends; treat it as not yet at full production parity. See Data Sources.

use_evdb=false bypasses the feature DB entirely and reads features directly from amulog at DAG-generation time (LogEventLoaderDirect), shown as the dashed path in the diagram above.

Phase 2 — DAG generation

Run via the logdag CLI (make-dag and related subcommands). See Generating DAGs.

  1. arguments.all_args() splits the whole analysis term (whole_term) into job windows by unit_term / unit_diff and enumerates jobs as (conf, dt_range, area).
  2. For each job, makedag.makedag_main() orchestrates the flow:
    • log2event.makeinput() reads all events from the evdb (or directly from amulog when use_evdb=false), bins them (sequential / slide / radius), and builds a pandas DataFrame with rows = time bins and columns = event IDs, together with an EventDefinitionMap (the eid <-> event-definition mapping). If there is no data it returns (None, None).
    • Optionally, merge_sync_event() merges identically-timed events into a MultipleEventDefinition.
    • pknowledge.init_prior_knowledge() builds the prior-knowledge graph / constraints (see below).
    • makedag.estimate_dag() dispatches on the [dag] cause_algorithm setting to the appropriate method adapter.
  3. The result (a NetworkX DiGraph) is packaged in showdag.LogDAG and saved to <output_dir>/<area>_<date>/ as pickle or JSON.

Phase 3 — Post-processing

  • Display & statistics (logdag show-* / plot-*): load a LogDAG, apply showdag.apply_filter with the filter functions in showdag_filter (no_isolated, directed, across_host, ate_prune, ...), and emit edge listings, statistics, or plots. See Viewing and Filtering DAGs.
  • Evaluation (logdag.eval): manage trouble tickets with trouble.TroubleManager and match DAG edges against tickets with match_edge.match_edges().
  • Visualization & comparison (logdag.visual): TF-IDF edge scoring, DAG similarity and clustering (edge_search), set operations between two DAG configurations (comparison), and graph drawing (draw).

Causal discovery via method adapters

makedag.estimate_dag() is a single dispatch point that selects a causal-discovery method based on [dag] cause_algorithm. Each method lives in its own adapter module so that the algorithm-specific details (library calls, parameters, prior- knowledge feed) are isolated:

cause_algorithm Adapter module Method
pc pc_input.py PC algorithm (via pcalg)
pc-corr pc_input.py PC with correlation-based variant
lingam lingam_input.py LiNGAM (ICA-LiNGAM / DirectLiNGAM)
lingam-corr lingam_input.py LiNGAM correlation variant

See Causal Discovery Methods for parameters (independence-test choices, thresholds) and details of each method.

Prior knowledge

pknowledge lets you constrain the causal search with domain knowledge such as network topology. init_prior_knowledge() builds a PriorKnowledge object that holds edge / no-edge / path / no-path constraints, and exposes them to the adapters through two channels: pruned_initial_skeleton() (an initial graph for PC) and lingam_prior_knowledge() (a constraint matrix for LiNGAM). This is optional; without it the search runs unconstrained.

Where each capability acts

The pipeline is the organizing spine: each tunable belongs to exactly one stage. This table places the main capabilities by the stage they affect (config keys in parentheses; see Configuration Options for the full set).

Stage Capability Config
Source log events (amulog) and/or SNMP metrics (RRD) [general] log_source / snmp_source, [dag] source
Event definition & granularity aggregation axes — log-template (ltgid/ltid) and host (host-group tier) [database_amulog] event_gid, host_tier
Feature extraction periodicity / linear-trend filtering, diff / anomaly post-processing [filter] rules, evpost
Feature storage (evdb) backend sql / influx (v1) / influx3, or bypass and read amulog directly [general] evdb, use_evdb
Input matrix time binning (sequential / slide / radius), sync-event merge [dag] ci_bin_method, merge_syncevent
Prior knowledge topology / imported constraints [prior_knowledge] methods
Causal discovery method (PC / LiNGAM), independence test [dag] cause_algorithm, ci_func
Post-processing DAG filters, ticket matching, visualization showdag_filter, logdag.eval, logdag.visual

The two aggregation axes are siblings (both in [database_amulog]): host stratification (host_tier) is not a "data source" but the host-axis counterpart of log-template grouping (event_gid). Coarser granularity on either axis shrinks the causal-discovery problem.

See Data Sources for input sources and evdb backends in detail, and Configuration Options for every config key.

Where to go next

Clone this wiki locally