Architecture Overview

This page is the developer-facing map of logdag: how the package is laid out, what each module is responsible for, the core data structures that flow through the pipeline, and the external dependencies that the design leans on. If you are a user who just wants to run logdag, start with Overview and Pipeline and Generating DAGs; this page assumes you intend to read or extend the code. To actually add a method or a data source, see Extending logdag.

Grounding: everything here is written against the current source tree (logdag.__version__ == "0.3.1") — logdag/makedag.py, log2event.py, showdag.py, arguments.py, pknowledge.py, the method adapters (pc_input.py, lingam_input.py), and the logdag/source/ subpackage. Contracts that are not 100% obvious from a single read are flagged inline.

1. What logdag is, structurally

logdag turns time-series events (primarily syslog templates produced by amulog, optionally SNMP/RRD numeric metrics) into a causal DAG among those events. The output is a networkx.DiGraph wrapped in a LogDAG container. It is a batch-oriented research/operations CLI tool, not a service.

The processing splits into three phases, and the package layout mirrors them:

Feature extraction (input layer) — the logdag.source subpackage reads a source (amulog log DB or SNMP/RRD), turns occurrences into per-(host, group) time series, optionally filters them, and stores them in a feature database (evdb). See Data Sources.
DAG generation (core) — the main package (makedag / log2event / pknowledge / the *_input adapters) loads features into a binned pandas DataFrame, applies a causal-discovery algorithm, and saves the resulting graph. See Generating DAGs and Causal Discovery Methods.
Post-processing — showdag / showdag_filter (inspection, filtering, statistics), logdag.eval (matching edges against trouble tickets), and logdag.visual (comparison, scoring, drawing). See Viewing and Filtering DAGs.

logdag depends heavily on amulog (amulog>=0.5.0): it delegates config loading, CLI dispatch (amulog.cli), the log database access layer (amulog.log_db), SQL helpers (amulog.db_common), anonymization, and host-group resolution. logdag does not re-implement these; understanding logdag means knowing that the data layer and config layer are amulog's.

2. Package layout

logdag/                      # main package — DAG generation & presentation
├── __init__.py              # __version__ = "0.3.1"
├── __main__.py              # CLI: `logdag` (make-dag, show-*, plot-*, …)
├── arguments.py             # config loading, job enumeration, path resolution
├── makedag.py               # per-job DAG-generation orchestration + dispatch
├── log2event.py             # event-definition classes, evmap, input DataFrame
├── pknowledge.py            # prior-knowledge generation (topology, import, …)
├── pc_input.py              # PC-algorithm adapter (pcalg)
├── lingam_input.py          # LiNGAM adapter
├── showdag.py               # LogDAG container + result presentation
├── showdag_filter.py        # named DAG filter functions (CLI `-f`)
├── dtutil.py                # datetime ranges & discretization (pure logic)
├── data/config.conf.default # default config values (the config "schema")
├── source/                  # input sources & feature DB (evdb)
│   ├── __main__.py          # CLI: `logdag.source` (build evdb)
│   ├── evgen_common.py      # EventLoader ABC + evdb backend factory
│   ├── evgen_log.py         # log feature loading (LogEventLoader[Direct])
│   ├── evgen_snmp.py        # SNMP feature loading
│   ├── filter_log.py        # time-series filtering (periodic / linear / …)
│   ├── sqlts.py             # TimeSeriesDB ABC + SQLite3/MySQL backend
│   ├── influx.py            # InfluxDB v1 backend
│   ├── influx3.py           # InfluxDB v3 Core backend (stdlib urllib)
│   ├── src_amulog.py        # amulog log-DB adapter (AmulogLoader)
│   ├── src_rrd.py           # RRD source adapter
│   ├── evpost.py            # feature post-processing (diff / anomaly)
│   └── convert.py           # timestamp <-> DataFrame helpers
├── eval/                    # CLI: `logdag.eval` (ticket matching)
└── visual/                  # CLI: `logdag.visual` (comparison / scoring / draw)

The main package owns the DAG-generation core; source is the input side, eval and visual are downstream tools. The four __main__.py files are the four console scripts (logdag, logdag.source, logdag.eval, logdag.visual); each one defines a DICT_ARGSET of subcommands and hands it to amulog.cli.main. See CLI Commands for the full command list.

3. Module responsibilities (core package)

Module	Key symbols	Responsibility
`arguments.py`	`ArgumentManager`, `open_logdag_config()`, `all_args()`, `args2name()`	Open and merge the config (defaults + user). Enumerate jobs `(conf, dt_range, area)` by splitting `whole_term` into `unit_term`/`unit_diff` windows × areas. Serialize the job list (`args` file) and resolve all output/cache paths.
`makedag.py`	`makedag_main()`, `estimate_dag()`, `make_input()`	Run one job end-to-end: build the input, build prior knowledge, dispatch to the selected causal-discovery method, and package the result as a `LogDAG`. `estimate_dag` is the algorithm dispatcher (`cause_algorithm`).
`log2event.py`	`EventDefinition`, `MultipleEventDefinition`, `EventDefinitionMap`, `makeinput()`, `load_event()`, `merge_sync_event()`	The data model. Defines what an "event" is, maps event ids ⇄ definitions, and builds the binned input `DataFrame` from the loaded series. Detailed in §4.
`pknowledge.py`	`PriorKnowledge`, `KnowledgeGenerator`, `init_prior_knowledge()`	Build optional prior-knowledge constraints (network topology, imported DAGs, host independence). Exposes two feeds: an initial skeleton graph (PC) and a LiNGAM prior-knowledge matrix.
`pc_input.py`	`pc()`, `estimate_dag()`, `binarize_input()`, `_build_skeleton_args()`	Wrap `pcalg`. Choose the conditional-independence test, optionally binarize, estimate a skeleton, and return a CPDAG (a `DiGraph` in which an undirected edge is two opposite arcs).
`lingam_input.py`	`estimate()`, `estimate_corr()`	Wrap the `lingam` library (ICA / Direct LiNGAM). Returns a weighted `DiGraph` (edge `weight` = coefficient).
`showdag.py`	`LogDAG`, `apply_filter()`, `iter_results()`, `show_edge*()`, `stat_groupby()`	The result container and all presentation logic. Detailed in §4.3.
`showdag_filter.py`	`FUNCTIONS`, `no_isolated`, `directed`, `across_host`, `ate_prune`, …	The named DAG filter functions selected from the CLI / config; resolved by name in `showdag.apply_filter`.
`dtutil.py`	`range_dt()`, `discretize_sequential/slide/radius()`, `iter_term()`	Pure datetime / discretization helpers used during binning. No I/O.

4. Data model

This is the part of the architecture worth reading carefully: four data structures carry everything through the pipeline.

4.1 The job tuple `args = (conf, dt_range, area)`

A single DAG-generation unit is identified by a 3-tuple, used positionally everywhere:

conf — the merged configparser-style config object.
dt_range — a (start, end) pair of tz-aware datetimes (one time window).
area — a string scoping which hosts participate ("all", "each", or a named host group; resolved by log2event.AreaTest).

arguments.all_args(conf) enumerates the full job list by stepping whole_term in unit_diff increments, each window of length unit_term, crossed with every configured area. arguments.args2name(args) derives a stable job name "<area>_<YYYYmmdd[_HHMMSS]>" used for the on-disk output directory. The job name is parsed back to a job tuple by jobname2args (the regex tolerates an area that itself contains underscores, e.g. host_xxx).

Contract: args is a plain tuple accessed by position (args[0] is conf, etc.). There is no dataclass/namedtuple wrapper; callers unpack it as conf, dt_range, area = args.

4.2 Event definitions and the event map (`log2event.py`)

An event is one time series — a (source, host, group, …) combination whose occurrences over time become one column in the input matrix.

EventDefinition (ABC) — base class carrying source, host, group. Concrete subclasses live in the source loaders, e.g. evgen_log.LogEventDefinition (adds gid, the amulog template-group id) and the SNMP equivalent. Each subclass implements:
- identifier — a stable string id used for equality and as the evmap key (for log events: "<host>:<gid>").
- event() — the source-agnostic event label (for log: str(gid)).
- series() — returns (measure, tags) used to read/write the evdb (for log: ("log_feature", {"host": ..., "key": ...})).
- all_attr(key) — returns the set of values for an attribute (a set so that MultipleEventDefinition can hold several).
MultipleEventDefinition — a composite event produced by merge_sync_event: several member events whose time series are identical (and optionally share source/host/group). Its identifier is the sorted join of member identifiers; all_attr returns the union over members. Code that inspects an event's host must use all_attr("host") rather than .host, because a composite has no single host.
EventDefinitionMap (evmap) — the bidirectional map between integer event ids (eid, 0-based, dense) and event definitions. add_evdef assigns the next id; evdef(eid) and get_eid(evdef) convert each way (the reverse map is keyed by identifier). The evmap is the bridge between graph nodes (integers) and their meaning. It is pickled per job to <job_dir>/evdef.pickle (dump(args) / load(args)); load falls back to a legacy path if the modern pickle is unreadable.

4.3 The input DataFrame (binarization / binning)

log2event.makeinput(conf, dt_range, area) is the core of input construction:

For each configured source it iterates event definitions (load_event_all), reads each series from the evdb (or directly from amulog when [general] use_evdb = false), and bins it into a one-column DataFrame via load_event.
The binning method is [dag] ci_bin_method:
- sequential — fixed, non-overlapping bins of width ci_bin_size.
- slide — overlapping bins stepping by ci_bin_diff < ci_bin_size.
- radius — bins centered on the step grid with a ±0.5·ci_bin_size radius. Empty / all-zero series are dropped.
Each surviving series is assigned an eid in the evmap, its single column is renamed to that eid, and the columns are concatenated.

The result is the input DataFrame: rows = time bins (a tz-aware DatetimeIndex), columns = event ids (eid), values = per-bin occurrence counts. This matrix is what every causal-discovery method consumes.

Empty-data contract: when no series survive, makeinput returns (None, None). Every caller must handle that — makedag_main returns None, and aggregating callers must skip None (with sparse data most windows have no events).

Binarization is method-specific, not part of makeinput: the PC adapter with a binary CI test calls pc_input.binarize_input(df), which maps each value to 1 if x >= 1 else 0. LiNGAM uses the raw counts.

Optional sync-event merging ([dag] merge_syncevent) collapses columns with identical value vectors into one MultipleEventDefinition, controlled by merge_syncevent_rules (which of source/host/group must also match). This reduces redundant nodes before causal discovery.

4.4 The result container `LogDAG` (`showdag.py`)

LogDAG wraps the estimated graph together with the context needed to interpret it:

LogDAG(args, graph=None, evmap=None) — holds the job tuple (and unpacks conf, dt_range, area), the networkx.DiGraph, and (lazily) the evmap.
Persistence — dump() / load() write/read <output_dir>/<jobname>/dag.{pickle|json} ([dag] output_dag_format). The JSON form uses networkx.node_link_data / node_link_graph. The evmap is stored separately (evdef.pickle); a freshly loaded LogDAG lazily loads the evmap on first use.
Edge directedness — the graph encodes undirected edges as a pair of opposite arcs. edge_isdirected((u, v)) is True iff the reverse arc is absent; helpers like edges_by_direction() and the directed/undirected filters split a graph on this. "No-duplication" edge iteration (remove_edge_duplication) yields each undirected edge once.
Node ⇄ event resolution — node_evdef(node), evdef2node(evdef), edge_evdef(edge) translate between integer nodes and event definitions via the evmap. With [database_amulog] use_anonymize_mapping = true, the evmap is remapped on read to restore original (de-anonymized) host names (restore_host per source loader); original=True/False selects the restored vs. the as-analyzed view.
Presentation — node_str, edge_str, edge_detail, show_edge*, show_subgraphs, stat_groupby, etc. format edges/nodes for the CLI.

Module-level helpers complete the read API: iter_results(conf, area=None) iterates every saved LogDAG for a config, apply_filter(ldag, names, th) applies a chain of named filters, and empty_dag() returns an empty DiGraph.

5. Data flow

                 amulog log DB                    SNMP / RRD
                      │                                │
        src_amulog.AmulogLoader              src_rrd / evgen_snmp
                      │                                │
            evgen_log.LogEventLoader  ── filter_log ──┘
                      │  (feature extraction)
                      ▼
          evdb backend (TimeSeriesDB)
          sql / influx (v1) / influx3            ── Phase 1: build evdb
                      │                              (CLI: logdag.source)
   use_evdb=false ───┤  (bypass evdb, read amulog directly)
                      ▼
        log2event.makeinput()
        → input DataFrame (rows=bins, cols=eid) + evmap
                      │
        pknowledge.init_prior_knowledge()   (optional constraints)
                      │
        makedag.estimate_dag()  ── cause_algorithm ──┐
                      │                               │      ── Phase 2: estimate
            pc_input / lingam_input      (CLI: logdag make-dag)
                      ▼
        showdag.LogDAG  (networkx DiGraph + evmap)
        → <output_dir>/<jobname>/dag.{pickle|json}
                      │
        ┌─────────────┼─────────────────┐            ── Phase 3: post-process
        ▼             ▼                  ▼
   showdag /     logdag.eval        logdag.visual
   show-*,-f     (match_edge)       (comparison / scoring / draw)

Phase 1 is optional in the sense that use_evdb = false skips the evdb entirely and reads amulog at generation time; otherwise Phase 1 populates the evdb and Phase 2 reads from it.

6. Dispatch points (why extension touches three files)

Three independent if/elif dispatchers select behaviour from config; adding a capability means editing the matching branch. These are exactly the seams that Extending logdag walks through:

Causal-discovery method — makedag.estimate_dag switches on [dag] cause_algorithm. Reachable values today: pc, lingam, pc-corr, lingam-corr.
Source / feature loader — log2event.init_evloader switches on the source class ("log" / "snmp").
evdb backend — source/evgen_common.EventLoader._init_evdb switches on [general] evdb (influx, influx3, sql/sqlite/mysql).
Prior knowledge — pknowledge.init_prior_knowledge switches on each name in [prior_knowledge] methods (import, topology, multi-topology, independent, additional-source).

The method adapters do not share a single uniform signature — estimate_dag adapts arguments per branch, and prior knowledge is consumed differently by PC (initial skeleton graph) vs. LiNGAM (constraint matrix). Keep that asymmetry in mind when extending.

7. The TimeSeriesDB abstraction (evdb)

All feature-DB backends implement the TimeSeriesDB ABC in source/sqlts.py. The abstract contract (the methods every backend must provide) is:

list_measurements()                          -> list of measurement names
list_series(measure)                         -> list of tag-dicts (distinct series)
list_fields(measure)                         -> list of field names
add(measure, d_tags, d_input, columns)       -> write rows
commit()
get_items(measure, d_tags, fields, dt_range) -> iterator of (datetime, np.array)
get_df(measure, d_tags, fields, dt_range,
       str_bin=None, func=None, fill=None, limit=None) -> DataFrame | None
get_count(measure, d_tags, fields, dt_range) -> int
drop_measurement(measure)

Concrete backends: SQLTimeSeries (SQLite3/MySQL, in sqlts.py), InfluxDBv1 (influx.py), and InfluxDBv3 (influx3.py, the most recent backend; a stdlib-urllib client against the InfluxDB v3 Core HTTP API, needing no extra Python package). Each is constructed by a factory (init_sqlts / init_influx / init_influx_v3) selected by evgen_common._init_evdb.

Two contract details are shared and tested (tests/contract/test_tsdb_contract.py):

get_df(func=None) over an empty range returns None, not an empty frame.
get_count over an empty range returns 0.
get_df returns a wide frame with a tz-aware local DatetimeIndex and field-named columns.

The contract test suite is the safety net that keeps the three backends behaviourally equivalent. Note that some convenience methods (e.g. has_data) exist only on some backends; rely on the ABC methods above for cross-backend code and check the concrete class for anything beyond them. See Data Sources for usage and Extending logdag for adding a backend.

8. External dependencies

Required (requirements.txt / setup.py):

Package	Role
`amulog>=0.5.0`	Log DB, config loading, CLI dispatch, anonymization, host groups — the base every module builds on.
`pcalg>=0.1.9`	PC-algorithm skeleton + CPDAG estimation.
`gsq>=0.1.6`	G-squared CI test for binary data.
`networkx>=2.1`	Graph representation (the DAG is a `DiGraph`).
`pandas`	The binned input matrix and time-series handling.
`numpy` / `scipy`	Numerics, FFT, statistics.
`scikit-learn`	Clustering / feature processing (mainly `visual`).
`lingam`	LiNGAM algorithms.
`statsmodels`, `python-dateutil`	Statistics, datetime/timezone handling.

Optional / conditional (imported lazily where used):

Package	When needed
`influxdb` (`extras_require["influx"]`)	InfluxDB v1 backend (`evdb = influx`). The v3 backend needs no extra package.
`pygraphviz` (+ system graphviz)	Graph drawing in `visual/draw.py` and `LogDAG.graph_nx`.
`Hawkes` (`extras_require["testdata"]`)	Synthetic causal test data (vendored `logdag.causaltestdata`, lazy import).

See Installation for how to install the required set and the optional extras.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Architecture Overview

Architecture Overview

1. What logdag is, structurally

2. Package layout

3. Module responsibilities (core package)

4. Data model

4.1 The job tuple `args = (conf, dt_range, area)`

4.2 Event definitions and the event map (`log2event.py`)

4.3 The input DataFrame (binarization / binning)

4.4 The result container `LogDAG` (`showdag.py`)

5. Data flow

6. Dispatch points (why extension touches three files)

7. The TimeSeriesDB abstraction (evdb)

8. External dependencies

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

Uh oh!

Architecture Overview

Architecture Overview

1. What logdag is, structurally

2. Package layout

3. Module responsibilities (core package)

4. Data model

4.1 The job tuple args = (conf, dt_range, area)

4.2 Event definitions and the event map (log2event.py)

4.3 The input DataFrame (binarization / binning)

4.4 The result container LogDAG (showdag.py)

5. Data flow

6. Dispatch points (why extension touches three files)

7. The TimeSeriesDB abstraction (evdb)

8. External dependencies

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

4.1 The job tuple `args = (conf, dt_range, area)`

4.2 Event definitions and the event map (`log2event.py`)

4.4 The result container `LogDAG` (`showdag.py`)