IDSLab 0.3.2: a shorter README and a figures guide

Documentation only; the code and the evaluation are unchanged, and finished runs are reused.

• A shorter README: overview, quickstart for every system, guides, versions and citation.
• docs/how-it-works.md explains a run with its figures: the design, the splits, the accuracy after
  each stage, the frontier gap and the flow map.
• The project site describes the current builder, results page and fingerprints, and links the DOI.

Install

pip install idslab==0.3.2

Every change is in CHANGELOG.md. Each release is archived on Zenodo with its own DOI; 10.5281/zenodo.23435536 lists them all.

IDSLab 0.3.1: finished runs survive releases; on PyPI

• Finished run units survive releases that do not change the evaluation. Fingerprints carry a results
  version (idslab.results_version.RESULTS_VERSION, now 1) instead of the package version, and it
  changes only with a release that changes what a run computes. Units from 0.2.0 and 0.3.0 count as
  results version 1, so this release reuses them; DONE also records the release that wrote the unit.
• IDSLab is on PyPI: pip install idslab. The install guides for Windows, macOS and GNU/Linux use
  it, so they no longer need the source code; installing from source is described separately for
  changing IDSLab itself.

Install

pip install idslab==0.3.1

Every change is in CHANGELOG.md. Each release is archived on Zenodo with its own DOI; 10.5281/zenodo.23435536 lists them all.

IDSLab 0.3.0

IDSLab is an open-source framework for building and evaluating staged (cascade) network intrusion
detection systems. A design is described as a graph: the stages (binary classifiers, anomaly
detectors and attack-type classifiers), the model in each stage, its decision rule, and the routes
that carry flows between stages. IDSLab fits the design under a fixed protocol and reports what
each stage adds.

Protocol

• Data is split per seed into FIT, CAL, VAL and TEST. Every model, threshold and on/off choice is
  made on VAL, and TEST is read once, after the cascade is frozen.
• The cascade is compared with controls built from its own fitted first stage: at the cascade's
  choices, at its best validation cutoff, and at the cascade's alarm budget. A paired bootstrap per
  run gives the uncertainty.
• Results also include the effect of removing each stage, the path of every test flow, and the
  frontier gap. This compares the cascade's detection rate with the best the first stage can reach
  at the same false-alarm rate by changing its threshold.
• Held-out attack families and attack types seen only in TEST measure detection of unseen attacks.
• Models can come from scikit-learn style libraries, XGBoost, LightGBM, PyOD, PyTorch, custom
  adapters, or score files produced elsewhere.

This version

Version 0.3.0 rebuilds the local web dashboard around experiments: a config file together with
the results written under its name.

• Experiments home: each design is listed with its state and the next step. Results are matched
  to their config by name and stored fingerprints, never by guesswork.
• Guided setup: Data, Cascade, Evaluation, and Review & run, sharing one draft with a YAML view
  that keeps hand-written comments.
• Visual builder: routing by drag, click or keyboard. Missing routes, loops and unreached
  stages are shown while the design stays editable.
• Results page: reads the cascade against its first stage with the per-run bootstrap, in
  percentage points or as a cost. A small positive mean is not reported as an improvement.
• Jobs page: progress taken from the run's own log, and past activity read from the
  workspace files.

The evaluation is unchanged from 0.2.0: the same config and data give the same per-flow
predictions, frozen configurations and controls. Because the version is part of each run's
fingerprint, results produced with 0.2.0 are computed again once under 0.3.0.

Use

Python 3.10 or newer. Install from the tagged source:

pip install git+https://github.com/hashtagRR/IDSLab.git@v0.3.0
idslab example demo
idslab run demo/r1_router_gate.yaml --out demo/runs
idslab dashboard --workspace demo --open

The example writes a small synthetic flow dataset and three reference designs. Its results
demonstrate the method; they are not findings about real network traffic. IDSLab is a research
tool for evaluating designs, not a network sensor.

• Source: https://github.com/hashtagRR/IDSLab
• Documentation: https://hashtagRR.github.io/IDSLab
• Install guides: https://github.com/hashtagRR/IDSLab/blob/main/INSTALL.md
• Changes: https://github.com/hashtagRR/IDSLab/blob/main/CHANGELOG.md

Licence: GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later).

IDSLab 0.2.0: comparison and sweeps

Visual pipeline builder; cost objectives and stage cost; greedy_fixed selection; plugins; idslab compare with paired per-seed differences and idslab sweep; fixes for chained score dependencies and external-score freshness. Runs from 0.1.0 rerun.

Changes

Finished run units from 0.1.0 are not reused: the version is part of every unit's fingerprint, and the
fixes below change what a fingerprint covers.

Added:

• Visual pipeline builder in the dashboard: drag stages onto a canvas, wire outputs to stages or
  terminals, drag score edges for features, and edit every stage setting in a panel (models from a
  catalog with parameter forms, search spaces, pipeline steps, decisions, training rows). A dataset
  panel reads a sample of the data and suggests the label, type and categorical columns. The config is
  written with the loader's rules and validated on every change; node positions are saved next to the
  config (idslab.config.dump_yaml writes configs that read back unchanged).
• Cost objectives: protocol.objective: {cost: {missed_attack: A, false_alarm: B}} chooses on VAL by
  the lowest average cost per flow; {recall_at_fpr: X} by the highest recall within a false alarm
  cap. The objective appears as its own column in every report.
• Stage cost: each run records the share of TEST flows reaching every stage and its scoring time; a
  stage may declare cost:. Reports add the expected cost per flow of the cascade against its first
  stage alone.
• protocol.search: greedy_fixed: later stages with a fixed decision stay on while earlier stages are
  judged. On the synthetic R2 example it raised VAL and TEST accuracy in each of 3 seeds.
• Plugins through the entry point group idslab.plugins: models and pipeline steps for the builder
  catalog, and named objectives.
• idslab compare: experiments side by side, and for experiments on the same data and splits a
  paired comparison per unit on the same TEST flows, with paired bootstrap intervals. Writes
  compare.md, compare.json and compare.html.
• idslab sweep: one base config with named variants or a grid of dotted-key changes; every variant
  is checked first, written as its own config, run, reported and compared.
• Dashboard: Compare and Sweeps pages; compare and sweep run as dashboard jobs.

Fixed:

• Chained score dependencies (a stage that feeds another stage while using a third stage's scores)
  are refused at config load. Their out-of-fold scores were not fully out of fold: the inner stage's
  fold refits used features from models that had seen the held-out fold.
• External score sidecars (.json) are part of the unit fingerprint, so editing a sidecar reruns the
  unit and its provenance is checked again; before, the finished unit was skipped.
• Reports check each unit's external score files against the current ones and leave out units whose
  files changed, naming the reason; before, stale results from replaced files stayed in the report.
• Dashboard on Windows: Stop ends the job's worker processes too, and a path on another drive or with
  a colon is refused with a message instead of an error.

Documentation:

• A short README with an overview figure and a table of guides; INSTALL.md with step-by-step guides
  for Windows, macOS and GNU/Linux from the download to the dashboard; docs/architecture.md,
  docs/limitations.md, docs/compare-and-sweeps.md; mini READMEs for the example designs and the
  dashboard source. Figures are drawn by docs/img/make_figures.py from a real run.
• A project site (docs/index.html, served by GitHub Pages from docs/).
• IDSLab is not on PyPI yet; the guides install it from GitHub
  (pip install git+https://github.com/hashtagRR/IDSLab.git).

Install

This version is not on PyPI; install it from its tag:

pip install "git+https://github.com/hashtagRR/IDSLab.git@v0.2.0"

The current version installs with pip install idslab. Every change is in CHANGELOG.md.

IDSLab 0.1.0: first public release

First public release. Any acyclic cascade of classifier, type and detector stages; greedy selection on VAL with an off option; first-stage controls, frontier gap and paired bootstrap; held-out family and natural novelty modes; external scores with provenance checks; Markdown, JSON and HTML reports; the web dashboard.

Changes

First public release.

• Documentation in docs/: getting started, config reference, protocol and controls, models, outputs,
  command line, dashboard.
• examples/custom_adapter.py (a model behind its own adapter class) and
  examples/external_score_job.py (an outside training job that writes an accepted score file),
  both covered by the test suite.
• Package metadata for PyPI, a citation file and a test workflow.

Install

This version is not on PyPI; install it from its tag:

pip install "git+https://github.com/hashtagRR/IDSLab.git@v0.1.0"

The current version installs with pip install idslab. Every change is in CHANGELOG.md.