configuration concepts

Configuration Concepts

You will learn how the update-ipsets catalog is organized, how individual YAML files combine into a working configuration, and where each type of definition lives on disk.

Catalog as a directory

The catalog is a directory of YAML files — not one monolithic file. Each feed, merge, and artifact has its own file. The loader reads all fragments recursively, merges them, normalizes them, and validates the result.

Directory structure

configs/firehol/
  runtime.yaml          Daemon behavior, concurrency, paths, health thresholds
  categories.yaml       Feed categories for public browsing
  defaults.yaml         Provider defaults (ASN, GeoIP)
  renames.yaml          Old-name → new-name state migrations
  deleted.yaml          Historical names whose local state is removed
  critical_asn_context.yaml  Secondary ASN context entries

  sources/              One YAML per feed, grouped by category subdirectory
    intrusion/
      dshield.yaml
      feodo.yaml
      ...
    scanners/
      misp_shodan_scanners.yaml
      ...
    special_use/
      bogons.yaml
      ...
    asn/
      maxmind_geolite2_asn.yaml
      ...
    geolocation/
      geolite2_country.yaml
      ...
    ...

  merges/               One YAML per merge feed
    firehol_level1.yaml
    firehol_level2.yaml
    ...

  artifacts/            One YAML per downloadable artifact parent
    dronebl.yaml
    ...

The installed configuration directory also contains templates/markdown/. These files are copied from the repository's configs/templates/markdown/ directory during installation. They are not YAML catalog fragments; they control the public Markdown artifacts generated for feeds, countries, ASNs, and maintainers.

How loading works

1. The loader walks the catalog directory recursively.
2. Every YAML fragment is read and merged into a single configuration tree.
3. The merged tree is normalized: outputs are canonicalized, derivatives are expanded, synthetic sources are injected.
4. Validation runs on the final merged result — not on individual fragments.

Cross-file references

Individual files do not need to be self-contained. A source file can reference a category defined in categories.yaml, an artifact defined in artifacts/, or other feeds defined in other source files. The loader resolves all references after merging.

Adding a new feed

Create a new .yaml file in the right category subdirectory under sources/. Add the category definition to categories.yaml if the category is new. Reload with SIGHUP or restart the daemon.

Adding a new merge

Create a new .yaml file in merges/. Reference existing feed names in the sources and optional exclude lists. The referenced feeds can live anywhere in the catalog. Add history: only when the derived windows should be based on the merged output.

Shared registries

File	Purpose
runtime.yaml	All daemon-level settings: concurrency, cadence, health thresholds, web URLs
categories.yaml	Public taxonomy: labels, descriptions, colors, sort order
defaults.yaml	Canonical ASN and GeoIP provider selection
renames.yaml	Old-name to new-name cleanup migrations for existing local state
deleted.yaml	Historical names whose existing local state should be removed
critical_asn_context.yaml	Secondary ASN-level context for blast-radius analysis
templates/markdown/	Public Markdown templates installed beside the catalog

renames.yaml and deleted.yaml are cleanup registries, not public API aliases. During normal scheduler processing, the daemon applies them before the pipeline run: rename entries move old local outputs, public artifacts, history, library state, and cache entries to the new name when possible; deleted entries remove the same classes of local state for retired names.