Document the global Chart.js plugin trap
dadGauge is registered globally, so it ran on the two 24-hour pies and drew
the word "undefined" at their centre. Chart.js hands an unconfigured plugin a
truthy resolver Proxy, so `if (!opts)` is not a "does this plugin apply?" test.
Development: the mechanism, why dadBands escaped it by accident, and the
regression test. Interface: a pointer for anyone who sees the symptom.
Co-authored-by: openhands <openhands@all-hands.dev>
Document the ordered nav bar and Windows dropdown
The bar is no longer $_models.sort. Add a "The nav bar" section to Interface
covering the declared order (Event, Search, System, Service), the Windows
dropdown (AD, Windows), why it is <details>/<summary> rather than JS, and the
is_active tableize fix.
Correct the present-tense claims in Access-Control (hidden? decides whether a
tab appears, not where it sits or what it is called) and Development (the
navbar ul's overflow: hidden became a clearfix, because overflow also clips an
absolutely positioned submenu).
Historical entries in What Changed Since 2016 are left as written.
Co-authored-by: openhands <openhands@all-hands.dev>
Document that each container is its own system, not one host and six services
The Docker stack reported one host: every container's log driver sends from the
bridge gateway, so all five services arrived carrying the same address and
System.find_or_add -- which keys identity on systems.address -- gave them one row,
while each container name was filed as a Service.
Architecture gains "Which host an event came from": the frame shapes, the pid-slot
rule SyslogSource uses to tell an application from a hostname, and why the pid in
the Docker frame is dockerd's rather than the container's.
Also recorded: the same rule fixes a non-Docker mis-parse. RFC 3164 as rsyslog
forwards it does carry a hostname slot, and the positional read filed web01 as the
Service -- the Service sanitizer strips digits, so the row said web -- and dropped
sshd entirely.
The rename is gated on the address being one of this container's own gateways AND
the importer being containerized; without the second gate a systemd deployment
would rename syslog from a LAN router. Noted in Installation's env table that the
escape hatch is therefore inert under systemd, so nobody goes looking for a knob
they do not need.
Development's frame-format bullet corrected: the tag lands where the application is
read from, not where the Service is read from. The docker CI job description now
mentions the step asserting per-container Systems, which passed on the runner with
3 hosts and none collapsed onto the gateway.
Co-authored-by: openhands <openhands@all-hands.dev>
Match the wiki to PostgreSQL 18 and the bind-mounted database
Three pages said `docker compose down -v # discard the database too`. That is
now false, and it was the description of a defect rather than of intended
behaviour: the database lived in a Compose named volume, which `down -v`
deletes, and which is namespaced by the project name so renaming the checkout
started a second empty database. Both reproduced; the stack now bind-mounts
./data/postgres.
The engine moved 17 -> 18. The pin had no reason behind it, and PostgreSQL has
no LTS tier (every major gets five years), so the newest major is the
longer-support choice.
Also recorded the two 18-specific traps, both hit while verifying this: the
bind mount target is /var/lib/postgresql rather than .../data, and
lc_collate/lc_ctype no longer exist as GUCs. Collation behaviour itself is
unchanged, so the varchar_pattern_ops reasoning still holds.
Version claims that only restated the pin are now written without a number
where the behaviour is version-independent, so this page stops going stale on
the next bump. Claims that record an actual measurement ("measured on 17.11")
are left as measured.
New: an "Upgrading the PostgreSQL major version" section on Installation --
the dump-and-restore procedure was executed end to end (17 cluster ->
pg_dumpall -> fresh 18 -> rows intact) before being written down.
This wiki edit was created by an AI agent (OpenHands) on behalf of the user.
Record why the collector must not forward its own logs
The exemption read as tidiness ("it would otherwise feed itself") and a reader
could reasonably delete it as an inconsistency. Measured with DAD_SYSLOG_DEBUG=1,
the loop is unbounded: the staging file grew 5.7GB to 13.9GB in 20 seconds,
about 411MB/s, until the container was killed.
Also names dad-windows among the forwarding services and notes that CI now
asserts both halves of the rule against the resolved config with all profiles
enabled.
Co-authored-by: openhands <openhands@all-hands.dev>
Fix nine false or stale claims, verified against the tree
A page-by-page audit of all 23 pages against repo master found seven claims that
are simply false and two that are stale. Each was re-checked against the tree
before being changed rather than taken on trust.
False:
- /events/recent as the dashboard feed endpoint. No such route: events is index +
collection { get :search }, and there is no recent.js.erb. The feed refreshes via
GET /events.js; EventsController#recent is unreachable dead code. (Interface.md.)
- "Every management index screen is paginated at 25 rows". systems is deliberately
unpaged, as are services, words (fixed 20), alerts, events, polling_priorities.
- The list of screens pagination replaced: that commit touched twelve controllers,
systems was not one of them, and hunks -- which it did touch -- was omitted.
windows_targets and active_directory_connections inherited it later.
- A Search row as a string "plus its time window". searches carries string,
user_id, description, short_description and nothing else; no timeframe column has
ever existed, so it is chosen again at run time.
- "the systemd units set /var/DAD/offline_storage, mode 0700". No unit sets the
variable; the path comes from the default plus WorkingDirectory. And nothing in
the app sets 0700 -- EventPartitionBackup uses mkdir_p, so the directory gets the
umask default. This one is load-bearing: it told operators a directory of raw log
lines was permission-protected when nothing protects it.
- db:seed "seeds the two maintenance jobs" -- it seeds four.
- JobHandler::HANDLERS as "seven operations" -- eight since poll_windows_events.
Stale:
- "Below 720px" in three pages; the media query is max-width: 768px.
- Statistical-Alerting.md crediting a Math monkey patch; the helpers are
Stats.mean/variance/standard_deviation, and the patch was removed.
Also documents the new self-logging stack: every service but syslog ships its own
stdout/stderr to the collector, the two measured traps that decide whether anything
is stored at all (the log driver resolves syslog-address in the daemon's namespace,
and the frame format decides the Service), the resulting single-System attribution,
and the CI step that asserts Service names rather than merely a row.
Checked and left alone: the Viewer rights row. An earlier audit flagged it for
omitting preferences, but preferences is Admin-gated in CONTROLLER_RIGHTS, so the
row is correct as written.
Co-authored-by: openhands <openhands@all-hands.dev>
Correct the suite counts and the PollingPriority nav claim
Development.md said '581 unit/controller runs plus 40 system runs, 1969
assertions in total'. That mis-splits a single number: rake test excludes
test/system by design, so its 621 runs are all non-system and the 40 system
runs are additional. Scanning every commit in history, the non-system count
was never 581 -- it was 542 up to b6aafd7 and 621 after. The page now gives
the counts per command, measured on Ruby 3.3.8 / PostgreSQL 17.11: 621 runs
and 1969 assertions from rake test, 40 runs and 247 assertions from
test:system, 5 scenarios and 18 steps from cucumber, all green.
Access-Control.md said PollingPriority hides itself because it has no index
route. It does have one (only: [:index, :edit, :update]); it hides itself
because the bands are reached from the Windows Targets screens, which link
to polling_priorities_path. The same error was in a config/routes.rb
comment and an AdImportBatch comment in the repo, both now fixed.
This wiki edit was made by an AI agent (OpenHands) on behalf of the user.
Co-authored-by: openhands <openhands@all-hands.dev>
Document importing Windows hosts from Active Directory
Adds Active Directory Import, and updates the pages that the feature changes.
The new page leads with the part that is not obvious from the requirement: the
preview is persisted server-side and apply may only subtract from it, because a
WindowsTarget row makes DAD connect out to a named host carrying a domain
credential. It also records the three query behaviours that are load-bearing
(paging against AD's silent 1000-object cap, unclassifiable operating systems,
domain controllers), the two-credential model, and what is and is not covered by
CI.
Home: the feature is the most recent pass, and the chronology now reads in order
-- the Windows-source paragraph no longer claims to be "the most recent".
Access Control: new section on why two controllers are Admin-only at the
controller level, where reads are gated, unlike windows_targets. Corrects
hidden? to a class method.
Development: the two remote integrations and the seams they sit behind; suite
counts corrected to 581 + 40 runs / 1969 assertions.
Interface: the connection screens and the two-page import, including which rows
arrive pre-checked and which do not.
Installation: DAD_AR_ENCRYPTION_* and WINDOWS_SSL_VERIFY now name the second
model that reads them; one variable covers the LDAPS and WinRM trust decision
rather than two.
Windows Events: cross-links, and the LDAP gap added to "limits worth knowing".
What Changed Since 2016: new dated section, and the system-suite assertion count
corrected to 247.
Every claim here was checked against the code. Two were wrong as first written
and corrected: the LDAPS/StartTLS note is on the show page, not the index, and
the import result lands on the Windows targets screen.
Co-authored-by: openhands <openhands@all-hands.dev>
Update the wiki for direct WinRM polling of Windows hosts
DAD now polls Windows hosts itself with wevtutil over WinRM instead of
requiring OpenWEC on the Windows side plus a fourth daemon to forward
what it wrote. The OpenWEC path stays documented as the alternative,
because a subscription receives events pushed to it and a pull model
cannot recover records for a host DAD cannot reach.
Windows-Events.md is restructured around the two paths sharing one
pipeline rather than around OpenWEC specifically. New material: the
priority bands and the adaptive interval (clamp(500/eps, band_min,
band_max), EMA-smoothed rate, 25% deadband), per-host disable, the
credential model and why the encryption keys must not be rotated, the
60s delta overlap and the duplicate trade, and the failure behaviour
that keeps one bad host or channel from costing the others.
Also corrected along the way, independent of this feature:
- Access-Control.md never documented ACTION_RIGHTS at all, only
CONTROLLER_RIGHTS. That is the mechanism that makes reading a target a
Viewer action while writing or polling one is Admin, so it is now
spelled out with the reason for each override.
- Interface.md had no Windows targets section, and its paginated-list
count was twelve.
- Jobs.md listed seven handlers; there are eight. Added a note that the
Job frequency is not the polling interval, since the two numbers being
different is easy to misread as a bug.
- Development.md test counts were 488/40 runs, 1679 assertions.
- Installation.md was missing DAD_AR_ENCRYPTION_KEY,
DAD_AR_ENCRYPTION_SALT and WINDOWS_SSL_VERIFY, and had no warning that
rotating the credential keys makes stored credentials unreadable --
unlike SECRET_KEY_BASE, which is safe to rotate.
- Architecture.md and Home.md described Windows collection as OpenWEC
only.
- What-Changed-Since-2016.md gets an entry, since the older pages
describe a design that no longer holds.
Stated as limits rather than glossed over: the WinRM transport is not
exercised in CI (no Windows host; the transport sits behind a seam and
the parsing/delta/queueing logic is tested against recorded wevtutil
XML), and the pull path reads rather than subscribes, so records written
while a host is unreachable are only recovered if they are still in the
channel next poll.
This wiki update was prepared by an AI agent (OpenHands) on behalf of the
repository owner.
Co-authored-by: openhands <openhands@all-hands.dev>
Document Windows event ingest via OpenWEC
Adds [[Windows Events]]: the OpenWEC setup, the four output formats and
their field mapping, the environment variables, and a troubleshooting
table that names which symptom means which broken link.
The two things an operator can get wrong in a way that corrupts data
rather than erroring get their own headings -- do not point OpenWEC at
Logs/LogStaging, and why one record is not one line.
Home: new paragraph and a row in the start table. Architecture: the
unit table gains dad-windows, and the "nothing else should touch Logs/"
line now says *why* -- the collector's startup recovery moves every file
it finds in LogStaging, so a second writer there gets its in-progress
file renamed away mid-append. Installation: OpenWEC as an optional
requirement, the windows compose profile, and the five WINDOWS_*
variables. Development: the docker job's new Windows leg, and suite
counts corrected to 488 runs / 1679 assertions. What Changed Since 2016:
DAD is no longer syslog-only, with the two assumptions that stop holding.
Co-authored-by: openhands <openhands@all-hands.dev>
bin/docker-test runs three suites, not two
test:system became its own task because rake test excludes test/system.
Co-authored-by: openhands <openhands@all-hands.dev>
Document the browser-facing pass: system suite, assets, and two fixed defects
The wiki was current through the Rust collector but stopped before the six
frontend/UI changes (PRs #34-#39). This brings it up to date.
What Changed Since 2016: new section covering the test/system suite (and why
rake test excludes test/system by default), the scaffold stylesheet merge, the
viewport and the two over-wide layouts, the error-summary partial, the 209
duplicate ids, and the complete: $.ajax() stray-request bug.
Development: test counts corrected (438 unit/controller + 40 system runs); new
"Three suites, not one" explaining the DEFAULT_TEST_EXCLUDE trap and why the
system tests use rack-test rather than a browser; new "Frontend conventions
worth knowing" covering the single stylesheet, the digest/compile coupling, the
draw_* chart API, why coffee-rails cannot be removed despite there being no
.coffee files, the jQuery options-object trap, .js.erb comments being shipped to
the browser, unique ids, and the shared form partial.
Interface: the dashboard renders seven charts, not six, and draw_graph draws a
pie, not a bar -- both were wrong. Added how the dashboard stays live (the
60-second poll) and a short responsive note.
Home: status paragraph for the browser-facing pass.
Co-authored-by: openhands <openhands@all-hands.dev>
Update the wiki for PostgreSQL-only dev/test and the deleted provision.sh
SQLite is no longer the development and test engine, and Vagrantfile and
provision.sh are gone, so most pages described a setup that no longer exists.
- Installation: drop the Vagrant quick-start, make the manual path create a
Postgres database per environment and use db:migrate_from_scratch, and move
the secret-file guidance off provision.sh onto the systemd units.
- Development: replace "Two databases, and why CI has both" with "PostgreSQL
everywhere", documenting why db:migrate_from_scratch is required, why
maintain_test_schema is now unconditionally false (its old DATABASE_URL
condition inverted once Postgres became the default), and what
test/support/scratch_database.rb and query_plan.rb are for. Suite counts
updated to 436 runs / 1237 assertions / 0 skips.
- Search: expand the collation finding to the actual mechanism -- the
[term, term + U+10FFFF) range and why no sound upper bound exists under
en_US.UTF-8 -- with an explicit "do not reintroduce" note.
- Architecture, Retention, Jobs, Access-Control, Home: correct the
engine claims and provision.sh references.
- What Changed Since 2016: new "Removed: SQLite as the development and test
engine" section, including the two guards that look like dead SQLite
support but handle an unpartitioned Postgres install.
Co-authored-by: openhands <openhands@all-hands.dev>
Document the Rust syslog collector; correct the buffering claim
The collector is now syslog_collector/, a zero-dependency Rust crate; syslog.pl
is deleted (DAD PR #32).
Architecture, Home, Development, Installation and What-Changed updated: the
pipeline diagram, the unit/service tables, the CI job table (six -> seven, with
the new `collector` job), and the `DAD_SYSLOG_*` environment table, which now
documents the port/staging/processing/rotate knobs the Rust collector reads.
Corrected a claim that turned out to be wrong. Architecture said the 0-byte
staging file meant "the data is in the buffer, not lost", and Development said
"neither is data loss". Measured: 40 messages sent to the Perl collector left 0
bytes durable after both SIGTERM and SIGKILL, because nothing closed the handle.
It was data loss. The Rust collector flushes per record, so the staging file is
readable immediately.
Also documented three other deliberate departures from Perl -- orphan recovery at
startup, the 65507-byte datagram cap replacing silent truncation at 1540, and the
record format that Event#event_fields depends on -- plus the caveat that the
Perl/Rust equivalence check is not automated.
What-Changed-Since-2016 gains a "Changed: the syslog collector is Rust, not
Perl" section.
Co-authored-by: openhands <openhands@all-hands.dev>
Record the docker CI job that builds and drives the stack
Development: the Docker section said no CI job builds the image and told you to
run `docker compose build` yourself. There is now a `docker` job that builds it
and then drives the running stack -- migrate+seed, a real login, an asset fetch,
and a UDP probe that has to land in events. Documented the two things that cost
a debugging cycle each (the probe needs a syslog timestamp; the wait has to poll
because rotation is traffic-driven and the importer scans once a minute), plus
the `bash -e` trap where a bare `x=$(curl ...)` aborts a step on a transient
reset. CI table gains the sixth job.
What Changed Since 2016: "adding a build step to CI is still outstanding" is no
longer true.
Co-authored-by: openhands <openhands@all-hands.dev>
Document the Docker stack, the path coupling, and the rotation trigger
Development: bin/docker-test as the preferred test command, a "Running the
whole stack in Docker" section with the four things that cost an afternoon,
and the DatabaseCleaner remote-URL allowance cucumber needs against a
non-localhost host.
Installation: a Docker way between Vagrant and the manual install, and Docker
listed in Requirements.
Architecture: the pipeline diagram implied both halves resolve Logs/ paths the
same way. They do not. syslog.pl hardcodes /var/DAD/Logs/...; import_syslog.rb
uses paths relative to its working directory. They agree only because every
systemd unit sets WorkingDirectory=/var/DAD, and nothing validates it. Run
either from anywhere else and the collector accepts UDP packets and writes into
a tree the importer never reads, silently.
Also corrected in both diagrams: rotation is not "every 60s". RotateLog is
called from logit, so it fires on a 60s boundary *and only when a packet
arrives*. With no traffic the staging file is never renamed and the importer
has nothing to claim -- a quiet system looks like a broken importer. Verified
against syslog.pl.
What Changed Since 2016: the Docker stack, and the fact that the Dockerfile
that shipped could not build (a bare gem install line is not a Dockerfile
instruction) and had no JS runtime, which broke every view render through
ExecJS.
Home: status paragraph and a Docker row in the where-to-start table.
Co-authored-by: openhands <openhands@all-hands.dev>
Record the dead events_words index drop and close the secret_key_base item
Architecture: state that events_words carries exactly one index, the
composite covering index search reads, and why the standalone generated
index was removed.
Search: the plan note claimed joining drives events_words "through the
generated index" -- that index no longer exists. Reworded, and the
idx_scan evidence and identical timings recorded.
What Changed Since 2016: new section on the removed index, including the
two figures that did not survive re-measurement (the 2.3-185x search
claim, and ~6% vs 5.1% insert cost), and the partitioned-index drop
constraint. Marks the secret_key_base item closed: no live deployment
holds the key, so rotation is moot and history rewriting stays declined.
Development: only a development-env db:migrate rewrites schema.rb, since
test.rb sets dump_schema_after_migration = false. A test-env migrate runs
the migration and leaves the file stale.
Co-authored-by: openhands <openhands@all-hands.dev>
Document bulk word resolution and the ingest throughput win
Ingest resolved every word-cache miss against the words table one token at a
time: 186,089 statements and 46,515 transactions per 20k realistic syslog
lines, ~55% of ingest time. Tokens missing from the cache are now deferred
and resolved in bulk at flush time -- 360 statements, 67 transactions, 6.4x
cold / 2.36x warm, with identical events_words rows written and an unchanged
search plan.
Architecture: new "Word resolution" subsection with the measured numbers, and
the word-cache section now says a word resolved at flush is added to the
cache, which is what makes steady state cheap.
Development: rules for anyone touching the ingest path -- per-event token
attribution, all-or-nothing resolution, and the lowest-id rule for duplicate
words.text rows (words.text is not unique; pluck().to_h keeps the highest and
silently re-points ingest at a different word than existing index rows use).
Plus a note that throughput regressions here are silent, so the tests assert
mechanism rather than timing.
What Changed Since 2016: new "Changed: how words get into the index" section,
stating plainly that the index contents did not change.
Home: status paragraph for the throughput pass.
Co-authored-by: openhands <openhands@all-hands.dev>
Document the ingest batching and word-cache fixes
Brings the wiki in line with the code after the two ingest defects closed in
the write-path review.
Architecture: new "Batching" section - a flush chunks its multi-row INSERTs to
the adapter's bind-parameter ceiling (32,766 SQLite / 65,535 Postgres) and
commits them in one transaction, so a batch stays all-or-nothing. Explains why
events_words is the binding table (3 columns, ~16 rows per event) and that a
2000-event batch needs ~96k parameters, which is past both ceilings. New "The
word cache" section: pruned by TTL, not size, triggered past 100,000 cached
words, lifetime starting at 320s.
Development: two rules under "Before touching the ingest path". All ingest state
must be initialised in resetCaches - both defects came from variables it did not
cover. And some ingest limits are invisible to an in-process run, because
DEFAULT_BULK_INSERT_SIZE is 1 in dev/test; only RAILS_ENV=development subprocess
tests cross the parameter ceiling. Same SQLite-green/production-broken shape as
the SqlPlaceholders failure already documented on that page.
Jobs: prune_words row notes the 320s starting lifetime.
Home: status note for the ingest pass.
What Changed Since 2016: new section on both defects, including that the
extraction commit moved prune_words' reader and writer but dropped the
@@cachelifetime = 320 assignment, and that the three tests touching prune_words
each set the value by hand - which is why the suite stayed green.
Co-authored-by: openhands <openhands@all-hands.dev>
Document the 2026-11 integrity/scheduler/pagination pass, with screenshots
Brings the wiki in line with the code as of the four fixes that followed the
2026 review pass.
Jobs: the handler registry is seven operations, not six - prune_sessions is new
and seeded hourly. Failure behaviour rewritten: the loop body now lives in
JobScheduler, and a Job with no usable frequency is disarmed (next_run NULL)
rather than crashing the process. Documented why: the next_run line sat outside
the begin/rescue, so a blank Frequency field crash-looped the scheduler under
Restart=always with the row re-selected on every start, and no Job ran at all.
Access Control: session_hash unique index (the per-request cookie lookup was a
full scan), Session.purge_expired on logon plus the seeded prune_sessions job,
and username uniqueness - do_logon uses find_by, so a second account sharing a
name could never sign in, silently.
Interface: new "Paginated lists" section, 25 rows/page across all twelve
management indexes, ?page clamped at both ends, hand-rolled rather than a gem.
Two new screenshots (sessions, statistics) at the same 1440x1008 as the rest.
What Changed Since 2016: new section on the integrity constraints the
application never had, and how the migration repairs existing bad rows without
deleting any account.
Development: SQLite can create a CHECK but cannot ALTER TABLE ADD CONSTRAINT, so
the Postgres job is the only thing that exercises that branch.
Home: status note for the follow-up pass. Rails version in Interface corrected
to 8.1.4.1.
Co-authored-by: openhands <openhands@all-hands.dev>
Update wiki for the search-form fix and the 8.1.4.1 lock
Co-authored-by: openhands <openhands@all-hands.dev>
Bring the wiki in line with the current project
The wiki last changed in April 2016 and described an alpha that let you type
Ruby into the Jobs form and the display editor. Both of those designs are gone,
so the pages were not just stale but actively misleading.
New pages: Architecture, Installation, Search, Displays (rewritten), Jobs,
Retention and Archives, Access Control, Exclusions, Development, and What
Changed Since 2016, which maps the 2015 material onto the current code.
The eight alerting recipes are kept as a recipe archive with a banner and an
API-status table on Example Jobs: the analysis shape still ports, but they call
a Job design that no longer exists, and Math.mean/variance/standard_deviation
were removed in favour of Stats.
Home is now a landing page with a status summary and a task-based index.
Co-authored-by: openhands <openhands@all-hands.dev>