Document importing Windows hosts from Active Directory
Adds Active Directory Import, and updates the pages that the feature changes.
The new page leads with the part that is not obvious from the requirement: the
preview is persisted server-side and apply may only subtract from it, because a
WindowsTarget row makes DAD connect out to a named host carrying a domain
credential. It also records the three query behaviours that are load-bearing
(paging against AD's silent 1000-object cap, unclassifiable operating systems,
domain controllers), the two-credential model, and what is and is not covered by
CI.
Home: the feature is the most recent pass, and the chronology now reads in order
-- the Windows-source paragraph no longer claims to be "the most recent".
Access Control: new section on why two controllers are Admin-only at the
controller level, where reads are gated, unlike windows_targets. Corrects
hidden? to a class method.
Development: the two remote integrations and the seams they sit behind; suite
counts corrected to 581 + 40 runs / 1969 assertions.
Interface: the connection screens and the two-page import, including which rows
arrive pre-checked and which do not.
Installation: DAD_AR_ENCRYPTION_* and WINDOWS_SSL_VERIFY now name the second
model that reads them; one variable covers the LDAPS and WinRM trust decision
rather than two.
Windows Events: cross-links, and the LDAP gap added to "limits worth knowing".
What Changed Since 2016: new dated section, and the system-suite assertion count
corrected to 247.
Every claim here was checked against the code. Two were wrong as first written
and corrected: the LDAPS/StartTLS note is on the show page, not the index, and
the import result lands on the Windows targets screen.
Co-authored-by: openhands <openhands@all-hands.dev>
4993901
Update the wiki for direct WinRM polling of Windows hosts
DAD now polls Windows hosts itself with wevtutil over WinRM instead of
requiring OpenWEC on the Windows side plus a fourth daemon to forward
what it wrote. The OpenWEC path stays documented as the alternative,
because a subscription receives events pushed to it and a pull model
cannot recover records for a host DAD cannot reach.
Windows-Events.md is restructured around the two paths sharing one
pipeline rather than around OpenWEC specifically. New material: the
priority bands and the adaptive interval (clamp(500/eps, band_min,
band_max), EMA-smoothed rate, 25% deadband), per-host disable, the
credential model and why the encryption keys must not be rotated, the
60s delta overlap and the duplicate trade, and the failure behaviour
that keeps one bad host or channel from costing the others.
Also corrected along the way, independent of this feature:
- Access-Control.md never documented ACTION_RIGHTS at all, only
CONTROLLER_RIGHTS. That is the mechanism that makes reading a target a
Viewer action while writing or polling one is Admin, so it is now
spelled out with the reason for each override.
- Interface.md had no Windows targets section, and its paginated-list
count was twelve.
- Jobs.md listed seven handlers; there are eight. Added a note that the
Job frequency is not the polling interval, since the two numbers being
different is easy to misread as a bug.
- Development.md test counts were 488/40 runs, 1679 assertions.
- Installation.md was missing DAD_AR_ENCRYPTION_KEY,
DAD_AR_ENCRYPTION_SALT and WINDOWS_SSL_VERIFY, and had no warning that
rotating the credential keys makes stored credentials unreadable --
unlike SECRET_KEY_BASE, which is safe to rotate.
- Architecture.md and Home.md described Windows collection as OpenWEC
only.
- What-Changed-Since-2016.md gets an entry, since the older pages
describe a design that no longer holds.
Stated as limits rather than glossed over: the WinRM transport is not
exercised in CI (no Windows host; the transport sits behind a seam and
the parsing/delta/queueing logic is tested against recorded wevtutil
XML), and the pull path reads rather than subscribes, so records written
while a host is unreachable are only recovered if they are still in the
channel next poll.
This wiki update was prepared by an AI agent (OpenHands) on behalf of the
repository owner.
Co-authored-by: openhands <openhands@all-hands.dev>
b6396f1
Document Windows event ingest via OpenWEC
Adds [[Windows Events]]: the OpenWEC setup, the four output formats and
their field mapping, the environment variables, and a troubleshooting
table that names which symptom means which broken link.
The two things an operator can get wrong in a way that corrupts data
rather than erroring get their own headings -- do not point OpenWEC at
Logs/LogStaging, and why one record is not one line.
Home: new paragraph and a row in the start table. Architecture: the
unit table gains dad-windows, and the "nothing else should touch Logs/"
line now says *why* -- the collector's startup recovery moves every file
it finds in LogStaging, so a second writer there gets its in-progress
file renamed away mid-append. Installation: OpenWEC as an optional
requirement, the windows compose profile, and the five WINDOWS_*
variables. Development: the docker job's new Windows leg, and suite
counts corrected to 488 runs / 1679 assertions. What Changed Since 2016:
DAD is no longer syslog-only, with the two assumptions that stop holding.
Co-authored-by: openhands <openhands@all-hands.dev>
2f63f62
bin/docker-test runs three suites, not two
test:system became its own task because rake test excludes test/system.
Co-authored-by: openhands <openhands@all-hands.dev>
9182fd3
Document the browser-facing pass: system suite, assets, and two fixed defects
The wiki was current through the Rust collector but stopped before the six
frontend/UI changes (PRs #34-#39). This brings it up to date.
What Changed Since 2016: new section covering the test/system suite (and why
rake test excludes test/system by default), the scaffold stylesheet merge, the
viewport and the two over-wide layouts, the error-summary partial, the 209
duplicate ids, and the complete: $.ajax() stray-request bug.
Development: test counts corrected (438 unit/controller + 40 system runs); new
"Three suites, not one" explaining the DEFAULT_TEST_EXCLUDE trap and why the
system tests use rack-test rather than a browser; new "Frontend conventions
worth knowing" covering the single stylesheet, the digest/compile coupling, the
draw_* chart API, why coffee-rails cannot be removed despite there being no
.coffee files, the jQuery options-object trap, .js.erb comments being shipped to
the browser, unique ids, and the shared form partial.
Interface: the dashboard renders seven charts, not six, and draw_graph draws a
pie, not a bar -- both were wrong. Added how the dashboard stays live (the
60-second poll) and a short responsive note.
Home: status paragraph for the browser-facing pass.
Co-authored-by: openhands <openhands@all-hands.dev>
2f2a9f4
Update the wiki for PostgreSQL-only dev/test and the deleted provision.sh
SQLite is no longer the development and test engine, and Vagrantfile and
provision.sh are gone, so most pages described a setup that no longer exists.
- Installation: drop the Vagrant quick-start, make the manual path create a
Postgres database per environment and use db:migrate_from_scratch, and move
the secret-file guidance off provision.sh onto the systemd units.
- Development: replace "Two databases, and why CI has both" with "PostgreSQL
everywhere", documenting why db:migrate_from_scratch is required, why
maintain_test_schema is now unconditionally false (its old DATABASE_URL
condition inverted once Postgres became the default), and what
test/support/scratch_database.rb and query_plan.rb are for. Suite counts
updated to 436 runs / 1237 assertions / 0 skips.
- Search: expand the collation finding to the actual mechanism -- the
[term, term + U+10FFFF) range and why no sound upper bound exists under
en_US.UTF-8 -- with an explicit "do not reintroduce" note.
- Architecture, Retention, Jobs, Access-Control, Home: correct the
engine claims and provision.sh references.
- What Changed Since 2016: new "Removed: SQLite as the development and test
engine" section, including the two guards that look like dead SQLite
support but handle an unpartitioned Postgres install.
Co-authored-by: openhands <openhands@all-hands.dev>
f201fe4
Document the Rust syslog collector; correct the buffering claim
The collector is now syslog_collector/, a zero-dependency Rust crate; syslog.pl
is deleted (DAD PR #32).
Architecture, Home, Development, Installation and What-Changed updated: the
pipeline diagram, the unit/service tables, the CI job table (six -> seven, with
the new `collector` job), and the `DAD_SYSLOG_*` environment table, which now
documents the port/staging/processing/rotate knobs the Rust collector reads.
Corrected a claim that turned out to be wrong. Architecture said the 0-byte
staging file meant "the data is in the buffer, not lost", and Development said
"neither is data loss". Measured: 40 messages sent to the Perl collector left 0
bytes durable after both SIGTERM and SIGKILL, because nothing closed the handle.
It was data loss. The Rust collector flushes per record, so the staging file is
readable immediately.
Also documented three other deliberate departures from Perl -- orphan recovery at
startup, the 65507-byte datagram cap replacing silent truncation at 1540, and the
record format that Event#event_fields depends on -- plus the caveat that the
Perl/Rust equivalence check is not automated.
What-Changed-Since-2016 gains a "Changed: the syslog collector is Rust, not
Perl" section.
Co-authored-by: openhands <openhands@all-hands.dev>
28f10fc
Record the docker CI job that builds and drives the stack
Development: the Docker section said no CI job builds the image and told you to
run `docker compose build` yourself. There is now a `docker` job that builds it
and then drives the running stack -- migrate+seed, a real login, an asset fetch,
and a UDP probe that has to land in events. Documented the two things that cost
a debugging cycle each (the probe needs a syslog timestamp; the wait has to poll
because rotation is traffic-driven and the importer scans once a minute), plus
the `bash -e` trap where a bare `x=$(curl ...)` aborts a step on a transient
reset. CI table gains the sixth job.
What Changed Since 2016: "adding a build step to CI is still outstanding" is no
longer true.
Co-authored-by: openhands <openhands@all-hands.dev>
30ce21b
Document the Docker stack, the path coupling, and the rotation trigger
Development: bin/docker-test as the preferred test command, a "Running the
whole stack in Docker" section with the four things that cost an afternoon,
and the DatabaseCleaner remote-URL allowance cucumber needs against a
non-localhost host.
Installation: a Docker way between Vagrant and the manual install, and Docker
listed in Requirements.
Architecture: the pipeline diagram implied both halves resolve Logs/ paths the
same way. They do not. syslog.pl hardcodes /var/DAD/Logs/...; import_syslog.rb
uses paths relative to its working directory. They agree only because every
systemd unit sets WorkingDirectory=/var/DAD, and nothing validates it. Run
either from anywhere else and the collector accepts UDP packets and writes into
a tree the importer never reads, silently.
Also corrected in both diagrams: rotation is not "every 60s". RotateLog is
called from logit, so it fires on a 60s boundary *and only when a packet
arrives*. With no traffic the staging file is never renamed and the importer
has nothing to claim -- a quiet system looks like a broken importer. Verified
against syslog.pl.
What Changed Since 2016: the Docker stack, and the fact that the Dockerfile
that shipped could not build (a bare gem install line is not a Dockerfile
instruction) and had no JS runtime, which broke every view render through
ExecJS.
Home: status paragraph and a Docker row in the where-to-start table.
Co-authored-by: openhands <openhands@all-hands.dev>
a2b9bc9
Record the dead events_words index drop and close the secret_key_base item
Architecture: state that events_words carries exactly one index, the
composite covering index search reads, and why the standalone generated
index was removed.
Search: the plan note claimed joining drives events_words "through the
generated index" -- that index no longer exists. Reworded, and the
idx_scan evidence and identical timings recorded.
What Changed Since 2016: new section on the removed index, including the
two figures that did not survive re-measurement (the 2.3-185x search
claim, and ~6% vs 5.1% insert cost), and the partitioned-index drop
constraint. Marks the secret_key_base item closed: no live deployment
holds the key, so rotation is moot and history rewriting stays declined.
Development: only a development-env db:migrate rewrites schema.rb, since
test.rb sets dump_schema_after_migration = false. A test-env migrate runs
the migration and leaves the file stale.
Co-authored-by: openhands <openhands@all-hands.dev>
11b7be3
Document bulk word resolution and the ingest throughput win
Ingest resolved every word-cache miss against the words table one token at a
time: 186,089 statements and 46,515 transactions per 20k realistic syslog
lines, ~55% of ingest time. Tokens missing from the cache are now deferred
and resolved in bulk at flush time -- 360 statements, 67 transactions, 6.4x
cold / 2.36x warm, with identical events_words rows written and an unchanged
search plan.
Architecture: new "Word resolution" subsection with the measured numbers, and
the word-cache section now says a word resolved at flush is added to the
cache, which is what makes steady state cheap.
Development: rules for anyone touching the ingest path -- per-event token
attribution, all-or-nothing resolution, and the lowest-id rule for duplicate
words.text rows (words.text is not unique; pluck().to_h keeps the highest and
silently re-points ingest at a different word than existing index rows use).
Plus a note that throughput regressions here are silent, so the tests assert
mechanism rather than timing.
What Changed Since 2016: new "Changed: how words get into the index" section,
stating plainly that the index contents did not change.
Home: status paragraph for the throughput pass.
Co-authored-by: openhands <openhands@all-hands.dev>
b597100
Document the ingest batching and word-cache fixes
Brings the wiki in line with the code after the two ingest defects closed in
the write-path review.
Architecture: new "Batching" section - a flush chunks its multi-row INSERTs to
the adapter's bind-parameter ceiling (32,766 SQLite / 65,535 Postgres) and
commits them in one transaction, so a batch stays all-or-nothing. Explains why
events_words is the binding table (3 columns, ~16 rows per event) and that a
2000-event batch needs ~96k parameters, which is past both ceilings. New "The
word cache" section: pruned by TTL, not size, triggered past 100,000 cached
words, lifetime starting at 320s.
Development: two rules under "Before touching the ingest path". All ingest state
must be initialised in resetCaches - both defects came from variables it did not
cover. And some ingest limits are invisible to an in-process run, because
DEFAULT_BULK_INSERT_SIZE is 1 in dev/test; only RAILS_ENV=development subprocess
tests cross the parameter ceiling. Same SQLite-green/production-broken shape as
the SqlPlaceholders failure already documented on that page.
Jobs: prune_words row notes the 320s starting lifetime.
Home: status note for the ingest pass.
What Changed Since 2016: new section on both defects, including that the
extraction commit moved prune_words' reader and writer but dropped the
@@cachelifetime = 320 assignment, and that the three tests touching prune_words
each set the value by hand - which is why the suite stayed green.
Co-authored-by: openhands <openhands@all-hands.dev>
707faff
Document the 2026-11 integrity/scheduler/pagination pass, with screenshots
Brings the wiki in line with the code as of the four fixes that followed the
2026 review pass.
Jobs: the handler registry is seven operations, not six - prune_sessions is new
and seeded hourly. Failure behaviour rewritten: the loop body now lives in
JobScheduler, and a Job with no usable frequency is disarmed (next_run NULL)
rather than crashing the process. Documented why: the next_run line sat outside
the begin/rescue, so a blank Frequency field crash-looped the scheduler under
Restart=always with the row re-selected on every start, and no Job ran at all.
Access Control: session_hash unique index (the per-request cookie lookup was a
full scan), Session.purge_expired on logon plus the seeded prune_sessions job,
and username uniqueness - do_logon uses find_by, so a second account sharing a
name could never sign in, silently.
Interface: new "Paginated lists" section, 25 rows/page across all twelve
management indexes, ?page clamped at both ends, hand-rolled rather than a gem.
Two new screenshots (sessions, statistics) at the same 1440x1008 as the rest.
What Changed Since 2016: new section on the integrity constraints the
application never had, and how the migration repairs existing bad rows without
deleting any account.
Development: SQLite can create a CHECK but cannot ALTER TABLE ADD CONSTRAINT, so
the Postgres job is the only thing that exercises that branch.
Home: status note for the follow-up pass. Rails version in Interface corrected
to 8.1.4.1.
Co-authored-by: openhands <openhands@all-hands.dev>
cdf4a3d
Update wiki for the search-form fix and the 8.1.4.1 lock
Co-authored-by: openhands <openhands@all-hands.dev>
5943fc0
Say where job handler output actually appears
scheduler.rb logs the handler return value
Rails.logger.info "job #{job.name} -> #{JobHandler.perform(job.script)}"
so the disk_utilization reading lands in the journal, not on the Jobs screens.
The Jobs show page lists name, operation, owner, last/next run and frequency
only. Stated on Jobs, Interface, and DAD Disk Space Warnings, since all three
now tell the reader to check the job output.
Co-authored-by: openhands <openhands@all-hands.dev>
1d63845
Correct the dashboard chart list and the disk-space recipe
Interface: the dashboard renders six charts, not "four gauges and two
longitudinal charts" -- three gauges (7 day / 24 hour / 1 hour), two 24-hour
bar charts (by system, by service), and the two longitudinal rate charts.
Tabulated against the canvas ids in app/views/events/index.html.erb.
DAD Disk Space Warnings: the 2015 snippet now misleads in two ways.
Event.diskUtilization measures the archive partition rather than /, and
returns nil rather than 0 when df cannot measure it, so `used > 98` raises
NoMethodError on nil instead of reporting an empty disk. Adds the nil-safe
form, notes that the disk_utilization handler already ships this alert, and
points at archiveFreeSpacePercent as the thing that actually prevents
running out of space.
What Changed Since 2016: records the diskUtilization semantics change under
"Still true", since the method name survives but the contract does not.
Co-authored-by: openhands <openhands@all-hands.dev>
2ffa6c0
Add interface screenshots and correct disk/search documentation
Adds an Interface page with screenshots of the running UI (login, dashboard,
search results, jobs, users/roles/rights, preferences, alerts, displays,
exclusions, stored searches, systems) taken against a seeded demonstration
dataset, and threads the relevant ones into the pages they describe.
Corrects the record on disk utilization: there is no dashboard disk gauge. The
author commented it out in bec738c ("Hiding disk gauge", 2015-02-28); the live
surfaces are the disk_utilization job and preferences.archiveFreeSpacePercent.
Documents the open search-form bug (RECOMMENDATIONS 2.3): the Rails check_box
always submits timesearch_enabled[enabled]=0, and EventsController#search
branches on the presence of that key, so every form search takes the absolute
date branch with a now->now window and reports 0 events.
Co-authored-by: openhands <openhands@all-hands.dev>
a1f445e
Search: document what the index actually contains
stage_event downcases the line and replaces every character outside
- _ : @ * / . with a space before tokenising, so a search term holding any
other punctuation matches nothing at all. That reads as a broken search rather
than as the index working as designed. hunks keeps the original line, so the
index and the event views are built from different strings.
Co-authored-by: openhands <openhands@all-hands.dev>
74f08cf
Bring the wiki in line with the current project
The wiki last changed in April 2016 and described an alpha that let you type
Ruby into the Jobs form and the display editor. Both of those designs are gone,
so the pages were not just stale but actively misleading.
New pages: Architecture, Installation, Search, Displays (rewritten), Jobs,
Retention and Archives, Access Control, Exclusions, Development, and What
Changed Since 2016, which maps the 2015 material onto the current code.
The eight alerting recipes are kept as a recipe archive with a banner and an
API-status table on Example Jobs: the analysis shape still ports, but they call
a Job design that no longer exists, and Math.mean/variance/standard_deviation
were removed in favour of Stats.
Home is now a landing page with a status summary and a task-based index.
Co-authored-by: openhands <openhands@all-hands.dev>
c44e466
Update on optimization progress.
6049180
Completely revised display filters.
d570c01
Updated sudo activity (markdown)
ded927f
Updated sudo activity (markdown)
f4597a3
Created sudo activity (markdown)
457cf81
Updated Example Jobs (markdown)
38a1a28