Repository navigation
v0.3.0: Syslog live, Slack approvals and a planned failover and failback
The syslog path is live on the platform from both arrays, with the audit
and alert layouts parsed into fields; notifications reach Slack and
approvals come back from it through a bot with identities of its own; a
planned ActiveDR failover and failback runbook joins the orchestrator
workflows, now nine behind nineteen job templates, and tools/ao_sync.py
imports the definitions into a live orchestrator. Exercised against the
two Purity//FA 6.10.6 arrays, Ansible Automation Platform 2.7 and a live
orchestrator; docs/VERIFICATION.md has the
ledger.
- Every approval node names the group
approversbesideadmins, so the
people who decide and the Slack bot's orchestrator user can sit in a
group of their own; the orchestrator resolves approver names when it
creates an approval and takes users and groups, not service accounts. - A planned ActiveDR failover and failback runbook.
playbooks/dr_switchover.yml
demotes the link's source pod with quiesce, waits for the link to report
the target's copy complete, promotes the target pod and records both
pods' states and the link's new direction; the same playbook with the
sides swapped fails back. A pod cannot be demoted again while the
undo-demote pod of an earlier demotion exists, so with
eradicate_undo_demotetrue the playbook eradicates those pods first
and reports them inundo_demote_eradicated; without it the play
refuses and names them.orchestrator/dr_failover_failback.jsonruns
it behind two approvals and sets the variable, andaap/seed.yml
creates theFlashArray - DR switchoverjob template. tools/ao_sync.pyasks the orchestrator for/api/v1/service_accounts
(the path it used did not exist; the first live run found it) and
compares a file with its export withoutcredential_namesand without an
emptyextra_vars, neither of which the export carries. Verified live as
the service account: create, update and unchanged. The dry run says a
changed definition would publish a new version and no longer guesses
its number.purefa_syslogparses what Purity//FA 6.10.6 sends: lines without a
<PRI>, a pid on the program, the command layout ofpurity.audit
([user (local)]@array(id) command. Message ID: N UTC Time: ... Array Name: x, givingcommand,subcommand,arguments,action,
utc_timeandmessage_kind: command) beside the session layout, and
the severity from the text when there is no priority. Newprograms
option (varssyslog_programs): the array forwards its controllers'
whole system log, so onlypurity.alert,purity.auditand
purity.testare emitted by default and the rest is dropped and
counted.syslog.ymlskips session records bymessage_kind.tools/ao_sync.pyimports the orchestrator definitions into an
orchestrator the way orchestrator/README.md ("Importing") describes:
creates the workflows the project lacks, updates the existing ones by
name, fills theREPLACE_AT_SYNCplaceholders from--aap-credential,
--aap-integrationand--service-account, validates first, publishes,
reports workflows no file defines and never deletes;--dry-runwrites
nothing. A changed definition is published as the next version, the
live contract (a version's PATCH changes metadata only). Standard
library only;tests/unit/test_ao_sync.pydrives it against an
in-process mock of the API.tools/ao_lint.pyknows loop nodes:iterateandcompleteports, the
body's return edge (to_port: iterate) as the one allowed back edge
(loop_ports), and the loop parameter keys.- A second orchestrator route: a protection group whose replication is
overdue or stuck is handed over whenorchestrator_enabledis true, and
orchestrator/replication_overdue_approval.jsongates the immediate
replicated snapshot behind an approval (orchestrator_paths.replication). orchestrator_fallback_local(default false): a capacity handoff that
produced no execution raises the local request anyway.- Slack:
notify.ymlposts every notification to an incoming webhook when
SLACK_WEBHOOK_URLis set (env file, or the newSlack webhook
credential the seed attaches to the notify template); the seed also
creates a Slack notification template for job failures and the
controller-tier workflow's approval events when a bot token and channel
are set. slack/approval_bot.py: a Slack Socket Mode bot that posts pending
orchestrator and controller approvals as cards with Approve and Reject
buttons and carries the decision back with the Slack user's name
(SLACK_APPROVERSlimits who may click). Unit-tested with fakes.replicate_now.ymlpublishes job artifacts (pgroup,snapshot_suffix,
changed, ...), and a unit test checks that every${node.artifacts.key}
a definition reads is published by that template's playbook.purefa_syslogcounts the array's own service accounts as automation
(system_users, defaultpure_datainsights;puritywrites with auth
sourcesystem), so the human-action rule stays quiet for the array's
scheduled snapshots and tag housekeeping.- The ansible-builder definitions build:
python_interpreterunder
dependenciespins the base image's Python 3.12, and each collection
directory names its owndestunderadditional_build_files. The
decision environment fromdecision-environment/execution-environment.yml
is built on avfsimage store and runsalertmanager.ymlthrough a
posted alert to a notify record; the two supported-base definitions build
there as well (PACKAGING.md has the rootless-overlay caveat). aap/decision-environment.ymlexcludessystemd-pythonfrom the pip
stage:ansible.eda's requirements name it, it has no wheel, and the
supported base carries no Python headers or compiler to build it.- Links people can open:
aap/seed.ymlsets the platform's base URL for
notification links (TOWER_URL_BASE, fromaap_url_base, the gateway
URL by default), and the bot's controller cards link toAAP_UI_URL
when set. purefa_syslogparses thepurity.alertlayout Purity//FA 6.10.6 sends
(level word, subject,[code], description, support-case sentence,
Severity:, UTC and array time, array name, domain, knowledge base link,
suggested action) into fields, grades the event by the severity word and
stops the UTC trailer atArray Time:; the syslog alert rule grades an
info alert as info, names the code in the subject and passes
alert_code. The bot keeps the channel id Slack returns (chat.update
refuses the channel name) and a failed card update no longer stops the
poll.- The Slack text of
notify.ymlcarries a real line break between the
subject line and the detail (a'\n'literal inside a Jinja expression
is two characters under Ansible's backslash doubling), names the array
once, drops carriage returns and escapes&,<and>; a unit test
refuses escape literals inside Jinja expressions outsideregex_*
filters. The admin-authentication drift notice names the baseline values
beside the observed ones. - Verified since 0.2.0 (VERIFICATION.md): the syslog path live on the
platform from both arrays; the Alertmanager chain into the platform's
activation, with the capacity rule under a lab override and the
replica-link rule on a real pause; the DR drill schedule firing on its
own; the capacity handoff from a running activation through approval to
the reclaim; the overdue-replication route through its approval to the
replicated snapshot, once approved from Slack through the bot; the
notify template posting to Slack from the platform; an approval nobody
decided expiring and failing its execution at the approval node, with
fallback_decisionmaking no difference; livepurity.alertlines from
both arrays (a scratch pod stretched between them: an info line from
each and a critical one, parsed into fields and graded by the rebuilt
decision environment) and the array's test line over UDP, with the
arrays replaying queued lines after an 80-minute listener outage; the scrape-failure rule through Alertmanager to the
platform's activation, firing and resolved; the platform's Slack
notification on a controller-tier approval and the bot's card for it;
the alerts signal against a Purity//FA 6.12.4 array; the planned
failover and failback on the lab's ActiveDR link, both ways, with the
link reversed in between, then again with both approvals given from
Slack by the bot's user, the undo-demote pods of the first run
eradicated on the way and a duplicate launch of the failback (the
gateway answered 503 after the job was created) absorbed by the
playbook changing nothing the second time.