Skip to content

v0.3.0: Syslog live, Slack approvals and a planned failover and failback

Choose a tag to compare

@sdodsley sdodsley released this 09 Nov 14:32
· 13 commits to main since this release

The syslog path is live on the platform from both arrays, with the audit
and alert layouts parsed into fields; notifications reach Slack and
approvals come back from it through a bot with identities of its own; a
planned ActiveDR failover and failback runbook joins the orchestrator
workflows, now nine behind nineteen job templates, and tools/ao_sync.py
imports the definitions into a live orchestrator. Exercised against the
two Purity//FA 6.10.6 arrays, Ansible Automation Platform 2.7 and a live
orchestrator; docs/VERIFICATION.md has the
ledger.

  • Every approval node names the group approvers beside admins, so the
    people who decide and the Slack bot's orchestrator user can sit in a
    group of their own; the orchestrator resolves approver names when it
    creates an approval and takes users and groups, not service accounts.
  • A planned ActiveDR failover and failback runbook. playbooks/dr_switchover.yml
    demotes the link's source pod with quiesce, waits for the link to report
    the target's copy complete, promotes the target pod and records both
    pods' states and the link's new direction; the same playbook with the
    sides swapped fails back. A pod cannot be demoted again while the
    undo-demote pod of an earlier demotion exists, so with
    eradicate_undo_demote true the playbook eradicates those pods first
    and reports them in undo_demote_eradicated; without it the play
    refuses and names them. orchestrator/dr_failover_failback.json runs
    it behind two approvals and sets the variable, and aap/seed.yml
    creates the FlashArray - DR switchover job template.
  • tools/ao_sync.py asks the orchestrator for /api/v1/service_accounts
    (the path it used did not exist; the first live run found it) and
    compares a file with its export without credential_names and without an
    empty extra_vars, neither of which the export carries. Verified live as
    the service account: create, update and unchanged. The dry run says a
    changed definition would publish a new version and no longer guesses
    its number.
  • purefa_syslog parses what Purity//FA 6.10.6 sends: lines without a
    <PRI>, a pid on the program, the command layout of purity.audit
    ([user (local)]@array(id) command. Message ID: N UTC Time: ... Array Name: x, giving command, subcommand, arguments, action,
    utc_time and message_kind: command) beside the session layout, and
    the severity from the text when there is no priority. New programs
    option (vars syslog_programs): the array forwards its controllers'
    whole system log, so only purity.alert, purity.audit and
    purity.test are emitted by default and the rest is dropped and
    counted. syslog.yml skips session records by message_kind.
  • tools/ao_sync.py imports the orchestrator definitions into an
    orchestrator the way orchestrator/README.md ("Importing") describes:
    creates the workflows the project lacks, updates the existing ones by
    name, fills the REPLACE_AT_SYNC placeholders from --aap-credential,
    --aap-integration and --service-account, validates first, publishes,
    reports workflows no file defines and never deletes; --dry-run writes
    nothing. A changed definition is published as the next version, the
    live contract (a version's PATCH changes metadata only). Standard
    library only; tests/unit/test_ao_sync.py drives it against an
    in-process mock of the API.
  • tools/ao_lint.py knows loop nodes: iterate and complete ports, the
    body's return edge (to_port: iterate) as the one allowed back edge
    (loop_ports), and the loop parameter keys.
  • A second orchestrator route: a protection group whose replication is
    overdue or stuck is handed over when orchestrator_enabled is true, and
    orchestrator/replication_overdue_approval.json gates the immediate
    replicated snapshot behind an approval (orchestrator_paths.replication).
  • orchestrator_fallback_local (default false): a capacity handoff that
    produced no execution raises the local request anyway.
  • Slack: notify.yml posts every notification to an incoming webhook when
    SLACK_WEBHOOK_URL is set (env file, or the new Slack webhook
    credential the seed attaches to the notify template); the seed also
    creates a Slack notification template for job failures and the
    controller-tier workflow's approval events when a bot token and channel
    are set.
  • slack/approval_bot.py: a Slack Socket Mode bot that posts pending
    orchestrator and controller approvals as cards with Approve and Reject
    buttons and carries the decision back with the Slack user's name
    (SLACK_APPROVERS limits who may click). Unit-tested with fakes.
  • replicate_now.yml publishes job artifacts (pgroup, snapshot_suffix,
    changed, ...), and a unit test checks that every ${node.artifacts.key}
    a definition reads is published by that template's playbook.
  • purefa_syslog counts the array's own service accounts as automation
    (system_users, default pure_datainsights; purity writes with auth
    source system), so the human-action rule stays quiet for the array's
    scheduled snapshots and tag housekeeping.
  • The ansible-builder definitions build: python_interpreter under
    dependencies pins the base image's Python 3.12, and each collection
    directory names its own dest under additional_build_files. The
    decision environment from decision-environment/execution-environment.yml
    is built on a vfs image store and runs alertmanager.yml through a
    posted alert to a notify record; the two supported-base definitions build
    there as well (PACKAGING.md has the rootless-overlay caveat).
  • aap/decision-environment.yml excludes systemd-python from the pip
    stage: ansible.eda's requirements name it, it has no wheel, and the
    supported base carries no Python headers or compiler to build it.
  • Links people can open: aap/seed.yml sets the platform's base URL for
    notification links (TOWER_URL_BASE, from aap_url_base, the gateway
    URL by default), and the bot's controller cards link to AAP_UI_URL
    when set.
  • purefa_syslog parses the purity.alert layout Purity//FA 6.10.6 sends
    (level word, subject, [code], description, support-case sentence,
    Severity:, UTC and array time, array name, domain, knowledge base link,
    suggested action) into fields, grades the event by the severity word and
    stops the UTC trailer at Array Time:; the syslog alert rule grades an
    info alert as info, names the code in the subject and passes
    alert_code. The bot keeps the channel id Slack returns (chat.update
    refuses the channel name) and a failed card update no longer stops the
    poll.
  • The Slack text of notify.yml carries a real line break between the
    subject line and the detail (a '\n' literal inside a Jinja expression
    is two characters under Ansible's backslash doubling), names the array
    once, drops carriage returns and escapes &, < and >; a unit test
    refuses escape literals inside Jinja expressions outside regex_*
    filters. The admin-authentication drift notice names the baseline values
    beside the observed ones.
  • Verified since 0.2.0 (VERIFICATION.md): the syslog path live on the
    platform from both arrays; the Alertmanager chain into the platform's
    activation, with the capacity rule under a lab override and the
    replica-link rule on a real pause; the DR drill schedule firing on its
    own; the capacity handoff from a running activation through approval to
    the reclaim; the overdue-replication route through its approval to the
    replicated snapshot, once approved from Slack through the bot; the
    notify template posting to Slack from the platform; an approval nobody
    decided expiring and failing its execution at the approval node, with
    fallback_decision making no difference; live purity.alert lines from
    both arrays (a scratch pod stretched between them: an info line from
    each and a critical one, parsed into fields and graded by the rebuilt
    decision environment) and the array's test line over UDP, with the
    arrays replaying queued lines after an 80-minute listener outage; the scrape-failure rule through Alertmanager to the
    platform's activation, firing and resolved; the platform's Slack
    notification on a controller-tier approval and the bot's card for it;
    the alerts signal against a Purity//FA 6.12.4 array; the planned
    failover and failback on the lab's ActiveDR link, both ways, with the
    link reversed in between, then again with both approvals given from
    Slack by the bot's user, the undo-demote pods of the first run
    eradicated on the way and a duplicate launch of the failback (the
    gateway answered 503 after the job was created) absorbed by the
    playbook changing nothing the second time.