Conversation
…e ops report (#61) - Make `name` optional in `psa domain compare` (iterate all domains with --ops) - Add --commit flag to auto-confirm pushes to Ops - Extract _compare_domain_ops helper for per-domain compare+commit logic - Remove `psa ops report` command (replaced by domain compare --ops --commit) - Add tests for commit flow and all-domains mode, delete test_ops_report.py
psa.core.output holds verbosity in a module-global _verbosity. Tests that invoke CLI commands with --quiet (or call set_verbosity directly) leave it stuck at QUIET, silently suppressing print_info in subsequent tests and breaking output-dependent assertions. Adds autouse conftest fixture that resets to DEFAULT before/after each test. Clears 15 pre-existing test failures.
* Bump minimum Python to 3.9; document prereq in README * Drop Python 3.8 classifier (requires-python is >=3.9)
… + environment.conf (#68) * Rename PSA_CUST → DPK_CUST_HOME, hiera env/ → environment/ Hard rename in prep for psadmin.conf 2026 lab content: - env var $PSA_CUST → $DPK_CUST_HOME - config key psa_cust_path → dpk_cust_home - DEFAULT_PSA_CUST → DEFAULT_DPK_CUST_HOME (default /u01/app/io/dpk-cust) - _scaffold_psa_cust → _scaffold_dpk_cust_home - _resolve_cust_path → _resolve_dpk_cust_home - generated hiera.yaml dir env/ → environment/ (fact name unchanged) - help text + status output updated - tests renamed and assertions updated * Hide kit/ops/dpk data; add enable_psa_kit flag psa kit, psa ops, and psa dpk data are hidden from --help (still callable). psa dpk sync --data/--tier/--environments hidden too. New PsaConfig.enable_psa_kit flag (default False): - gates psa kit visibility in --help - _generate_hiera_yaml omits kit layer when False - _generate_puppet_conf omits kit modulepath segment when False Re-enable via 'enable_psa_kit: true' in ~/.config/psa/config.yaml; no code change needed. * Add psa dpk init; rename puppet.conf -> environment.conf; flat layout New 'psa dpk init' scaffolds DPK_CUST_HOME (flat layout) with: - data/{domain,server,environment,tier,zone}/ + READMEs - manifests/site.pp, modules/, common.yaml stub - generated hiera.yaml + environment.conf Requires 'psa init' to have run first. Refuses non-empty target. Renames _generate_puppet_conf -> _generate_environment_conf and _deploy_puppet_conf -> _deploy_environment_conf. environment.conf deploys to <DPK>/puppet/production/environment.conf with content: modulepath = <DPK_CUST_HOME>/modules:<DPK>/puppet/production/modules manifest = manifests/site.pp environment_timeout = unlimited Hiera cust datadir + environment.conf modulepath now use flat layout (<DPK_CUST_HOME>/data, <DPK_CUST_HOME>/modules). psa kit install scaffold delegates to shared helper for layout consistency. psa dpk sync --puppet-conf flag renamed to --environment-conf.
* Add psa dpk module install/list; SudoFileOps.write_text New 'psa dpk module' subgroup clones bolt-on Puppet modules from GitHub (HTTPS) into <DPK_CUST_HOME>/modules/<repo_name>: - install <org/repo>... [--branch] [--dry-run] — shallow clone (--depth 1), skips existing dirs, validates org/repo format - list [--json] — shows git remote/branch/short-commit per module Extends SudoFileOps with write_text(path, content, *, as_root=False). When as_root=True, uses 'sudo bash -c' (not sudo su -) to write to root-owned paths. Content is base64-encoded over the wire. Needed by upcoming 'psa dpk facts' for facts.d/server.yaml writes. * Add psa dpk facts init/set/list New 'psa dpk facts' subgroup manages Puppet Facter external facts in <facts.d>/server.yaml so server identity persists for any Puppet run (psa-cli, cron, manual, Ansible). Commands: - init: write server.yaml with explicitly-passed flags only; backs up existing to .bak; validates --role against known roles (app, appbat, web, prcs, mid, webapp) - set <key> <value>: update or add a single key; preserves other keys; warns on unknown keys but allows them - list [--json]: print current server identity facts.d resolution order: 1. <DPK_BASE>/psft_puppet_agent/facter/facts.d/ 2. /opt/puppetlabs/facter/facts.d/ 3. /etc/facter/facts.d/ Writes go through SudoFileOps.write_text(..., as_root=True), which tries direct write first and falls back to 'sudo bash -c' for root-owned paths. * Wire psa dpk apply to read facts.d/server.yaml (#71) apply now reads <facts.d>/server.yaml at runtime and treats it as a fact source between CLI flags (highest priority) and config defaults (lowest). When a fact is supplied by server.yaml and no CLI override is given, FACTER_<key> is left unset so Facter reads server.yaml directly — works for puppet runs from cron, Ansible, etc. Precedence: 1. CLI flag (--role/--env/--tier/--zone/--pillar) — always sets FACTER_* 2. server.yaml — read by Facter, no FACTER_* override 3. config.ops.* defaults — only when neither CLI nor server.yaml supplies Adds _read_server_facts(dpk_base, config) that searches <dpk_base>/psft_puppet_agent/facter/facts.d/server.yaml, <dpk_base>.parent/psft_puppet_agent/.../server.yaml, /opt/puppetlabs/facter/facts.d/server.yaml, /etc/facter/facts.d/server.yaml — first hit wins. Also broadens psa dpk facts _resolve_facts_d to try the .parent layout so both commands accept the same --dpk-path.
ops/kit hidden from CLI surface in #68; docs/init output still advertised them. Drop OPS sections from README, drop psa ops setup/report/status hints from psa init success output.
ncurses-compat-libs isn't packaged for EL 9+, so the old 'dnf install ncurses-compat-libs' recommendation never works. Detect OS major from /etc/os-release; on EL 9+, plan per-lib ln -s from the installed .6 to the missing .5 (installs ncurses-libs first if .6 also missing). EL 7/8 path unchanged. Also expands the .5 family checked beyond just libncursesw: adds libtinfo, libncurses, libform, libpanel, libmenu since the DPK installer fails partway on EL 9 hitting a different missing .5 the prereq check never flagged. Idempotent: re-running with all .5 libs present is a silent pass.
Drop the hard-fail gate that pointed at the non-existent 'psa init' command. dpk init already calls config.save() at the end (which creates parent dirs), so just let get_config() fall back to defaults when no file exists, and surface a hint to run 'psa config setup' for full PS path auto-detection. Also fix README quick-start to use psa config setup (not the phantom psa init) and the right --runtime-user flag.
Some repos (puppetlabs/puppetlabs-inifile) clone to a name Puppet's modulepath doesn't accept. Allow --as to set a custom directory name. Single-repo only; rejects path-escape characters.
server.yaml was being written to <dpk_base>/psft_puppet_agent/ facter/facts.d, which (a) is inside the DPK tree so a reinstall wipes server identity, and (b) isn't searched by Facter's default external-facts paths so the file got written but never read. Switch default to /etc/puppetlabs/facter/facts.d (Puppet 6+/PE convention), falling back to /etc/facter/facts.d. Replace --dpk-path with --facts-dir as the override on init/set/list. psa dpk apply's _read_server_facts now reads from the same candidates list (imported from facts.py) so writer and reader stay in sync. facts list now reports the searched paths when nothing is found.
In the 2-tier model (DPK_HOME + DPK_CUST_HOME), modules live in DPK_CUST_HOME/modules and are managed by 'psa dpk module install'. Sync's job is just refreshing the generated config files (hiera.yaml, site.pp, environment.conf) into DPK_HOME so Puppet's modulepath in environment.conf can pick them up. Removes --source/--modules flags, _get_source_path(), and _deploy_module_files() — all kit-era plumbing whose only caller was sync. Hard-fails when DPK_CUST_HOME is unset (env or config), pointing at 'psa dpk init'. Kit hiera/modulepath segments in the generators stay (gated on enable_psa_kit) for users still on that opt-in path.
Two related path concepts were conflated under DPK_BASE / --dpk-path:
- DPK_BASE = parent install dir (Oracle convention) — for setup,
cleanup, status, puppet binary lookup
- DPK_HOME = DPK_BASE/dpk = the dpk install dir — for sync, apply,
init (where puppet manifests/hiera/modulepath live)
Sync papered over the difference with auto-append; apply did the
reverse with .parent. Now separated cleanly:
flag --dpk-home / -d, env , derives from config.dpk_base/dpk
flag --base-dir / -b, env (unchanged on setup/cleanup)
flag --dpk-cust-home / -c, env , config.dpk_cust_home
Adds dpk_base field to PsaConfig (load/save/from_environment + SETTABLE_KEYS).
Renames --path -> --dpk-cust-home on dpk init and dpk module install
for symmetric naming. README env-vars section now documents all three.
Sync no longer auto-appends /dpk; takes the input as-is and fails with
a clear error if puppet/ isn't there. Existing -only setups
keep working since DPK_HOME derives from when not set.
opc running 'psa dpk sync' fails on shutil.copy2 because DPK_HOME is owned by psadm2. The codebase already has SudoFileOps (used by psa dpk facts) which falls back to 'sudo su - <runtime_user>' when direct writes are denied. Refactor the three _deploy_* helpers to use it: backup-by-read + write-via-fileops instead of shutil.copy2 + Path.write_text. Helpers take an optional fileops kwarg; sync injects a SudoFileOps(config) so production runs use sudo when needed. Tests pass nothing and get a direct-mode default — no test changes required for existing helper tests.
Sync was failing with just 'Failed to back up X -> Y' and no
clue why. Now SudoFileOps.write_text populates self.last_error
on failure with the actual cause (sudo exit code + stderr,
timeout hint, or direct OSError), and the dpk sync/facts
helpers print it after the high-level error. Common cases:
- 'sudo write failed (exit N): <stderr>' — runtime user
can't write the path, or sudo NOPASSWD missing
- 'sudo write timed out after 30s — likely waiting for
a password prompt. Configure passwordless sudo for the
runtime user.'
- 'direct write failed: <OSError>' — when sudo is disabled
and the path is unwritable
Existing callers unaffected (return type still bool); new
last_error attribute is opt-in via getattr(fileops, ...).
Oracle's psft-dpk-setup.sh runs as root, so DPK_HOME files are
typically root-owned. sudo'ing as runtime_user (psadm2) still
hits Permission denied. Add a one-step escalation: if the
runtime-user write fails, retry as root via sudo bash -c.
Practically: opc -> sudo su psadm2 fails (psadm2 not owner)
-> sudo bash -c succeeds (root can write).
New _write_with_escalation helper wraps the two-attempt
pattern; _backup_and_write uses it for both backup and target.
Files end up root-owned (matching the rest of the DPK install).
…tics Apply was reporting success when puppet hit catalog/Hiera failures because --detailed-exitcodes wasn't passed. Now passes the flag, maps 0/1/2/4/6 to distinct outcomes, streams stderr always, drops Info/Debug from stdout unless --verbose, warns on <0.5s catalog compile, announces ps_role class, counts changed resources. New psa.core.subprocess_runner.stream_subprocess (Popen + pump threads) also adopted by dpk setup/cleanup for consistent capture behavior.
Two follow-ups to the prior fix that Lab 2 still tripped on: 1. Puppet --noop with --detailed-exitcodes can exit 0 even when catalog compilation hits errors (missing Hiera key, unknown class, etc.) -- the error appears on stderr but the resource-state exit code says "no changes." Now scan captured stderr for ^Error:, lookup() did not find, Could not find class, Could not parse, Evaluation Error; if any match, override the exit code and fail. 2. stream_subprocess was routing stderr through Rich's error_console from pump threads; lines were not appearing in the user's terminal. Switch stdout/stderr pumps to write directly to sys.stdout/sys.stderr with explicit flush per line.
Without sudo, puppet apply runs as the invoking user. On lab boxes that's opc, who can't read the full hiera tree -- puppet silently produces an empty catalog and exits 0, hiding the real lookup() / Hiera errors that surface under sudo. Apply now mirrors setup/cleanup: prepend sudo when not root, not the configured runtime_user, and sudo_enabled. FACTER_* are passed via sudo's VAR=val arg form so they survive sudo's env strip. Bump to 0.2.4.
Wraps `puppet lookup <key>` with the same fact resolution and auto-sudo that `apply` uses. Defaults to --render-as yaml; --explain passes through to puppet for the hierarchy walk so users can see which level (if any) matched. Useful for debugging missing Hiera keys (e.g. the lab-2 oracle_client_version case that motivated the apply error-handling fix). Refactors: pulls _resolve_puppet_bin / _build_facter_env / _wrap_with_sudo out of apply's body so both commands share the same machinery. Behavior unchanged for apply -- existing 462 tests still pass; +9 new for lookup. Bump to 0.2.5.
Defaults block configures eyaml_lookup_key with pkcs7 keys under <dpk_home>/puppet/secure/keys/ so encrypted DPK passwords decrypt at lookup time instead of passing ENC[...] strings into Puppet types. Reordered delivered files: defaults -> customizations -> unix -> deployment -> configuration -> patches. Hiera stops at first match, so configuration (broad runtime values) now sits below customization layers that should override it.
eyaml_lookup_key replaces data_hash; keeping both is redundant and the extra data_hash entry can mask the eyaml backend on some hiera versions.
Layered filter on top of existing default. Drops in --summary: - Notify-resource echo lines (~50% of output): every meaningful Notice is duplicated immediately by puppet's Notify resource as 'Notice: /Stage[...]/Notify[<msg>]/message: defined ...'. Drop both single-line and multi-line variants, silently (info is on screen above). - 'Warning: Unknown variable' / 'Warning: ... is deprecated' / 'Warning: ModuleLoader' (counted) - 'Notice: Scope(...)' / 'Notice: Local environment:' (counted) Always preserves Error: lines, Compiled/Applied catalog notices, real Stage[] state changes (User/Exec/password/feature_settings/etc.), and DPK milestone messages (<DPK*>). End-of-run summary block prints hidden warning/notice counts when nonzero. --verbose or --debug override --summary with a one-line warning.
Puppet emits Warning/Notice/Error to stderr, not stdout, so the summary filter was a no-op for the lines it was meant to suppress. Add stderr_filter to stream_subprocess and pass the same callable for both streams in summary mode (shared counters). Default mode keeps stderr verbatim. Stderr capture for fatal-pattern scanning is unaffected.
Puppet wraps stderr lines in color escapes (e.g. \x1b[1;33mWarning:...\x1b[0m in yellow), which made our prefix-based filters miss every Warning/Notice/etc. Strip ANSI for the filter decision only; the original colored line still reaches the user's terminal.
Suppress 'Reading from server.yaml', 'Setting fact:', 'Resolved ps_role' and 'Running: ...' in summary mode. Replace with a single Rich rule showing the resolved env/tier/role. Dry-run is marked in the header. Default mode behavior is unchanged.
Switch from blacklist to allow-list for Notices. Only keep:
- '<DPK*>' milestone markers (DPKUSERS, DPKAPPDOM, DPKBOOTPIADOM, etc.)
- 'Compiled catalog' / 'Applied catalog'
- 'Applying pt|io_*::*' top-level phase markers
- Any Notice mentioning fail/failed/failure (failure context for errors)
Also drop blank lines silently so multi-line Notices that get partially
suppressed don't leave orphaned separators.
Drops 'Notice: /Stage[...]/User/Exec/Pt_ulimit_entry/feature_settings/etc'
per-resource state changes, 'Performing action sstatus', 'The user X is
present', and similar — all noise at summary level. Failure context
('Job for X.service failed') is still kept via the \bfail rule.
When a content-bearing Notice or Warning is dropped, also drop the following non-prefixed lines until a new Notice/Warning/Error/Info/Debug prefix starts a fresh logical line. This kills the multi-line value dumps that puppet emits for Pt_webserver_domain settings, site_list, config_settings, etc — the opening 'Notice: ...: defined X as [' was already dropped, but the value lines slipped through as continuations. Bare 'Notice:' lines (multi-line milestone openers like 'Notice:\n<DPKUSERS>...') are exempt: their content is on the next line and we want it kept. Tuxedo error context after an 'Error:' line is still kept because the Error: prefix clears the suppression flag.
…PK paths under /u01/app/psa
Removes just domain configs and DPK systemd units (psft-{appserver,prcs,pia}-*),
leaving PS_HOME, Tuxedo, WebLogic, DB client, and DPK install intact. Per-domain
flow: stop -> kill (if needed) -> psadmin -c|-p|-w delete -> rm -rf fallback ->
systemctl stop/disable/rm -> trailing daemon-reload. Legacy full cleanup path
unchanged when --domains-only not passed.
Adds --domain, --type, --keep-services, --force flags (scoped to --domains-only).
…g types find_domain returns the first match; with DPK installs that share one name across app+prcs+web (e.g. ihlab), only the app domain got cleaned. Switch to filtering run_discovery results so a bare --domain hits every type.
* Release 0.3.0 prep: LICENSE, SECURITY, CHANGELOG, version bump to 0.3.0/Beta; drop cache/secrets stubs * Add GitHub workflows (test+docs) and PR/issue templates (closes #53) * Add MkDocs Material docs site + public README * Commit uv.lock for reproducible installs + CI cache
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First public release of psa-cli. Merging
staging(40 commits ahead ofmain) intomainfor the v0.3.0 cut.Highlights
list,status,start,stop,restart,bounce,comparewith Rich + JSON output.stage → setup → init → sync → apply → cleanup, 3-tier Hiera, custom module install,dpk lookup,dpk facts.dpk apply --summary— filtered, readable Puppet output with surfaced errors and exit-code diagnostics.SudoFileOpstransparently escalates when the runtime user can't write.Known limitations (shipped as-is, fixes tracked for future)
psa domain --jsonincludes raw config content with encrypted password fields (#23)Full notes: CHANGELOG.md
Closes
Closes-milestone: v0.3