feat: centralized logging with --verbose/--quiet/--log-file/--output json CLI flags - #22
Merged
Conversation
…e CLI flags Create p2m/logging_config.py with configure_logging() that sets up: - RichHandler on sys.__stderr__ (bypasses OTel/Phoenix stderr wrappers) - Optional FileHandler for --log-file - Level mapping: default=INFO, --verbose=DEBUG, --quiet=WARNING - LiteLLM/httpx loggers pinned to WARNING regardless of verbosity Wire --verbose/-v, --quiet/-q, --log-file into the top-level CLI group so all subcommands inherit the logging configuration.
Replace root-level logging.warning()/logging.info() with module-level log = logging.getLogger(__name__) in: - p2m/core/transcript.py (2 sites) - p2m/stages/rollout.py (4 sites) - p2m/stages/judge.py (2 sites) Named loggers enable per-module filtering and show the source module in log output.
Remove the custom _progress() function (sys.__stderr__ workaround) and replace all 32 call sites with log.info(). The RichHandler from logging_config.py already writes to sys.__stderr__, preserving the same OTel/Phoenix bypass. Upgrade error messages to log.error() and route tracebacks through the logging system (exc_info=True) instead of traceback.print_exc(). Remove unused traceback import.
runner.py: 2 print() calls → log.error() for config errors config.py: 2 print() calls → log.warning() for validation warnings Remove unused sys import from config.py.
New test_logging_config.py covering: - Default/verbose/quiet level mapping - Console handler targets sys.__stderr__ - File handler creation and parent dir creation - LiteLLM loggers pinned to WARNING - Repeated calls don't duplicate handlers Add CLI flag acceptance tests to test_cli.py for -v/--verbose, -q/--quiet, and --log-file.
Tests previously patched sys.stderr/sys.__stderr__ to capture output.
Now that runner output goes through the logging system, switch to
assertLogs('p2m.runner') which captures log records directly.
Remove unused io import.
… retry noise
Replace raw sys.__stderr__.write() for per-seed rollout progress
lines with log.info() (success) / log.warning() (error). These lines
previously appeared without any level prefix (INFO/WARNING/ERROR).
Suppress openai and httpcore loggers to WARNING — the openai SDK logs
every retry attempt at INFO ('Retrying request to /chat/completions
in N seconds'), which floods output during rate limiting. p2m's own
model_client already handles retry reporting.
Remove unused sys import from rollout.py.
Convert %s/%d placeholder-style log calls to f-strings for readability. The performance difference is negligible at this call volume (~30 calls per pipeline run).
Add log = logging.getLogger(__name__) and debug-level log lines to: - stages/policy.py — model, behavior_count, web_search - stages/design.py — factor_sizes after generation - stages/seeds.py — kinds enabled, sample sizes, tool_source - stages/systematization.py — concept, model, mode, web_search - stages/systematization_convert.py — model, behavior_count_hint - core/model_client.py — model, api_mode, schema_name per LLM call All new log lines use log.debug() so they only appear with --verbose. The runner already handles INFO-level stage start/done banners.
The policy stage makes two sequential LLM calls (systematization then convert) that can each take 30-90s. Add INFO-level progress between sub-steps so the output doesn't appear frozen: [1/2] Researching risk taxonomy... [1/2] Risk taxonomy complete [2/2] Converting to structured policy...
Strip leading ' ' from all runner log messages — these were from the _progress() era when messages were written raw to stderr and needed manual indentation. RichHandler handles column alignment, so the extra spaces caused misalignment with stage-internal logs like rollout per-seed progress.
Add log_heartbeat() async context manager to async_utils.py that logs a 'still working, Ns elapsed' message every 15 seconds while an LLM call is in progress. Prevents the CLI from appearing frozen during the policy stage's two sequential calls (systematization + convert), which can each take 30-90 seconds.
# Conflicts: # p2m/core/model_client.py
…mand Users naturally type 'p2m run --verbose' but Click only accepts group options before the subcommand. Fix by adding the same flags to the run subcommand and re-configuring logging if they're passed there. Both forms now work: p2m --verbose run --config ... p2m run --verbose --config ...
Replace pre-call debug logs with a compact post-call summary: generate: model=azure/gpt-5.4, 1.2s, 850+2400 tokens, finish=stop Each generate/generate_structured/generate_with_tools call now logs model name, wall-clock latency (including retries), prompt+completion token counts, and finish_reason. All at DEBUG level.
The Responses API uses input_tokens/output_tokens (not prompt_tokens/completion_tokens) and status (not finish_reason). _normalize_usage now falls back to the Responses API field names, and finish_reason falls back to the status field. Previously showed '? tokens, finish=?' for all Responses API calls.
Apply consistent formatting to all user-facing log messages: - Add [stage_name] prefix to all stage messages - Use ✓ for success and ✗ for failure consistently - Capitalize first word after prefix - Pipeline-level messages use 'Pipeline' (capitalized, no prefix) - Standardize skip/error/done patterns Before: INFO Generating behavior taxonomy... INFO ✓ Generated 5 behaviors (41.2s) INFO rollout [4/5] ✔ [prompt] Multi-turn... INFO pipeline completed (162.3s) After: INFO [policy] Generating behavior taxonomy... INFO [policy] ✓ Generated 5 behaviors (41.2s) INFO [rollout] [4/5] ✓ [prompt] Multi-turn... INFO Pipeline completed (162.3s)
Add JSON log format for CI pipeline consumption via --output json. Each log record emits one JSON object with timestamp, level, logger, message, and optional exception fields. - New _JsonFormatter class in logging_config.py - --output flag (text|json) on the top-level CLI group - Tests for JSON format validity and exception inclusion
Apply consistent formatting to all user-facing log messages: - Add [stage_name] prefix to all stage messages - Use ✓ for success and ✗ for failure consistently - Capitalize first word after prefix - Pipeline-level messages use 'Pipeline' (capitalized, no prefix) - Standardize skip/error/done patterns Before: INFO Generating behavior taxonomy... INFO ✓ Generated 5 behaviors (41.2s) INFO rollout [4/5] ✔ [prompt] Multi-turn... INFO pipeline completed (162.3s) After: INFO [policy] Generating behavior taxonomy... INFO [policy] ✓ Generated 5 behaviors (41.2s) INFO [rollout] [4/5] ✓ [prompt] Multi-turn... INFO Pipeline completed (162.3s)
AaronAspinwall123
approved these changes
May 8, 2026
AaronAspinwall123
added a commit
that referenced
this pull request
May 8, 2026
Resolves conflicts in p2m/runner.py and p2m/stages/rollout.py introduced by main's centralized-logging refactor (PR #22) and policy/rollout error-handling improvements (PR #24). p2m/runner.py: kept the artifact-cache stage path (prepare_artifact_plan / activate_artifact_plan / override_cacheable_output_paths) and combined it with main's logging style. The two stage-skip messages now use log.info with the new '[stage] Skipped' prefix instead of the deleted _progress() helper, and _progress() itself was removed since main's logging configuration writes to sys.__stderr__ via RichHandler, neutralizing the original Phoenix sys.stderr-wrapping concern. p2m/stages/rollout.py: dropped the sys.__stderr__ rollout progress writer in favor of main's log.info / log.warning calls (same rationale). Kept the 're' import that the cache code uses for _VERSIONED_ARTIFACT_RE; dropped the now-unused 'sys' import. Verified: pytest passes for tests/test_artifact_cache.py, tests/test_runner_artifact_cache.py, tests/test_runner_progress.py, and tests/test_viewer_server_artifacts.py (56 passed, 2 skipped). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AaronAspinwall123
added a commit
that referenced
this pull request
May 8, 2026
…user save_dir override Addresses Jake's review on the artifact-cache PR. 1. Disk-leak on stage failure (Jake's #2): when a cacheable suite stage failed after prepare_artifact_plan allocated vNNNN/ but before finalize_artifact_plan wrote the sidecar, the partial directory stayed on disk forever. _next_version kept incrementing past abandoned slots and the stage_root accumulated dead version directories on every failed run. Adds discard_artifact_plan(ctx, plan) to remove the version directory and pop the orphaned ctx['artifact_versions'] entry. Wired into runner.py's stage exception path. No-ops for reused plans so a downstream failure cannot blow away a healthy upstream cache hit. latest.json is left untouched (finalize is its only writer for non-reused plans, so a discarded plan never touched it). 2. Silent override of user save_dir (Jake's non-blocking #2): override_cacheable_output_paths now log.warnings whenever it replaces a user-supplied save_dir/save_path with the versioned cache location. Customers who set save_dir in YAML get a clear actionable message instead of seeing their value silently ignored. No warning when no user value was set. 3. Concept-hash transitive flow (Jake's non-blocking #1): added a comment in _stage_descriptor explaining that concept_hash is computed only for policy and propagates through the dependency chain to design and seeds via _dependency_descriptor. Notes the safety invariant (every cacheable stage must depend on its upstream) so a future stage that breaks the chain triggers a code-review flag rather than silent stale-cache reads after a concept edit. Note: Copilot findings on _metadata_outputs_exist / _metadata_output_paths primary-key KeyError and basename validation (Jake's #1) were already addressed in de9a32a (round 4) — _metadata_output_paths now overlays metadata onto the canonical _output_paths default keyset and _metadata_outputs_exist verifies every expected file exists on disk via the merged path map. _is_safe_artifact_basename rejects unsafe filenames before they become Path components. Tests: +7 regression tests (4 for discard helper covering missing dir, reused plan, ctx cleanup, version-slot reuse; 2 for override warning behavior; 1 runner integration test that fails design mid-stage and asserts vNNNN is cleaned up plus next run reuses the freed slot). Verified: 585 passed, 14 skipped (2 pre-existing Windows logging tempdir flakes from main's PR #22 deselected). npm --prefix viewer run check: 0 errors / 0 warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Merged
tangym
added a commit
that referenced
this pull request
May 11, 2026
…cache Align artifact_cache.py and viewer_read_model.py with the centralized logging convention from PR #22. Replace 10 print(file=sys.stderr) calls with log.warning() using structured placeholders so they respect --quiet, --log-file, and --output json. Remove unused sys imports from both modules.
tangym
added a commit
that referenced
this pull request
May 11, 2026
…pecific error types (#34) * fix: add try-except for file I/O and YAML/JSON parse errors Wrap unguarded file reads and parse calls with specific exception handlers that include the file path in the error message: - config.py: load_config catches FileNotFoundError, PermissionError, yaml.YAMLError and re-raises as ConfigError - otel.py: _parse_otlp_json catches FileNotFoundError, JSONDecodeError - tools.py: load_toolset_file catches FileNotFoundError, yaml.YAMLError - systematization_convert.py: run_systematization_to_policy catches FileNotFoundError, JSONDecodeError * fix: add error handling for callable and HTTP session imports - CallableSession.open: catch ModuleNotFoundError, AttributeError with actionable messages including the callable ref string - OTelTracedSession.open: same pattern for OTel-traced callable imports - HTTPEndpointSession.run_turn: catch aiohttp.ClientResponseError and ClientError with endpoint URL and HTTP status in the message * fix: improve judge and policy stage error handling - judge.py: catch JSONDecodeError when loading policy file, add LLM error types and json/ValueError before the catch-all in the worker, log traceback via log.debug for all caught worker exceptions - policy.py: include model name and raw response text in the error message when policy generation returns non-JSON output * fix: improve rollout and systematization error handling - rollout.py worker: add LLM error types re-raise before catch-all, add ValueError/KeyError handler, log.debug with traceback and seed_id for all caught worker exceptions - rollout.py auditor: log.debug with traceback on each retry failure - systematization.py: catch JSONDecodeError on json.loads fallback with raw text snippet in the error message * fix: add specific error handling for Phoenix collector Catch ConnectionError separately from generic Exception in PhoenixCollector.get_spans with a targeted message about Phoenix connectivity. Validate 'context.trace_id' column existence before filtering. * test: add 21 test cases for exception handling paths Cover all new error handling across config, otel, tools, session, otel_session, judge, systematization_convert, rollout, and collector. Verify exception types, error messages include context (file paths, module names, endpoint URLs), and debug logging fires with tracebacks in worker failures. * fix: skip Phoenix collector tests when pandas is not installed CI environment doesn't include pandas (optional dependency). The PhoenixCollector.get_spans method does 'import pandas' at the top, which fails before our exception handling runs. * style: replace print(file=sys.stderr) with log.warning() in artifact cache Align artifact_cache.py and viewer_read_model.py with the centralized logging convention from PR #22. Replace 10 print(file=sys.stderr) calls with log.warning() using structured placeholders so they respect --quiet, --log-file, and --output json. Remove unused sys imports from both modules. * fix: address review feedback — hoist aiohttp import, use high port in test - Move 'import aiohttp' from run_turn to open(), store as self._aiohttp to avoid re-importing on every turn call - Use port 59123 instead of port 1 in HTTPEndpointSession test for better portability across CI/container environments
AaronAspinwall123
added a commit
that referenced
this pull request
May 14, 2026
Lifts the four product-code changes from PR #44 into our branch and removes the corresponding benchmark monkey-patches that were doing the same job out-of-tree. Changes: * CallableSession.run_turn now reclassifies user-callable provider errors via `_classify_llm_error` so litellm BadRequestError / ContentPolicyViolationError emerge as typed `LLMInputError` (with `__cause__` preserved) instead of bypassing the typed-exception layer the rollout/judge stages key off. Unclassified exceptions propagate untouched. (PR #44 commit 0184d8d) * `_run_prompt_seed` records a `[TARGET INPUT REFUSED: ...]` event in the transcript and sets `stop_reason='target_input_refused'` when the target's run_turn raises `LLMInputError`. Other classified errors still propagate. (PR #44 commit 82cf339) * `_run_auditor_target_loop` splits the combined classified-error catch into a typed `LLMInputError` branch that records `[TARGET INPUT REFUSED: ...]` / `[AUDITOR INPUT REFUSED: ...]` events and sets `stop_reason='target_input_refused'` / `'auditor_input_refused'` respectively. Auth / rate-limit / 5xx errors still propagate. (PR #44 commits 82cf339 + f265154) * judge `_worker` splits the combined catch so `LLMInputError` becomes a synthesized `score_row` with `judge_status='filter_skipped'` and `judge_error='judge_input_refused: ...'`. Rate-limit / provider 5xx still go to the per-row error path. Auth still fails fast. (PR #44 commit dcaa91f) * New `P2M_ROLLOUT_ERROR_FAIL_RATIO` env var (default 0.10) caps the rollout stage's tolerance for *untyped* worker errors. Below the threshold: warn and continue (existing soft-fail). Above: raise the first error. Typed refusals (target_input_refused, auditor_input_refused, target_error) are NOT counted toward the ratio — they're recorded data, not real failures. Inspired by PR #44 commit 15332c8 but scoped to untyped errors only instead of #44's blanket runtime_error catch-all (which would have hidden real bugs behind the same threshold). * scripts/benchmark.py: drops `_install_content_filter_tolerance`, `_install_judge_skip_blocked`, and `_is_content_policy_violation` helpers (~210 lines). The product-code typed handlers above replace them. The benchmark now scans `transcripts.jsonl` and `scores.jsonl` after the run completes and counts typed stop_reasons / judge_status values. CSV columns updated to typed names: `target_input_refused_count`, `auditor_input_refused_count`, `target_error_count`, `judge_filter_skipped_count` (replaces the legacy `content_filter_blocked` / `target_error_tolerated` aggregates). `--no-tolerate-content-filter` flag preserved as a no-op for back-compat. Tests: * test_run_turn_reclassifies_litellm_bad_request_as_input_error * test_run_turn_passes_through_unclassified_exceptions * test_run_prompt_seed_records_target_input_refusal * test_run_rollout_isolates_target_input_refusal_to_one_seed * test_run_rollout_isolates_auditor_input_refusal_to_one_seed * test_run_rollout_still_fails_fast_on_provider_5xx * test_run_rollout_fails_when_untyped_error_ratio_exceeds_threshold (new — covers the threshold introduced in this commit) * test_run_judge_isolates_input_refusal_to_one_seed * test_run_rollout_keeps_partial_successful_transcripts_when_later_worker_fails (updated — sets P2M_ROLLOUT_ERROR_FAIL_RATIO=0.6 since the original 2-seed/1-failure setup now exceeds the production 10% default) 654 passed, 14 skipped, 13 subtests, 2 deselected (pre-existing Windows tempdir flakes in test_logging_config.py from main's PR #22). 0 new test failures. viewer check: 0 errors / 0 warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tangym
added a commit
that referenced
this pull request
May 14, 2026
All other modules use 'log = logging.getLogger(__name__)' per PR #22.
tangym
added a commit
that referenced
this pull request
May 15, 2026
…ndling (#42) * fix: add try-except for file I/O and YAML/JSON parse errors Wrap unguarded file reads and parse calls with specific exception handlers that include the file path in the error message: - config.py: load_config catches FileNotFoundError, PermissionError, yaml.YAMLError and re-raises as ConfigError - otel.py: _parse_otlp_json catches FileNotFoundError, JSONDecodeError - tools.py: load_toolset_file catches FileNotFoundError, yaml.YAMLError - systematization_convert.py: run_systematization_to_policy catches FileNotFoundError, JSONDecodeError * fix: add error handling for callable and HTTP session imports - CallableSession.open: catch ModuleNotFoundError, AttributeError with actionable messages including the callable ref string - OTelTracedSession.open: same pattern for OTel-traced callable imports - HTTPEndpointSession.run_turn: catch aiohttp.ClientResponseError and ClientError with endpoint URL and HTTP status in the message * fix: improve judge and policy stage error handling - judge.py: catch JSONDecodeError when loading policy file, add LLM error types and json/ValueError before the catch-all in the worker, log traceback via log.debug for all caught worker exceptions - policy.py: include model name and raw response text in the error message when policy generation returns non-JSON output * fix: improve rollout and systematization error handling - rollout.py worker: add LLM error types re-raise before catch-all, add ValueError/KeyError handler, log.debug with traceback and seed_id for all caught worker exceptions - rollout.py auditor: log.debug with traceback on each retry failure - systematization.py: catch JSONDecodeError on json.loads fallback with raw text snippet in the error message * fix: add specific error handling for Phoenix collector Catch ConnectionError separately from generic Exception in PhoenixCollector.get_spans with a targeted message about Phoenix connectivity. Validate 'context.trace_id' column existence before filtering. * test: add 21 test cases for exception handling paths Cover all new error handling across config, otel, tools, session, otel_session, judge, systematization_convert, rollout, and collector. Verify exception types, error messages include context (file paths, module names, endpoint URLs), and debug logging fires with tracebacks in worker failures. * fix: skip Phoenix collector tests when pandas is not installed CI environment doesn't include pandas (optional dependency). The PhoenixCollector.get_spans method does 'import pandas' at the top, which fails before our exception handling runs. * style: replace print(file=sys.stderr) with log.warning() in artifact cache Align artifact_cache.py and viewer_read_model.py with the centralized logging convention from PR #22. Replace 10 print(file=sys.stderr) calls with log.warning() using structured placeholders so they respect --quiet, --log-file, and --output json. Remove unused sys imports from both modules. * fix: address review feedback — hoist aiohttp import, use high port in test - Move 'import aiohttp' from run_turn to open(), store as self._aiohttp to avoid re-importing on every turn call - Use port 59123 instead of port 1 in HTTPEndpointSession test for better portability across CI/container environments * Security hardening: validate dynamic imports, URLs, and credential handling Add p2m/core/security.py with centralized security validation: - validate_callable_ref(): block dangerous path segments in callable target references (FINDING-001, CWE-94) - validate_sys_path_addition(): scope sys.path additions to workspace boundaries (FINDING-002, CWE-427) - validate_endpoint_url(): block SSRF via private IPs and cloud metadata endpoints (FINDING-003, CWE-918) - sanitize_payload(): redact credentials from artifact files (FINDING-004, CWE-532) - validate_module_ref(): block dangerous segments in connector/tool module references (FINDING-006, CWE-94) - _validate_module_file_path(): enforce workspace boundaries on direct .py file loading (FINDING-010, CWE-94) - validate_dotenv_location(): warn on suspicious .env locations (FINDING-009, CWE-426) - P2M_NO_AUTO_ENV opt-out for auto_envvar_prefix (FINDING-005, CWE-15) - _require_within() on resolved artifact paths (FINDING-007, CWE-22) Opt-out environment variables for backward compatibility: - P2M_TRUST_CALLABLE=1: suppress callable import warnings - P2M_ALLOW_PRIVATE_ENDPOINTS=1: allow localhost/private endpoints - P2M_NO_DOTENV=1: disable automatic .env loading - P2M_NO_AUTO_ENV=1: disable P2M_* env var overrides Update test_exception_handling.py to use P2M_ALLOW_PRIVATE_ENDPOINTS for localhost endpoint test. * Fix SSRF DNS rebinding bypass, sanitization depth leak, and response credential leak - validate_endpoint_url(): resolve hostnames via socket.getaddrinfo() and validate resulting IPs against blocked ranges (FINDING-002, CWE-918) - sanitize_payload(): redact at max_depth instead of passing through unsanitized (FINDING-003, CWE-200) - HTTPEndpointSession: add _sanitize_response_text() to redact credential patterns before writing to transcripts (FINDING-004, CWE-532) * fix: add Azure Wireserver IP (168.63.129.16) to SSRF blocklist * refactor: remove vestigial dotenv/env-var security machinery Remove P2M_NO_DOTENV, P2M_NO_AUTO_ENV, P2M_TRUST_CALLABLE and associated code (should_load_dotenv, validate_dotenv_location, is_callable_trusted, P2MSecurityWarning). These addressed scenarios from an incorrect threat model — load_dotenv(override=False) already safe, CI owns its env vars, and callable import warnings block the primary workflow. Keep: format validation, blocklist, SSRF prevention, credential sanitization. * style: use 'log' instead of '_log' in session.py for consistency All other modules use 'log = logging.getLogger(__name__)' per PR #22. * test: add unit tests for security module 50 tests covering validate_callable_ref, validate_module_ref, validate_sys_path_addition, validate_endpoint_url (SSRF/DNS rebinding), sanitize_payload, and _sanitize_response_text.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces four competing output mechanisms (unconfigured
logging.*,_progress()→sys.__stderr__,click.echo(err=True),print(file=sys.stderr)) with a single centralized logging system.Before

After

Verbose


JSON

Changes
Phase 1: Logging bootstrap
p2m/logging_config.pywithconfigure_logging()—RichHandleronsys.__stderr__(preserves OTel/Phoenix bypass), optionalFileHandler--verbose(DEBUG),--quiet(WARNING),--log-file <path>,--output jsonPhase 2: Unify stdlib loggers
logging.warning()/logging.info()with namedlog = logging.getLogger(__name__)in transcript, rollout, judge (8 sites)Phase 3: Migrate
_progress()→log.info()_progress()function and replace 32 call siteslog.error()withexc_info=TruePhase 4: Migrate raw
print(file=stderr)runner.pyconfig errors →log.error()config.pyvalidation warnings →log.warning()Phase 5: Add logging to silent modules
log.debug()in policy, design, seeds, systematization, systematization_convert, model_clientlog.info()sub-step progress in policy stage (two sequential LLM calls)Phase 6: Structured JSON output
--output jsonflag for CI pipeline consumption_JsonFormatteremits one JSON object per log record with timestamp, level, logger, message, and optional exceptionAdditional improvements
[stage_name]prefix format with ✓/✗ symbolssys.__stderr__.write())"Retrying request to /chat/completions")log_heartbeat()for long-running LLM calls (15s periodic "still working" messages)input_tokens/output_tokens→prompt_tokens/completion_tokens)--verbose/--quiet/--log-file/--outputaccepted on bothp2mgroup andrunsubcommandTest results
544 passed, 13 skipped (1 pre-existing Docker permission failure)