Zscripts

Zscripts is a structured log collection, normalization, redaction, diagnostics, and reporting toolkit for developers and automation systems.

PUBLIC BETA — ACTIVE DEVELOPMENT

Zscripts is actively developed and is not guaranteed to parse every possible third-party log format. Adapter output, diagnostics, and generated reports should be reviewed before use in production or publication.

Redaction is a defense-in-depth feature, not a substitute for reviewing sensitive output. Never assume automated redaction has identified every credential, personal record, proprietary value, or organization-specific secret.

Public repository visibility is distinct from a tagged stable release. Until a stable release is explicitly published, the current main branch and any pre-1.0 artifacts should be treated as beta software.

Public narrative:

Zscripts converts raw development and CI logs into normalized, redacted, diagnosable, and reportable output through a reusable Python CLI and adapter architecture.

End-to-End Demonstration

Representative raw log input:

============================= test session starts =============================
FAILED tests/test_services.py::test_generate_report_applies_redaction - AssertionError: API_KEY=not-a-real-secret-redaction-fixture-1234567890abcdef leaked
=========================== short test summary info ===========================
1 failed, 24 passed

Normalized output (python cli.py --adapter ci parse --input examples/raw_to_report/raw.log):

{
  "tool": "pytest",
  "ecosystem": "python",
  "status": "failed",
  "summary": "No summary provided.",
  "errors": [
    {
      "message": "tests/test_services.py::test_generate_report_applies_redaction - AssertionError: API_KEY=not-a-real-secret-redaction-fixture-1234567890abcdef leaked"
    }
  ]
}

Redacted output (python cli.py redact --input examples/raw_to_report/raw.log):

... AssertionError: API_KEY=[REDACTED] leaked

Generated Markdown report (python cli.py --adapter ci report --input examples/raw_to_report/raw.log --format markdown --redact --output report.md):

# pytest Report

- **Status:** failed
- **Severity:** error

## Summary
[FAILED] pytest run for python | No summary provided. | Errors: 1

flowchart LR
    A[Raw Log File or External Command] --> B[Adapter]
    B --> C[Normalized Schema]
    C --> D[Redaction Pipeline]
    D --> E[Diagnostics Snapshot]
    D --> F[Markdown or JSON Report]

Loading

Highlights

• Unified CLI entry point (python cli.py) orchestrating collection, parsing, reporting, diagnostics, and extension tooling.
• Strictly typed configuration system (zscripts.config.ToolkitConfig) with support for JSON/TOML files and inline --set overrides.
• Observability pipeline with structured logging, Prometheus metrics, and diagnostics snapshots suitable for dashboards.
• Automation adapters (agents/, adapters/) that expose the CLI surface to external systems without shelling out.
• Comprehensive pytest suite covering operations, adapters, observability, and infrastructure layers.
• Required JSON Schema validation for normalized payloads; invalid payloads are rejected in editable and isolated-wheel installations.
• Experimental local repository review workspace with bounded, read-only Python AST analysis, atomic SQLite snapshots, and the complete Scan → Explore → Review → Compare → Handoff workflow across Overview, Symbols, Relationships, Findings, Compare, and Handoff views.

Quickstart

Run commands from the repository root. The top-level cli.py shim simply dispatches to zscripts.cli.main(), and installed environments also expose the zscripts console command.

# Inspect the active sandbox guardrails
python cli.py guardrails

# Equivalent installed entry point
zscripts guardrails

# Collect logs from an external command with redaction enabled
python cli.py collect --command pytest --redact

# Parse a log file into the normalised schema
python cli.py parse --input examples/python/sample.log

# Generate a concise summary for the collected log
python cli.py summarize --input examples/python/sample.log

# Produce a detailed explanation of failures and guardrails
python cli.py explain --input examples/python/sample.log

# Generate a Markdown summary and write it to disk
python cli.py report --input examples/python/sample.log --format markdown --output report.md

# Redact sensitive values from collected output using configured patterns
python cli.py redact --input examples/python/sample.log

# Capture diagnostics including metrics and extension inventory
python cli.py diagnostics --include-metrics --format json

# Scaffold a new extension skeleton
python cli.py extensions scaffold demo_extension --directory ./my_extensions

# Discover bundled example logs for each adapter
python cli.py examples --format json

# Inspect supported adapters, descriptions, and sample logs
python cli.py adapters --format json

Global flags such as --config, --set, --adapter, --enable-telemetry, --log-level, and --log-format are available to every command. See agents/cli_adapter.py for a machine-readable description of the surface area.

Experimental Repository Review Workspace

The workspace is a local-first product slice for scanning an ordinary Python repository without importing or executing it. It stores metadata-only snapshots outside the analyzed repository and serves responsive Overview, Symbols, focused Relationships, reviewable Findings, deterministic Compare, and bounded local Handoff views from one localhost-only process.

# Editable development checkout
python -m pip install -e ".[dev,workspace]"
pnpm --dir workspace-ui install --frozen-lockfile
pnpm --dir workspace-ui build
python scripts/build_workspace_assets.py

# Start the local workspace at http://127.0.0.1:8765
zscripts workspace

# Or scan through the experimental CLI
zscripts experimental analyze /path/to/repository --json

Default discovery excludes likely credentials, generated/cache directories, binaries, ignored files, and symlinks. Scans have file-count, per-file, and total-byte limits; cancelled or failed attempts never appear as completed snapshots. Ordinary use makes no outbound network request.

This is PUBLIC BETA — ACTIVE DEVELOPMENT. Static syntax evidence cannot prove runtime behavior, architectural intent, code safety, or framework semantics. Relationship exploration is deliberately bounded and reports ambiguous or unresolved evidence instead of guessing. Findings are deterministic static-analysis signals, not diagnoses: the workspace shows their metric, threshold, confidence, source evidence, lifecycle, and explicit local review decision without changing the analyzed repository. The Findings view uses the repository's current lifecycle even when an older supported snapshot is selected, and it visibly skips automatic resolution when truncation or parse gaps make absence unreliable. Compare matches immutable evidence by conservative logical keys and never infers renames. Handoffs include only explicitly selected bounded evidence; review notes require a separate per-finding opt-in and remain local. Review the complete repository review guide for installation, API, storage/deletion, privacy, exclusions, limitations, and validation details.

Helpers (Legacy and Optional)

The helper collection under zscripts/helpers is not part of the strict core identity and is treated as legacy/optional utility code. Phase 2A keeps all 154 tracked helper modules in the wheel without changing their behavior. Seven registry-exposed modules are temporary import/registry compatibility points; all other helpers are legacy, unsupported, and temporarily wheel-included.

Helper modules are grouped by domain:

• Image processing (zscripts.helpers.pillow)
• Web crawling (zscripts.helpers.web_crawl)
• Data manipulation (zscripts.helpers.pandas)
• And more

Install helpers extras only when needed:

• pip install .[helpers]
• pip install .[helpers-web]
• pip install .[helpers-ml]

The helpers-ml extra installs only the repository-managed scikit-learn and tiktoken dependencies. It intentionally does not provision Torch or TorchText. The unchanged historical ML source remains wheel-included as unsupported compatibility material; anyone deliberately running it must provision and own a separate environment.

The Phase 2A compatibility window begins when Phase 2A merges and lasts until both 90 calendar days and one documented public-beta deprecation cycle have completed; the later condition controls. Phase 2B requires consumer review and separate owner approval. This public-source beta commitment is not a stable semantic-version guarantee. See docs/operations/LEGACY_HELPER_COMPATIBILITY.md for the enforceable contract, docs/operations/LEGACY_HELPER_CONSUMER_REVIEW.md for the evidence-backed consumer review, docs/operations/LEGACY_HELPER_DEPRECATION_NOTICE.md for the active public-beta deprecation notice, and docs/operations/LEGACY_ML_DEPENDENCY_POLICY.md for the managed legacy-ML dependency boundary.

Removing Torch from project-managed profiles under issue #130 changes only the dependency boundary. It does not remove or modify helper source, change package discovery or registry targets, complete the deprecation cycle, or authorize Phase 2B.

Use the registry system to call helpers by tag:

from zscripts.helpers.registry import call

# Add watermark to image
call("pillow.add_watermark", input_path="img.png", output_path="out.png", text="Demo")

Repository Layout

The root directory intentionally stays small. Consult the README in each folder for deeper context.

• adapters/ – Adapter interfaces and language-specific integrations.
• agents/ – Automation-friendly wrappers that describe the CLI for AI clients.
• configs/ – Version-controlled configuration defaults (see configs/README.md).
• docs/ – Documentation hub (docs/INDEX.md links architecture, automation, and planning material).
• examples/ – Sample projects and captured artifacts used in guides/tests.
• schemas/ – JSON schema definitions for normalised logs.
• scripts/ – Developer utilities for scaffolding, diagnostics, and releases.
• tests/ – Pytest suite mirroring the runtime modules.
• zscripts/ – Core package containing the CLI, runtime services, extensions, and observability infrastructure. Includes helpers/ for domain-specific Python utilities.

Support and usage references:

• Adapter support matrix: docs/adapters/SUPPORT_MATRIX.md
• Raw log to normalized/redacted walkthrough: docs/guides/RAW_LOG_TO_REDACTED_REPORT.md
• GitHub Actions usage examples: docs/guides/GITHUB_ACTIONS_USAGE.md

Governance and contribution references:

• CONTRIBUTING.md – Setup, quality gates, and pull-request expectations.
• docs/TASKLIST.md – Historical task ledger; current active work is tracked in GitHub issues.
• CHANGELOG.md – Chronological record of notable changes.

Development Workflow

Create a virtual environment and install the optional tooling extras:

python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install .[dev,helpers,workspace]  # Includes local workspace development

Execute the fast contributor gate with:

make check  # formatting, lint, mypy, security, pytest

Use make quality for the complete hosted-CI contract (audit, binary scan, coverage ≥85%, documentation, packaging, zipapp, and diagnostics), and make release for the local release contract (quality plus redaction, tracked-worktree/full-history Gitleaks, and a final clean-worktree check). All three targets delegate to scripts/quality_gate.py and can be invoked directly on Windows.

Common individual commands:

• ruff check / ruff format – lint and format the Python codebase.
• python scripts/quality_gate.py type – strict static type checks for the centrally defined maintained runtime surface.
• python scripts/quality_gate.py bandit / python scripts/quality_gate.py audit – source and dependency security checks.
• pytest – run the automated test suite (see tests/README.md).
• python scripts/collect_quality_metrics.py – emit complexity and dependency metrics.

One-Command Flows

• make setup – bootstrap the environment (falls back to --skip-install if package installation fails, useful in restricted sandboxes).
• make dev – execute the complete quality profile.
• make test – run the full pytest suite.
• make build – build the zipapp bundle at artifacts/build/zscripts.pyz.
• make deploy – smoke the packaged CLI by running guardrails and saving the JSON snapshot to artifacts/build/guardrails.json.

Operational and quality snapshots are documented in:

• docs/operations/BASELINE.md – runtime, automation, and dependency inventory.
• docs/operations/QUALITY_AUDIT.md – lint/type/test/security/coverage audit results and profiling notes.

Configuration and Extensions

Configuration defaults live in zscripts/config.py with a JSON mirror in configs/zscripts.config.json. Supply a TOML or JSON file via --config or pass inline overrides with --set key=value pairs. Telemetry can be toggled per-invocation using --enable-telemetry.

Extensions implement ToolkitExtensionProtocol from zscripts/extensions/base.py. Use python scripts/scaffold_module.py extension <name> to generate a telemetry-aware skeleton or python scripts/scaffold_module.py health <name> to create a reusable health check provider. Extension and health-check contributions must follow zscripts/extensions/AGENTS.md and agents/AGENTS.md.

Further Reading

• docs/architecture/ARCHITECTURE.md – Component relationships, flows, and extension guidance.
• docs/guides/ – How-to guides for extending adapters and running automation.
• docs/helpers/LEGACY_OPTIONAL_HELPERS.md – legacy helper policy and migration plan.
• docs/releases/RELEASE_NOTES.md – Narrative release history.
• docs/SUPPORT.md / SECURITY.md – Support channels and vulnerability reporting.

Track current work in GitHub issues, preserve historical task records in docs/TASKLIST.md, and log notable upgrades in CHANGELOG.md.