Skip to content

v0.5.1-beta

Pre-release
Pre-release

Choose a tag to compare

@21tmccauley 21tmccauley released this 02 Sep 20:38
· 702 commits to main since this release
3ad7f05

Fixed

  • An AWS failure now says why, not just which call broke. 0.5.0-beta got
    all 80 AWS fetchers reporting through $FETCHER_STATUS_FILE, but 250 of their
    CLI calls ran as aws ... 2>/dev/null and the matching log line recorded only
    a label. Of 310 failure records, 8 carried the AWS error text and 302 carried
    none
    — so expired credentials, a missing IAM permission and throttling all
    arrived looking identical, while being three unrelated fixes. All 80 also
    passed a hardcoded partial_failure, so metadata.error_code carried no
    information for the largest category in the repo.

    Both are fixed. metadata.error now carries the AWS error text behind the
    calls that failed, and metadata.error_code is derived from it:

    {
      "error": "2 AWS API failure(s); first: aws sts get-caller-identity failed; aws ec2 describe-security-groups (list) failed (exit=254) -- aws: [ERROR]: An error occurred (InvalidClientTokenId) ... The security token included in the request is invalid. | aws: [ERROR]: An error occurred (AuthFailure) ... AWS was not able to validate the provided access credentials",
      "code": "auth_failed"
    }

    That code is what the not-enabled classification work and the
    deleted-mid-scan decision were both waiting on.

    No call site was rewritten. aws in
    fetchers/aws/_shared/aws.sh is now a shell
    function shadowing the CLI, so all 250 calls keep their exact text — including
    2>/dev/null — and still have their stderr captured. stdout, stderr and the
    exit code reach the caller unchanged, so the 2>"$_ERR" + grep pattern the
    not-enabled check depends on still works, synchronously. The one constraint
    this adds: a fetcher must reach the CLI as plain aws, never command aws or
    an absolute path, or its calls go unclassified. A test enforces that.

    Classification is regex over botocore's wording, which is the only signal a
    CLI-based fetcher has — the SDK-based categories read a typed exception
    instead. A future AWS rewording can therefore downgrade a code to
    partial_failure; that is the safe direction, and
    tests/test_aws_failure_classification.py pins the current wordings verbatim.

  • A not-enabled service still exits 0. The wrapper captures the
    SubscriptionRequiredException stderr like any other, so this was the
    regression risk worth naming: had it been recorded as a failure, every account
    without Macie, Inspector, or Shield would have started reporting a spurious
    partial_failure. aws_service_unavailable is still checked first and
    short-circuits, verified end-to-end and pinned by a test.

  • Failure reasons read correctly again. The separator was built with
    tr '\n' '; ', and tr maps one character to one character — so it emitted
    ; with no space and left a trailing one before the (+N more) suffix.

Changed

  • Two Azure SDK pins widened, and CI can now tell whether that is safe.
    azure-mgmt-network moves from <32 to <33 and
    azure-mgmt-recoveryservicesbackup from <11 to <12, in both
    pyproject.toml and requirements.txt. Anyone installing .[azure] gets the
    new majors.

    The reason this is worth an entry: CI installed .[dev] and .[dev,tui] but
    never .[azure], so no test in the suite had ever imported an Azure SDK.

    Every azure-mgmt major that has broken this category broke it silently —
    wrong or empty evidence rather than an exception — which meant a dependency PR
    widening an Azure pin went green on evidence that proved nothing about Azure.
    One did
    exactly that: it relaxed the load-bearing azure-mgmt-monitor<7 and passed all
    twelve checks, while 7.0.0 removes the diagnostic_settings operation group
    that three fetchers read.

    A new azure-sdk (surface) job is now the only one that installs the Azure
    extra, and it pins the surface in the shapes a major bump actually takes it
    away: relocated client imports, vanished operation groups, vanished methods,
    and the model-field renames that produce wrong rather than empty evidence — 56
    checks across all 27 Azure fetchers. It asserts the extra really resolved
    before running, because the module opens with pytest.importorskip and a
    failed install would otherwise skip and report green. azure-mgmt-monitor
    majors are held back until the operation group returns; a test fails when it
    does, so the pin and the hold get lifted together instead of going stale.

  • The container base image moves to Python 3.14 (deploy/Dockerfile:
    python:3.12-slim → python:3.14-slim), and CI now tests on 3.14 as well
    as 3.10–3.13. The interpreter the Dockerfile and the dev environments actually
    run was previously the one nothing gated.

  • Corrected the 0.5.0-beta entry below. It said metadata.error "now carries
    the actual cause", and illustrated it with a report rich in AWS error text.
    That example came from one of the 8 call sites that captured stderr, out of
    310 — it was not representative, and for AWS the accurate claim at 0.5.0-beta
    was that the report names the call that failed. The entry now says so, and
    this release is what makes the original claim true.


Full changelog: v0.5.0-beta...v0.5.1-beta

Install by checking out the tag — a plain clone tracks moving main:

git clone https://github.com/paramify/paramify-fetchers.git
cd paramify-fetchers && git checkout v0.5.1-beta
python -m venv .venv && source .venv/bin/activate
pip install -e '.[all]'      # or `pip install -e .` for the headless CLI only

Azure and GCP need their own extras (.[azure], .[gcp]) — see the README.