Skip to content

OCI Capacity-Aware Provisioner v0.1.0

Latest

Choose a tag to compare

@its-spark-dev its-spark-dev released this 05 Aug 16:57
· 4 commits to main since this release
824160e

OCI Capacity-Aware Provisioner v0.1.0

Summary

First public release.

Creating a VM.Standard.A1.Flex instance in a popular Oracle Cloud region
usually fails with Out of host capacity — not a misconfiguration, just Oracle
having no free Ampere host at that moment. The common workaround is a shell loop
retrying every ten seconds, which gets tenancies throttled, can create duplicate
instances when a request times out after the server already accepted it, and can
quietly provision resources outside the free allowance overnight.

This is a CLI that handles the same situation carefully: it rotates availability
domains one request at a time, waits at least five minutes between rounds, stops
the instant it succeeds, refuses to create a second instance with the same name,
and refuses to send any request that would exceed the Always Free allowance.

Highlights

Availability-domain rotation. One LaunchInstance request at a time. On a
capacity shortage it moves to the next domain rather than retrying the same one,
and the starting domain shifts between rounds so a permanently full AD-1 does not
absorb every round's first attempt.

Bounded, deliberate pacing. 10 minutes between rounds by default, with
one-sided jitter. The floor is 300 seconds, validated at config load and clamped
in code, with no flag to lower it. Runs are bounded by max_runtime_hours.

Duplicate prevention that actually holds. A server-side display-name query
runs before every launch request — not once at startup, because a run can last
two days. Every request carries an opc-retry-token derived from the request's
identity, so a timed-out launch repeated with the same token returns the original
instance instead of creating a second. A machine-wide OS file lock prevents two
processes racing.

Cost guardrails with no bypass. Before launching, the tool totals every
non-terminated A1 instance and every live boot and block volume across the
tenancy, adds the pending request, and compares against the Always Free
allowance. If it cannot complete that measurement, it refuses to launch — not
knowing is never treated as zero.

Error classification that fails closed. Decisions use exception type, HTTP
status, and the OCI service error code together. A host-capacity shortage is
retried; authentication failures, permission errors, quota exhaustion, billing
problems, and anything unrecognised stop the run immediately.

doctor. Twenty-odd checks that find the private subnet, the x86 image, the
region-mismatched image OCID, the missing IAM policy, or the exhausted allowance
in five seconds — each with a remedy — instead of forty minutes into a retry
loop. A single FAIL blocks launch.

--dry-run that tells the truth. It resolves the image, the availability
domains, the current usage, and the duplicate check, then prints the exact
request object that would be sent. Identifiers masked; the SSH key never printed.

Clean shutdown. SIGINT or SIGTERM during a ten-minute wait is noticed
immediately: the in-flight call finishes, state is saved, the lock is released,
and the process exits 130.

Structured logs with central redaction. A JSON Lines audit trail with UTC
timestamps, per-attempt outcomes, OCI error codes, and masked request IDs. Every
field routes through one redaction module — OCIDs masked to head and tail,
private keys and webhook URLs never emitted at all.

Optional generic webhook. Fires on terminal events only — success, a fatal
error, or running out of time — never once per capacity miss. A webhook failure
never changes a provisioning outcome. The URL is read from the environment, never
from the config file.

Safety guardrails

Guardrail Can a user turn it off?
Only VM.Standard.A1.Flex — no fallback shape at any level No
Max 2 A1 OCPUs tenancy-wide No
Max 12 GB A1 memory tenancy-wide No
Max 200 GB combined boot and block storage No
Boot volume 50–200 GB No
Minimum 300-second retry interval No
One provisioner process per machine No
One launch request in flight at a time No
Unknown errors are fatal No
Unmeasurable usage blocks the launch No
Bounded total run time Within a 168-hour cap

There is no --allow-paid, no --allow-over-free-tier, and no environment
variable that relaxes any of these. Raising a ceiling requires a source change
and a release. See
ADR 0003.

Safety fixes before release

A pre-publication audit reviewed the whole branch against the OCI SDK and found
ten defects, all in paths the test suite did not yet cover. They are listed
here rather than left in the commit log, because two were fail-opens in the cost
guardrail — the promise this project is built around — and because each one
explains why a guardrail has its current shape. All ten are fixed, and every one
has a regression test.

Cost guardrail (fail-open, now closed)

  • Usage was measured in one compartment. The Always Free allowance is
    tenancy-wide, so an A1 instance in any compartment consumes it. Pointing the
    tool at a different compartment reported zero usage and would have permitted a
    second 2-OCPU launch — taking the tenancy to double the allowance, billable on
    a pay-as-you-go account. Usage is now swept across every compartment the caller
    can enumerate, deduplicated by OCID, with doctor reporting allowance.scope
    and warning when the sweep cannot be proven complete.
  • Restricting availability_domains narrowed the usage sweep. Boot volumes
    are listed per availability domain, so launching into a subset while measuring
    only that subset hid storage in the rest and undercounted the 200 GB ceiling.
    Measurement now always covers the region; launching still honours the subset.

Duplicate prevention and request pacing

  • Unknown lifecycle states were treated as free. Detection used an allowlist,
    so a state Oracle adds later would let a same-named instance through and would
    not count its OCPUs. Only TERMINATED now releases resources.
  • Repeated token rejection could loop unbounded. A recurring
    409 InvalidatedRetryToken retried the same availability domain with no cap
    and no backoff, capable of producing exactly the stream of LaunchInstance
    calls this tool exists to prevent. Rotation is now capped per domain.

Reporting and diagnosis

  • Fatal errors exited 5 instead of 1. The systemd unit in these docs sets
    SuccessExitStatus=0 5 130, so an authentication, permission, quota, or
    billing failure was being reported to the supervisor as a successful run.
  • Pre-flight failures could escape unclassified. A throttle during image
    resolution or availability-domain discovery surfaced as "Unexpected error" with
    no remedy, and a failure to confirm image/shape compatibility propagated raw
    instead of blocking the launch.
  • A throttled image read was blamed on the region, sending users to debug the
    wrong thing.
  • The internet-gateway check searched the wrong compartment, reporting a
    correctly-routed subnet as having no gateway when the VCN lived in a shared
    network compartment. An invisible gateway is now "cannot confirm", not "absent".
  • token_expired crashed on a naive timestamp, raising TypeError from the
    aware/naive subtraction outside its ValueError guard.
  • version output was not machine-readable. Rich's highlighter inserted
    ANSI inside the version token whenever colour was enabled, so
    VERSION=$(oci-capacity-provisioner version) captured escape sequences rather
    than 0.1.0 — caught by the release workflow's own version check. The ssh
    command users copy was affected the same way. Both are now plain text.

Installation

Requires Python 3.12 or newer.

pipx install git+https://github.com/its-spark-dev/oci-capacity-aware-provisioner@v0.1.0
uv tool install git+https://github.com/its-spark-dev/oci-capacity-aware-provisioner@v0.1.0

Or download oci_capacity_aware_provisioner-0.1.0-py3-none-any.whl from this
release and:

pip install ./oci_capacity_aware_provisioner-0.1.0-py3-none-any.whl

Verify the download against SHA256SUMS.txt:

sha256sum -c SHA256SUMS.txt

Not on PyPI. Install from a git tag or a release wheel.

Quick start

oci-capacity-provisioner init              # create config.toml interactively
oci-capacity-provisioner doctor            # check the environment
oci-capacity-provisioner launch --dry-run  # see the exact request
oci-capacity-provisioner launch            # provision, retrying patiently
oci-capacity-provisioner status            # see what exists

Try it with no Oracle Cloud account at all:

git clone https://github.com/its-spark-dev/oci-capacity-aware-provisioner
cd oci-capacity-aware-provisioner
make install && make demo

Supported platforms

Platform Status
Linux (x86-64, arm64) Supported, tested in CI
macOS 13+ (Intel, Apple silicon) Supported, smoke-tested in CI
Windows via WSL2 Best effort
Windows native Not supported — POSIX signals and file locking differ

Python 3.12 and 3.13 are tested. Authentication is via OCI API key
(~/.oci/config); instance principals and resource principals are not supported
in this release.

Known limitations

  • API key authentication only. No instance principal, resource principal, or
    security token support.
  • One instance per run. Provisioning several means several configs with
    different display names — and the allowance rarely permits it.
  • Ubuntu-oriented image resolution. Other operating systems work by setting
    operating_system and operating_system_version, but only Canonical Ubuntu
    aarch64 is exercised.
  • The Compute Capacity Report is advisory. It does not reserve capacity, can
    be stale, and requires its own IAM permission. It only ever reorders a round;
    it never skips a launch attempt.
  • Provisioning only. The tool creates the instance and stops. It does not
    install software, run cloud-init scripts, configure anything inside the VM, or
    terminate anything.
  • The lock is per-machine. Two machines running the same config can still
    race; the pre-launch query and the retry token cover almost all of that, but
    not by construction.
  • No cost monitoring after launch. Set a budget alert in the OCI Console.
  • Windows-native staleness detection is degraded. Stale-lock diagnosis reads
    process command lines via ps.

Security notes

  • Credentials are read from ~/.oci/config by the OCI SDK. This tool never
    reads, copies, stores, or logs private key material.
  • Generated files (config.toml, state, lock metadata) are written 0600.
  • Every log line, webhook payload, and state file passes through one redaction
    module. OCIDs are masked; request IDs truncated; private keys, webhook URLs,
    and authorization headers never emitted.
  • The webhook URL is environment-only (OCI_CAPACITY_WEBHOOK_URL), deliberately
    not a config field, because config files get pasted into bug reports.
  • Run state holds identifiers, timings, and a derived idempotency token — never
    key material, never the webhook URL, never a copy of the config.
  • CI runs with no OCI credentials. Integration tests against a real tenancy
    require an explicit opt-in; tests that would create resources require a second,
    independent one.

Report vulnerabilities privately through
GitHub Security Advisories.
Never paste a credential into a public issue — see
SECURITY.md.

Upgrade notes

None — this is the first release.

For future upgrades: config.toml is validated strictly, so an unknown key is an
error rather than a silent no-op. Run state carries a schema version and is
discarded rather than misread if the schema changes.

Cost warning

This tool does not guarantee Oracle's Always Free policy. It enforces the
ceilings it was built with. Oracle owns the actual policy and can change it at
any time without changing this tool.

On a pay-as-you-go tenancy, anything beyond the free allowance is billed. Read
Always Free
Resources

and check your own tenancy limits before an unattended run.

Full changelog

https://github.com/its-spark-dev/oci-capacity-aware-provisioner/blob/main/CHANGELOG.md