Akash Network
[Proposal] shoestring - GPUs on a budget with one command #1508
Replies: 6 comments 4 replies
|
Provider perspective — I run a small au-syd provider and I'm in the pilot dataset, so read this as interested rather than neutral. I checked the pilot data against my own records first. Of my 48 leases in the set: 47 reachable, 29.88s median time-to-ready, identical bid price on all 48 rows across the month. The numbers in the proposal are accurate. The single failure (2026-07-30, On M2, the reliability canaryThis is the part I'd most want to exist. There is currently no independent, continuous source of provider reliability data on Akash. Audit attestation is the closest thing available, and it isn't that — it's a point-in-time assessment against an auditor's criteria and scope, both of which legitimately change over time. Nothing published lets a tenant distinguish "unaudited and unreliable" from "unaudited and fine." I'm a live example of the gap rather than a hypothetical one. My attestation was removed on 18 Aug when the auditor's scope shifted; nothing about the hardware, uptime, or price changed. Console's default flow and trial accounts filter to audited providers, so the practical effect was disappearing from the main discovery funnel. A tenant with ten completed leases against me asked in Discord this week why I'd stopped appearing. The canary found me in the same period and measured me as the most reliable provider in its cheapest tier. To be clear about what I'm not arguing: audit does things measurement can't — verifying identity, hardware claims, operational seriousness. Continuous reliability data doesn't replace that and shouldn't. But a gate and a measurement answer different questions, and right now only the gate is published. Two things I'd push onA gap in the pilot data. Continuity. The proposal treats this as a named gate rather than a hope — daily CSV exports in a public repo, a budgeted 12 months, a handoff plan, and an explicit shutdown-and-archive commitment rather than letting stale data masquerade as live. That's the right instinct, and it's the failure mode that kills reliability datasets: not failing to launch, but going stale while still looking current. Worth holding to. On the named-blacklist question in M1Replacing a named list with criteria-based filtering over measured reachability is the right call for a community-funded tool. I say that as someone who maintains a named list in the other direction — a public blacklist of tenants who've run mining workloads in violation of deployment terms. Two things from that: Evidence standards have to be written before the first entry, not after the first dispute. Mine documents what counts as evidence and what doesn't, and it's still the part that gets tested. Dispute handling matters more than detection accuracy. Self-healing when a provider improves, plus a published contest process, covers this better than a static list can — but someone will eventually be filtered on data they disagree with. Deciding now how that gets adjudicated, and that the decision is public, is worth more than getting the threshold right. Happy to compare notes on either. What I'd want from this as a providerPer-provider historical reachability I can point tenants at, and the GPU truth data. The second is quietly the most valuable thing in the proposal — on-chain GPU attributes are self-declared and there is no published check against reality. A dataset that does that check continuously would be worth funding on its own. |
|
Thank you for the reply @jjozzietech and for your thoughtful feedback. Nice catch on the gap in the audit data. Indeed, gpu_model was missed due to a bug in the initial code, it's been updated, and the canary is still running and gathering data including the gpu_model metadata that was missed in the initial study. I will attach a repository and update the dataset in GitHub daily as an initial pass to make the findings available to the community until my funds run out (I am about halfway through my initial investment) I think long term, a public facing website with an api layer that integrates nicely with the Akash ecosystem is a good way to land the requirement going forward. It would give something for you to point your tenants at, to showcase your historical performance metrics and real value per dollar spent. The current study costs me real money (akt) to spin up and validate leases, as does the hosting infrastructure to make this dataset public. Agreed on the blacklist criteria protocol, and self healing opportunities. I would ask the community for ideas of how to go about remediating a provider's reputation using provable criteria.
Open to any and all suggestions about how to structure an evaluation of provider quality. The two main levers in my initial study were cost/availability, with a strict focus on low-cost providers because of the nature of my self-funding efforts and well, the entire purpose of the initial project, to run gpu workloads on a "shoestring" budget 😋 The canary workloads could run at a regular interval against providers that may be blacklisted or otherwise ranked lower, in order to allow incremental remediation of any of the evaluation criteria within a reasonable timeframe. I propose that a framework for provider evaluation be drafted and frozen on-chain once the community has decided on a reasonable framework. The evaluation canary service and associated dataset publication would be a service for the community allowing more rapid onboarding of providers that meet basic user criteria for availability and price, without the formal audit process, and thus presents a viable on-ramp to new providers and tenants, given basic criteria. This is why the shoestring tool is useful - it presents a simple on-ramp using measured data for tenants curious about the network, and the canary evaluation system provides a simple on-ramp to winning bids for providers who pass the "sniff test" in real terms, but may not yet be audited. I believe an engagement to begin a full scale canary audit (against more than the cheapest providers) and the resulting continuous publication of M2 dataset would represent a meaningful deployment of the treasury towards something explicitly targeting network growth and service quality. |
|
Follow-up with receipts on yesterday's promises, plus some new data: Daily publication is live. data/canary_live.csv in the repo now refreshes automatically every day at 13:35 UTC (GitHub Action), with per-row provenance documented in data/README.md. The pilot CSV stays frozen so the proposal's numbers remain reproducible against it. gpu_model, concretely: the fixed canary deployed today — rows from 2026-08-30T16:00Z onward carry the model measured from the winning bid's on-chain attributes. For history, I backfilled 99 of 222 rows where market snapshots showed the provider advertising exactly one model within ±6h of the run; the other 123 are honestly NULL rather than guessed. Chain-based recovery turned out to be impossible — closed deployments age out of the Console API within days — so the repo now records the lesson: provenance has to be captured at write time. Since you know your hardware: confirm the 3090 and I'll mark your remaining rows operator-confirmed — I think that's a legitimate third evidence class next to measured and inferred. The scorecard discussion turned into code this afternoon. The canary now cycles three profiles (gpu, cpu-micro, cpu-heavy), persists every bid received to data/bids.csv — so each probe doubles as a marketplace census — explores by least-recently-tested bidder rather than randomly, and times a small CPU benchmark on every lease (first step toward your linpack/price-performance bullet). First cpu-micro probe ran today: seven bidders, five of them providers that never once appeared in a month of GPU-only probing. "How big is Akash really" is now a measurable question, which was rather the point. Your failed lease, for the investigation you offered: 2026-07-30T12:07:39Z (≈10:07pm AEST). Your standard bid won, the lease went active and an ingress URI existed, but the workload's HTTP response never arrived within the ~3.5-minute probe window. If your logs show anything for 12:03–12:12Z I'll attach it to the row. And one live data point for the remediation design: the 0-for-8 provider from the pilot recorded its first measured success this morning. The self-healing filter now has a test case. Standing offer from yesterday holds — I'd like to co-draft the evaluation framework with your evidence-standards principle as its first line. |
The 30 Jul failure — root cause foundThanks for the timestamp precision, it made this findable. Your row is the visible symptom of a silent network partition on the GPU node that ran roughly 16 Jul – 1 Aug. The node's kubelet kept posting status successfully, so the API server showed it Sequence that produced exactly what you observed:
Resolved 1 Aug by an on-site power cycle. Two detection changes are now scheduled off the back of it: a Ceph OSD status cross-check in the daily health script, which catches a partition on any node carrying an OSD, and an outbound heartbeat cron per node, which catches it regardless of Ceph presence — including GPU nodes that may have no OSD at all. The second is the one that would have caught this specific case. Provider-side logs from 30 Jul are gone — the pod has restarted many times since — so I can't give you a provider log excerpt. The node's kubelet journal establishes it cleanly on its own, and I'd rather say that plainly than reconstruct something. Why this matters for the framework rather than just for my rowThis is a distinct failure class worth naming: silent partition. A node retains just enough kubelet function to stay Three things follow for the scorecard:
Happy for this whole diagnosis to go in the dataset notes if it's useful — a documented failure with a named cause is worth more to the framework than a clean row. |
|
Correction to my previous comment — please don't build on it yet. Revisited the diagnosis and found it was over-reached. Correcting fast rather than letting it stand. What still holds:
What I stated as fact and shouldn't have: That your specific canary lease landed on that partitioned node. The evidence was high kubelet log volume in your window — which on second look is the retry-loop signature of a partitioned node, not evidence that a workload was scheduled onto it. The six-step sequence I posted is a plausible mechanism, not an established chain. If the lease actually landed on a healthy node, the failure has a different cause we haven't found. So: partition confirmed, causation not confirmed. I published the second as though it were the first. What would settle it: if you have the canary's tenant address for that run, or a dseq, we can look up which node the lease landed on and give you a definitive answer. I know the pilot rows lack dseqs — the tenant address alone may be enough on our side. One thing worth taking from this for the framework, since it's the same lesson you hit with That argues for the canary recording enough at write time — dseq, provider, and ideally the resolved node or ingress target — that attribution never depends on someone reconstructing it afterward from memory and log volume. You already concluded provenance has to be captured at write time for GPU model. Same principle, and failure attribution is the case where it matters most, because that's the data the scorecard actually scores on. Apologies for the noise. Better a correction an hour later than a wrong story in the dataset notes. |
|
@jjozzietech The dseq you need: Your rows are updated: all 48 now carry rtx3090, logged as an operator-confirmed provenance event (a third evidence class alongside measured and inferred — your confirmation created the category). Both your comments — diagnosis and correction — are now in the dataset notes (data/NOTES.md), recorded exactly as far as the evidence goes: partition log-evidenced, causation unresolved pending the dseq lookup. Your three scorecard principles are adopted there verbatim as framework lessons: failure cause-classes over booleans, retry as cheap node-vs-provider discrimination, recency weighting over raw incidence. And your write-time attribution point is integrated: as of an hour ago the canary records the ingress endpoint per row, alongside the dseq it already captures — attribution will never again depend on anyone reconstructing it from log volume and memory. This is the same principle that fixed GPU attribution. Last thing, said plainly: the correction you posted is worth more to this project than the diagnosis was. A dataset like this lives or dies on whether people can distinguish "confirmed" from "plausible" under social pressure to have answers — you just modeled that distinction in public, against your own post, within the hour. That's the epistemic standard the framework doc should open with, right next to your evidence-standards-before-first-entry rule. Thanks for that! |
|
Node lookup done. The answer is "cannot determine", but the chain lookup turned up something more useful than the node would have been. The lease resolves fine. dseq Node placement: not determinable, and I'll say plainly why. The provider pod has restarted several times since (most recently 2 Sep), so pod-side logs for the window are gone. Kubelet doesn't log dseq or tenant address in searchable form — the lease namespace would appear, but namespaces are torn down within minutes of lease close, and a journald search for the dseq across all workers 34 days later returns nothing. What I can confirm about cluster state: one node was in a sustained silent-partition state across the entire day of 30 Jul, not just your window — kubelet log flood at ~140 lines/minute against a healthy baseline of 1–5, continuously from 11:00 to 14:00 UTC and beyond. That matches the documented 16 Jul – 1 Aug incident, recovered by power cycle. So: if the workload landed on that node it would have hit the partition. If it landed on any of the ~19 other eligible workers it should have run normally. I can't tell you which, and I'm not going to guess again. Suggested annotation for the row: failure occurred; provider-side records aged out; root cause unconfirmed. A documented silent-partition incident on one node overlapped the window and could account for it if the workload was scheduled there. Post-incident detection has since been added for this failure class. Three things for the canary, in order of usefulness1. Record the namespace, not just the dseq. The string preceding 2. Your probe window and the lease duration don't quite line up. You described a ~3.5 minute probe window; the lease lived ~138 seconds before your close. Not necessarily wrong — block times vary and I may be converting badly — but worth checking that the close isn't firing before the stated window elapses. If it is, some fraction of "never reachable" rows are workloads that were still starting. 3. I have two One note on my own detection gap, since I raised the failure class. The silent-partition check I've added cross-references Ceph OSD status against Kubernetes node-Ready state — divergence flags a partition candidate. It would not have caught this specific node, which hosts no OSD. Outbound heartbeat per node is the follow-up that closes that gap, and it isn't built yet. Stating that rather than implying the problem is solved. On the framework co-draftThank you for the offer, and for putting the evidence-standards principle at the top of it — that's the right place for it. I should be straight about capacity rather than leave it open: I can't commit to co-authoring this quarter. I'm mid-way through a hypervisor relocation project on my own infrastructure and my time for community work is a couple of hours a week, which isn't enough to be a reliable co-author on a document that matters. What I can do, and would like to: review drafts properly and answer specific questions from the provider side. If you write it, I'll read it carefully and give you real feedback rather than a rubber stamp. The principles are yours to use regardless — they came out of this thread and they aren't mine to withhold. One disclosure while I'm here, since it should be on the record rather than inferred: my provider is in your dataset, and if this proposal passes, some of the $800 marketplace test spend flows to providers including me. It's a trivial amount in absolute terms, but I'd rather state it than have someone notice it later. Read everything above with that in mind. |
|
Great suggestions @jjozzietech ! I will get to work on the ingress records and I will trace the missing data as well. I am happy to update the annotations on the failure cases with your provided notes as well. I am running this system on a very thin (read: 8gb ram and 4 core) kubernetes cluster that is already running some other production services. I believe a few canary workloads may have terminated early due to resource contention and I will investigate this hypothesis to ground it in fact. I may modify my canary to actually run on Akash itself to "dogfood" the service itself! Which came first, the canary or the coalmine? :-) As far as the framework goes, honestly, your communication and thoughtful responses to what I've built here has been incredibly helpful thus far and I appreciate your input in whatever form it comes! I am happy to know that at least one other person finds this project useful to the community and I will do my best to build it in a way that adds value for providers like you! Your feedback in all forms is very welcome. Thanks for your help so far. |
|
All three landed as changes — details in data/NOTES.md (https://github.com/themailman05/shoestring/blob/main/data/NOTES.md), short version here:
Every other unmatched lease was identified and published with its purpose in data/nonprobe_leases.csv — workloads and tests from the same wallet, including the ML training lease your provider ran in July. That file also answers your 60-vs-48: your extra 12 are actual training-run leases that I used to train my model, an Aug 13 test sweep, and post-pilot probes. Your two manifest_timeouts will slot into that reconciliation — a provider volunteering failures the dataset missed is exactly the direction this should run, and they'll be annotated as operator-contributed. Thanks for that! On capacity: review-not-coauthor is plenty — honestly it's the more valuable role, since the framework needs an adversarial reader more than a second author. And the conflict disclosure is noted and appreciated; it'll be stated in the framework doc's contributor notes rather than left for someone to discover. For symmetry: my own disclosure is already in the proposal — the tool was built for my own workloads first, and the community gets the generalization. |
|
The two Lease 1
Lease 2
Both classified by the chain, no ambiguity in either. These were one incident, not two. Created 9m 54s apart, both ran the full manifest-acceptance timeout before the chain-side close, both closed at exactly 52 blocks. My provider was in some degraded state for that ~10-minute window that made both manifest submissions fail identically. Cause unknown — provider pod has restarted many times since, logs for that window are gone, and there's no incident record in my ops notes for 18 Jul 15:53–16:08 UTC, which is 01:53–02:08 my local time. Not reconstructable 63 days later. Zero Two things for your reconciliation, both from checking the numbers rather than from the incident. Price. Both of these bid at A dseq check that may be useful to you. Both dseqs, read as unix-ms, land ~18–21s before their on-chain One observation, your data not mine: my lease count from your tenant is still 60, unchanged since the 3 Sep check. Sixteen days, no new probe against my provider. You mentioned probes moving to public CI around then, so this may be intended — flagging in case it isn't, since a canary that silently stops probing a provider is the failure mode the dataset would be least able to see in itself. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Community Pool Proposal: shoestring — GPUs on a budget with one command
Draft for GitHub Discussions review, following the proposal best practices. Not yet submitted on-chain. The image above was generated by this proposal's own prototype: a FLUX.1-schnell instance booted onto an Akash a100 lease by
shoestring.py --engine comfyui-flux, prompted through a tailnet-only ComfyUI, and iterated across a few batches of seconds-per-image candidates — total marketplace cost for the entire logo session, well under one dollar. The tool illustrated itself.TL;DR
shoestring is a single-file, MIT-licensed tool that turns the cheapest GPU on Akash into your private AI box in one command: a 27B coding model served for $0.16/hour, an image-generation studio in ~12 minutes, and the training rig that shipped a production ML model for ~$30 — all tailnet-private, all measured on real leases, all torn down with one command. We request $41,200 in AKT to run and publish a continuous provider-reliability + marketplace-census dataset (the canary is already live), harden the tool to v1.0, ship
shoestring train, and make Akash the easiest place on the internet to rent a GPU.Repo: https://github.com/themailman05/shoestring — already public, including the raw canary pilot dataset (
data/). The pilot schema did not record per-lease dseqs — an honesty gap we note rather than hide; the M1 dataset records dseqs from day one so every row is verifiable against the chain, and the worked examples in this proposal publish theirs.Goal / Mission
Make Akash the default answer to "where do I run an open-weights model cheaply?" by making the first deploy trivial, private, and honest about costs — and by publishing the reliability data that lets users trust the marketplace's cheapest bids.
Overview
The gap between "Akash has the cheapest GPUs anywhere" and "people actually use them" is the first hour of user experience: SDL authoring, under-documented manifest/lease failure modes, GPU attributes that don't disclose VRAM, a price denom most users misread. We hit every one of these building shoestring — and encoded the fixes so the next person doesn't have to.
What exists today, working end-to-end:
shoestring.py deploy→ bids surveyed, cheapest reliable provider selected, model served ~6 minutes later;shoestring.py close→ billing stops.--tailscalejoins the container to the user's tailnet as an ephemeral node; the model server binds to loopback and is unreachable from the public internet. WireGuard end-to-end.gpu_modelcolumn in all 176 rows (a recording bug, since fixed — the live canary now captures it, and rows recovered from chain data are being backfilled). The shippedcanary.pycloses all of these schema gaps, and M1 establishes the finding rigorously, in public. To be explicit: the pilot evidences reachability, pricing, and time-to-ready; the GPU truth-data capability (advertised attribute vsnvidia-smireality) begins with the live canary, not the pilot.--engine comfyui-fluxboots ComfyUI + FLUX.1-schnell (Apache-2.0) for image generation on an a100-class lease — validated live (tailnet-only UI serving in ~12 minutes cold at $1.84/hr). One tool, three workload classes: LLM serving, small-model training, image generation. This engine also debuted the flight recorder: a crashed workload serves its own boot log on the workload port instead of restarting blind, so failed leases debug themselves.Beyond inference, the same machinery has done real ML training on Akash: a small production audio model, distilled from a much larger teacher entirely on Akash leases — Jupyter-driven runs, canary-informed provider selection, automatic checkpoint salvage — for roughly $30 total, less than one hour of reserved hyperscaler H100 time. It ships today in a production iOS/macOS app built by the proposer (name and model details withheld while a patent application is pending; available privately to reviewers — disclosed plainly: M3 generalizes infrastructure built for our own product, and the community gets the generalized tool under MIT). M3's worked example makes the claim reproducible by anyone, lease dseqs published.
All of the above was self-funded, and no compensation is requested for any pre-proposal work. It is presented as evidence of execution capability and as de-risking: this proposal funds the hardening and generalization of a thing that already works, not a plan.
How this differs from existing Akash tooling
Akash Console, its templates, and awesome-akash are real on-ramps, and where a capability belongs upstream we will contribute it there (see M4) rather than maintain a parallel implementation. What none of them do today, and shoestring does: VRAM-adaptive context sizing (working around the chain's silence on GPU memory), tailnet-only private serving, measured-reliability provider selection, agent-ready config output (Claude Code / opencode) on every deploy, and training runs with automatic checkpoint salvage.
There is a direct precedent for this trajectory in a sibling ecosystem. In Sia, the community-built sia-host-dashboard (Sia Central) pioneered host health monitoring, metrics, and income analytics as an independent tool — and proved the demand so thoroughly that the Sia Foundation built the functionality into its official hostd daemon with an embedded web UI, at which point the community tool was gracefully archived ("deprecated and replaced by the Sia Foundation's hostd", per its own README). That is the intended endgame here: shoestring's canary and reliability data exist to prove out what marketplace-trust tooling Akash needs; the win condition is Console or the provider stack absorbing it upstream, not a permanently parallel community tool. The proposer watched this pattern from inside the Sia ecosystem — it is how healthy decentralized-infra communities metabolize good ideas.
Why this benefits Akash: every shoestring user is new GPU demand (inference and training), the tool's field notes double as ecosystem documentation, and the canary methodology gives the community something it currently lacks — independent, continuous, published reliability data on providers.
Detailed Deliverables
M1 — Provider reliability canary + marketplace census (public dataset + dashboard)
Promoted to first priority during review: the sole substantive community feedback to date — from a provider/validator who independently verified the pilot data against his own records — named this deliverable "the part I'd most want to exist" and the GPU truth data "worth funding on its own."
data/NOTES.md, beginning with a provider-contributed silent-partition analysis from the review thread — cause classes instead of booleans, retry-based node-vs-provider discrimination, recency weighting, and a three-classgpu_modelprovenance scheme (measured / inferred / operator-confirmed).gpu/cpu-micro/cpu-heavyprofiles, and every bid received is persisted — thecpu-microuniversal profile makes each probe a census of who is actually alive on a network with ~1,800 registered providers. First census probe (2026-08-30): seven bidders, five of them providers never seen in a month of GPU-only probing.nvidia-smireality — built new here; the pilot does not evidence this capability).canary.pyas the runnable artifact that makes the handoff real. If no operator accepts, we publish a shutdown-and-archive plan rather than letting stale data masquerade as live — the sia-host-dashboard → hostd absorption (see above) is the model: prove the need independently, then hand the pattern upstream and archive gracefully.M2 — shoestring v1.0 (hardened tool)
--gputargeting flag, multi-model presets beyond Qwen (Llama, DeepSeek, GLM families),--list(show your open leases), resume/reattach.comfyui-fluximage-generation prototype (workflow presets, output persistence) alongside the LLM engines.uvx shoestring, Homebrew tap; CI that dry-run-validates every SDL variant./v1/modelsauth gap and peers).M3 —
shoestring train: small-model training utilityshoestring trainboots a GPU Jupyter/script box with (a) dataset staging from any S3-compatible bucket, (b) automatic checkpoint/artifact salvage to the user's bucket on every epoch and at teardown (leases are ephemeral; our workflow has already survived a destroyed 81GB dataset with zero checkpoint loss), (c) canary-informed provider preference, and (d) the same tailnet-only privacy mode as inference.M4 — Adoption content
Timeline and Milestones
One quarter, four milestones, payment tranches tied to delivery:
Gates are written to be checkable by strangers, not self-attested:
shoestring trainshipped + worked distillation example rerun by a named third party from the docs alone, costs and dseqs publishedProgress reported monthly in the Community Pool Spend Reporting repo per the standard process.
Funding Amount Requested
$41,200, paid in AKT (tranche-dated conversion; see below).
Cost breakdown
The rate reflects senior consulting work delivered solo; all code MIT-licensed, all data CC-BY. Notably, the marketplace test spend flows straight back to Akash providers.
Price risk and non-delivery
Research and Ideation (already done)
This proposal follows a month of self-funded groundwork:
data/directory. Pilot rows lack dseqs — the live canary schema records them).shoestring trainwill generalize.Supporting References
Revisions during review
Kept as a changelog so reviewers can see what changed and why.
2026-08-30 — Canary v2 shipped: marketplace census (profile matrix + full bid persistence), coverage-driven exploration, per-lease CPU benchmark. Daily automated dataset publication live (
data/canary_live.csv+ provenance notes).gpu_modelgap in the pilot stated plainly (per review feedback), 99/222 historical rows backfilled via unambiguous market inference, the rest left NULL rather than guessed; the live canary records the field measured from the winning bid. Milestones reordered: the canary/census is now M1, reflecting review feedback from the provider side.2026-09-02 — Probes now run 6x/day in public CI, committing rows to the repo. Per-provider 24h GPU-probe cooldown shipped same-day after provider feedback. Write-time attribution extended (dseq + ingress endpoint per row) after an operator's own post-hoc analysis — posted and then honestly self-corrected in this thread — demonstrated that failure attribution cannot be reconstructed later, even provider-side.
data/NOTES.mdbegins the annotated-failure practice; firstgpu_modeloperator confirmation recorded. First census probes: the universal CPU profile drew seven bidders, five never seen in a month of GPU-only probing.Review Plan
About
Liam Sargent — decentralized-compute veteran: winner of several Sia decentralized-storage hackathons and formerly the first employee at Filebase, where he worked on file-packing efficiency and protocol-level integrations across decentralized storage providers. Currently a solo founder/engineer running production workloads and this GPU tooling on Kubernetes + Akash. GitHub: @themailman05.
All reactions