Repository navigation
Proxy URL Sources
This guide covers feeding the provider a live proxy list URL instead of (or alongside) a static proxy.txt file — useful if you pull proxies from a service that publishes a rotating list of fresh entries.
Some proxy-list endpoints publish new ip:port entries on a rolling basis — some every few minutes. Without this feature, picking up fresh entries means manually re-downloading the list, re-importing it, and hoping you don't duplicate proxies you already added. There's also no way to automatically prune entries that go dead without touching proxies you added by hand.
Point the provider at a URL and it will:
- 🌱 Fetch on an interval (default every 15 minutes) and add any genuinely new proxies — entries already running, by address, are skipped. This never disturbs already-warmed-up proxies.
- 🧹 Optionally clean up dead entries on a much slower, separate cadence (default once a day) — and only the ones that came from a URL, by default. Proxies you added yourself via a file or
proxy addare left alone unless you explicitly widen the cleanup scope. - 🤝 Coexist with your existing proxy file. A URL source is additive — you can run
--proxy_fileand--proxy_urlat the same time. They share one hot-reload pipeline (see Proxy Management & Hot-Reloading).
Proxies that pass stage-0 (the SOCKS5 greeting and API CONNECT) are not yet
trusted with client traffic. Each one is graded against a sampled block of
the backend's destination table — dialed through the proxy itself at :443
— before it reaches the auth queue.
Scoring: a pass samples sample_width hosts (default 12) from the
backend's ~127-host health table, disjoint from the previous pass, and
dials each at :443 with a per-target timeout of timeout_ms (default
4000ms). The proxy's score is the fraction of successful dials in the
pass. Only a SynAck counts as success; a timeout or refusal is not treated
as proof the proxy is bad, since the target host itself could be
unreachable for unrelated reasons. Hostnames are resolved through the
box's own DNS, not through the proxy being probed, so a proxy with broken
DNS is not penalized for a resolution failure that isn't its fault.
A grade is only written when the pass is decidable — a quorum of the sampled hosts answered and the probe context wasn't cancelled. An empty, cancelled, or resolver-gutted pass leaves the proxy's previous grade in place rather than overwriting it with a false failure.
pass_bar: a proxy must score at or above pass_bar (default 0.6) to
be admitted to the auth queue. A score at or above preferred_bar
(default 0.9) marks the proxy as preferred tier. Proxies that clear
stage-0 but fail stage-1 never spawn, and are reported separately from
dead proxies in the auth gate's summary.
Every URL-source proxy that clears the stage-1 gate is assigned a letter grade from its score:
| Grade | Score |
|---|---|
| A | >= 0.9 |
| B | >= 0.8 |
| C | >= 0.7 |
| D | >= 0.6 |
| F | < 0.6 |
- Admission funnel: candidates from all sources are pooled and added best-first up to the cache cap, instead of per-source in whatever order sources happen to be processed.
- Best-overall cache eviction: when the cache is full, eviction compares candidates across all sources by tier — a full cache keeps the fleet's highest-tier proxies regardless of source.
-
Per-cycle grade breakdown is logged (
admitted by tier,probe grade breakdown,cap eviction,reaper: refreshed grade). - Fetch probing: each cycle probes only newly-seen addresses; the reaper's stale sweep refreshes grades on already-cached proxies.
- Cross-source duplicates are table-probed once per cycle.
Proxies from --proxy_file or the internal config bypass the URL
admission gate by construction. A background sweep grades every non-URL
proxy the box serves with the same stage-1 table probe on the same 1-3h
stale cadence, persisting Score/Graded/Failed/LastGraded into
proxy.state. Read-only by construction: grades never gate admission,
never evict, and never feed give-up/cleanup — a graded F keeps serving
exactly as it did before. proxy_probe.json enabled=false skips the sweep
entirely.
Credentialed URL entries: RFC 1929 auth (host:port:user:pass) is
carried through both the stage-0 and stage-1 probes, so paid/credentialed
URL-source entries are graded on the same footing as free ones.
Kill switch: create ~/.urnetwork/proxy_probe.json with
{"enabled": false} to disable stage-1 entirely and fall back to
stage-0-only admission. See Configuration for the
full set of proxy_probe.json knobs.
Persistence: Score, Graded, and Failed are stored per proxy in
proxy_url.json, alongside the existing cache fields.
Start the provider with a live source:
urnetwork provide --proxy_url=https://example.com/your-proxy-list.txtOr manage sources at runtime without restarting:
urnet-tools proxy add-source https://example.com/your-proxy-list.txt
urnet-tools proxy remove-source https://example.com/your-proxy-list.txtadd-source triggers an immediate fetch and persists the URL so it survives restarts — you don't need to keep passing --proxy_url by hand.
docker run -d \
--name=urnetwork \
--restart=unless-stopped \
--cap-add=NET_ADMIN \
--cap-add=NET_RAW \
--sysctl net.ipv4.ip_forward=1 \
-e BUILD=jwt \
-e PROXY_URL='https://example.com/your-proxy-list.txt' \
-v /path/to/your/proxy.txt:/app/proxy.txt \
ghcr.io/full-bars/meso-miner:latest YOUR_AUTH_CODE_HEREPROXY_URL and -v .../proxy.txt can be used together — the URL source adds on top of whatever's in the mounted file.
Tip
Multiple URLs in Docker: there's no PROXY_URL_2/PROXY_URL_3 — repeating -e PROXY_URL=... just overwrites itself, since Docker env vars aren't additive. Put all your sources in one comma-separated PROXY_URL. The repeatable --proxy_url=<url> --proxy_url=<url> form is only available on the binary/CLI side, where docopt flags can be passed more than once.
| Flag | Env var | Default | What it controls |
|---|---|---|---|
--proxy_url=<url> |
PROXY_URL |
— | The live source. Pass multiple times (or comma-separate the env var) for more than one source. |
--proxy_url_refresh=<duration> |
PROXY_URL_REFRESH |
1h |
How often to fetch and add new entries. |
--proxy_url_max=<n> |
PROXY_URL_MAX |
500 |
Caps total URL-sourced proxies. 0 = unlimited. Once hit, new entries are skipped until cleanup or restart frees room — existing proxies are never evicted to make space. |
--proxy_dead_cleanup_scope=url|all|none |
PROXY_DEAD_CLEANUP_SCOPE |
url |
Which proxies the automatic cleanup is allowed to remove. url (default) means only URL-sourced dead proxies are cleaned. none disables it entirely. all treats every source the same. Manual proxy remove-dead always works regardless. |
--proxy_dead_cleanup_interval=<duration> |
PROXY_DEAD_CLEANUP_INTERVAL |
6h |
How often the automatic cleanup runs, when scope isn't none. |
Tip
If you're pulling from a free/public list: the default url scope is what you want — the provider keeps adding fresh entries on the refresh interval and quietly retires ones that never panned out, but it will never touch proxies from your own hand-curated file.
v1 supports plain-text lists only, one proxy per line — the same format --proxy_file already accepts, plus an optional socks5:// prefix:
1.2.3.4:1080
1.2.3.4:1080:myuser:mypass
socks5://1.2.3.4:1080
socks5://myuser:mypass@1.2.3.4:1080
Blank lines and # comments are ignored. Lines with a non-socks5:// protocol prefix are skipped with a warning (this fork is SOCKS5-only) — one bad line doesn't fail the whole fetch. CSV and JSON list formats are not supported yet.
Every proxy is tagged internally with where it came from: url, file, or internal (i.e. added via proxy add). The --proxy_dead_cleanup_scope flag controls which of those tags the automatic daily job is allowed to act on:
| Scope | Behavior |
|---|---|
none |
Automatic cleanup never runs. You prune dead proxies yourself with urnet-tools proxy remove-dead. |
url (default) |
Automatic cleanup only ever removes dead proxies that came from a --proxy_url source. Your file/proxy add entries are never touched automatically. |
all |
Automatic cleanup treats every source the same — equivalent to running proxy remove-dead --all once a day. |
Manual urnet-tools proxy remove-dead is unaffected by this setting in every case — it always lets you choose interactively, regardless of source.
URL-sourced lists are often noisy — an entry can be reachable on the wire but fail auth forever (e.g. a live SOCKS5 endpoint behind a stale password). Older builds retried these on a flat 15-minute cycle indefinitely, keeping the auth rate limiter busy and filling logs with give-up messages for hopeless addresses.
Each URL-sourced address now gets an escalating per-address backoff after it gives up on auth:
| Give-up cycle | Retry delay (±20% jitter) |
|---|---|
| 1st | 15m |
| 2nd | 30m |
| 3rd | 1h |
| 4th | 2h |
| 5th | 4h |
| 6th | 8h |
| 7th | 16h |
| 8th+ | 24h (capped) |
After 4 give-up cycles, the address is permanently evicted: removed from the URL cache and written to a blacklist persisted in proxy_url.json. Blacklisted addresses are skipped at the only add path (mergeProxyURLEntries), so a hopeless proxy can never re-enter the auth lottery — even across provider restarts or if the source list keeps republishing it.
Note
Eviction is permanent and survives restarts. If a previously-blacklisted address genuinely comes back to life (e.g. the upstream fixes the password), remove it from the Blacklist map in proxy_url.json and restart, or it will keep being skipped.
This is independent of the daily dead-proxy cleanup (--proxy_dead_cleanup_scope): backoff/eviction is driven by repeated auth give-ups, while cleanup acts on proxies marked dead by the health tracker.
Concurrent fetch cycles for the same URL are now prevented. If a fetch is already in progress when the refresh interval fires, the new cycle is skipped with a log line ([proxy-url] fetch already in progress for <url>, skipping). This prevents accidental thundering-herd when multiple triggers fire near-simultaneously (e.g., a --proxy_url interval coinciding with a add-source command).
The provider periodically prunes internal data structures to control memory growth over long runtimes with large proxy lists:
- Failure history: Per-proxy failure counters and last-error timestamps for proxies that have been removed or replaced are freed after the cleanup cycle.
- Proven set: The internal set of addresses that have been validated (proven working) is periodically pruned of entries that are no longer in the active proxy list, preventing unbounded growth.
This ensures that a provider running for weeks with high proxy churn doesn't accumulate stale metadata that bloats heap usage.
The URL fetch subsystem now uses a dedicated HTTP client with sensible timeouts, rather than relying on the provider's default transport:
- Connection timeout: 30 seconds
- Response timeout: 60 seconds
-
User-Agent:
urnetwork-proxy-url-fetcher/1.0
This prevents a slow or hanging proxy list URL from blocking the provider's control-plane transport. The dedicated client is used exclusively for --proxy_url / PROXY_URL fetches and is independent of the provider's WebSocket/QUIC transports.
Will this duplicate proxies already in my file?
No — new entries are deduplicated by address against everything currently running (URL, file, and internal sources share one address space). If the same ip:port shows up in both your file and a URL source, the most recently applied one wins, same as today's hot-reload behavior.
What happens if the URL is unreachable? The fetch cycle is skipped with a logged warning. Already-added proxies from that source keep running — a stale list is better than wiping working proxies because of a transient network blip. After several consecutive failures, the provider logs a louder warning suggesting the source may be dead, but it won't remove the source for you.
What happens when I run proxy clear?
proxy clear (which calls proxy remove --all) now wipes the entire proxy_url.json — cache, blacklist, and source URLs — alongside the internal config and proxy.state. After a clear, no URL-sourced proxies will be fetched unless a source is re-added via urnet-tools proxy add-source <url>. This ensures a clean slate when you want only the proxies you explicitly add.
Does this validate proxies before adding them?
No — newly added proxies go through the same warmup and health-tracking lifecycle as any other proxy (see Proxy Management & Hot-Reloading). If a fetched proxy never connects, it'll show as dead in proxy health and get swept up by cleanup (if scope allows) or by a manual remove-dead. An address that connects but repeatedly fails auth is handled separately — it backs off on an escalating schedule and is permanently evicted after 4 give-up cycles (see Auth Give-Up Backoff & Permanent Eviction).