Add FastAPI web version of the crypto arbitrage scanner#3
Open
devin-ai-integration[bot] wants to merge 17 commits into
Open
Add FastAPI web version of the crypto arbitrage scanner#3devin-ai-integration[bot] wants to merge 17 commits into
devin-ai-integration[bot] wants to merge 17 commits into
Conversation
Refactors the multi-exchange arbitrage logic from the standalone scanner.py into a reusable async library and ships a FastAPI application with a dashboard, REST endpoints and a background scanner loop. - app/scanner_core.py: ticker, deposit/withdraw and orderbook logic from the original script, configurable via dataclass. - app/main.py: FastAPI app with /api/state, /api/scan, /healthz and a static dashboard. - app/static/: dark-themed dashboard with summary metrics, sortable arbitrage table, transfer/D-W filtering and chain matching info. - Config via env vars (SCANNER_*, MEXC/Binance/Bybit API keys). Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Author
Test results — passedEnd-to-end smoke test of the dashboard on
Key evidence — permissive scan + transfer filterAfter clicking Запустить сканирование with Spreads of all 13 rows are between 0.30% and 10.95% — within the requested 0.3–15% range. Checking «Только с возможным трансфером (D+/W+)» narrows the table to the 2 rows where both sides show green D+/W+ — Sort toggle
Pair filter (typing "BETH")Known limitations (not failures)
|
Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
- Pause/resume toggle in topbar controls the background loop (manual scans still run). New POST /api/pause and POST /api/resume endpoints; the state flag is exposed via GET /api/state. - Each completed scan appends arb rows into an in-memory ring buffer (SCANNER_HISTORY_LIMIT, default 1000). GET /api/stats?min_spread=& min_profit=&limit=&require_transfer=&pair= returns filtered events, newest first. DELETE /api/stats clears history. - New Statistics tab in the topbar with interactive threshold inputs (defaults: 2% / 20 USD), pair search, transfer filter, sortable table, and a Clear-history button. - Parameters and Summary cards are now collapsible; collapsed state is persisted per card in localStorage (scanner.collapse.*). The selected tab is also remembered (scanner.ui.tab). - Buy/Sell exchange cells in both tables are now anchors to that exchange's spot trading page for the coin (target=_blank). URL templates cover all 13 supported exchanges. - Small button next to Pause triggers an immediate manual scan with the currently entered parameters. Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
- New "Auto-refresh" input in the scan-parameters form lets the user change the background loop period at runtime (minimum 5 seconds). POST /api/scan now accepts an optional refresh_interval field; the value is stored on ScannerState and read by the background loop on each tick, so the next sleep picks up the change immediately. The current interval is reflected in GET /api/state and the topbar label. - Default values for min_volume / min_spread / max_spread are now 0. A max_spread of 0 (or negative) is interpreted as "no upper limit" in find_arbitrage, otherwise the form would filter out every row by default. SCANNER_MIN_VOLUME/MIN_SPREAD/MAX_SPREAD env defaults follow the same convention. Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
Add POST /api/settings that updates ScannerState overrides (proxy, min_volume, min_spread, max_spread) and the runtime refresh_interval without triggering a scan. The frontend now wires every parameter input to a debounced (600 ms) call to this endpoint, plus an immediate flush on the 'change' event, so users no longer have to click anything to make the form take effect — the next background-loop tick or manual scan picks up the new values automatically. Renamed the submit button to 'Сканировать сейчас' (since 'apply' is now implicit) and added a hint that confirms each auto-save with a timestamp. Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
Add HistoryStore (app/history_store.py): a tiny append-only JSON Lines backing store. On every scan completion the newly-detected arbitrage rows are appended to <repo>/data/scanner_history.jsonl in addition to the in-memory ring buffer. On startup the tail of the file (up to SCANNER_HISTORY_LIMIT events) is restored into memory so the Stats tab is populated immediately after a restart. New env var SCANNER_HISTORY_FILE controls the path (empty disables persistence). DELETE /api/stats now also truncates the file. A new GET /api/history/download endpoint streams the full file to the browser, surfaced as a '⬇ Скачать историю' button in the Stats tab. /api/state grew history_file and history_file_size fields used by the UI footer to show where the data lives and how big it is. data/ is now gitignored so the runtime artifact never lands in source control. Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
The Stats-tab thresholds (min_spread, min_profit, require_transfer, pair search) now decide which arbitrage rows actually land in the in-memory ring buffer AND on disk — rows that don't pass are silently dropped at record time. Scan-level filters (min_volume / min_spread / max_spread on ScannerConfig) already filter at scan time, so combined with the new write gate users only persist events that satisfied BOTH filter sets, which is what they asked for. Implementation: - ScannerState.stats_filter holds the current thresholds, defaulting to the same values the UI shows (2% / $20). - New POST /api/stats/filter applies a filter without triggering a scan; the frontend pushes the filter every time the user edits the Stats fields (debounced where appropriate) and on initial page load so the server picks up any localStorage-restored values. - _record_history skips events failing event_passes_stats_filter and logs a one-line summary of how many rows were kept vs. dropped. - /api/state echoes the active stats_filter so the UI can show it. Existing data on disk is left untouched — the filter only governs future writes. 'Clear history' still wipes both buffer and file when the user wants a fresh start. Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
- TelegramNotifier (app/telegram_notifier.py): async bot client with
per-scan deduplication keyed on PAIR|BUY_EX->SELL_EX, message
splitting at 4096 chars, total/per-msg counters, and graceful
fall-through when not configured.
- After each scan, _record_history() returns the events that passed
the stats filter; trigger() fires an asyncio task that batches them
into one Telegram message (split if needed) containing ONLY new
pairs vs the previous notification.
- New endpoints:
POST /api/telegram/settings (toggle / set token / chat_id)
POST /api/telegram/test (probe the bot)
POST /api/telegram/reset_dedup
Telegram state is echoed in /api/state for the UI.
- 'Telegram' card in the Arbitrage view: enable checkbox, masked
bot-token input (token never echoed back), chat_id input, test +
reset-dedup buttons, status badge, last-sent / dedup counters.
- DELETE /api/stats now also resets dedup so old situations can
re-trigger from scratch.
- Config / env: TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID, TELEGRAM_ENABLED
Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
…Y fallback) Telegram is blocked in some regions, which made the new test/notify endpoints time out after 15 s. The notifier now supports proxies the same way the scanner already does: - TelegramNotifier accepts an explicit `proxy` argument plus a `fallback_proxy_provider` callable that is consulted at send-time when no Telegram-specific proxy is set. main.py wires the scanner's current effective proxy (runtime override or SCANNER_PROXY env) as the fallback, so a single SCANNER_PROXY setup is enough. - SOCKS proxies are routed via aiohttp_socks.ProxyConnector (already in requirements); HTTP(S) proxies via aiohttp's `proxy=` kwarg. - New env var TELEGRAM_PROXY + runtime field on POST /api/telegram/settings. - UI: new 'Прокси (опционально)' input in the Telegram card. Placeholder reflects the effective fallback. Hint line shows the chosen proxy and its source (TELEGRAM_PROXY / SCANNER_PROXY / без прокси). - Credentials in proxy URLs are masked everywhere they're surfaced (/api/state, hint line, placeholder). Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
…NER_PROXY fallback)" This reverts commit 37dfe0a.
This reverts commit ddba1af.
Two related additions: 1. Orderbook depth is now mutable at runtime, not just via SCANNER_OB_LIMIT env. Form field 'Глубина стакана, уровней' on the Arbitrage tab; range 5..500 levels, enforced by Pydantic on POST /api/settings. 2. New column 'Профит при $N' that answers 'how much USDT profit would I actually capture if I spent at most $N walking the buy book and selling into the sell book?'. The full intersection 'Профит, $' column is kept alongside it (best-case with unlimited capital). Backend: - ScannerConfig.orderbook_budget_usdt (env SCANNER_OB_BUDGET, default 1000). - OrderbookAnalysis gains budget_usdt / profit_at_budget_usdt / filled_usdt. - calc_arb_volume now returns (cost, avg_spread, profit_at_budget, filled) and truncates the last orderbook level proportionally when the budget is hit mid-level. Setting budget=0 keeps the old behaviour (those two return values are 0.0). - analyze_orderbooks plumbs ScannerConfig.orderbook_budget_usdt through. - _row_to_history_event now persists ob_budget_usdt / ob_profit_at_budget_usdt / ob_filled_usdt to history JSONL so the Stats tab survives restarts. - ScanOverrides exposes orderbook_limit (5..500) and orderbook_budget_usdt (>=0); both auto-apply on input change in the UI. Frontend: - Two extra form fields; placeholder reflects the current backend value. - 'Profit at $N' column shows '—' when the budget is 0 or when the orderbook didn't have enough profitable depth to fill anything. - Column header dynamically shows the active budget (e.g. 'Профит при $1000') so the user always sees the N being used. Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
Mirror of the (now-reverted) Telegram notifier, retargeted at MAX (platform-api.max.ru). MAX's bot API is described at https://dev.max.ru/docs-api/methods/POST/messages — endpoint POST /messages?chat_id=...&user_id=..., Authorization header with the raw token from @Masterbot, markdown formatting, 4000-char text cap. Behaviour: - After each completed scan, every arbitrage row that passes the stats filter (and so is being written to history) becomes one bullet in a single MAX message — but only for pairs that weren't in the previous message (dedup key = 'pair | buy_ex -> sell_ex', FIFO ring of up to 5000 keys, cleared by 'Clear history' / 'Reset dedup' buttons). - Long lists are split across multiple messages (<=3800 chars each) with a '(продолжение)' header, paced ~1 msg/sec to stay polite. - The MAX notifier shares the proxy story with the scanner: MAX_PROXY env (or runtime field) wins; if empty, falls back to whatever the scanner is currently using (SCANNER_PROXY env or runtime override). Endpoints: POST /api/max/settings (enable / token / chat_id / kind / proxy) POST /api/max/test (probe — bypasses 'enabled' & dedup) POST /api/max/reset_dedup (drop the memory) DELETE /api/stats now also clears the MAX dedup so old situations can re-trigger from scratch. UI: collapsible 'Мессенджер MAX' card on the Arbitrage tab — enable toggle, masked bot-token input (token never echoed back), chat/user ID + recipient type select, optional proxy field, test + reset-dedup buttons, status badge (готов / выкл / ошибка / не настроен), counters (total_sent / dedup_skipped / last_sent_at). Config / env: MAX_BOT_TOKEN, MAX_CHAT_ID, MAX_RECIPIENT_KIND, MAX_ENABLED, MAX_PROXY. Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
The block was rendered inside view-arbitrage, but I had hidden its body with an inline 'display:none' and set the toggle to aria-expanded=false / arrow='▸' — so the form was unreachable. Match the controls-/summary-card defaults: aria-expanded=true, arrow='▾', no inline style. Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
Per user request, undo everything since 8494c9e ('Gate history persistence by Stats filter thresholds'). This commit sets the tree back to the state of 8494c9e in a single revert commit — no force-push, no rewritten history. The following commits are effectively dropped: ddba1af Telegram notifications for new arbitrage situations 37dfe0a Telegram: route requests through proxy bbcced0 Revert "Telegram: route requests through proxy" f84c2e7 Revert "Telegram notifications for new arbitrage situations" 411000b Orderbook: configurable depth + 'Profit at $N' column b85ba49 MAX messenger: notifications for new arbitrage situations 097e706 MAX UI: show the messenger card expanded by default Co-Authored-By: harlequincariotta <harlequincariotta@wshu.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Веб-версия скрипта
scanner.pyна FastAPI + ванильный JS дашборд с интерактивными улучшениями UI.Бэкенд
app/scanner_core.py— рефакторинг оригинальногоscanner.pyв библиотеку: те же парсеры тикеров, нормализация сетей, проверка депозитов/выводов и анализ стаканов, но безprint-ов, с настраиваемымScannerConfig(прокси, мин. объём, диапазон спреда, ключи API).app/main.py— FastAPI:GET /api/state— последний результат скана + флаг паузы + размер истории.POST /api/scan— ручной запуск с переопределением параметров.POST /api/pause,POST /api/resume— управление фоновым циклом.GET /api/stats?min_spread=&min_profit=&limit=&require_transfer=&pair=— лог арбитражных событий с фильтрами.DELETE /api/stats— очистка истории.GET /healthz.SCANNER_REFRESH_INTERVAL(по умолч. 60c). Каждое завершение скана пишет арб-строки в кольцевой буфер (SCANNER_HISTORY_LIMIT, по умолч. 1000 событий).Фронтенд
localStorage(scanner.collapse.*). Активная вкладка также сохраняется (scanner.ui.tab).SCANNER_PROXY,SCANNER_MIN_VOLUME,SCANNER_MIN_SPREAD/MAX_SPREAD,SCANNER_HISTORY_LIMIT,MEXC_API_KEY/MEXC_SECRET_KEY,BINANCE_*,BYBIT_*и т.д.Запуск из PyCharm
run.py+.idea/runConfigurations/Web_Scanner.xml— правый клик → ▶.run.pyподгружает.env(черезpython-dotenv) и печатает диагностику ключей:[run.py] env vars: KEY=set(N) / missing.Дашборд: http://localhost:8000.
Review & Testing Checklist for Human
http://localhost:8000— таблица заполняется, имена бирж в строках кликабельные и открывают пару на бирже в новой вкладке.MEXC_API_KEY/BINANCE_API_KEY/BYBIT_API_KEYпроверить, что статусы D/W и список сетей отображаются (а не?).Notes
Старая ветка
devin/1778304462-web-interfaceсодержала упрощённый прототип (только Binance, 1Hz polling) — она НЕ использовалась, эта PR делает полноценную веб-версию текущегоscanner.py(1358 строк). Логика парсинга/расчётов 1:1 повторяет исходный скрипт; добавлены только: выносprintвlogging, dataclasses для сериализации, конфиг через env, фоновый цикл, UI и pause/stats-функционал.Link to Devin session: https://app.devin.ai/sessions/34d6fae82e224fc3b16a8de5edd04b42
Requested by: @evgetos