Extend mt5-httpapi with backtest endpoints on top of v4 config

[Marinski]

Check brief video explanation below:

Summary

This PR extends mt5-httpapi with HTTP-driven MT5 Strategy Tester support and an opt-in per-terminal execution mode (live / backtest).

mt5-httpapi already runs real MT5 terminals inside a Windows VM and exposes them over HTTP for live trading and market data. This PR keeps that model intact and adds a second execution mode focused on isolated tester runs, served behind the same /<broker>/<account>/... routing, the same Docker / Windows VM runtime, and the same single-file YAML config.

The branch lands as 2 commits on top of v4.0.1:

1. feat(backtest): MT5 Strategy Tester HTTP API + mode-aware terminal launch
2. docs(backtest): document mode field, endpoints, and asset workflow

Motivation

The reason mode exists at all is structural: MT5 is single-instance per portable data directory. If terminal64.exe is already running to back the live SDK on a portable dir, a Strategy Tester subprocess against the same dir exits silently with exit code 0 and produces no report, no journal entry, nothing. There is no way to run a tester on a directory currently held by a live terminal.

The fix is to declare intent up front:

• a mode: live terminal keeps terminal64.exe running so the MT5 SDK stays initialized for live trading endpoints
• a mode: backtest terminal prepares the same portable dir but does not auto-launch terminal64.exe, leaving the data dir free for the tester subprocess that the /backtest endpoint spawns on demand

A single installation can run both kinds of terminal side by side.

What this PR adds

1. Per-terminal execution mode

A terminal entry in config.yaml can now declare a mode. Default is live (fully backwards-compatible).

terminals:
  - broker: roboforex
    account: main
    port: 6542
    utc_offset: "3h"
    # mode: live (default) — keeps terminal64.exe running for live SDK calls

  - broker: roboforex
    account: tester
    port: 6543
    utc_offset: "3h"
    mode: backtest   # don't auto-launch terminal64.exe; reserved for /backtest jobs

The mode flows config.yaml → config_helper.py → start.bat → api_runner.bat → mt5api --mode → MODE constant in config.py. main.py branches on it: backtest-mode skips init_mt5, the background init thread, and the live monitor; sweep_orphans still runs.

2. Backtest HTTP endpoints

Routes are registered on every terminal (the Flask blueprints don't change shape per terminal), but POST /backtest only runs to completion on a mode: backtest terminal — see Motivation above. On a mode: live terminal the tester subprocess will be denied the data dir and the job will fail.

Endpoints introduced by this PR:

• POST /backtest/build-ini — stateless helper that turns a small JSON spec (symbol, timeframe, expert, date range, modelling, latency, deposit, currency, leverage, set, report name) into a fully-formed tester.ini. Optional; you can also write the INI yourself.
• POST /backtest — multipart submission. Returns 202 Accepted with a job payload and Retry-After: 60.
• GET /backtest/<job_id> — job state + parsed summary on completion.
• GET /backtest/<job_id>/report — the MT5 HTML report.
• GET /backtest/<job_id>/log — the terminal log captured for the run.

GET /ping echoes back {"status":"ok","mode":"<live|backtest>"} so callers can verify a terminal is configured the way they expect.

Concurrency: only one tester runs at a time per API process (serialized by an internal RUN_LOCK); additional submissions queue.

POST /backtest

Multipart fields:

• ini — required. Submitted as UTF-8 text. The runner re-encodes to UTF-16-LE + BOM + CRLF before handing it to MT5, because MT5 silently rejects [Tester] Login under UTF-8.
• expert — optional uploaded .ex5
• expert_name — optional filename resolved from experts
• set — optional uploaded .set
• set_name — optional filename resolved from sets

Rules:

• exactly one of expert or expert_name is required
• set / set_name are optional
• if both upload and name are provided for the same file type, upload wins
• path traversal in *_name is rejected
• the INI's [Common] Login / Password / Server are always overwritten with the URL-selected account's credentials

GET /backtest/<job_id>

Status lifecycle: queued → running → completed | failed. When status: completed, the payload includes a summary parsed from the HTML (netProfit, profitFactor, recoveryFactor, expectedPayoff, sharpeRatio, maxDrawdown, totalTrades, profitTrades, lossTrades, …). Jobs left running when the API restarts are marked failed on the next startup.

3. MT5 tester execution flow

The backtest handler:

• loads the URL-selected account's credentials from config.yaml
• parses and validates the submitted tester INI (configparser with optionxform = str so MT5 case sensitivity is preserved)
• injects Login / Password / Server into [Common]
• normalizes Report to Reports\<jobId>.htm under the portable dir
• stages EA / set / normalized INI artifacts under the job's working dir
• spawns terminal64.exe /portable /config:<ini> under the run lock
• persists job state (queued/running/completed/failed + paths + summary) for polling
• returns the report HTML and the run log on demand

4. Optional host-managed asset pool

For repeated runs of the same EA, this PR adds an optional asset model:

assets/
  experts/   # *.ex5 — host-managed expert advisors (mounted read-only)
  sets/      # *.set — host-managed parameter files

Mounted via ./assets:/shared/assets:ro in docker-compose.yml, referenced from POST /backtest via expert_name / set_name. Inline upload still works exactly as documented; the two are interchangeable per request.

5. Integration with the v4 single-file config model

Uses the v4 config.yaml model. config_helper.py gained a terminals command that emits broker / account / port / utc / mode for start.bat and api_runner.bat to consume.

API behavior

All terminals, regardless of mode, expose the same Flask blueprints. The difference is runtime state, not route surface:

• mode: live — terminal64.exe is up, MT5 SDK is initialized, all live-trading + market-data endpoints function normally. POST /backtest will fail because the live process owns the portable dir.
• mode: backtest — terminal64.exe is not running, MT5 SDK is not initialized. Live-trading and market-data endpoints will return errors (no SDK). POST /backtest works because the data dir is free for the tester subprocess.

Example requests

Build INI from a JSON spec

curl -sS -X POST -H "Authorization: Bearer $MT5_API_TOKEN" \
  -H "Content-Type: application/json" \
  http://localhost:8888/roboforex/tester/backtest/build-ini \
  -d '{
    "symbol": "NZDJPY",
    "timeframe": "M15",
    "expert": "EA Studio NZDJPY M15 1615044595.ex5",
    "lastYears": 5,
    "modelling": "open-prices",
    "latencyMs": 5,
    "expertParameters": "ea studio nzdjpy m15 1615044595.set"
  }' > tester.ini

Submit with uploaded files

curl -X POST -H "Authorization: Bearer $MT5_API_TOKEN" \
  -F 'ini=@tester.ini;type=text/plain' \
  -F 'expert=@dist/MyStrategy.ex5' \
  -F 'set=@dist/MyStrategy.set' \
  http://localhost:8888/roboforex/tester/backtest

Submit with host-managed assets

curl -X POST -H "Authorization: Bearer $MT5_API_TOKEN" \
  -F 'ini=@tester.ini' \
  -F 'expert_name=MyStrategy.ex5' \
  -F 'set_name=MyStrategy.set' \
  http://localhost:8888/roboforex/tester/backtest

Poll status, fetch report and log

curl -H "Authorization: Bearer $MT5_API_TOKEN" \
  http://localhost:8888/roboforex/tester/backtest/$JOB

curl -H "Authorization: Bearer $MT5_API_TOKEN" \
  http://localhost:8888/roboforex/tester/backtest/$JOB/report -o report.htm

curl -H "Authorization: Bearer $MT5_API_TOKEN" \
  http://localhost:8888/roboforex/tester/backtest/$JOB/log -o run.log

Example queued response

{
  "jobId": "4e3a7f5a1b0d4f6b8c9d2e1f3a4b5c6d",
  "status": "queued",
  "broker": "roboforex",
  "account": "tester",
  "submittedAt": "2026-05-08T12:34:56Z",
  "reportName": "backtest-report.htm",
  "reportPath": "C:\\...\\Reports\\4e3a7f5a....htm",
  "logPath": "C:\\...\\logs\\backtest-roboforex-tester-4e3a7f5a....log",
  "statusUrl": "/backtest/4e3a7f5a1b0d4f6b8c9d2e1f3a4b5c6d",
  "reportUrl": "/backtest/4e3a7f5a1b0d4f6b8c9d2e1f3a4b5c6d/report",
  "logUrl": "/backtest/4e3a7f5a1b0d4f6b8c9d2e1f3a4b5c6d/log",
  "pollAfterSeconds": 60,
  "queuePosition": 0
}

POST /backtest also sets Retry-After: 60.

Configuration changes

config.yaml

New optional terminal field:

• mode: live (default) | backtest

Optional host-managed asset folders

• experts
• sets

Mounted into the VM via ./assets:/shared/assets:ro. Used only when the caller supplies expert_name / set_name.

Documentation updates included in this PR

• README.md — new Backtest section with the two-stage flow, asset sources, worked NZDJPY example, full endpoint reference, and the terminals[].mode field documented in the config example with an explanation of the single-instance lock that makes mode: backtest mandatory for tester runs.
• SKILL.md — same structural updates so the skill stays in sync with README.
• setup.md — minor reference fixes.
• docker-compose.yml.example — adds the ./assets:/shared/assets:ro mount.

Compatibility

Fully additive:

• mode defaults to live; existing config files keep working as-is.
• Live terminals retain their full API surface and runtime behavior.
• /<broker>/<account>/... routing is unchanged.
• Multi-terminal architecture is unchanged.
• Docker / Windows VM deployment model is unchanged.
• The host-managed asset pool is opt-in.

Validation

• Unit tests: full pytest suite green — 121 passed, including 28 new tests covering the backtest handler, INI builder, and job lifecycle.
• Static checks: python3 -m py_compile across mt5api, bash -n run.sh, docker compose -f docker-compose.yml(.example) config -q.
• End-to-end tester run against a real Darwinex demo terminal in mode: backtest:
  • Job 8992f6ec… completed successfully.
  • terminal64.exe exit code 0, wall time 114.5 s.
  • backtest-report.htm written, 276,090 bytes.
  • Parsed summary: netProfit: 129.89, profitFactor: 1.34, totalTrades: 123.
  • Journal confirms Tester automatical testing started → last test passed with result successfully finished.
  • GET /ping on the same terminal returned {"status":"ok","mode":"backtest"}.

Review notes

The PR crosses a few layers (API routes, MT5 execution flow, startup scripts, compose, docs) but the core idea is narrow: let a terminal declare whether it's there to serve live trading or to run tester jobs, because MT5's single-instance lock means a single portable dir can't do both at once.

The branch is structured as feat: then docs: so the implementation can be reviewed independently of the documentation surface.

[psyb0t]

Hey, @Marinski !
Awesome PR you have here! I'll merge it.

Just one thing before tho:
BACKTEST_TIMEOUT_SECONDS <- this is currently hardcoded. should be like this: DEFAULT_BACKTEST_TIMEOUT=6h in code, configurable BACKTEST_TIMEOUT in config.py and also supported as a POST argument when executing it.
So like, if not POST arg -> get BACKTEST_TIMEOUT from config. if not there -> use hardcoded default.
Also notice the format i used. parse_duration_to_seconds check this func in config.py that's already used for utc_offset.

Once that's fixed, I'll merge xD

[Marinski]

Hey @psyb0t , very happy that you like the PR idea.

I implemented the requested backtest timeout change with the fallback chain you asked for: POST override first, then config, then the hardcoded 6h default, and it reuses the existing duration parser instead of introducing a new format.

Besides that, while testing the backtester flow I fixed a few runtime issues I hit in practice: per-terminal symbol suffix support so the same test inputs can run across multiple brokers with different symbol naming (in case someone wants to bulk backtest same EA/set against multiple terminals), startup now respects reboot_interval overrides so backtest instances are not interrupted when config sets it to 0, nginx upload/body limits are configurable for larger hosted EA/set uploads, and the report parsing now includes bars/ticks so missing history is visible in the parsed result.

[psyb0t]

@Marinski awesome! Merged! Happy backtestin'!

[psyb0t]

@Marinski also bumped version + updated the clawhub skill btw