Epic/02 remote http management api#2
Merged
diegoparrilla merged 7 commits intomainfrom Apr 30, 2026
Merged
Conversation
Server (RP2040):
- New rp/src/http_server.{c,h}: HTTP/1.1 server on lwIP raw TCP API.
Per-connection state machine (read-headers, dispatch, write-response)
with a 4-slot static pool, 2 KB header buffer per conn, 1 KB response
buffer, 16 s idle timeout via tcp_poll. Validates Host:, rejects
Transfer-Encoding: chunked with 411. HEAD handled at the dispatcher
so every GET route gets HEAD for free. Centralised response writer
always emits Server: md-devops/<v>, Connection: close, Content-Type
and Content-Length. Dispatch returns 405 with Allow: on verb
mismatch and 404 on unknown route.
- One route registered: GET /api/v1/ping returns
{"ok":true,"version":"<RELEASE_VERSION>","uptime_s":<n>}.
- emul.c calls http_server_init() once Wi-Fi connect attempts complete
and adds an "API : http://<host>.local/ (<ip>)" line to the menu
under the GEMDRIVE block.
- lwipopts.h: dead LWIP_HTTPD block removed; mDNS responder enabled
(LWIP_MDNS_RESPONDER, LWIP_IGMP, LWIP_NUM_NETIF_CLIENT_DATA,
MDNS_RESP_USENETIF_EXTCALLBACK, MDNS_MAX_SERVICES,
MEMP_NUM_SYS_TIMEOUT) so the existing #if LWIP_MDNS_RESPONDER
blocks in network.c (mdns_resp_init / add_netif / add_service for
_http._tcp) actually compile.
- CMakeLists.txt links pico_lwip_mdns and adds http_server.c to the
source list.
- mDNS hostname comes from existing gconfig PARAM_HOSTNAME (default
"sidecart") — no new aconfig key.
CLI (cli/sidecart.py):
- Single-file Python CLI, stdlib-only (Python >= 3.10). argparse with
subcommand dispatch, --host / --json / -q global flags,
SIDECART_HOST env override, granular exit-code map (0..8).
- ping subcommand wired end-to-end via urllib.request.
Tests (cli/test_sidecart.py):
- 14 unittest cases covering ping (human, --json, -q),
status-code -> exit-code mapping (200/404/503/500/network),
host-resolution precedence, base-url construction.
- In-process http.server fake on 127.0.0.1:0 captures verb/path/
headers/body and serves canned responses.
Build: rp.uf2 produced, BOOT.BIN unchanged, all 14 CLI tests green.
Server (rp/src/http_server.c):
- Route table with {path, methods_mask, handler}; dispatcher returns
405 with the correct Allow header on verb mismatch, 404 when no
path matches. HEAD is allowed wherever GET is.
- New helpers: url_decode (percent-encoding), query_get (URL-decoded
?key=value extractor), normalize_rel (collapse repeated slashes,
strip trailing slash, resolve ".", reject "..", NUL, non-ASCII,
control chars), resolve_jail (prepend GEMDRIVE_FOLDER),
fat_to_iso8601 (FatFs DOS date/time -> YYYY-MM-DDTHH:MM:SS),
body_appendf (vsnprintf with overflow detection), format_allow.
- GET /api/v1/volume: f_getfree -> {ok, total_b, free_b, fs_type}.
503 busy when SD is unmounted.
- GET /api/v1/files?path=<rel>: f_opendir/f_readdir, JSON entries
{name, size, is_dir, mtime}, 1000-entry cap, truncated:true on
overflow, 422 is_file when path resolves to a file, 404 not_found
when missing.
- Per-conn response buffer 1 KB -> 8 KB (4 conns x 8 KB = 32 KB)
so a realistic listing fits as a single tcp_write. True chunked
streaming deferred to S5 where multi-MB downloads need it.
CLI (cli/sidecart.py):
- New subcommands `volume` and `ls [PATH]`. volume prints
"free <human> / total <human> (FAT32)" or --json passthrough. ls
aligns name/size/type/mtime columns, URL-encodes the path, surfaces
the truncated flag.
Tests (cli/test_sidecart.py):
- 8 new cases (volume human/JSON/503, ls default-root/explicit-path/
truncated/404/422). Now 22 tests, all green.
Build: rp.uf2 produced; CLI tests pass.
Server (rp/src/http_server.c, .h):
- Body-read state machine (HC_READ_BODY) with 256-byte per-conn body
buffer for short JSON (rename {"to":"..."} fits comfortably). 413
on Content-Length > buffer; 411 on missing CL or chunked TE.
- 8.3 validator (validate_8_3_last): stem ≤ 8, ext ≤ 3, ASCII only,
rejects FAT-illegal chars, leading/trailing dot or space, multiple
dots.
- Hand-rolled JSON extractor (json_extract_string) ~80 LOC, parses
flat string-valued objects with \" \\ \/ \n \t \r escapes; rejects
non-string-on-target-key with bad_json/unprocessable. No JSON lib.
- write_response_ex helper with optional extra-headers block (used
for Location: on 201 and Allow: on 405).
- POST /api/v1/folders/<rel>: f_mkdir, 8.3 enforced, 409 if exists,
404 if parent missing, 201 with Location:.
- DELETE /api/v1/folders/<rel>: f_unlink, 404 not_found / is_file
(with hint), 409 not empty, 409 root.
- POST /api/v1/folders/<rel>/rename: requires application/json
(415), reads {"to":"..."}, jails source and target, validates 8.3
on target, detects cycle (422 unprocessable), no-op on from==to
(200), 409 target exists, 404 target parent missing.
- Route table prefix dispatcher recognises /api/v1/folders/<rel> and
the trailing /rename action suffix; emits 405 with Allow: on verb
mismatch and 404 for empty <rel>.
- Content-Type captured during initial header walk (was previously
rescanned post-parse, which fails since the parser nul-terminates
in place — every JSON request was incorrectly rejected as 415).
- All 43 functions wrapped with __not_in_flash_func so callbacks
fired from cyw43_arch_poll don't take XIP cache misses that would
stall the cartridge bus.
Server: streaming directory listing
- HC_STREAM_LISTING state. f_opendir is held open across recv/sent
callbacks; conn_close releases it on completion or abort.
- Response uses HTTP/1.1 chunked Transfer-Encoding so listings of
hundreds of entries work without growing per-conn buffers. Each
tcp_sent ack drives the next chunk via stream_listing_drive;
tcp_sndbuf checked to avoid ERR_MEM. Closing envelope and
terminator sent once the dir is exhausted.
Server: RAM pressure fixes
- Per-conn buffers shrunk to fit alongside the cartridge ROM mirror
in the same SRAM bank (memmap_rp.ld puts heap right before
ROM_IN_RAM): HEADER 2048->1024, RESPONSE 8192->1024,
REQUEST_BODY 1024->256, MAX_CONNECTIONS 4->2. BSS contribution
dropped ~47 KB -> ~5 KB.
Server: 8.3 uppercase enforcement
- normalize_rel case-folds ASCII letters to uppercase as it copies
segments, so every FatFs call (mkdir/unlink/rename/stat/opendir)
goes through with the canonical 8.3 form. Pre-existing mixed-case
entries still match because LFN matching is case-insensitive.
- Listing output uppercases finfo->fname before formatting so the
wire always shows uppercase regardless of how each entry was
stored.
Atari ST side (target/atarist/src/gemdrive.s):
- "GEMDRIVE active." banner replaced with "Starting DevOps
Microfirmware..."; the three feature-bullet lines removed.
CLI (cli/sidecart.py):
- New subcommands `mkdir REMOTE`, `rmdir REMOTE`, `mvdir FROM TO`.
mvdir builds JSON via json.dumps and sets Content-Type:
application/json. Output: ok <path> on create, ok on delete,
ok <from> -> <to> on rename, or --json passthrough.
Tests (cli/test_sidecart.py):
- 8 new cases covering mkdir 201/409/400, rmdir 204/409/404 is_file,
mvdir 200/422. Now 30 tests, all green.
Build: rp.uf2 produced; BOOT.BIN at 7044/8192 bytes; CLI tests pass.
Server (rp/src/http_server.c):
- handle_file_delete: DELETE /api/v1/files/<rel>. f_stat to verify
existence and regular-file type, f_unlink, 204 No Content. 404
not_found on missing, 404 is_directory (with hint pointing at
/folders/<rel>) when path resolves to a directory.
- handle_file_rename: POST /api/v1/files/<rel>/rename. Requires
Content-Type: application/json (415 otherwise) and Content-Length
(411 otherwise). Reads {"to":"..."} via the hand-rolled JSON
extractor, jails source and target, validates 8.3 on target's
last segment, no-op (200) on from==to, 409 if target exists, 404
on missing source or target parent, 404 is_directory if source
resolves to a directory.
- route() extended with /api/v1/files/<rel> prefix dispatch, including
the trailing /rename action suffix. 405 with the correct Allow:
header on verb mismatch (DELETE for plain, POST for /rename).
Empty <rel> -> 404 not_found. Exact-match /api/v1/files (listing)
remains served by the existing exact-match table.
CLI (cli/sidecart.py):
- New _files_url helper mirroring _folders_url.
- New subcommands `rm REMOTE` and `mv FROM TO`. mv builds JSON via
json.dumps and sets Content-Type: application/json. Output: ok
on delete, ok <from> -> <to> on rename, or --json passthrough.
Tests (cli/test_sidecart.py):
- 6 new cases (rm 204 / 404 is_directory / 404 not_found, mv 200 /
409 conflict / 404 is_directory). Now 36 tests, all green.
Build: rp.uf2 produced; CLI tests pass.
Server (rp/src/http_server.c): - New HC_STREAM_DOWNLOAD state + per-conn FIL handle, stream_pos / stream_end offsets, holds_body_lock flag. - Range: header parsed during the same loop that captures Host / Content-Length / Content-Type. Three forms: bytes=N-M (closed), bytes=N- (open), bytes=-N (suffix). Multi-range (comma) -> 416. - handle_file_download: validates and jails path, f_stat (404 not_found / 404 is_directory + hint pointing at ?path=), resolves Range against file size (416 + Content-Range: bytes */N on bad range), opens file, seeks, acquires the body-stream lock, sends status (200 or 206) + Accept-Ranges + Content-Length + optional Content-Range, then transitions to streaming. HEAD returns headers only. - stream_download_drive: pumps as many TCP_SND_BUF-sized chunks as fit per tcp_sent ack; on tcp_write ERR_MEM, f_lseek rewinds so the next ack retries the same bytes. On completion releases the lock, closes the FIL, conn_close. - Body-stream lock (g_body_stream_busy): acquired by streaming GET download (and S6 PUT upload). Second body request returns 503 + Retry-After: 1. conn_close releases the lock and the FIL handle on normal completion, idle timeout, or peer-abort. - /files/<rel> prefix dispatch extended: GET / HEAD invoke the download handler. Verb-mismatch 405 Allow updated to "GET, HEAD, DELETE". CLI (cli/sidecart.py): - New `get REMOTE [LOCAL] [-r/--resume]`. Streams via urllib.request.urlopen in 8 KB chunks. With -r, sends Range: bytes=<existing>- and appends only when the server returns 206; falls back to overwrite if the server returns 200. Progress counter on stderr (`<done> KB / <total> KB`, \r-rewrite), suppressible with -q. Exit code map: 200/206 -> 0, 404 -> 3, 416 -> 5, 5xx -> 7, network -> 8. Tests (cli/test_sidecart.py): - 5 new cases (full download, custom local path, resume sending Range: header and appending the slice, 404, 416). Now 41 tests, all green. Build: rp.uf2 produced; CLI tests pass.
Server (rp/src/http_server.c):
- New HC_STREAM_UPLOAD state + per-conn upload fields (FIL,
upload_file_open, upload_was_overwrite, upload_received,
upload_norm for Location, upload_abs for cleanup unlink).
- has_content_length flag added to the parser so we can distinguish
"no header" (PUT -> 411) from "Content-Length: 0" (legal).
- HTTP_MAX_UPLOAD_BYTES = 4 MB cap.
- parse_and_dispatch intercepts PUT-on-/files/-prefix BEFORE the
generic body-bearing block (whose 256-byte cap is wrong for
uploads). It computes leftover header-segment body bytes and
hands them to handle_file_upload_init.
- handle_file_upload_init: 503 busy if body-stream lock held;
PUT-on-/rename -> 405 POST; missing CL -> 411; > 4 MB -> 413;
path normalize+jail+8.3; root -> 409; ?overwrite=0/1/other ->
400 bad_query; target-is-directory -> 409 is_directory; target
exists no overwrite -> 409 conflict. f_open with FA_CREATE_ALWAYS,
parent missing -> 404, acquire body-stream lock, write any
leftover header-segment bytes.
- srv_recv_cb gains HC_STREAM_UPLOAD branch: pbuf_copy_partial into
per-conn 1 KB scratch buffer, f_write chunk-by-chunk, accumulates
upload_received. On completion -> upload_finish_ok which closes
FIL (with explicit f_close error check + unlink + 500 surface so
a failed close doesn't silently leave a FatFs FF_FS_LOCK entry),
releases lock, sends 200 (overwrite) or 201 + Location header
(new file).
- conn_close interrupted-upload cleanup: closes FIL and unlinks
upload_abs so peer-aborts/timeouts/write-errors don't leave a
half-written file on disk.
- Hardening from delete-fails-after-upload investigation:
- upload_finish_ok captures f_close return code; surfaces 500
+ unlink + DPRINTF if it fails.
- handle_file_delete maps FR_DENIED -> 409 conflict ("File is
open elsewhere or marked read-only") instead of generic 500.
- handle_folder_delete FR_DENIED message widened to mention all
three causes (non-empty, locked, read-only).
- handle_file_download empty-body (Content-Length 0) branch
now releases the body-stream lock.
CLI (cli/sidecart.py):
- New `put LOCAL [REMOTE] [-f/--force]`. Uses
http.client.HTTPConnection so we can interleave a progress
counter with each chunk send (urllib.request would buffer the
whole body before reading the response). Sets
Content-Type: application/octet-stream and Content-Length.
--force adds ?overwrite=1. Default REMOTE = basename(LOCAL).
Progress on stderr (`<sent> KB / <total> KB`, \r-rewrite,
-q suppresses).
Tests (cli/test_sidecart.py):
- 4 new cases (put 201 create with proper headers + body, --force
adds ?overwrite=1, 409 conflict without -f, 503 busy). Now 45
tests, all green.
Build: rp.uf2 produced; CLI tests pass.
Server (rp/src/http_server.c):
- Final pass on error envelopes: every site uses the locked
{ok:false, code, message} shape with a status from the documented
matrix. One inconsistency unified — the suffix-range edge case
(bytes=-N against an empty file) now goes through write_response_ex
with Content-Range: bytes */0 like the other 416 paths.
Docs (docs/api.md, new):
- Quick start + curl/sidecart examples for the most common verbs.
- Conventions: no auth (LAN-only), jailed root, FAT 8.3 only,
HTTP/1.1, Host: mandatory, no chunked TE on uploads, body-stream
lock, response header rules, error envelope.
- Status code matrix and full error code vocabulary.
- Per-endpoint reference for ping / volume / ls / get / put / rm /
mv / mkdir / rmdir / mvdir, each with HTTP shape, success / error
responses, curl example, sidecart example.
- CLI exit code map.
CLI (cli/sidecart.py):
- Top-of-file docstring rewritten as a complete usage reference:
flags, env-var, full subcommand -> endpoint table, exit codes,
examples.
README.md:
- New "Remote HTTP Management API" section pointing at sidecart.local,
the CLI, and docs/api.md, with the LAN-only / no-auth caveat.
Menu / countdown (rp/src/emul.c, rp/src/term.{c,h}):
- Indentation in menu argument labels: removed two spaces before the
":" in F[o]lder / [D]rive / [R]eloc addr / Mem[t]op / URL /
IP address so the column is tighter.
- Split the API line into URL and IP address rows for readability.
- Any keystroke now halts the autoboot countdown — bound OR unbound
— matching md-drives-emulator. term.c sets a static
anyKeyPressedFlag on every incoming keystroke and exposes
term_consumeAnyKeyPressed() (consume-on-read); the main loop polls
this and sets haltCountdown = true.
- When haltCountdown flips false -> true the main loop repaints the
bottom strip once with "Countdown stopped. Press [E] or [X] to
continue." instead of leaving the strip frozen on whatever counter
value it last drew.
Build: rp.uf2 produced; CLI tests still 45/45 green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.