feat(logloader): download logs over MAVLink FTP only

[dakejahl]

Summary

Supersedes the transport half of #28. Replaces the classic MAVLink log protocol with MAVLink FTP for both the log listing and the bulk transfer, and adds ArduPilot support as a consequence rather than as a special case.

#28 should stay open for its other two commits — the upload-spam fix and the Flight Review API key — which are independent of the transport and good as-is (minus the 401/403 blacklisting, see the review notes on that PR).

Problem

LOG_DATA (msg 120) has no target_system/target_component. A router between logloader and the autopilot has no addressing information to work with, so it copies every chunk to every endpoint it serves, telemetry radio included. On a node running mavlink-router a log download saturates links that have no interest in it.

LOG_ENTRY also forces a second problem: it is the only thing that identifies a log, and it reports an id and a timestamp rather than a file. Anything that downloads over FTP while listing over the log protocol has to map one onto the other by size, mtime or list ordering, and ArduPilot's list-entry numbering, its LOG_MAX_FILES wrap and its .BIN naming all have to be modelled to do it. That mapping layer is where the complexity and the failure modes live: when it guesses wrong, the wrong file is transferred in full and then discarded.

Solution

Use MAVLink FTP for both halves. FILE_TRANSFER_PROTOCOL (msg 110) is addressed and both stacks reply to the requesting sysid/compid, so transfers are unicast. The log protocol is gone entirely — no LOG_REQUEST_LIST, no LOG_ENTRY, no LOG_DATA, no MAVSDK LogFiles plugin.

Identity moves to the listing: path below the log root plus size. There is no cross-protocol mapping left to get wrong, and ArduPilot needs no special handling — a directory listing just reports files. This is the shape QGC's Onboard Logs (FTP) tab already uses.

The root is probed at @MAV_LOG (the virtual directory from the FTP spec), then /fs/microsd/log and /APM/LOGS for older firmware. Timestamps come from the listing when the vehicle implements ListDirectoryWithTime, otherwise from the start time PX4 encodes in the path; ArduPilot logs have none and nothing needs one.

MAVSDK's Ftp plugin discards size and mtime from list replies, so FtpListClient runs the listing over MavlinkPassthrough and keeps the full entry. Bulk transfers still use the plugin. Worth upstreaming to MAVSDK later so this can be deleted.

Pre-FTP databases keyed logs on the LOG_ENTRY timestamp, which FTP cannot reproduce. Those rows are set aside as logs_legacy on first start and matched to the listing by size, so a fleet upgrading to this does not re-download and re-upload its history.

Along the way: a log is only queued once two consecutive listings agree on its size (so the log being written right now is never fetched at a size it will not keep); downloads are staged and size-checked before being moved into place; a log that cannot be fetched sorts to the back of the queue instead of blocking everything behind it; FTP sessions are reset at connect, which is what left PX4 refusing transfers until reboot after logloader was killed mid-download.

Verified against PX4 SITL (main, MAVSDK 3.17.1)

• @MAV_LOG probe, session reset, index, burst download; files byte-identical to the vehicle's, staging directory clean afterwards.
• Timestamps via ListDirectoryWithTime, and via the path-derived fallback when the opcode is forced off (the path ArduPilot takes).
• remote_log_directory override against a physical path.
• Actively-written log correctly held back (3 of 4 indexed, 1 still being written) instead of being downloaded and discarded.
• Legacy migration across all three cases: downloaded+uploaded (not refetched), downloaded-not-uploaded (keeps its existing file), never-downloaded (fetched fresh).

Follow-ups

• Bump the ARK-OS submodule pointer after this merges.
• Confirm ARK's shipped PX4 configs start the companion MAVLink instance with -x; FTP is now load-bearing.
• Large-file FTP timeouts that Download logs over MAVLink FTP, add ArduPilot support #28's hardware test hit are unaddressed here; the queue reordering keeps one bad log from blocking the rest, but resume is still worth doing.

[dakejahl]

Superseded by #32, which contains this branch's commits unchanged and with authorship intact — merging #32 merges this.

Reviewing this stack turned up structural problems in the code underneath it, and fixing those meant rewriting the parts these PRs sit on. Keeping them open would have meant reviewing #29's ServerInterface changes and then reviewing their replacement in the same sitting, so it is one branch now.

Nothing here is dropped: the FTP transport, the reachability cooldown and the API-key headers are all in #32. The one deliberate change is that 401/403 no longer blacklist a log — see the note in #32.