CHANGELOG

Changelog

v1.0.1

Bug fixes

• save() had a no-file crash window: the bulk rewrite path did LittleFS.remove(USERS_DB_PATH) then LittleFS.rename(tmp, USERS_DB_PATH) as two separate calls. A power loss between them left no users.bin at all, and begin() would silently recreate an empty database on the next boot. Fixed: the tmp file is now verified by re-reading it and comparing its CRC32 against the in-memory database before touching the canonical file, and finalization prefers a direct LittleFS.rename() over the existing destination (a single atomic commit) rather than remove-then- rename, falling back to the old two-step sequence only if that direct rename is ever rejected outright.

• save() stack overflow: the fixed-width binary database format (below) was first implemented with its ~8KB record-flush buffer as a stack array. The default Arduino loop task stack is 8192 bytes total, so that one buffer alone consumed nearly all of it, leaving nothing for the rest of save()'s call chain -- crashed with a Guru Meditation Error on the first save(). Moved to a PSRAM heap allocation (same allocator already used for users_/uidIndex_), freed via an RAII guard so every early return still cleans it up.

• TWDT reboot every ~10.25s: RFID_Access_Control.ino subscribes loopTask to the Task Watchdog Timer in setup() but nothing in the normal loop() path ever called esp_task_wdt_reset(), so the TWDT panicked ~10.2s after boot regardless of load. Fixed by adding esp_task_wdt_reset() at the top of loop().

• list reboots with >~15000 users: sendUserList() streams the entire user list inside a single loop() iteration -- at USB-CDC speeds, tens of thousands of users take longer than the TWDT timeout. Fixed by calling esp_task_wdt_reset() every 256 users inside the streaming loop.

• removeUser() after fs::File::truncate() build error: the arduino- esp32 core's fs::File has no truncate() member (compile error, not runtime). saveSuffixFrom_() now falls back to a full save() when the array shrinks (removeUser), since there is no way to shrink a file in place. rename/renew (no size change) still get true single-record seek+write.

• import didn't accept .bin files: any file not ending in .json fell into the CSV parser, which choked on binary bytes. Fixed: import now detects .bin by extension, validates the header, and sends the bytes straight through to import_bin.

• export ... .bin produced a headerless file: cmd_export() saved the raw bytes from the export_bin wire transfer as-is, which omits the 7-byte file header (magic "RUD1" + version + record size) since that data travels in the JSON control line, not the raw stream. The resulting .bin file would fail _parse_import_bin() on re-import ("missing/bad header"). Fixed: cmd_export() now prepends convert.header_bytes() to the output. Verified end-to-end: a round-trip export->import now produces 0 skips, 5/5 matching entries.

New features

• sync command: makes the device database exactly match a local JSON/CSV/.bin file via a merge-diff (remove/add/replace) against a compact (uid, per-record CRC32) manifest, instead of import's additive-or-fully-destructive model. One raw binary transfer for the whole diff, one flash write, and a final db_crc32 round-trip so the host can confirm the two databases actually match rather than just trusting an "ok" status. See sync_begin/sync_manifest/sync_apply in the Serial Protocol Specification.

• Binary user database: users.json (NDJSON) replaced with a fixed-width binary format (users.bin) -- see DatabaseManager.h for the full record layout. Existing users.json databases are migrated automatically on first boot with this firmware and kept as users.json.bak. Per-record CRC32 preserves the old format's graceful-degradation property (a corrupted record is skipped, not the whole database).

• Sorted array + targeted save strategies: users_ is now kept physically sorted by UID at all times. Lookups use binary search (O(log n)) directly over the sorted array -- the old uidIndex_ std::map was removed as redundant. Save strategies depend on the operation: renameUser()/renewUser() use a single-record seek+write (saveSingleRecord_); addUser() rewrites only from the insertion point onward (saveSuffixFrom_); removeUser() and bulk operations still do a full rewrite.

• Raw binary import/export sub-protocol: new import_bin and export_bin serial commands transfer the database as raw fixed-width records with no JSON framing or ArduinoJson parsing on the device. The CLI defaults to this path for both import and export, with --json-transport falling back to the older JSON-based pipeline. .bin files are sent/received byte-for-byte with no re-encoding.

• CRC32 fingerprint skip: status now includes db_crc32 (a CRC32 over every encoded record in sorted order). The CLI computes the same CRC locally and skips the entire transfer + save if it matches -- common when re-importing the same export.

• find by UID (O(log n)): find --uid does a device-side binary search over the sorted array. Response includes search_us (micros spent in the search only, excluding serial I/O).

• find_name (device-side): find --name now filters on-device (case-insensitive substring match on name) instead of transferring the full database to the CLI for client-side filtering. Only matching records cross Serial. Response includes scan_us.

• Backup format prompt: remove --force and import --clear now prompt for backup format (JSON or .bin) before wiping. --no-backup skips the backup entirely.

• Import accepts .bin files: import users.bin reads the on-disk binary format directly and sends it byte-for-byte to import_bin (no decode-then-re-encode round trip).

• Export to .bin: export users.bin writes the raw on-disk format (header + records) directly from the export_bin wire transfer. Any other extension (.json, .txt, etc.) always produces JSON -- never raw binary under a misleading filename.

• File encoding fallback: import now tries utf-8-sig, then cp1252, then latin-1 for JSON/CSV files, with a warning when a non-UTF-8 encoding is used. Fixes "can't decode byte 0xe9" failures from Excel-exported CSVs on Windows/French locales.

• import_bin/export_bin raw binary transfer: the device reads/writes fixed-width records directly off Serial with no JSON framing. A stalled transfer is detected (short read) and reported as an error requiring reconnect -- there is no mid-stream resync for raw binary on a shared serial line.

• ImportProfiler instrumentation: device-side timing of every phase during an import (JSON parse, batch loop, ACK serialize, save breakdown, transport wait). Reported in the import_result response. Not part of the stable protocol contract.

Changed

• MAX_USERS: 10000 -> 70000 (Config.h).

• Serial baud rate: 921600 -> 2000000 (SERIAL_BAUD in Config.h). The Python CLI auto-matches this rate; a v1.0.0 CLI talking to v1.0.1 firmware (or vice versa) will see garbled output until both sides are updated.

• Line buffer: kLineBufCapacity raised from 4096 to 16384 to support ~100 users per batch_add line. Serial RX ring buffer is now sized from this constant so the two can't diverge.

• Batch import size: IMPORT_BATCH_SIZE raised from 15 to 100 (matches the new 16KB line buffer with >2.5x headroom).

• Boot wait: CLI _wait_for_ready() timeout raised from 15s to 45s to cover the first boot after flash where LittleFS.format() on the 12MB partition can take 10-30s. Post-boot backlog drain added to prevent stale status replies from contaminating the next real command.

• partitions.csv: coredump partition added (64KB at 0xF10000).

v1.0.0

Bug fixes

• JSON escaping: a user name containing " or \ corrupted users.json on the next save, which then failed to parse on the next boot and silently wiped the entire database. Names are now properly JSON-escaped everywhere they're written (DatabaseManager::save(), SerialProtocol::sendUserList()). Control characters (newline/tab/etc.) are now rejected at validation time since they'd still garble the LCD even when escaped correctly.

• O(log n) lookups: every badge scan, remove, rename, and tag-renew did a linear O(n) scan over the user list despite an index existing -- uidIndex_ is now a real uid-to-position map, so those operations are O(log n).

• Idempotency-aware retries: the Python CLI could report a false failure ("Duplicate UID" / "UID not found") for an add/remove/rename that actually succeeded, if the device's reply arrived after the 2s timeout and got retried. Retries are now aware these commands aren't idempotent.

New features

• Automatic backup before wipe: remove --force and import --clear now save an automatic timestamped backup of the current device database to python_cli/backups/ before wiping anything (best-effort -- a backup failure is logged but does not block the operation you asked for).

• Anti-brute-force lockout: after MAX_CONSECUTIVE_DENIALS (default 5) consecutive denied badges, the reader stops accepting cards for LOCKOUT_DURATION_MS (default 30s). Both tunable in Config.h. The serial/CLI link stays fully usable during a lockout.

• Runtime timezone: timezone is now configurable at runtime via python cli.py timezone --offset SECONDS [--dst SECONDS] -- persisted on the device (NVS), no reflash required. Config.h's NTP_GMT_OFFSET_SEC/NTP_DAYLIGHT_OFFSET_SEC are now only the first-boot default.