Phase 39 extmod opt ins

Phase 39 — Additional extmod feature opt-ins (in progress)

Part of the Amiga port design log.

Status: in progress. Steps 1–4 shipped; Steps 5–6 remain.

Audit of extmod/ against the current Amiga port config turned up a handful of cross-port modules and feature flags that aren't currently enabled but would plausibly pay off on AmigaOS. This phase is a menu; which items actually land is to be decided when the phase is picked up. Each candidate below carries a rough cost/benefit so the choice can be made in context.

The current baseline (via MICROPY_CONFIG_ROM_LEVEL_EXTRA_FEATURES plus the port-local opt-ins) already includes binascii, hashlib (SHA-256 only), heapq, random, re, deflate decompress, uctypes, cmath, framebuf, errno, select, plus port-local json, socket, ssl, and VfsAmiga. The candidates here are the gaps left by that baseline that look genuinely useful on Amiga (i.e. not GPIO / radio / micro-flash territory).

Candidates

1. Wall-clock time surface — time.time(), time.time_ns(), time.gmtime(), time.localtime(), time.mktime(). Currently the time module only exposes ticks_* and sleep* (defaults for MICROPY_PY_TIME_GMTIME_LOCALTIME_MKTIME and MICROPY_PY_TIME_TIME_TIME_NS are both 0 and the port hasn't overridden them). Backing: dos.library/DateStamp() (day + minute
   • tick since 1978-01-01) for the wallclock, plus optionally battclock.resource for sub-minute granularity on machines that have a battery-backed clock. Implementation lives in a port-local MICROPY_PY_TIME_INCLUDEFILE. Probably the highest-impact item on the list — a lot of stdlib-flavoured Python assumes time.time() exists. ~150 lines of glue. The Amiga epoch offset is 2922 * 86400 seconds (8 years from 1970-01-01 to 1978-01-01); locale handling stays minimal (UTC; locale.library integration is out of scope).
2. hashlib.md5 + hashlib.sha1 — currently gated on upstream MICROPY_PY_SSL (which this port doesn't set; it uses MICROPY_PY_AMIGA_SSL instead), so both digests are off despite the C sources being compiled in for free with hashlib. One-line opt-in via #define MICROPY_PY_HASHLIB_MD5 (1) and MICROPY_PY_HASHLIB_SHA1 (1). Cheap and routinely needed for file checksums, HTTP digest auth, and content-addressed lookups.
3. btree (extmod/modbtree.c + the bundled lib/berkeley-db-1.xx submodule, already a git submodule of the repo). Pure C, no networking, no GPIO — just a persistent K/V store over the file API. Gives Amiga scripts a real on-disk dict without writing flat-file plumbing. ~30 KB binary cost. The berkeley-db submodule must be initialised (it isn't pulled by default on a fresh clone). Pairs naturally with VfsAmiga.
4. MICROPY_PY_DEFLATE_COMPRESS — extends the existing deflate module from decompress-only to a full read/write surface (gzip / zlib output). Single-flag opt-in. Useful for log compression, network payload packing, and .tar.gz / .zip generation paired with binascii.crc32 (already on).
5. marshal — bytecode (de)serialisation. Disabled below ROM_LEVEL_EVERYTHING. Cheap to enable; only useful if a use case shows up for serialising code objects (tooling, .mpy introspection beyond what mpy-tool already covers). Optional / low priority.
6. websocket + webrepl (extmod/modwebsocket.c, extmod/modwebrepl.c). Layers on the existing port-local socket module to give a REPL over the network — a meaningful quality-of-life win on AmigaOS where the local console is the classic Shell window. Caveats: webrepl wants a dupterm hook and the MICROPY_ENABLE_SCHEDULER, which this port has deferred along with asyncio (see MICROPY_PY_ASYNCIO (0) and MICROPY_ENABLE_SCHEDULER (0) in mpconfigport.h). So this item is the most work of the lot — likely needs a small cooperative poll-from-VM-hook shim rather than a real scheduler. Defer until after the asyncio story is resolved.

Explicitly out of scope

• asyncio — already deferred port-wide pending threads / scheduler.
• bluetooth, lwip, cyw43, wiznet5k, ninaw10, esp_hosted, nimble, btstack, network_ppp_lwip — hardware-specific (radio / Ethernet MAC drivers) with no Amiga fit. Networking on this port is bsdsocket.library-only and that's already wired.
• machine_* (adc, pwm, i2c, spi, uart, timer, wdt, usb_device, i2s, can, pulse, bitstream, signal, pinbase) — no Amiga peripheral equivalent.
• onewire — GPIO-based.
• vfs_fat, vfs_lfs, vfs_posix, vfs_rom, vfs_blockdev — VfsAmiga already covers the filesystem layer via dos.library. Mounting FAT/LFS images would be a separate, large project.
• modopenamp — multi-core (M4/M7) RPMsg. N/A.
• modplatform — platform-identification strings. Already covered by platform.amiga_info() (Phase 33).

Decision deferred

Which subset to actually implement is decided when the phase is picked up. Likely landing order if all are taken:

1. Wall-clock time (biggest API-surface gain).
2. hashlib MD5 + SHA-1 (one-line, lots of use cases).
3. MICROPY_PY_DEFLATE_COMPRESS (one-line).
4. btree (medium effort, niche-but-real use case).
5. marshal (optional).
6. websocket + webrepl (waits on a scheduler-lite story; arguably its own phase rather than part of this one).

Each item lands as a discrete commit so a partial Phase 39 can stop cleanly between any two.

Files (per item, expanded when the phase starts)

ports/amiga/mpconfigport.h           — feature-flag flips
ports/amiga/modtime.c (new)          — for item 1 (time wallclock)
ports/amiga/Makefile                 — SRC_C / SRC_QSTR additions
tests/ports/amiga/test_phase39_*.py  — per-item smoke tests

Variants: all three. Cumulative size impact depends on which items land — rough upper bound if everything except webrepl is taken is ~40 KB text (btree dominates).

Step 1 — Wall-clock time surface ✅

Shipped:

• time.time() and time.time_ns(), backed by timer.device GetSysTime() (already opened at startup by amiga_timer.c for ticks_/delay_; no additional resource open).
• time.gmtime() / time.localtime() / time.mktime() via extmod/modtime.c's timeutils path (MICROPY_EPOCH_IS_1970 selected to match CPython; the 1978-1970 offset of 2922 * 86400 = 252,460,800 seconds is applied in ports/amiga/modtime.c).

battclock.resource was considered and rejected: the AmigaOS system clock is synchronised from battclock at boot and tracks any user Date edits thereafter, so GetSysTime() is the authoritative runtime source. Reading battclock directly during runtime would race the boot synchronisation and skip the user's edits.

There is no timezone database on AmigaOS, so time.gmtime(t) and time.localtime(t) return the same broken-down tuple for the same t. Whatever the user has set their AmigaDOS clock to (local time or UTC) is reflected directly; the port doesn't try to add a timezone-offset layer.

vamos doesn't emulate GetSysTime (the call returns d0=0 with the destination buffer untouched), so the smoke test test_time_wallclock_smoke.py gates the "is this a real recent wall-clock?" assertions behind a runtime detection and only verifies static surface coverage (gmtime(0), gmtime(amiga_epoch), mktime round-trip, time.time() / time_ns() return-type) on that host. Full wall-clock validation lands on Amiberry / real hardware.

Variants: all three (no FPU sensitivity). Build sizes:

Variant	text
standard	586,660 bytes
68020fpu	557,244 bytes
68040	567,828 bytes

(Size delta from Phase 38: ~6-8 KB across variants, dominated by shared/timeutils/timeutils.c which now compiles into the port for the first time, plus the conditional gmtime/mktime/time/time_ns helpers in extmod's modtime.c.)

Step 2 — hashlib MD5 + SHA-1 ✅

Shipped:

• hashlib.md5 and hashlib.sha1 are now in the module alongside hashlib.sha256. All three follow the standard upstream API: one-shot hashlib.foo(b"..."), streaming update(...) / digest(), and a final "hash is final" ValueError after the first digest().

The upstream extmod/modhashlib.c only implements MD5 / SHA-1 via MICROPY_SSL_MBEDTLS or MICROPY_SSL_AXTLS. We don't ship mbedtls and AmiSSL is an OpenSSL-shaped runtime library (wrong API for both backends), so the port routes through the axtls code path:

• mpconfigport.h flips MICROPY_SSL_AXTLS=1, MICROPY_PY_HASHLIB_MD5=1, MICROPY_PY_HASHLIB_SHA1=1.
• MICROPY_PY_SSL stays 0, which keeps modtls_axtls.c and modcryptolib.c dormant — only the algorithm sources are linked, not the rest of axTLS (no tls1, no x509, no bigint...).
• ports/amiga/Makefile adds lib/axtls/crypto/md5.c and lib/axtls/crypto/sha1.c directly to SRC_C, along with -I flags for lib/axtls/ssl, lib/axtls/crypto, and extmod/axtls-include. A per-file CFLAGS override silences axtls's older -Wall-noisy style (mirrors extmod.mk's rule).
• A minimal ports/amiga/axtls-include/axtls_os_port.h shadows the upstream shim. The upstream copy pulls arpa/inet.h (for tls1's SOCKET_READ macro), which on AmigaOS drags in exec/types.h via netinet/in.h → sys/socket.h → devices/timer.h. That collides with lib/crypto-algorithms/sha256.h's BYTE / WORD typedefs (AmigaOS defines them as signed int8_t / int16_t; crypto- algorithms defines them as unsigned char / unsigned int). The port-local shim provides just #include <stdint.h> — enough for md5.c / sha1.c, and they don't use any of the upstream shim's socket / SHA-256-alias macros anyway.

tests/ports/amiga/test_hashlib_smoke.py pins each algorithm against canonical vectors:

• MD5: RFC 1321 ("", "abc", "message digest").
• SHA-1: FIPS 180-1 ("", "abc", the 56-byte "abcdbcdecdefdefg..." vector).
• SHA-256: FIPS 180-2 ("", "abc").

Also covers streaming (split update(...) chains must equal the one-shot digest) and the final-state contract (update / digest after digest() must raise ValueError).

Side note from the test-vector verification: an early version of the test had a transcribed-wrong SHA-256(b"abc") expected value, which made the Amiga output look 4-bit-shifted vs the (typo'd) expected. SHA-256 was always producing the correct FIPS digest — confirmed by running the same input through host CPython. No SHA-256 bug exists.

Variants: all three. Build sizes:

Variant	text	Δ vs Step 1
standard	514,324 bytes	~+3 KB
68020fpu	488,204 bytes	~+3 KB
68040	497,952 bytes	+2,764 B

(Δ is the cost of md5.c + sha1.c plus the additional dispatch in modhashlib.c. The 68040 baseline was re-measured against HEAD~1 for the delta column; the other two are within a few hundred bytes of the same figure.)

Step 3 — deflate compress (write) surface ✅

Shipped:

• deflate.DeflateIO(stream, format) now writes as well as reads for all three formats (deflate.RAW, deflate.ZLIB, deflate.GZIP), so scripts can both decompress and compress.
• The decompress half has been on since the port joined EXTRA_FEATURES; Step 3 just flips MICROPY_PY_DEFLATE_COMPRESS to 1 in mpconfigport.h. Single-line change.

No Makefile work was needed: extmod/moddeflate.c #includes lib/uzlib/lz77.c (which in turn #includes defl_static.c) under #if MICROPY_PY_DEFLATE_COMPRESS, so the compressor source is already pulled into the existing moddeflate.o translation unit.

Test coverage in tests/ports/amiga/test_deflate_compress_smoke.py:

• Compress→decompress round-trip for RAW, ZLIB, and GZIP against a repetitive payload, asserting both bytes-out < bytes-in and exact match after decode.
• Streaming write split across multiple write(...) calls (including a zero-length write) yields the same compressed bytes as a single concatenated write.
• write(...) on a closed DeflateIO raises OSError.
• Documents the uzlib quirk that compressing an empty input produces an empty byte stream rather than a header-only frame. (CPython's zlib emits a 2-byte zlib header + 4-byte Adler32 trailer; uzlib only emits anything if data has actually been written. The test pins the current behaviour rather than treating it as a bug — anyone needing CPython-compatible framing can prepend the header by hand.)

Variants: all three. Build sizes:

Variant	text	Δ vs Step 2
standard	515,976 bytes	+1,652 B
68020fpu	489,876 bytes	+1,672 B
68040	499,648 bytes	+1,696 B

(Δ is the incremental cost of lz77.c + defl_static.c once they compile into moddeflate.o.)

Step 4 — btree (Berkeley DB persistent K/V) ✅

Shipped:

• import btree opens a B-tree-on-stream key/value store via btree.open(stream, ...). Keys and values are arbitrary bytes (anything implementing the buffer protocol works for both); the store layers on top of whatever stream the caller provides, so it works equally well against an io.BytesIO, a VfsAmiga file handle (open(path, "w+b")), or any other MicroPython stream.
• Supports the standard dict-ish surface: db[key] / db[key] = val / del db[key] / key in db, db.get(k[, default]), db.put(k, v), plus db.keys() / db.values() / db.items() (with start, end, and btree.INCL / btree.DESC flags), and db.flush() / db.close().

Wiring in this port:

• ports/amiga/Makefile sets MICROPY_PY_BTREE = 1 as a make variable before including extmod.mk. That make-side flag is what triggers extmod.mk to pull in lib/berkeley-db-1.xx/btree/*.c + mpool/mpool.c and add the matching -I path. The corresponding C macro is set automatically by extmod.mk via CFLAGS_EXTMOD += -DMICROPY_PY_BTREE=1.
• ports/amiga/mpconfigport.h flips MICROPY_STREAMS_POSIX_API = 1 so the mp_stream_posix_{read,write,lseek,fsync} shims (declared in py/stream.h, defined in py/stream.c) are visible to extmod/modbtree.c. Those shims bridge berkeley-db's read(fd, buf, n) / write(fd, buf, n) / lseek(fd, ...) calls onto a Python stream object's MP_STREAM_OP_* methods.
• No port-local Berkeley DB config file is needed -- the upstream extmod/berkeley-db/berkeley_db_config_port.h builds cleanly against AmigaOS clib2, which provides sys/file.h, sys/param.h, sys/stat.h, fcntl.h, etc. with the expected signatures.

(Going through the stream protocol rather than direct file I/O is also what lets the test exercise the whole module against an in-memory io.BytesIO without touching the filesystem.)

Test coverage in tests/ports/amiga/test_btree_smoke.py:

• open() / put / __getitem__ / __contains__ / __delitem__, including the KeyError paths for missing keys.
• Buffer-protocol arguments (memoryview slices used for both key and value).
• keys(), items(), items(start, end), items(None, None, btree.DESC), plain for k in db -- all asserted against the sorted key list. (Note: btree shares a single iteration cursor across all calls on the same DB object, so you can't run keys() and values() concurrently via zip(). The test cross-checks values via db[k] lookup instead.)
• flush() + close(), plus the "use after close" guard which raises either OSError or ValueError("database closed") depending on the call site (currently ValueError, but the test accepts both for forward compatibility).

Variants: all three. Build sizes:

Variant	text	Δ vs Step 3
standard	535,292 bytes	+19,316 B
68020fpu	509,192 bytes	+19,316 B
68040	519,152 bytes	+19,504 B

(Δ is dominated by the 13 berkeley-db source files [bt_*.c + mpool.c] plus modbtree.c itself, against the MICROPY_STREAMS_POSIX_API shim weight.)

Steps remaining

• Step 5: marshal (optional).
• Step 6: websocket + webrepl (defer until scheduler-lite story resolved — likely promoted to its own phase).