Spike plan: split builtin Pointer into Pointer + UnsafePointer (#4925)

[SeanTAllen]

This is the plan for a spike to test the Pointer / UnsafePointer split proposed in #4925. It is a proof-of-feasibility, not a merge-ready change: the goal is to show the split works end-to-end (compiler + standard library + ponylang/ssl all compile and pass), then open a draft PR for review and refinement. Not merging short-term.

Goal

Introduce a new builtin type UnsafePointer[A] alongside Pointer[A]. The two have identical runtime/codegen behavior but different static semantics, enforced by the type system (which signatures accept which). This gives FFI authors a way to represent two genuinely different pointer provenances instead of overloading one Pointer for both.

The semantic model

• Pointer[A] (safe) — backed by Pony-managed memory. Source: .cpointer() / .cstring() on a Pony Array/String, Pointer[A]._alloc, addressof, or an FFI function that returns/fills pony_alloc'd memory. It is the only kind that may be used or adopted in place without copying (an Array/String backing store, or a network read buffer C fills and Pony then reads directly). Passed into FFI with the capability matching the real C semantics (val/box if C only reads, ref if C writes).

• UnsafePointer[A] — NOT backed by Pony's guarantees. Two sub-cases: (a) trust-asserted raw foreign memory; (b) an opaque foreign handle that came from FFI and is later passed back into FFI (FILE*, DIR*, addrinfo*, SSL*), often wrapped in a class with a finalizer. The ONLY way to land its data in a safe Pony object is to copy (e.g. String.copy_cstring, which allocates fresh memory and copies). Never adopted in place; should never be exposed in a public interface.

Distinguishing test for from-FFI pointers: does the code adopt the pointer as a backing store without a copy (→ Pointer, source must be pony_alloc'd) or copy the bytes into freshly Pony-allocated memory (→ source may be UnsafePointer)? Example: String.from_cstring does _ptr = str (adopt, no copy) → takes Pointer. String.copy_cpointer copies into a fresh allocation → takes UnsafePointer.

Subtyping: distinct, non-interchangeable. No implicit coercion either direction. This is what makes the eventual migration a breaking change.

Key decision: void* mapping

A C void* parameter (declared Pointer[None] in Pony today) means nothing to us, so we map it to whichever raw pointer the Pony signature actually declares — the two kinds do not interchange across a void* boundary. Concretely, void_star_param enforces a kind match: a Pointer[None] param accepts Pointer[A]/NullablePointer[A]/USize; an UnsafePointer[None] param accepts UnsafePointer[A]/USize. This keeps the distinction honest at the FFI boundary and makes FFI declarations self-documenting about provenance.

Compiler approach (small)

Because the split does not change codegen (identical caps, identical lowering), almost no compiler site needs to tell the two types apart:

• genprim_pointer_methods resolves each intrinsic by method name on the reach type it is handed, so UnsafePointer reuses the same generator verbatim.
• Codegen recognizes the type at exactly two spots (gentype.c: the use/mem-type assignment, and the intrinsic-method dispatch) — both gain an || str_UnsafePointer.
• The roughly eleven is_pointer() gates all mean "this is a raw pointer, treat it as opaque." They move to a combined is_raw_pointer() helper (= is_pointer || is_unsafe_pointer).
• The single exception is void_star_param, which keeps is_pointer / is_unsafe_pointer separate to enforce the kind match above.
• addressof always yields Pointer; no mirror needed there.

Phases

1. Compiler — add UnsafePointer as a recognized raw-pointer type (packages/builtin/unsafe_pointer.pony, the three predicates, the two codegen spots, the void* kind-match). Non-breaking: UnsafePointer exists but is unused. Committed test exercises its intrinsics.
2. Stdlib migration — reclassify pointer usages per the inventory (~255 sites across ~36 files; heaviest in builtin, files, net, process) and retype the UnsafePointer-bucket sites (opaque foreign handles + copy sources, plus the void* FFI-declaration params in lockstep). This is where the breaking change lands.
3. ponylang/ssl testbed — migrate ssl as the real-world validation that the model is expressible and ergonomic on third-party FFI-heavy code; report friction.

The UnsafePointer near-clone of Pointer has one deliberate exception: _copy_to's destination stays typed Pointer, because copying is precisely how unsafe data is brought into Pony-managed memory.

Divergences / breaking changes

• Breaking for downstream FFI users. Because the two types are non-interchangeable, external code that today passes a Pointer to a stdlib API we retype to UnsafePointer (notably String.copy_cpointer/copy_cstring), or that stores/pattern-matches the retyped FFI return values, will stop compiling. Intended (it is the safety win), but it is an ecosystem migration. The stdlib + ssl pass measures that cost.
• No deprecation shim in the spike.

Out of scope for the spike (follow-up once the design is settled)

• Lint rule forbidding UnsafePointer in public interfaces (it must unfold type aliases, mirroring is_literal, or type Foo is UnsafePointer[X] evades it).
• Trimming/reshaping either pointer's API.
• Final release notes (added before any merge).

Open questions to resolve during review

• Per-site classification of a handful of ambiguous FFI pointers (env.argv, the pony_ctx handle, the Windows find-data allocation, hash-block Pointer[None] params).
• Whether Pointer and UnsafePointer should later diverge in API surface.
• Hard split vs. a transitional deprecation period before any real merge.

[SeanTAllen]

Spike results so far

The compiler + standard-library half of the spike works. UnsafePointer[A] is in as a recognized raw-pointer type and the model holds across every pointer pattern I migrated.

Compiler (small, as predicted): a new packages/builtin/unsafe_pointer.pony; is_unsafe_pointer() + a combined is_raw_pointer() used at the ~11 sites that treat any raw pointer identically; void_star_param enforcing the kind match (the one site that distinguishes the two); str_UnsafePointer + two gentype spots reusing the existing genprim_pointer_methods verbatim.

Standard library migrated and passing (501 stdlib + 42 files + 13 net tests green, plus a new direct UnsafePointer intrinsics test and a copy_* test that copies real malloc'd foreign memory):

• String.copy_cstring / copy_cpointer → take UnsafePointer (copy is how foreign data enters a Pony String). Exercises _copy_to/_apply on real data.
• Env._create argv/envp → UnsafePointer[UnsafePointer[U8]] (foreign C-runtime memory, copied into Pony strings). ABI-safe — the runtime glue resolves Env._create by name and passes ptr args.
• AsioEventID type alias → UnsafePointer[AsioEvent] tag. Propagated to all 9 users with zero friction — the cleanest result.
• std_stream FILE* handle + its void* params → UnsafePointer, while the buffer param stays Pointer (Pony String/Array data). One FFI call carries both kinds — exactly the intended split.
• net/dns addrinfo* and files/directory DIR*/find-data handles → UnsafePointer, while ponyint_unix_readdir's return stays Pointer[U8] iso^ and is adopted by from_cstring (the adopt path) — adopt-vs-copy working in one loop.

The model is closed: provenance is permanent, copy is the only bridge

There is no conversion between the two pointer types in either direction, and that is the point:

• Foreign memory (C-allocated) → UnsafePointer, permanently. The only way it becomes safe is copy_cpointer/copy_cstring, which reads the bytes into a fresh Pointer-backed String/Array. You never adopt foreign memory; copy isn't a conversion, it builds a new Pony object.
• Pony-allocated memory → Pointer, permanently — even when you hand it to C and C writes through it. You give C memory from a Pointer and get it back; Pony owns it the whole time. It never touches UnsafePointer.

Because there is no UnsafePointer → Pointer operation, you can never claim foreign memory is Pony-managed. If you want Pony-managed memory, you allocate it as a Pointer yourself.

Migrating libponyc-wrapping tools (and ssl)

copy_* taking UnsafePointer makes the libponyc-wrapping tools (pony-compiler/lint/lsp/doc) stop compiling, because they pull raw char*s out of libponyc. That's a mechanical migration, not a design problem — each site classifies the same way:

• A pointer that is a foreign libponyc-owned C string (ast_name, ast_get_print, package paths, …) → UnsafePointer, and the existing copy_* call keeps working.
• A Pony buffer that C fills (e.g. ponyint_pool_alloc_size scratch that gets a path written into it) stays a Pointer and is built with from_cpointer (adopt) — it was Pony memory the whole time and never should have gone through copy_*.

ssl carries the same two cases: OpenSSL handles (_SSL/_BIO/X509/…) are foreign → UnsafePointer (clean, like asio/dns), and the read/write buffers are Pony data → Pointer.

Status / next

• Branch carries the compiler change + the stdlib migration above; tools are still pristine (they go red purely from the copy_* signature change — the blast radius). Migrating them is the mechanical pass described above. ssl follows.

[SeanTAllen]

Status snapshot

Checkpoints pushed for both repos so nothing is lost while we decide how to proceed.

• ponyc — draft PR Spike: split builtin Pointer into Pointer and UnsafePointer #5431 — branch unsafe-pointer, HEAD 51afd715b (spike 7b7759d60 + the UnsafePointer API trim)
• ssl — draft PR Migrate to the Pointer / UnsafePointer split ssl#65 — branch unsafe-pointer, commit 5a2c3f2, 8 files (+98 / −97)

Everything compiles and all tests pass

Suite	Result
libponyrt (gtest)	39 passed
libponyc (gtest)	1640 passed
full-program (debug + release)	199 + 199 passed
stdlib-release	501 passed
examples	all build + link clean
pony-compiler / lint / lsp / doc	7 / 392 / 316 / 41 passed
ssl (against the split ponyc)	54 passed

Zero failures.

What's migrated

• Compiler: UnsafePointer recognized as a raw-pointer type; is_unsafe_pointer / combined is_raw_pointer predicates; void_star_param enforces the kind match; codegen reuses the existing genprim_pointer_methods.
• Stdlib: String.copy_*, Env.argv/envp, AsioEventID, std_stream (FILE*), dns (addrinfo), directory (DIR*/find-data). Plus a direct UnsafePointer intrinsics test and a copy_*-from-foreign-memory test.
• Tools (libponyc wrappers): foreign libponyc C-strings → UnsafePointer; the three copy-pasted _find_exe_directory helpers rewritten to the idiomatic Pony-buffer pattern (Array[U8] .> undefined + .cpointer() + truncate + from_array) instead of routing a Pony→Pony move through the FFI copy bridge.
• ssl (downstream testbed): opaque OpenSSL handles (_SSL, _BIO, _SSLContext, X509, _X509Name, _GeneralName, _EVPMD, _EVPCTX, _OpenSslInitSettings) and the void*-disguised handles (BIO_s_mem, TLS_method) → UnsafePointer; foreign data copied into Pony (ASN1 cert data, ALPN protocol names) → UnsafePointer; read/write/digest buffers stay Pointer. pony_os_ip_string takes a foreign src but returns pony_alloc'd memory adopted via from_cstring, so its return stays Pointer — the mixed case in one call.

API surface review (done)

Pointer and UnsafePointer are not duals. Determined empirically (trim, then rebuild stdlib + tools + ssl and add back only what the compiler demands):

• UnsafePointer trimmed to 6 methods — it is only ever held as an opaque handle or read so its bytes can be copied into Pony memory:
  • create (null handle), is_null (the dominant op), usize (non-null sentinel checks, e.g. Windows INVALID_HANDLE_VALUE), eq (matching a held handle against another — e.g. process_monitor dispatching an ASIO event to its owner), and the private _apply / _copy_to used by String.copy_*.
  • Dropped (17): _alloc, _realloc, _unsafe, _convert, _update, _offset, offset, _element_size, _insert, _delete, lt, ne, le, ge, gt, hash, hash64. All operate on Pony-managed memory (allocate / arithmetic / mutate / array-manipulate), which is Pointer's job.
• Pointer kept full. Only lt/le/ge/gt/hash/hash64 have no in-repo use, but it is the established public general-purpose raw pointer; trimming its comparison/hash operators would be a gratuitous breaking change for ~no benefit. Everything else is exercised by Array/String/FFI.

Outstanding / polish

1. Remaining stdlib handle sites (process, time) — a few opaque-handle FFIs are still typed Pointer; they compile and pass as-is, so this is a completeness sweep, not a blocker.
2. Opaque libponyc handle types in the tools (_AST, _Token, _Symtab, _Package, …) — still Pointer[_X]. Semantically these are foreign handles that should arguably be UnsafePointer (same as AsioEventID). Large mechanical ripple; deferred.
3. Windows-only certificate-store paths in ssl (X509_STORE, _CertContext, d2i_X509) — left on Pointer; not compiled or tested on Linux, so migrating them blind would be untested; deferred.
4. @pony_ctx — left as Pointer[None]; leans UnsafePointer (opaque runtime handle), low stakes.
5. Lint rule — forbid UnsafePointer in public interfaces; must unfold type aliases (mirror is_literal's TK_TYPEALIASREF handling) or type Foo is UnsafePointer[X] evades it. Deferred post-spike.
6. Final release notes (breaking change, both repos) — held until the design is settled.
7. Squash the spike checkpoint commits before any real PR (both repos).

Settled (for the record)

Provenance is permanent; the only bridge is copy_* (foreign → fresh Pony object). Pony memory stays Pointer (use String/Array buffers for "C fills a buffer"). No Pointer ↔ UnsafePointer conversion in either direction, no trust primitive, no copy_from(Pointer) — all considered and dropped. Raw memcpy-family FFI is a foreign-memory tool and naturally takes UnsafePointer; Pony memory is moved with Array/String methods, so there is no genuine "both provenances" need. The model held across the entire migration — stdlib, tools, and ssl — with no amendments.

Future work (separate from this spike)

A runtime FFI sanitizer mode, opt-in the way a sanitizer build is. It checks any FFI call that passes a Pointer and confirms at runtime that the referenced memory was really allocated by Pony. If it wasn't — foreign memory wearing a Pointer the boundary never got to check — it panics. The static split says Pointer means Pony-managed memory, but the FFI boundary is exactly where that promise stops being enforceable; this puts a check back on it. It could extend to other FFI sanitization down the line. Worth its own discussion if the split lands.