Fix SWI-Prolog 10 stack exhaustion and improve memory management

[hoijnet]

This branch addresses a class of stack overflow crashes that occur under SWI-Prolog 10 when large transaction_object dicts are carried through recursive Prolog call chains. It also enables correct LRU cache eviction in terminus-store and replaces the system allocator with jemalloc to reduce memory fragmentation in long-running server processes.

Also ensures the integration tests are optimized between each test for well-controlled optimization, instead of auto-optimizer that operates between transactions with 10% probability.

With these changes, there is an approx 15% performance improvement when running a WOQL-heavy set of integration tests and has sustained performance for long runs thanks to auto-optimizer, stack and memory usage improvements, tabling optimization, jemalloc and many other improvements made across the codebase to support swipl 10 and stability fixes.

Background

SWI-Prolog 10 introduced stricter stack segment management. The trie_gen_compiled/2 primitive, used internally by tabled predicates, asserts gTop+1 <= gMax && tTop+2 <= tMax at entry. When recursive predicates carried the full transaction_object — a deeply nested dict containing schema, instance, and inference graphs plus all metadata — each stack frame consumed significantly more space than necessary. This leaves insufficient headroom for trie operations, triggering hard crashes on databases with moderately complex schemas.

The core fix is straightforward: extract the lightweight Schema (a simple list of read-write objects) once at each entry point, then thread it through the recursive chain instead of the full transaction object. Wrapper predicates that extract schema internally — like is_subdocument/2, class_predicate_type/4, oneof_descriptor/3 — are replaced with their direct schema_* equivalents where the schema is already available.

Changes

Prolog stack pressure reduction

inference.pl — The inference chain (infer_type, infer_range, infer_object_type, and related predicates) previously threaded Database through every recursive frame. Refactored to extract Schema at entry points and pass it through the entire chain. Prefix merging moved to entry points to avoid repeated computation.

json.pl — Two recursive chains fixed:

• json_assign_ids extracts Schema once, delegates to json_assign_ids_ which carries the lightweight schema through ID generation for nested subdocuments. New json_idgen_schema and get_field_values_ variants take Schema directly.
• get_document extracts Schema and Instance once, delegates to get_document_ to avoid carrying the full transaction object through document retrieval.

migration.pl — strip_nonconforming_ids extracts Schema once at the entry point. The recursive strip_nonconforming_value_ uses schema_is_subdocument and schema_key_descriptor directly, eliminating the large transaction object from convlist lambda closures that previously captured it on every frame.

instance.pl — The refute_instance validation chain threads Schema from the entry points (refute_instance/2, refute_instance_schema/2) through refute_subject, refute_subject_1, refute_typed_subject, refute_cardinality, refute_cardinality_new, refute_object_type, and refute_object_type_. Each predicate now uses direct schema_* calls (schema_class_predicate_type, schema_oneof_descriptor, schema_is_abstract, is_schema_foreign, etc.) instead of wrappers that re-extract the schema. Old-arity predicates are kept as thin wrappers where external callers depend on them.

schema.pl — Exports is_schema_foreign/2, schema_class_predicate_type/4, schema_class_subsumed/3, and is_schema_simple_class/2 to support direct schema-based lookups from other modules.

Memory allocator

terminusdb-dylib — Replaces the system malloc with jemalloc via tikv-jemallocator. The default glibc allocator on Linux tends to fragment memory in long-running processes with many small allocations (common in layer cache operations). Jemalloc uses thread-local caches and size-class-based arenas that significantly reduce fragmentation. Configured with background_threads for asynchronous purging and disable_initial_exec_tls for compatibility as a dynamically loaded library. Only enabled on non-MSVC targets.

Test infrastructure

test_utils.pl — Improved spawn_server_1 to collect stderr lines during server startup and retry on a different port when the spawned server fails to start. Previously, server_has_no_output threw past the between/3 retry loop, causing flaky push/pull tests when ports were temporarily unavailable.