Scolta 1.0.0-rc1

First stable release — all features from 0.3.x promoted to 1.0 API surface.

Fixed

• scolta.js initPagefind() now uses a module-level pagefindInstance guard to prevent calling pagefind.init() more than once per page. Pagefind uses a SharedWorker that persists across navigations; calling init() a second time (e.g. on Drupal behavior re-attachment after a language switch) corrupts the WASM pointer permanently, causing "No pointer" errors and zero search results for the remainder of the tab session. The guard is stored in the outer IIFE scope so it spans all createInstance() calls on the same page.
• Summarize endpoint no longer returns HTTP 400 on large result sets. The context parameter validation limit has been raised from 50,000 to 100,000 characters. The client truncates to 49,000 characters before sending, so this server-side limit acts as a safety net only.
• IndexBuildOrchestrator now reports failure when the atomic swap or post-build sanity check fails. Two silent failure modes are closed: (1) atomicSwap() now throws RuntimeException if any rename() call returns false, so a failed filesystem rename can no longer cause a false-success result; (2) after the swap, verifyOutputHasFragments() checks that the pagefind output directory contains at least one .pf_fragment file when pages were processed — zero fragments after a non-empty build is treated as a hard failure. Both errors surface through the existing try/catch and are returned as success: false with a descriptive error string. (scolta-php#46)
• ContentExporter::filterItems() no longer crashes on CachedContentReference objects during re-index. When a prior build's timestamp manifest exists, gather() yields a mix of ContentItem and CachedContentReference objects (cache-hit markers for unchanged posts). filterItems() previously accessed $item->bodyHtml on every item, which is not a property of CachedContentReference. Added instanceof CachedContentReference type guard to pass cached items through without inspection. Fixes PHP fatal error on WordPress and any other platform on any site with a prior build.
• AI summary max_tokens raised from 512 to 1024. The previous limit caused the AI to truncate mid-markdown on multi-result summaries — the last result was particularly vulnerable. The new default halves truncation frequency. The value is now configurable via ScoltaConfig::$aiSummaryMaxTokens.
• cleanBrokenMarkdown() added to scolta.js before formatSummary() renders markdown to HTML. Mirrors the existing PHP MarkdownRenderer::cleanBrokenLinks() salvage logic: unclosed markdown links become bold text, unclosed bold/italic/backtick delimiters are closed. Prevents truncated AI output from producing broken HTML in the browser.
• AiClient now auto-appends /v1/chat/completions to OpenAI base_url values that have no path. LiteLLM proxies (including Amazee.ai) return a base URL without a path component. Passing that URL directly caused 405 Method Not Allowed errors. When the base_url has no path (or only /), the standard OpenAI chat completions path is appended automatically. URLs that already contain a path are used as-is.
• AmazeeClient::provisionTrial() and signIn() now parse the nested API response format, and token validation is removed from the provisioning path. The Amazee.ai /auth/generate-trial-access endpoint now returns credentials nested under a key object, and signIn wraps the access token under a token object. Both methods now check for the nested format first and fall back to the legacy flat format for backwards compatibility. The post-provisioning validateToken() call is removed from provisionTrial() because the /auth/me endpoint no longer exists on Amazee.ai's LiteLLM proxy — the token issued by the provisioning API is trusted directly.
• scolta.js result URLs resolved against pagefind base path instead of site root. pagefindBase was stored as an absolute URL (with origin), but pagefind.js returns root-relative URLs after applying its baseUrl (a path, not an origin). The startsWith check never matched, so the pagefind prefix was never stripped — result links pointed to /wp-content/uploads/scolta/pagefind/product/… instead of /product/…. pagefindBase is now stored as a path-only value by stripping the origin via new URL().pathname when the pagefind path is absolute.
• Adapter install — Drupal CI job failed because the release-validation workflow replaced the entire repositories array (wiping the packages.drupal.org/8 Drupal Packagist entry) instead of only swapping the local path repo for the GitHub VCS repo. The PHP inline script now filters out only the path-type scolta-php entry and appends the VCS repo, leaving other repository sources (including the Drupal Packagist) intact.
• Search status message text no longer overflows on narrow viewports. .scolta-results-header now carries overflow-wrap: break-word and word-wrap: break-word, and its first-child <span> gains min-width: 0 so the flex item can shrink below its content size. Long messages (e.g. "— no exact matches found, showing partial matches") wrap correctly at 320 px, 768 px, 1024 px, and 1440 px viewport widths. (#51)
• Memory profile documentation claimed "peak RSS ≤ 96 MB" for the conservative profile, but 96 MB is Scolta's internal allocation budget — total process RSS also includes the PHP runtime baseline (typically ~60 MB for Laravel CLI, ~80 MB for WordPress, ~130 MB for Drupal) and I/O overhead. PHPDoc for MemoryBudget::conservative(), MemoryBudgetSuggestion::suggest() reason strings, and checkProfileFit() warning messages now use "internal allocation budget" rather than "peak RSS". The README Memory and Scale section is updated with a platform baseline table and corrected estimates for total expected RSS per profile. The balanced and aggressive README comments ("~200 MB peak RSS" and "~384 MB peak RSS") are corrected to match the actual internal budget values (384 MB and 1 GB). (scolta-php#47)
• HealthChecker no longer reports ai_configured: true when the API key is whitespace-only. The previous !empty() check treated strings containing only spaces or tabs as configured. Changed to trim() !== '' so only non-empty, non-whitespace keys (including Amazee.ai tokens stored as the API key) report as configured.
• AI endpoints return HTTP 200 with empty data instead of 503 when no API key is configured. A new ApiKeyMissingException is thrown by AiClient when no API key is set. AiEndpointHandler catches it specifically — before the generic exception handler — and returns a graceful empty response: handleSummarize returns {} (no summary shown), handleExpandQuery returns the original query (no expansion), and handleFollowUp returns an empty response. Sites without AI configured no longer produce 503 console errors on every search. The scolta.js fetch catch blocks also now suppress TypeError (network unreachable / offline) silently rather than logging a warning or showing an error state, so search works normally when the AI endpoint is unreachable. (#50)
• Tests added for CATEGORY and VARIETY rules in the summarize prompt template, and for {SITE_NAME} placeholder presence in all three templates. Both CMS adapters delegate to DefaultPrompts::getTemplate() at runtime; these tests lock in the template contracts so prompt drift is caught immediately if the canonical text is removed or emptied. (#49)
• scolta.js result links no longer double the path on subdirectory installs. Result display URLs now prefer data.meta?.url (the verbatim URL stored in data-pagefind-meta by the binary indexer) over Pagefind's resolved data.url. Pagefind's JS client resolves stored root-relative paths against the pagefind base directory when building data.url, which on a subdirectory install (/drupal/web/) produces paths like /drupal/web/sites/default/files/scolta-pagefind/drupal/web/node/42. Using data.meta?.url avoids this resolution entirely; resolveUrl(data.url) is kept as the fallback for the PHP indexer path where data.meta.url is undefined. (scolta-drupal#40)
• IndexBuildOrchestrator::build() now returns error: 'memory_abort' when MemoryTelemetry fires. The catch block detects RuntimeException messages containing "exceeds safe threshold" and returns a structured StatusReport with error: 'memory_abort', the number of chunks already committed, and the committed page count from the build manifest. Framework adapters can now programmatically distinguish a memory abort from other failures and spawn a fresh --resume process automatically.
• Memory telemetry now measures actual RSS instead of PHP's allocator-reported memory. MemoryTelemetry reads VmRSS and VmHWM from /proc/self/status on Linux, falling back to memory_get_usage(true) / memory_get_peak_usage(true) when /proc is unavailable (macOS, Windows). Also reads cgroup v2/v1 memory limits (/sys/fs/cgroup/memory.max, /sys/fs/cgroup/memory/memory.limit_in_bytes) to determine the effective ceiling — on containerised/shared hosting the cgroup limit is often lower than memory_limit, and either one can SIGKILL the process. The 90% abort threshold and the heap-full guard in IndexBuildOrchestrator now use actual RSS against the effective limit. StatusReport.peakMemoryBytes now reports the RSS high-water mark (VmHWM) rather than PHP's monotonic memory_get_peak_usage(true).
• IndexBuildOrchestrator tail chunk now calls gc_collect_cycles(). The regular chunk loop already called gc_collect_cycles() after each commit; the tail chunk (items that don't fill a complete chunk) was missing it. Circular reference chains from the last batch of ContentItem objects now get collected before the merge phase begins.
• WordPress gatherer now flushes all relevant cache groups and calls gc_collect_cycles(). Previously only flushed the posts cache group; post_meta, terms, and term_relationships accumulated across the entire build. Also added gc_collect_cycles() between batches (matching the Drupal gatherer's existing cleanup pattern) and a wp_cache_flush() fallback for WordPress < 6.1.
• IndexBuildOrchestrator::build() now detects a full-heap condition after indexing and defers the merge to a fresh process. After all chunks are committed, memory_get_usage(true) (total allocated PHP segments, not live objects) is compared against the PHP memory_limit. If segments reach ≥75% of the limit, the heap is too fragmented to run the multi-pass pre-merge safely — even a 12 KB allocation can fail. The build returns early with error: 'index_only_complete' and the chunks remain intact on disk. Framework adapters (e.g. drush scolta:finalize) can then run the merge in a fresh process with a clean heap.
• InvertedIndexBuilder now caps stored positions at 200 per weight bucket per page. Common stop words ("the", "and", "of") appear hundreds of times per Wikipedia article. Without a cap, serializing merged position arrays across thousands of pages during the multi-pass pre-merge required 25+ MB of contiguous heap per common term, triggering OOM in constrained environments (512 MB PHP memory limit). 200 positions per weight bucket is sufficient for phrase-proximity scoring; additional occurrences are counted in wordCount but their positional data is discarded. Title positions are not capped (titles are short).
• IndexBuildOrchestrator::build() now calls gc_collect_cycles() after each chunk commit. Framework adapters (Drupal, WordPress, Laravel) use Drupal entities and similar OOP graphs that contain circular references. PHP's reference-counting GC does not reclaim circular reference groups automatically — they accumulate until the cycle collector runs. Explicitly running it after each chunk ensures circular references from the just-committed chunk are reclaimed before the next chunk begins, keeping live memory bounded on long-running builds.
• Chunk accumulator no longer holds bodyHtml across the full chunk. IndexBuildOrchestrator::build() and PhpIndexer::processChunk() previously stored the full ContentItem (including bodyHtml, which can be several MB for large articles) in the chunk accumulator for the entire chunk duration. With chunkSize=10, this kept up to 10 full HTML bodies in memory simultaneously even though bodyHtml is no longer needed after tokenizeItem(). The chunk now stores a slim proxy containing only the six scalar metadata fields that buildFromTokenData() actually uses (id, url, date, siteName, language, filters), so bodyHtml is freed as soon as the generator advances to the next page.
• PageWordCache write buffer now flushes on byte budget, not just entry count. When pages contain thousands of tokens (e.g. long encyclopedia articles), a single serialize() call on the write buffer could try to allocate 30+ MB and exhaust the PHP memory limit. The write buffer now tracks an estimated byte footprint per entry (token count × 80 bytes + content length) and flushes when either the entry count or the byte budget is reached. The byte budget comes from MemoryBudget::tokenCacheChunkBytes(): 4 MB for conservative, 16 MB for balanced, 64 MB for aggressive. The PageWordCache constructor default is also 4 MB, so code that constructs the cache without a budget is protected automatically.
• Stemmer memoization cache is now capped at 100,000 entries. The cache grew without bound across a build, accumulating every unique word form encountered in the corpus. For large multi-subject corpora (e.g. Wikipedia with 500k+ unique word forms), this added tens of megabytes of permanent heap growth per build. The cap allows the most common words — always encountered first — to stay memoized, while rare long-tail vocabulary is re-stemmed on each occurrence with negligible performance impact.
• Memory-budget abort threshold now uses current live RSS, not monotonic peak. MemoryTelemetry previously computed the threshold percentage from memory_get_peak_usage(), which only ever increases. On large-corpus builds with many small chunks (e.g. Wikipedia with --chunk-size=10), the accumulated peak from earlier chunks crossed 90% of the PHP memory limit long before actual live memory was under pressure, causing a false-positive abort. The threshold now uses memory_get_usage() (current live memory) instead, so the abort fires only when the process is genuinely close to its limit.
• Fixed OOM on large corpora caused by pageWordCache holding all token data in RAM. The page-word cache previously deserialized the entire cache file (~2+ GB for large sites) into a single PHP array at construction. Fix: replaced the monolithic in-memory cache with a chunked disk-backed architecture. A lightweight manifest (hash → chunk number, ~5 MB for 44k pages) stays in memory; chunk files are loaded on demand one at a time. Peak cache memory drops from O(corpus size) to ~15 MB regardless of site size. Includes automatic migration from the legacy single-file format. Extracts duplicated cache logic from IndexBuildOrchestrator and PhpIndexer into a shared PageWordCache class.
• Search result links now point to the correct page URL. Pagefind's fullUrl() prepends its own base path (e.g. /sites/default/files/scolta-pagefind/) to every stored root-relative URL before returning data.url. The new resolveUrl() helper strips that prefix back off, so result links navigate to /faculty/x instead of /sites/default/files/scolta-pagefind/faculty/x. pagefindBase is computed once in initPagefind() as the directory parent of the pagefind/ folder.
• Language facets now appear when using the Pagefind binary indexer. initPagefind() only loaded the page's primary language index; each per-language Pagefind index only contains its own language's pages, so search.filters.language had a single value and renderFilters suppressed it. initPagefind() now reads pagefind-entry.json after init and calls pagefind.mergeIndex() for every other language. The index path is passed as an absolute URL (new URL(basePath, window.location.href).href) to bypass Pagefind's same-index dedup guard, which compares string prefixes and would silently skip relative-path calls to the same index.
• Language and content-type facets now appear in the search sidebar. computeFilterCounts() was reading r.data.filters on individual Pagefind result objects, which do not carry a filters property. Pagefind only exposes aggregate filter counts at the search-response level (search.filters). doSearch() now assigns filterCounts = primarySearch.filters || {} directly from the Pagefind search response, so facet checkboxes render correctly for all indexed filter dimensions (language, site, content_type, etc.).
• InvertedIndexBuilder now emits language as a Pagefind filter dimension. The PHP indexer builds filter metadata from $item->siteName and $item->filters, but never included $item->language. The binary path worked because PagefindExporter emits data-pagefind-filter="language:..." HTML attributes which Pagefind reads at crawl time. The PHP-only path skips HTML entirely, so language was silently dropped from every index entry. $item->language is now merged into the filters array alongside site, making language faceting functional for the PHP indexer path.

Added

• ScoltaConfig::$aiSummaryMaxTokens (default 1024) controls the AI token budget for summary responses. Pass ai_summary_max_tokens in config arrays (Drupal, Laravel, WordPress) to override. AiEndpointHandler reads the value from ScoltaConfig via AiControllerTrait::createHandler().
• scolta.js.sha256 checksum file for cross-repo sync verification. scolta-drupal and scolta-wp CI verify their committed copies match this checksum so drift fails CI automatically.
• AutoProvisioner::ensureAiAvailable() provisions a free Amazee.ai trial on first use. CMS install hooks and lazy-init paths call this static method with a ConfigStorageInterface and an optional $hasExplicitApiKey flag. It is idempotent: it skips provisioning when an explicit API key is already configured, or when credentials are already stored. On success, credentials are persisted via the storage adapter and an optional $onModelsResolved callback is called so the CMS can save the resolved model names in its own config system. Provisioning failures are caught internally and returned as false so the install path degrades gracefully. AmazeeClient::provisionTrial() and AmazeeTrialProvisioner::provision() now accept an optional $email parameter (defaulting to '') for anonymous provisioning.
• scolta.js auto-filters search results by the current page language. A new optional currentLanguage config key in window.scolta accepts a 2-letter ISO language code. When set, every fresh search is pre-filtered to that language; users can uncheck the language facet in the sidebar to search all languages. Falls back to the <html lang> attribute when currentLanguage is not provided. URL filter params (f_language=...) take precedence over both — a shared or bookmarked URL always wins. CMS adapters pass currentLanguage at render time so results are automatically scoped to the page's content language. The auto-filter is guarded by ai_languages: it only activates when the detected language is in the configured list and the list has more than one entry, so monolingual sites are unaffected and a detected language without indexed content never produces empty results.
• IndexerResolver class for explicit indexer selection logging. Adapters can construct an IndexerResolver with a PagefindBinary and PSR-3 LoggerInterface and call resolve('php'|'binary'|'auto') to get the effective backend ('php' or 'binary') with a notice-level log message emitted at selection time. When 'binary' is configured but no binary is found, the resolver falls back to 'php' and logs "[scolta] Falling back to PHP indexer: binary not available.". When 'auto' is used, it probes the binary and logs which backend was auto-detected. IndexBuildOrchestrator::build() now also emits "[scolta] Using PHP indexer." at the start of every PHP-path build. (#48)
• Amazee.ai auto-configuration: AmazeeModelResolver picks the best available Claude model for each inference role. After a successful trial provision, the provisioner calls resolve() against the provisioned LiteLLM /model/info endpoint to select the highest-versioned Sonnet (ai_model) and Haiku (ai_expansion_model) available in the account. Models are returned in ProvisioningResult via new nullable $aiModel and $aiExpansionModel fields. The resolver degrades gracefully (returns null for both) if the endpoint is unreachable or returns no matching models. AmazeeTrialProvisioner accepts an optional ?AmazeeModelResolver $modelResolver parameter.
• AmazeeTrialProvisioner accepts an optional $hasExistingProvider callable. When provided and returns true, provision() returns ProvisioningResult::skippedExistingProvider() without making any API calls. ProvisioningResult gains three STATUS_* constants (STATUS_PROVISIONED, STATUS_SKIPPED_EXISTING_PROVIDER, STATUS_FAILED) and a $status string property (defaulting to STATUS_PROVISIONED for backward compatibility).
• Timestamp-based rebuild optimization: skip unchanged entities entirely. Two new classes enable gatherers to bypass loadMultiple() / WP_Query for entities whose changed timestamp has not changed since the last indexed build:
  • TimestampManifest — disk-backed manifest (timestamp-manifest.php in the state directory) mapping entity key → {ts, items[{hash, id, url, date, siteName, language, filters}]}. put() stores a new/updated entry, markSeen() prevents it from being pruned, and pruneAndSave() atomically removes entries for deleted entities and persists the manifest.
  • CachedContentReference — marker object yielded by gatherers for unchanged entities. Carries the pre-computed content hash and all metadata needed to build a chunk entry without re-loading the entity body.
• IndexBuildOrchestrator now exposes getTimestampManifest() and handles CachedContentReference in the build loop. The build loop checks instanceof CachedContentReference: on a token-cache hit the reference is added to the chunk and markSeen() is called; on a cache miss the entry is silently dropped, causing pruneAndSave() to remove it so the entity is treated as changed on the next build. pruneAndSave() is now called on both the PageWordCache and TimestampManifest in all three build-completion paths (deferred merge, successful merge, and the implicit prune that follows). On a 44k-page site with 10 changed pages this reduces rebuild time from minutes to seconds.
• Amazee.ai integration: core PHP library (Tag1\Scolta\AiProvider\Amazee\). Seven new classes supporting the full trial provisioning and account upgrade flows for Amazee.ai's LiteLLM-compatible API:
  • AmazeeClient — HTTP client for the Amazee.ai control-plane (provisioning, sign-in, region listing, private key creation, token validation). Uses the existing Guzzle dependency; no new composer requirements.
  • AmazeeApiException — thrown on API errors; carries the HTTP status code.
  • AmazeeBudgetExceededException — thrown by BudgetAwareProviderDecorator when Amazee.ai rejects a request with "Budget has been exceeded!".
  • ConfigStorageInterface — abstract storage backend implemented by each CMS adapter (Drupal, WordPress, Laravel) to persist LiteLLM credentials.
  • ProvisioningResult / UpgradeResult — immutable value objects returned by the provisioning and upgrade flows.
  • AmazeeTrialProvisioner — orchestrates trial provisioning: calls the API then stores credentials.
  • AmazeeAccountUpgrader — orchestrates the three-step upgrade flow: request OTP → sign in → provision private key.
  • BudgetAwareProviderDecorator — wraps AiClient and converts Amazee budget-exceeded errors into AmazeeBudgetExceededException.
• Multi-dimensional filter faceting in the search sidebar. Language, site, content type, and any other Pagefind filter dimensions are now rendered as separate grouped checkboxes instead of a single hardcoded "Site" filter. Language codes are mapped to full display names (English, Spanish, etc.) via a LANGUAGE_NAMES map. Dimensions with only one unique value are hidden; visible dimensions are ordered language first, site second, then others alphabetically. Filter state is preserved in the URL via f_<dim>=<value> query parameters and restored on page load and browser back/forward navigation.
• Page-word cache for smart rebuilds. PhpIndexer::processChunk() and IndexBuildOrchestrator::build() now consult a per-item token cache keyed by contentHash(item) (URL + body SHA-256). Items whose content has not changed since the last build skip HTML cleaning and tokenization entirely — only new or modified pages are re-tokenized. The cache is serialized to {stateDir}/page-word-cache.php and automatically pruned to the set of items seen in each build, so deleted pages never accumulate stale entries. Pass $force = true (or --force from a CLI command) to bypass cache lookups while still repopulating the cache with fresh token data.
• InvertedIndexBuilder::tokenizeItem() — new public method that extracts a single item's token data (title, body, URL tokens; word count; cleaned title; content string) without building the index. Returns null for items whose body is too short to index. The return value is safe to serialize to a persistent cache.
• InvertedIndexBuilder::buildFromTokenData() — new public method that assembles a partial inverted index from pre-tokenized item data. Item metadata fields that are not part of the content hash (date, siteName, filters) are always read from the current ContentItem, not from cached data.

Changed

• Added extra.branch-alias (dev-main → 1.0.x-dev) so consumers can resolve this package with ^1.0@dev from a VCS repository.
• indexer: auto now always uses the PHP indexer. Previously auto tried the Pagefind binary first and fell back to PHP. The PHP indexer works on all PHP hosting environments without exec() or Node.js, uses less memory, and supports fast incremental re-indexing. Use indexer: binary to keep the old binary-first behaviour.