Release v0.29.0

[lchoquel]

Release v0.29.0

Bumps version from 0.28.0 to 0.29.0.

Changelog

[v0.29.0] - 2026-05-20

Added

• gemini-3.5-flash model added to the google backend. Adaptive thinking mode, supports text / images / pdf in and structured outputs, max_prompt_images = 3000, costs { input = 1.5, output = 9.0 } per million tokens.

• New PipeStructure operator that turns text into a structured concept via a single LLM call. Takes one Text-compatible input (or a domain concept that refines = "Text"), produces any structured output with the usual multiplicity options (Foo, Foo[], Foo[N]). Useful whenever the source text comes from a PDF extraction, a search result, an upstream pipe, or any non-LLM origin. Documented at building-methods/pipes/pipe-operators/PipeStructure.md.

• Custom search attributes on the Temporal namespace, with a strict prerequisite at worker boot. Pipelex sets five custom Keyword search attributes on every workflow start — PipeCode, PipelineRunId, SessionId, UserId, DomainCode — so the Temporal dashboard can filter, group, and audit workflows by their actual semantic identity rather than by opaque workflow ids. A real Temporal cluster rejects every StartWorkflowExecution RPC that references an unregistered attribute, so the check at worker boot is a hard fail (raises SearchAttributeRegistrationError) on a reachable namespace — the previous "warn and continue" framing was dishonest because workflows would then fail at every dispatch with a much less actionable error. The exception message embeds both the pipelex setup-temporal-namespace invocation and the equivalent raw temporal operator search-attribute create command. RPCError (control-plane unreachable) stays a soft fail; the in-process test server is registered up front by the integration conftest. Documented at distributed-execution/cluster-setup.md.

• [temporal.search_attributes] config block — master enabled toggle and an attributes subset selector for the five built-ins. enabled = false skips both the worker-boot registration check and the per-workflow attribute attachment (use this when the namespace policy is owned by a third party who will not register attributes). The validator rejects unknown attribute names so typos like "PipelineRunID" surface at config load instead of silently producing no attribute. The Pydantic model lives in pipelex/temporal/config_temporal.py so the validator can reference BUILTIN_SEARCH_ATTRIBUTES without pulling temporalio into the config-load path.

• pipelex setup-temporal-namespace CLI command — wraps OperatorService.AddSearchAttributes against the configured server profile so operators don't need a separate temporal / tcld install for the common case. Reads the same [temporal.search_attributes].attributes list and [temporal.temporal_config] block the worker uses, so the names registered here can never drift from the names the worker will dispatch with. --dry-run prints the equivalent raw temporal operator search-attribute create command without executing. --server <profile> targets a non-default server profile. On Temporal Cloud namespaces where the worker API key lacks OperatorService.AddSearchAttributes permission, the command catches RPCError(PERMISSION_DENIED) and prints the fallback runbook (raw temporal CLI command, tcld for Cloud, Cloud UI link).

• Per-workflow static summary and details in the Temporal dashboard. Every top-level workflow start now sets static_summary (200-byte Markdown like translate_doc — Translate a document from English to French) and static_details (Markdown table of pipe code, domain, pipeline run id, user, session, optional library crate, optional inputs). Every workflow.execute_activity(...) call sets a per-call summary= carrying the call-specific meaning (LLM text · pipe=translate_doc · model=gpt-4o, Img gen N× · pipe=cover · model=fal-flux-dev · n=3, etc.). The formatting policy lives in the new pipelex/temporal/tprl/observability.py module.

• Documented render_js and include_raw_html on PipeExtract. Both fields were already shipping on PipeExtractBlueprint; only docs were missing. render_js = true asks the extraction backend to render JavaScript before fetching web-page content; include_raw_html = true populates each extracted Page's raw_html field with the fetched HTML. Added to building-methods/pipes/pipe-operators/PipeExtract.md.

• Documented XHIGH value on ReasoningEffort. The enum already shipped seven levels; the under-the-hood/reasoning-controls.md table listed only six. XHIGH sits between HIGH and MAX and maps to provider-specific xhigh values where supported.

• Bounded fan-out concurrency for PipeBatch. A PipeBatch over many items no longer spawns every branch — every coroutine, every deep-copied working memory, every inference call — at once. Branches now run in bounded chunks driven by the new [pipelex.pipeline_execution_config] setting max_concurrency (default 8; set to the literal "unbounded" for unbounded fan-out, the previous behavior). This is the resilience-without-Temporal pillar: it keeps a large workload (one pipe over thousands of documents) from overwhelming asyncio, memory, and provider rate limits. Results still preserve input order, and a branch failure still propagates (first error by input index wins). When a batch fans out over a large number of items, an advisory log line points at the Temporal track as the durable, rate-limited path. PipeParallel, which fans over a fixed pipe-defined branch set, is unchanged.

• Authoritative error_domain → HTTP-status mapping. pipelex/base_exceptions.py now owns the mapping that downstream HTTP APIs (pipelex-relay, pipelex-back-office) need to render an ErrorReport as an HTTP response: error_domain_to_http_status() (the pure domain table — INPUT → 422, CONFIG/RUNTIME/unknown → 500) and the ErrorReport.http_status property, which adds a provider-429 passthrough so the API can emit a Retry-After header from provider_metadata.retry_after_seconds. The library stays HTTP-agnostic — no web-framework dependency, just the mapping table; downstream FastAPI handlers call the helper instead of reinventing the contract.

• Category-aware error boundary on Temporal activities. Every in-scope Temporal activity is now decorated with @convert_pipelex_errors (new module pipelex/temporal/tprl/activity_error_boundary.py), which converts a PipelexError raised inside the activity into a TemporalError at the activity boundary. This packs the structured ErrorReport into ApplicationError.details and derives non_retryable from the error's InferenceErrorCategory — so the workflow-side TemporalError.from_app_error keeps error_category, user_action, model and provider instead of landing in its error_report is None fallback, and a non-retryable CogtError (CONFIGURATION, CONTENT, CAPACITY) is no longer retried. Resilience for inference failures lives entirely on the Temporal track — direct (non-Temporal) execution makes a single pipeline-level attempt. act_assemble_graph is deliberately left unwired — it is best-effort observability that swallows every failure and degrades to None. As part of this, TemporalError._log_critical / _log_error now select activity_log vs workflow_log based on activity.in_activity(), since workflow.logger raises _NotInWorkflowEventLoopError outside a workflow event loop.

• AMBIGUOUS inference-error category for outcome-uncertain failures. New InferenceErrorCategory.AMBIGUOUS for failures where the error type is known but the outcome is not — the operation may or may not have committed (e.g. a connection dropped mid-request). It is non-retryable, like CONFIGURATION / CONTENT / CAPACITY, but semantically distinct from UNKNOWN, which means the error could not be classified at all. The azure_rest image-generation worker now raises AMBIGUOUS for mid-request transport failures — ReadError / WriteError / RemoteProtocolError and ReadTimeout / WriteTimeout — so Temporal does not auto-retry a non-idempotent, billable image submit whose outcome is unknown; pre-request failures (ConnectError / ConnectTimeout / PoolTimeout) stay TRANSIENT and retryable.

• Explicit, uniform Tier 1 transport retry across every inference worker. A new [cogt] setting transport_max_retries (default 2) is now wired explicitly into every inference SDK client factory — Anthropic, OpenAI / Azure OpenAI, the Portkey-backed gateway clients, Mistral, and Google — instead of each factory silently inheriting whatever retry posture its SDK happens to default to. The two SDK families that default to no transport retry are brought up to the same floor: the Mistral client gets a bounded-backoff RetryConfig (retry_connection_errors=True) and the Google GenAI client gets HttpOptions(retry_options=...). The genuinely SDK-less path — the azure_rest image-generation worker, which talks to Azure over raw httpx — gets a tenacity-based transport-retry wrapper (new module pipelex/cogt/inference/transport_retry.py) that retries connection failures and transient HTTP statuses (408/409/429/5xx) and honors Retry-After. When no Retry-After header is present, the fallback backoff uses full jitter (wait_random_exponential) so a burst of failures retried together does not re-fire in lockstep. On a non-idempotent submit-style POST it narrows the retry to failures that prove the server did no billable work — a request that was never delivered, or a 408/429 rejection — withholding an ambiguous 5xx, a 409 conflict, and a post-delivery timeout such as a ReadTimeout. Transport retry is now a deliberate, configured, uniform policy rather than a per-provider accident. This is "Tier 1" of the retry model — direct (non-Temporal) execution still makes a single pipeline-level attempt on top of this transport floor; durable resilience remains the Temporal track. (The portkey-ai SDK does not expose a retry knob — it carries its own internal retry — so the gateway's AsyncPortkey client is left as-is; only its underlying OpenAI clients are wired.)

• Offline mode for Pipelex Gateway setup and dry-run. When the gateway is enabled but the remote config service is temporarily unreachable, Pipelex now falls back to a previously primed on-disk cache (~/.pipelex/cache/remote_config.json, schema-versioned) instead of failing setup outright. Dry-run, validation, and pipelex-agent run bundle --dry-run complete normally; only the actual inference call still needs the network at runtime. The cache is primed on every successful fetch and on pipelex init while online. When the gateway is disabled (BYOK), no remote fetch is attempted at all — setup is fully offline. A new RemoteConfigStaleWarning (UserWarning) is emitted whenever stale cache is in use; the agent CLI surfaces it on the JSON envelope as warnings: [{"type": "RemoteConfigStale", ...}]. Telemetry is suppressed (no-op) when running on a cached config so stale model identities don't pollute metrics. The doc/fixture generators (pipelex-dev update-gateway-models, preprocess_test_models_cmd) refuse the cache fallback via a new require_fresh=True flag, so committed reference docs and test fixtures never bake in stale data.

• GatewayUnknownModelError (pipelex.cogt.exceptions). Raised at setup time when the active model deck references a gateway model handle that isn't present in the (fresh or cached) gateway specs. Carries the model name and the config source (RemoteConfigSource.FRESH | CACHED); the message branches on source so a cached-source failure suggests pipelex init while online and a fresh-source failure points at deck/typo fixes. Wired through both the Rich CLI (handle_gateway_unknown_model_error in error_handlers.py) and the agent CLI (AGENT_ERROR_HINTS / AGENT_ERROR_DOMAINS).

• RemoteConfigUnavailableError (pipelex.system.pipelex_service.exceptions). User-facing offline-mode error: raised only when the network fetch fails AND no usable cached fallback exists. The message names the cache file path and the two remediation paths (run pipelex init while online to prime the cache; or disable pipelex_gateway in backends.toml for permanent BYOK operation). Distinct from the internal RemoteConfigFetchError, which is kept as the retry-layer exception.

• PIPELEX_REMOTE_CONFIG_URL environment variable. Overrides the default remote-config URL. Useful for staging/testing environments; defaults to the production URL when unset.

Changed

• Inference error handling refactored to Extract / Classify / Render (internal). Every inference worker's SDK-exception block now collapses to metadata = extract_*_metadata(exc); classification = classify_inference_error(metadata); raise render_*_error(...) from exc. New modules: pipelex/cogt/inference/error_classify.py (single shared classify_inference_error() returning ClassificationResult(category, user_action_kind, is_model_not_found)), pipelex/cogt/inference/error_render.py (single shared render_llm_error / render_img_gen_error / render_extract_error / render_search_error, picking the CogtError subclass from an InferenceErrorFamily tag plus the is_model_not_found flag), and pipelex/cogt/inference/provider_name.py (the ProviderName enum keying the extract-fn registry). ProviderErrorMetadata gains a message field plus is_quota_exhaustion / is_content_policy_violation / is_network_error @property accessors; the per-provider extract_*_metadata functions are now the only plugin-local piece — Classify and Render live once. Mistral and the gateway-search worker specialize HTTP 404 to ExtractModelNotFoundError / SearchModelNotFoundError; Azure img-gen keeps two worker-specific AMBIGUOUS branches for mid-request transport failures. Removed: the per-provider *_error_classification.py modules and their tests, AnthropicCredentialsError, GatewayFactory.classify_error_category / make_user_action_from_portkey_error / make_error_summary_from_portkey_error, and every inline _classify_*_error / _raise_categorized_* worker method. New unit-test tests/unit/pipelex/cogt/inference/test_provider_classification_parity.py walks every ProviderName against the extract-fn registry + worker-family map, so an unwired new provider fails fast. No user-facing API change — the structured ErrorReport contract is unchanged. Documented at under-the-hood/error-model.md.

• instructor's structured-output retry no longer re-runs completions on transport errors. Passed a bare int, instructor's max_retries builds a retry loop whose predicate retries any exception — so the structured-output path (PipeLLM / PipeStructure) was re-running the whole completion on transport / API errors, a second retry loop nested on top of the SDK client's own transport retry. Each instructor call site now passes a tenacity.AsyncRetrying (built by the new pipelex/cogt/llm/instructor_retry.py helper) whose retry predicate matches only validation failures (pydantic.ValidationError, json.JSONDecodeError, and instructor's own validation-error types). A transport error now propagates immediately as the raw SDK exception for the worker's except clause to classify — transport retry is the SDK client floor (Tier 1) alone, and instructor's retry is confined to genuine schema re-ask. As part of this the Google and Mistral structured-generation workers gained an httpx.TransportError except clause: those SDKs let raw connection / timeout errors propagate outside their own exception hierarchies, so with instructor no longer wrapping them they must be caught and classified directly. The Mistral structured path also now gets schema re-ask at all — it previously passed no max_retries.

• Inference schema-retry setting moved and renamed: [cogt.llm_config.llm_job_config] max_retries → [cogt.llm_config] schema_reask_max_attempts (breaking). The setting is instructor's schema re-ask budget for structured-output validation failures. The old name max_retries gave no hint of that scope and collided conceptually with the new top-level cogt.transport_max_retries — which counts retries beyond the initial attempt, whereas this one is a total attempt count (stop_after_attempt). The new name makes both the scope (schema re-ask) and the unit (attempts, not retries) explicit. The single-key [cogt.llm_config.llm_job_config] sub-table is also dropped — the setting now sits directly under [cogt.llm_config]. Projects that override this key in their own pipelex.toml must move and rename it.

• Agent CLI run / validate / init default to markdown output, with independent success/error format options (breaking). These commands previously always emitted JSON; they now accept --format markdown|json and default to markdown, matching models / check-model / doctor. A second flag --error-format markdown|json controls error reporting on stderr independently from success output — it defaults to the value of --format, so --format json still flips both, but the two can now be set separately (e.g. --format markdown --error-format json for human-readable success with machine-parseable errors). Internally, only the error format is carried in a ContextVar; the success format is threaded explicitly to agent_success_formatted(). The inputs / concept / pipe / fmt / lint / accept-gateway-terms commands are unaffected (always JSON / raw passthrough).

• pipelex-agent validate bundle graph-format option renamed --format → --graph-format (breaking). The --format/-f flag that selected the graph renderer (mermaidflow / reactflow / both) is now --graph-format/-f, freeing --format to be the uniform markdown/json output-format flag across every agent-CLI command.

• Top-level Workflow ID shape change (breaking). Workflow IDs go from {env}{session5}-{rand5}-{ClassName} (e.g. EdgdJ-HR5fd-TemporalPipeRun-pipe-router) to {env_prefix}{pipeline_run_id} (e.g. ut-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c). The session-id and random-id components and the calling class name are gone; identity flows entirely from Pipelex's existing JobMetadata.pipeline_run_id. Operational tooling that grepped for the old shape must update.

• Pipeline run chain semantics shift (breaking, behavioral). Because the Workflow ID is now derived from pipeline_run_id, callers that pass a stable pipeline_run_id to PipelineFactory.make_pipeline(pipeline_run_id=...) and re-execute now land on the same Temporal Workflow Execution Chain (with a fresh run_id per execution under the SDK-default ALLOW_DUPLICATE reuse policy). The previous behavior produced a fresh workflow_id per execution by accident, via the truncated session id and random shortuuid components — not by design. This is now documented behavior, not a bug.

• Child Workflow ID separator change (breaking). Child workflow ids use / instead of - as the separator. Examples: the fixed-role wf_pipe_router child of wf_pipe_run is {parent}/pipe-router; a dynamic sub-pipe spawned by a router is {parent}/{pipe_code}-{8-hex-chars} (the 8 hex chars come from workflow.uuid4() for replay-safety). Operational tooling parsing the nested-id format must update.

• Activity ID change (breaking). Pipelex no longer customizes activity_id. The Temporal Python SDK auto-assigns deterministic sequential integers ("1", "2", …) per workflow run, which guarantees per-(workflow_id, run_id) uniqueness by construction and is replay-safe (assigned by history position). Per-call meaning that previously lived in activity_id now lives in the per-activity summary=. Anything that filtered or grouped Event History by the old semantic strings ("craft-text", "craft-object-direct", "jinja2-text", "extract-pages", etc.) must read the per-activity summary or the Activity Type instead.

• wfid parameter removed (breaking). The wfid: str | None = None parameter is dropped from PipeRunProtocol, PipeRouterProtocol, ContentGeneratorProtocol, and every implementation (PipeRun, PipeRouter, DryPipeRouter, TemporalPipeRun, TemporalPipeRouter, ContentGenerator, ContentGeneratorDry, ContentGeneratorInWorkflow). The parameter was a legacy artifact — wfid originally seeded child workflow ids, was later co-opted as a default activity_id, and had no production callers. Identity now flows entirely via JobMetadata.pipeline_run_id; observability is auto-derived from pipe_code / domain_code. No deprecation cycle; per Pipelex's no-backward-compat policy, callers update at once.

• ContentGeneratorInWorkflow LRU + replay short-circuit deleted. The worker-singleton _seen_activity_ids: OrderedDict[(workflow_id, run_id), set[str]] cache, the _MAX_SEEN_RUNS bound, and _record_activity_id (with its workflow.unsafe.is_replaying() short-circuit) are gone. They existed solely to defend against the duplicate-activity-id failure mode that no longer exists now that Pipelex does not customize activity_id. Future contributors: do not reintroduce worker-singleton state on the activity-dispatch path — the determinism guarantee is the SDK assigning ids by history position.

• structuring_method = "preliminary_text" on PipeLLM works again, via build-time elaboration. When PipelexInterpreter parses a .mthds file with a PipeLLM carrying structuring_method = "preliminary_text", the new BundleElaborator rewrites it before any pipe runs into a PipeSequence of two synthetic pipes: a PipeLLM producing Text (step 1, inheriting the original prompt + inputs + model) and a PipeStructure producing the original output (step 2, optionally using model_to_structure). The synthetic codes are tracked in a PipelexBundleBlueprint.elaboration_metadata side-table (excluded from serialization). Output multiplicity is preserved: step 1 always emits a single Text; step 2 emits Foo, Foo[], or Foo[N] according to the original output. Two LLM calls are issued per invocation. The user-facing pipe code is unchanged, so callers, main_pipe, and the run API keep working as before. Mechanism documented at under-the-hood/build-time-elaboration.md.

• PipeLLM runtime no longer knows about structuring_method. The field is now a pure build-time directive. The runtime PipeLLM class no longer carries structuring_method, the validator that rejected mismatched output concepts has moved to PipeLLMBlueprint (where it fires at parse time), and the NotImplementedError previously raised at runtime when preliminary_text was selected is gone — the elaborator handles it. PipeLLMSpec exposes structuring_method so AI agents authoring via specs can still opt in.

• StructuringMethod enum import path moved. The enum is now defined in pipelex.pipe_operators.llm.pipe_llm_blueprint (it lives next to the blueprint that consumes it) instead of pipelex.pipe_operators.llm.pipe_llm. Direct importers must update their from pipelex.pipe_operators.llm.pipe_llm import StructuringMethod line to from pipelex.pipe_operators.llm.pipe_llm_blueprint import StructuringMethod.

• Collapsed the tprl_content_generation/ workflow layer. Each content-generation call (LLM text/object/object-list, image generation, Jinja2 rendering, document extraction, PDF page-view rendering) previously went through a dedicated child workflow (WfMakeLLMText, WfMakeObject, WfMakeImages, WfMakeJinja2Text, WfMakeExtract, WfRenderPageViews) that wrapped a single act_* activity. The wrappers added one workflow-scheduled / started / completed history-event triplet per call without changing durability guarantees — every retry still happens at the activity level. ContentGeneratorInWorkflow now calls workflow.execute_activity(act_*, ...) directly from inside WfPipeRouter, deleting all WfMake* / WfRenderPageViews workflows, both ContentGeneratorTop and ContentGeneratorChild generators along with their factories, and the content_generator_models.py assignment-types module. Per-step meaning in the Temporal Web UI now comes from per-activity summary= (and Activity Type), replacing the old semantic activity_id strings. Page-views augmentation (should_include_page_views) now works in Temporal mode: make_extract_pages dispatches act_render_page_views for document_uri inputs and builds an inline single-element [ImageContent(url=...)] list for image_uri inputs, then attaches each page-view to its PageContent. Previously the direct-mode generator handled both branches but the child-workflow path (WfMakeExtract) only ran the extract activity and silently skipped the page-views step, so should_include_page_views=True was a no-op under Temporal.

• Per-activity, per-handle Temporal task-queue routing. Replaced the provisional WorkerConfig.inference_task_queue (one-off LLM-text override) with a general activity_queues: dict[str, ActivityRouteConfig] table. Each entry declares a per-activity default queue and an optional by_handle map keyed by runtime handle (LLM model handle for text/object/object-list, img_gen_handle for image generation, extract_handle for document extraction). At dispatch time, WorkerConfig.resolve_queue(activity_name, routing_key) walks three layers: (1) activity_queues[activity_name].by_handle[routing_key], (2) activity_queues[activity_name].default, (3) the worker-wide default_task_queue default. Every ContentGeneratorInWorkflow call site now passes task_queue=worker_config.resolve_queue(...) uniformly — the asymmetric "LLM-text-only kwarg" is gone, the _inference_dispatch_kwargs stopgap is deleted, and WorkerConfig.inference_task_queue is removed (no backward-compat shim). Deployments can now scale OpenAI vs Anthropic worker pools independently, isolate image generation onto dedicated runners, or route specific OCR backends to dedicated queues — all via TOML config, with no code change required. Default pipelex.toml ships with an empty activity_queues table, so existing single-queue deployments are unaffected.

• Per-queue submitter options and named worker-runtime profiles. Layers on top of the per-activity routing above. (1) [temporal.queue_options.<queue>] declares per-queue start_to_close_timeout, schedule_to_close_timeout, schedule_to_start_timeout, heartbeat_timeout, retry_policy_config, and cluster-wide max_task_queue_activities_per_second. (2) activity_queues.<activity>.handle_options.<handle> declares rare per-handle overrides for a single model/backend variant. (3) [temporal.worker_runtime_profiles.profiles.<name>] declares concurrency slots, pollers, heartbeat throttles, worker-local rate cap, and graceful shutdown timeout; one worker process selects one profile via pipelex worker --profile <name>. The new WorkerConfig.resolve_dispatch(activity_name, routing_key) composes baseline → queue_options[resolved_queue] → handle_options[routing_key] last-wins for scalars, and unions non_retryable_error_types additively across all three layers (baseline main list + queue _extra + handle _extra). The RetryPolicyConfig schema split into a baseline class (owns non_retryable_error_types) and an overlay class (owns non_retryable_error_types_extra); extra="forbid" rejects the wrong field on each layer so the additive composition rule is enforced at config load. Every workflow.execute_activity(...) site in ContentGeneratorInWorkflow now splats worker_config.resolve_dispatch(...).to_execute_kwargs() — fixing a long-standing bug where the workflow-level workflow_execution_timeout was used as every activity's start_to_close_timeout (a 1h budget on a jinja2 render, same 1h on a slow PDF extract).

• Strict --task-queue validation at worker CLI startup. pipelex worker --task-queue X now fast-fails with WorkerTaskQueueUnknownError (and a Levenshtein "Did you mean?" suggestion) when X isn't declared in default_task_queue, any activity_queues entry, or any queue_options key. Strict counterpart to the lenient config-load WARN on routing entries that name unknown queues.

• Dispatch resolution tracing. Set temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced = true to emit one INFO log line per workflow.execute_activity call, including the resolved queue, timeout, retry attempts, and the layer (baseline / queue_options / handle_options) each scalar came from. Off by default.

• Specialized worker scopes. New runner-llm, runner-img-gen, runner-extract, runner-jinja2 scopes under [temporal.worker_scopes.scopes] for deployment manifests that want one worker pool per backend class. Each scope registers only its activities (e.g. runner-llm registers act_llm_gen_text, act_llm_gen_object, act_llm_gen_object_list). act_render_page_views is registered under both runner-img-gen and runner-extract (belt-and-suspenders for the two paths that need it).

• temporal.worker_config.task_queue renamed to default_task_queue. Migration map entry under [migration.migration_maps.config] maps the old key. New required field default_activity_start_to_close_timeout (baseline activity timeout, default "0:10:00") replaces the previous accidental reuse of workflow_execution_timeout for activities.

• Cluster-wide queue rate cap moved from Python constant to TOML overlay. TemporalTaskManager.make_worker previously hardcoded max_task_queue_activities_per_second=1000 on the Temporal Worker(...) constructor for every queue. It now reads this knob from queue_options[task_queue].max_task_queue_activities_per_second; the shipping default [temporal.queue_options.temporal_task_queue] sets it to 1000, so out-of-the-box behavior on the default queue is unchanged. Deployments using non-default queue names should add their own [temporal.queue_options.<queue>] entries with the cap appropriate for that backend pool. The per-worker max_activities_per_second = 1000 lives on the runtime profile and is unchanged.

• Removed dead code: pipelex/temporal/wrapper/. The start_tprl_activity wrapper had zero callers.

• Retry removed from the gateway workers; [cogt.tenacity_config] removed (breaking). The tenacity-based retry inside GatewayExtractWorker and GatewaySearchWorker is gone, along with the [cogt.tenacity_config] config block and its TenacityConfig model. Because config models forbid unknown keys, an existing ~/.pipelex/pipelex.toml (or any layered override) that still carries [cogt.tenacity_config] will fail to load — remove that block from your config. Transient transport blips are left to the provider SDK clients' own retry; the tenacity library itself is still used elsewhere (FAL job polling, remote-config fetch) and remains a dependency.

• PipeRouter.run() now reports CogtError failures to the observer. A CogtError raised out of pipe execution previously propagated past run() without an observe_after_failing_run notification (only PipeRunError was caught). It is now observed on the failing path, then re-raised as-is — the cause chain is preserved and it is not wrapped into PipeRouterError (only the PipeRunError path still wraps).

• Provider HTTP 404s now raise dedicated *ModelNotFoundError across LLM, image-gen, extract, and search. A model-or-deployment-not-found 404 from any LLM or image-gen provider, or from the Pipelex Gateway extract / search workers, now raises a dedicated *ModelNotFoundError (LLMModelNotFoundError, ImgGenModelNotFoundError, ExtractModelNotFoundError, SearchModelNotFoundError — all ModelNotFoundError subclasses, CONFIGURATION category) instead of a generic LLMCompletionError / ImgGenGenerationError / ExtractJobFailureError / GatewaySearchResponseError. Because these are siblings of the generic errors — not subclasses — a 404 propagates past the operator's generic except to except ModelNotFoundError in PipeOperator._live_run_pipe, which re-raises PipeOperatorModelAvailabilityError carrying the unavailable model_handle.

• RemoteConfigFetcher.fetch_remote_config() now returns a RemoteConfigResult carrying config, source (fresh | cached), and cached_at, instead of a bare RemoteConfig. Callers unwrap .config for the payload and may branch on .source to know whether the config is fresh or restored from cache. The fetcher accepts a new keyword-only require_fresh: bool = False — when True, a cached fallback raises RemoteConfigUnavailableError instead. RemoteConfigValidationError is never satisfied by the cache (server-side schema breaks must surface loudly).

• ModelManager.setup() and BackendLibrary._load_gateway_model_specs() accept a new gateway_config_source: RemoteConfigSource | None parameter. Passed through from Pipelex.setup() so the deck-level gateway membership check can branch its error message on FRESH vs CACHED. GatewayConfig itself stays extra="forbid" and source-free — provenance is plumbed alongside, not baked in.

• RemoteConfigUnavailableError message branches on whether the cache was refused vs absent. When require_fresh=True (dev-CLI generators) refuses to fall back to an existing cache, the message now reads "the local cache at <path> was refused because a fresh fetch is required" instead of the previously misleading "no local cache is available at <path>". The cache-truly-missing path keeps its original wording.

Fixed

• Inference-failure ErrorReports now carry model and provider in production. A real LLM / image-gen / extract / search failure used to surface an ErrorReport with model = None and provider = None: the leaf errors (LLMCompletionError, ImgGenGenerationError, ExtractJobFailureError, SearchJobFailureError, ExtractOutputError) carry no model or provider of their own, and CogtError.to_error_report() only duck-typed whatever attributes happened to be set on the exception. Each inference worker family now fills model_handle / backend_name from the worker — where both are unambiguously known — at its public-method chokepoint (gen_text / gen_object, gen_image / gen_image_list, extract_pages, search_sourced_answer / search_structured), via the new CogtError.fill_model_and_provider(). The fill never overwrites a value an inner error already set and skips the "unknown" placeholder external plugins report. model_handle / backend_name are now declared on CogtError (so to_error_report() reads them directly instead of getattr), making them uniformly str | None across the exception hierarchy.

• Two worker error-classification miscategorizations corrected. LinkupNoResultError (a search or fetch that returned nothing) had no explicit branch in either Linkup worker's _classify_linkup_error and fell through to the TRANSIENT catch-all — marking a query that cannot succeed by retrying as retryable. It is now classified CONTENT + CHANGE_INPUT in both the search and the extract worker. Behavior change: a no-result search is now non-retryable (CONTENT.is_retryable is False), so Temporal's activity retry policy no longer retries a query that returns nothing on every attempt. Separately, the FileNotFoundError branch in the Docling and pypdfium2 extract workers was classified CONFIGURATION — inconsistent with its sibling branches (CONTENT + CHANGE_INPUT) and with the rest of the codebase, where CONFIGURATION is reserved for setup problems. A missing input file is a content problem; it is now CONTENT. This second change does not alter retry behavior — CONFIGURATION and CONTENT are both non-retryable.

• Wrapped exceptions now surface the underlying inference error's classification. PipelexError.to_error_report() enriches its report from the __cause__ chain, so a PipelineExecutionError — or any wrapper around a transient CogtError — now reports error_category, retryable, model, and provider instead of dropping them. Previously the agent-CLI JSON / markdown error output for a failed pipeline run lost the worker's category and retryability once the error had been wrapped by the pipe operator → router → runner layers, leaving an agent unable to tell a transient failure from a fatal one.

• MTHDS JSON schema no longer leaks the elaboration_metadata side-table. PipelexBundleBlueprint.elaboration_metadata carried Field(exclude=True) so it stayed out of model_dump() / model_validate() round-trips, but Pydantic v2's exclude=True does not affect model_json_schema() — so derived/mthds_schema.json ended up declaring a top-level elaboration_metadata property plus $defs/ElaborationMetadata and $defs/StepRole entries, even though the side-table is process-local in the reference runtime. Wrapped the field type with SkipJsonSchema[...] so it is hidden from the schema. After regenerating, the three entries are gone from the public schema, matching the spec's silence on them.

• Cross-process Temporal decode of ListContent payloads for dynamic concepts. WorkingMemory.dump_for_temporal() was tagging each list item with __class__ / __module__ markers so the receiving worker could rebuild the right subclass for Anything[] outputs. Those keys are also kajson's universal-decoder protocol, so the Temporal data converter tried to eagerly bind dynamic-concept classes (e.g. structured_output_test__Invoice) at the payload boundary — before the child workflow's per-workflow ClassRegistry was loaded. In a true 3-process topology the class is not in the global registry (the child workflow tore down its scoped registry in finally), so kajson.loads raised KajsonDecoderError → RuntimeError: Failed decoding arguments and any --temporal run that produced a list of dynamic-structured-concept items hung. Renamed the markers to the pipelex-private namespace __pipelex_class__ / __pipelex_module__ in WorkingMemory.dump_for_temporal() and _hydrate_list_item(); kajson's decoder gates strictly on __class__ (kajson/json_decoder.py:132) so pipelex's nested dicts now pass through untouched and class binding stays inside pipelex's hydrator where the per-workflow registry lives. Also extended CLEAN_JSON_FIELDS_TO_SKIP in pipelex/tools/misc/json_utils.py to strip both marker families from user-facing JSON output.

• A directory containing a .pipelex/ config dir is now recognized as a project root. .pipelex was added to PROJECT_ROOT_MARKERS, so project-level config (e.g. .pipelex/inference/backends.toml) is honored even when the directory has no .git, pyproject.toml, or other source-project marker. Previously such a directory fell through to the global ~/.pipelex/ config, silently ignoring the project's own overrides — so a backend disabled in the project's backends.toml could still demand credentials because the global config (where it was enabled) was loaded instead. The home directory remains excluded from project-root detection, so the global ~/.pipelex/ is unaffected.

• PipeLLM outputting a concept that refines the native JSON concept no longer crashes with NameError: name 'Any' is not defined. On a LIVE run, such a concept resolves to a structured-output model carrying a dict[str, Any] field inherited from JSONContent. SchemaToModelFactory generates that model as source with from __future__ import annotations (every annotation becomes a string) and then rebuilds each class to resolve the string annotations. The rebuild namespace was assembled from the exec'd user types plus a hand-listed Literal, but typing.Any is a special form, not a type, so it was filtered out — model_rebuild then raised PydanticUndefinedAnnotation evaluating "dict[str, Any]". The rebuild namespace is now the exec namespace itself (minus __builtins__), so it carries exactly the names the generated source was written against and cannot drift as codegen emits other typing constructs. Fixed in pipelex/cogt/content_generation/schema_to_model_factory.py; covers both the sender path (make_from_json_schema) and the cross-process receiver path (make_types_from_source).

---

Summary by cubic

Release v0.29.0 focuses on reliability, observability, and control for both direct and Temporal runs. It adds PipeStructure, offline Gateway setup, per-activity task routing, uniform transport retry, and a safer error model — with a few breaking CLI and Temporal changes.

• New Features

  • PipeStructure operator + build-time elaboration: PipeLLM with structuring_method = "preliminary_text" is rewritten into PipeLLM(text) → PipeStructure(object); no runtime changes needed.
  • Temporal upgrades: strict custom search attributes (with pipelex setup-temporal-namespace), human-readable workflow/activity summaries, per-activity/per-handle routing ([temporal.worker_config.activity_queues]), per-queue submitter options and named worker runtime profiles, and collapsed content generation to direct activities.
  • Gateway offline mode: uses a primed on-disk cache for setup/validate/dry-run when remote config is unreachable; new GatewayUnknownModelError and RemoteConfigUnavailableError; PIPELEX_REMOTE_CONFIG_URL override.
  • Uniform Tier‑1 transport retry: [cogt].transport_max_retries applied across providers and raw httpx paths; honors Retry-After.
  • Resilience without Temporal: bounded PipeBatch fan‑out via [pipelex.pipeline_execution_config].max_concurrency (default 8), plus in-process transient retry in PipeRouter.
  • Error model and delivery: authoritative error_domain → HTTP mapping, ErrorReport now carries model/provider, new AMBIGUOUS category, category‑aware Temporal retry and details bridge, consistent provider metadata and user actions across workers.
  • Agent CLI UX: markdown is the default; independent --format and --error-format; validate ... --graph-format replaces --format for graph selection.
  • Model updates: add Google gemini-3.5-flash (text/images/pdf; structured output; adaptive thinking).
  • Reasoning controls: ReasoningEffort adds XHIGH between HIGH and MAX (mapped where supported).

• Migration

  • Temporal
    • Register search attributes once per namespace: run pipelex setup-temporal-namespace (or use the printed temporal operator/tcld command).
    • IDs/observability: workflow IDs now derive from pipeline_run_id; child IDs use /; activity IDs are SDK-assigned integers — update any dashboard tooling that parsed the old shapes or relied on semantic activity_id.
    • Config: worker_config.task_queue → default_task_queue; add/adjust [temporal.worker_config.activity_queues], [temporal.queue_options.<queue>], and [temporal.worker_runtime_profiles.profiles.<name>]; pipelex worker now supports --profile; --task-queue is strictly validated.
  • CLI
    • Agent: default output is markdown; use --format json for machine reads, or set --error-format independently. validate ... --graph-format replaces --format for graph renderer selection.
  • LLM structuring and retry
    • Rename/move: [cogt.llm_config.llm_job_config].max_retries → [cogt.llm_config].schema_reask_max_attempts (total attempts). Remove [cogt.tenacity_config] (deleted). Use [cogt].transport_max_retries for transport and [pipelex.pipeline_execution_config] for transient retries.
    • StructuringMethod import path moved to pipelex.pipe_operators.llm.pipe_llm_blueprint.
  • Gateway
    • To enable offline setup/validate/dry-run, run pipelex init while online to prime the cache; set PIPELEX_REMOTE_CONFIG_URL for non-prod endpoints.
  • Docs
    • Distributed execution: corrected config path references to .pipelex/pipelex.toml.

^{Written for commit bfcb011. Summary will update on new commits. Review in cubic}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 37e7c4fdbe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[lchoquel]

@greptileai

[cubic-dev-ai]

3 issues found across 567 files

<sub>Note: This PR contains a large number of files. cubic only reviews up to 100 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 3 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[lchoquel]

@greptileai

[lchoquel]

@greptileai review this

[thomashebrard]

congrats