Skip to content

v0.6.15

Latest

Choose a tag to compare

@rstrahan rstrahan released this 13 Nov 20:14
· 584 commits to develop since this release

[0.6.15]

Added

  • Experimental Terraform deployment. Each release can also be deployed with Terraform or OpenTofu (1.7 or later), from a module generated from the same CloudFormation templates and published beside them as a zip with a SHA-256 checksum. Use it from your own configuration (module "idp" { source = "<url>" }) or with idp-cli deploy --terraform, which can also build from a local checkout (--from-code .), assume a deployment role (--terraform-assume-role-arn) and refuses a plan that would delete or replace a data-holding or retained resource unless given --terraform-allow-replace-stateful. A read-only CloudFormation label stack lets idp-cli and the SDK address the deployment by --stack-name; every IAM role it creates takes the PermissionsBoundaryArn boundary, and feature-platform extensions still install as CloudFormation stacks. The principal that runs terraform apply needs kms:GenerateDataKey and kms:Decrypt on the stack's customer-managed key. See Terraform Deployment.

  • Move a genai-idp-terraform deployment onto the built-in Terraform module in place. idp-cli terraform migrate-from-genai-idp-terraform (and client.terraform.migrate_from_genai_idp_terraform in the SDK) has a new deployment adopt the old one's buckets, tables, KMS key and Cognito user pool, so documents, configuration, users and passwords stay. Each phase is a dry run until --apply, and it refuses anything it cannot prove safe. See Migrating from genai-idp-terraform.

  • Bulk Skip All Reviews. Finish many pending production reviews at once, from a Document List or review-queue selection or by configuration profile and submitted-date range, with a required reason and a dry run first. Large requests run as background jobs, listed under the new Home › Jobs page with cancel and delete, and an Inbox message on completion. A Resubmit for downstream processing option (on by default) can be unticked to skip without re-running Summarization, Evaluation and the post-processing hook. Admins and Reviewers can use it within their scope; Start Review and Release Review moved into the Document List's new Review menu. New setting hitl.bulk_skip_sync_limit. See Skip All Reviews for many documents and Review Queues.

  • Send to review. Put a finished production document, or a selection, into the Default review queue by hand with a reason that reviewers see on the item. See Send to review.

  • MCP service credentials, managed on a new MCP Configuration page. Admins create client-credentials apps for MCP clients such as Amazon Quick workflows or the MCP Connector, each with its own role, configuration profiles, write-tools setting and expiry; the secret is shown once and stored in Secrets Manager, and credentials can be rotated with overlap or revoked. Also available as idp-cli mcp credentials and IDPClient.mcp. Deleting the stack, switching MCP off or downgrading revokes every credential. See MCP Server.

  • Add documents to a test set from the Document List. Admins select processed production documents and choose Add to test set, which opens Test Studio's From processed documents dialog with the selection already made. See Adding documents to a test set. (#812)

  • Comments in prompt templates. {# ... #} anywhere in a prompt documents it and is removed before the prompt is sent; Prompt Preview shows both the sent and the as-written forms. ⚠️ Action: on upgrade, text between an existing {# and a later #} in a stored prompt is no longer sent; search your configuration for {# first. See Comments in a prompt. (#1653)

  • The reporting lake counts document runs as well as documents. metering_docs_hourly and metering_docs_daily gain n_runs, which rises on reprocessing and sums exactly over any range, unlike n_docs; the analytics agent knows the difference. Action: the stack update replays the rollup window (30 days by default), which takes tens of minutes of Athena on a busy stack. ⚠️ The replay rebuilds all four rollup tables, so the cost rollups, dashboard cost widgets and analytics agent read low until it finishes; raw metering is unaffected, and older partitions keep n_runs empty. See Reporting SQL layer and the migration runbook.

  • Info panels on the Home pages. The info icon on Inbox, Tasks, Queues, Jobs and a job's page explains what the page is for and who uses it.

Changed

  • EnableMCP now defaults to false. New stacks have no MCP server unless you set EnableMCP=true; the console and idp-cli deploy keep an existing stack's value. ⚠️ Action: if you update with the AWS CLI, idp-cli deploy --terraform, the Terraform module or another tool that sends only named parameters, pass EnableMCP=true to keep MCP on — turning it off revokes every MCP service credential, and they are not restored. See MCP Server.

  • MCP tools act as the caller who signed in, with that caller's role and configuration scope. process/reprocess need Admin or Author, status/get_results any role but Annotator, and search Admin, Author or Viewer with no profile scope. Under Strict isolation, IdpClaim or Api tenant grants, MCP is now scoped like the UI rather than refused. Action: an MCP user in no group, or whose role lacks a tool, is now refused; add them to the group they need. See Multi-Tenancy.

  • MCP client secrets are in Secrets Manager and no longer printed as stack outputs. MCPClientSecret and MCPConnectorClientSecret are replaced by MCPClientSecretArn and MCPConnectorClientSecretArn; the stack-created client appears on the MCP Configuration page as "Default (stack-created)" so it can be narrowed, rotated or revoked, and starts disabled under Strict isolation or IdpClaim/Api grants. Action: read a secret with aws secretsmanager get-secret-value --secret-id <MCPConnectorClientSecretArn> instead of the output; existing client IDs and secrets stay valid. See MCP Server.

  • MCP results that carry document text say so. get_results, status and search add a provenance field naming fields that hold document-derived text, and the MCP Connector passes that note to the client's model. See Using MCP clients.

  • Code Intelligence and External MCP agents are opt-in per conversation. External MCP agents are now off by default and enabled per conversation ("Enable External MCP agents", or enableExternalMcpAgents: true in the API); neither opt-in agent is used in a conversation once Analytics or Error Analyzer results are in it. Agent memory now expires after DataRetentionInDays. Action: deployments using External MCP agents must turn them on in each conversation that needs them. See Opt-in agents.

  • The analytics agent's tools check what they run. Its Athena tool accepts exactly one read-only statement, results reach the Code Interpreter only from the agent's last query, and Python it writes is checked for imports and file names before running. Its functions' permissions are narrowed to match. See Agent Analysis, which also now states that the Code Interpreter can reach Amazon S3 in its own Region (network reach).

  • The web UI's content policy names where each kind of resource may come from. Both hosting modes send default-src 'self' with explicit, narrower sources for images, frames, styles, fonts and media, so an HTML document in the file viewer no longer loads images from other sites.

  • Template security hardening, visible on upgrade. With the API's WAF enabled, requests are logged to aws-waf-logs-<StackName>-api. The UI and container-image CodeBuild projects log to stack-owned /<StackName>/codebuild/<ProjectLogicalId> groups (the old /aws/codebuild/... groups are left in place), and the UI build runs unprivileged. The UI and MCP content buckets gain tighter lifecycle rules, and the BDA results function loses unused Parameter Store access.

  • A field whose value is part of a longer OCR line highlights just the value in the Visual Editor, using word boxes from Textract and BDA OCR, in every language including Arabic. Backends without word boxes, and values that occur twice on a line, keep the whole-line box. Documents processed earlier keep their boxes until reprocessed. See Extraction & Confidence.

  • Test Studio "From files in S3" adds Input-bucket documents without ground truth as unlabeled instead of skipping them, matching the other add routes; an all-unlabeled create now succeeds instead of leaving a FAILED set. See Test Studio.

  • Re-extracting a test-set section deletes that section's saved draft, so the editor shows the re-extracted values instead of edits made over the old ones.

  • Configuration page: the Configuration tab is now Processing Settings, and Prompt Preview is a Preview prompts button that opens in a dialog. See Prompt Preview.

  • Multi-instance Section Detection can be set in Advanced extraction mode from the Web UI, and its descriptions say it applies there (except to sharded sections). See Extraction and confidence.

  • The Test Set Generator extension (idp-data-generator, 0.1.7) renders documents from local resources only and pins seed-data 0.0.7, which also restores scan/fax effects in newly built images. Update the extension to pick it up. See What the generator's model can do.

  • Inbox: the messages table fits the page instead of scrolling sideways on long titles.

Fixed

  • Chat history and agent memory belong to the signed-in user. Another user can no longer read, add to or delete a session, and streamed chat now applies the Agent Chat group restriction and document-chat profile scope like the REST API. Action: reload the web UI after upgrading; existing conversations continue without their earlier turns as agent context, though their history still displays. See Chat sessions belong to the signed-in user.

  • A tenant grant binds to the signing-in account's Cognito sub, so two accounts that share an email address each keep their own tenant record. setUserTenant and idp-cli tenancy set-user-tenant refuse an address held by more than one record. Action: update the PII Anonymization extension (0.2.5) if installed; for an address two accounts already share under Api or Strict Manual, see Accounts that share an email address.

  • Discovery honours configuration-profile scope. Class, policy and multi-document discovery, section auto-detection and several test-set operations refuse a profile outside the caller's allowedConfigVersions, and Replace is refused on the active profile. Discovered classes and rules are saved as plain text with URLs, images and HTML removed. See RBAC and Discovery.

  • Rendered markdown loads images only from the app's own origins, shows any other image as a link, and no longer applies embedded <style> blocks; printing a report writes the document name as text.

  • A configured Bedrock guardrail applies to agentic extraction, Chat with Document, discovery and Z3 rule translation, which previously ran without it. OpenAI GPT-5.x models do not receive it, and a warning is logged. See Bedrock Guardrails.

  • idp-cli config-validate and config-upload catch more configurations that fail at runtime. ⚠️ A configuration that validated before may now be refused: every model ID (escalation, per-class, confidence, chat, agent, discovery, evaluation and others) is checked against pricing.yaml, so a retired model such as Claude 3 Haiku is refused; the confidence prompt's placeholders are checked; a duplicated {DOCUMENT_IMAGE} or empty confidence prompt is refused; and agentic extraction is decided from extraction.mode, refusing a GPT-5.x extraction or escalation model under mode: advanced. See config-validate. (#1899, #1915, #1922)

  • Cost reports include usage that was billed but not reported. ⚠️ Reported cost goes up where it applies: failed Advanced-mode extractions (#1923), LLM evaluation comparisons, unparsed escalation calls, and the Evaluation step's Lambda usage are now metered. Escalation is reported under one context, ExtractionEscalation, in both modes. See Cost Considerations.

  • Claude Sonnet 5 cost reports use Sonnet 5's published price, two-thirds of what they charged. AWS billing is unchanged, and costs before and after the upgrade are not comparable. ⚠️ Action: a stack that saved the Pricing editor while it held the Sonnet 5 rows keeps the old rates; choose Restore default (All) on the Pricing page, then re-apply your own changes. (#1830)

  • A saved pricing table is merged over the defaults per model and unit, so models a later release adds are priced on stacks that have saved the Pricing page, and Restore default (All) is enabled for every customisation. Prices must be non-negative decimals, checked on save; a missing price is refused rather than stored as free, and the cost reports and web UI now read prices identically. See Editing the pricing table. (#1886, #1903)

  • LLM-based evaluation sends the configured model, prompts and sampling. The comparator ignored reasoning_effort, top_p and max_tokens, and a field with its own comparator config was judged by Claude Haiku 4.5 instead of evaluation.llm_method.model. Evaluations on models with an effort control now send medium by default, changing their cost and latency; an empty system or task prompt is reported as a failed evaluation. See Evaluation. (#1902, #1910)

  • Library code defaults match the system defaults. ⚠️ A deployed stack runs as before except that a numeric setting stored as null now resolves to the shipped value (for example discovery temperature 0.0, top_p 0.0), and agent fallback models are now Region-translated (an EU stack gets eu. profiles). For code that builds configuration without the merge — IDPConfig() or services handed a partial dict, as in notebooks and the standalone library — this is a model and cost change to the shipped models (for example extraction from Nova Pro to Claude Sonnet 5). See the configuration library notes.

  • Fallback models follow the stack's Region in EU and Canada. A cleared classification model, multi-document discovery's reflection report, synthetic data generation, local SDK/CLI discovery, the PII Anonymizer hook's default and the Test Set Generator's suggestion endpoint used us. profiles an EU stack cannot invoke. See EU Region Model Support.

  • A slow container-image build no longer leaves an upgrade stuck in UPDATE_ROLLBACK_FAILED. Image builds now wait up to CloudFormation's one-hour limit, and a rollback reuses the previous images instead of rebuilding; this protects rollbacks to this release or later. The Test Set Generator extension (0.1.6) waits the same way. Action: recover a stack already stuck as described in Troubleshooting. (#1824)

  • An upgrade rebuilds the multi-document discovery functions whenever their code changes, including changes only in idp_common, which previously left them on the previous release's image. (#1856)

  • A re-uploaded file no longer stays in "malware scanning" forever. With malware scanning on, a new upload under a previously processed key could have its GuardDuty verdict discarded and stay in MALWARE_SCANNING. Action: upload such a file again. See Malware scanning.

  • idp-cli shows why a document was refused at ingest under strict tenant isolation, instead of "Unknown error". See Strict mode.

  • Attaching several documents, or a zip, to a Quick Start chat works again, and Quick Start names the screens it sends people to correctly and is not offered when EnableQuickStartWidget=false. See Quick Start.

  • Adding documents to a test set from a bucket pattern no longer overwrites documents already in it, and skips matching .zip files that used to mark the set failed.

  • Test Studio's Edit Page Grouping no longer gives a new section a deleted section's fields, and the "Fields still reflect the previous class" note names the right class, follows its section and survives a reload. See Test Studio. (#1816, #1895)

  • History › Review Activity lists the legacy review's events, including section completions and Skip All Reviews. See Review History.

  • The configuration editor's code views run their background workers again, which the content policy blocked, making the UI sluggish.

  • Example notebooks no longer name end-of-life Bedrock models, so their LLM evaluation runs again, and the evaluation notebooks print the configured model. (#1924)

  • Documentation matches the product and the v0.6.14 benchmark. Configuration recommendations, cost and classification figures and the Simple-extraction list-size limit (complete to about 400 rows) now come from measured results (Configuration Guidance, extraction scaling guide); the EU model, knowledge base, evaluation, MLflow and extraction pages were corrected; and the Permission Matrix documents the Annotator role as enforced. (#1833, #1834, #1838)

Templates

  • us-west-2: https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/genai-idp/idp-main_0.6.15.yaml
  • us-east-1: https://s3.us-east-1.amazonaws.com/aws-ml-blog-us-east-1/artifacts/genai-idp/idp-main_0.6.15.yaml
  • eu-central-1: https://s3.eu-central-1.amazonaws.com/aws-ml-blog-eu-central-1/artifacts/genai-idp/idp-main_0.6.15.yaml
  • ca-central-1: https://s3.ca-central-1.amazonaws.com/aws-bigdata-blog-replica-ca-central-1/artifacts/genai-idp/idp-main_0.6.15.yaml