Skip to content

feat: add Azure Content Understanding with resumable lakehouse writes - #2698

Open
Rana Singh (ranadeepsingh) wants to merge 4 commits into
microsoft:masterfrom
ranadeepsingh:feat/content-understanding
Open

feat: add Azure Content Understanding with resumable lakehouse writes#2698
Rana Singh (ranadeepsingh) wants to merge 4 commits into
microsoft:masterfrom
ranadeepsingh:feat/content-understanding

Conversation

@ranadeepsingh

@ranadeepsingh Rana Singh (ranadeepsingh) commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

What changes are proposed in this pull request?

Add Azure Content Understanding to SynapseML through a Scala transformer and
generated Python API. Analyze PDFs, DOCX files, and other service-supported
inputs. Save each accepted operation and completed result to a Fabric lakehouse
or Spark table so a later document failure does not discard earlier work.

The implementation calls the Content Understanding REST API directly through
SynapseML's JVM HTTP infrastructure. It does not require a Content Understanding
Python SDK. The default is GA 2025-11-01; 2026-06-01-preview is an explicit
opt-in.

Python quickstart

import notebookutils
from synapse.ml.services.contentunderstanding import ContentUnderstanding

key = notebookutils.credentials.getSecret(
    "https://<your-vault>.vault.azure.net/", "<your-secret-name>"
)
documents = (
    spark.read.format("binaryFile")
    .load("Files/documents/")
    .selectExpr("path AS documentId", "content")
)

analyzer = (
    ContentUnderstanding()
    .setEndpoint("https://<your-resource>.cognitiveservices.azure.com")
    .setSubscriptionKey(key)
    .setAnalyzerId("prebuilt-read")
    .setDocumentBytesCol("content")
    .setDocumentNameCol("documentId")
    .setOutputCol("analysis")
    .setErrorCol("requestError")
)

# Lazy DataFrame API, like other SynapseML cognitive-service stages:
results = analyzer.transform(documents)

# Eager processing with per-document commits:
latest = analyzer.writeToTable(
    documents,
    idCol="documentId",
    tableName="content_understanding_operations",
    format="delta",
    batchSize=1,
)

# If a later document fails, read already-committed work without service calls:
partial = analyzer.readTable(spark, "content_understanding_operations")

# Rerun the same manifest against the same journal to resume saved handles:
latest = analyzer.writeToTable(
    documents, "documentId", "content_understanding_operations"
)

Use immutable document versions and stable unique string IDs. The example uses
the path as an ID, so files at those paths must not change during processing or
recovery. Configure setDocumentUrlCol instead of setDocumentBytesCol when the
service can read the source URL directly. Never configure both sources.

API signatures

All methods below are instance methods on ContentUnderstanding.

Python signature Return value
transform(dataset) DataFrame with the input columns, response struct at outputCol, and errorCol. Lazy until a Spark action.
writeToTable(dataset, idCol, tableName, format="delta", batchSize=1) DataFrame with the latest state of every ID in the destination journal. Processing and persistence are eager.
writeToPath(dataset, idCol, path, format="delta", batchSize=1) Same behavior, with a lakehouse/filesystem path instead of a catalog table.
readTable(spark, tableName) Latest persisted state per ID, without contacting the service.
readPath(spark, path, format="delta") Latest persisted state from a path, without contacting the service.
createAnalyzer(definition: dict | str, allowReplace=False) -> str Analyzer definition as JSON. Explicit driver-only provisioning.
getAnalyzer() -> str Current analyzer definition as JSON.

The response fields are operationLocation, id, status, httpStatus,
rawResponse, and error. Writer results expose those fields at the top level,
plus documentId, requestHash, and sequence. rawResponse retains the full
operation JSON, including usage, extraction fields, warnings, and preview
metadata. Input document bytes are not copied into the journal.
Writers accept existing input columns named outputCol or errorCol because
they return a separate journal schema. transform still rejects those
collisions rather than overwrite input.

Scala exposes the same operations:

def transform(dataset: Dataset[_]): DataFrame
def writeToTable(dataset: Dataset[_], idCol: String, tableName: String,
                 format: String = "delta", batchSize: Int = 1): DataFrame
def writeToPath(dataset: Dataset[_], idCol: String, path: String,
                format: String = "delta", batchSize: Int = 1): DataFrame
def readTable(spark: SparkSession, tableName: String): DataFrame
def readPath(spark: SparkSession, path: String, format: String = "delta"): DataFrame
def createAnalyzer(definitionJson: String, allowReplace: Boolean): String
def getAnalyzer(): String

Configure request options

Service options accept scalar and column setters, such as setRange("1-2")
or setRangeCol("pageRange"). This also applies to the analyzer ID, input
source, name, MIME type, model deployment map, string encoding, processing
location, API version, and authentication. Polling budgets, timeouts, response
size, and transform concurrency are scalar execution controls.

invoice = (
    analyzer.copy({})
    .setAnalyzerId("prebuilt-invoice")
    .setRangeCol("pageRange")
    .setModelDeployments(
        {
            "prebuilt-analyzer-completion": "<your-completion-deployment>",
            "prebuilt-analyzer-embedding": "<your-embedding-deployment>",
        }
    )
)

# Save accepted handles without waiting for their results:
analyzer.setOperationMode("submit").writeToTable(
    documents, "documentId", "content_understanding_operations"
)
# Later, poll the saved operations without a new POST:
analyzer.setOperationMode("analyze").writeToTable(
    documents, "documentId", "content_understanding_operations"
)

Poll-only transform uses setOperationMode("poll") and
setOperationLocationCol("handle"). It needs neither the original document
columns nor the original API-version column; the saved URL supplies its version.
The durable writers require the original manifest to verify request identity.

Custom extraction, layout, and segmentation settings belong in an analyzer
definition:

custom = analyzer.copy({}).setAnalyzerId("purchase-order-v1")
custom.createAnalyzer(
    {
        "baseAnalyzerId": "prebuilt-document",
        "config": {"returnDetails": True},
        "fieldSchema": {
            "name": "PurchaseOrder",
            "fields": {
                "Supplier": {
                    "type": "string",
                    "description": "The supplier's legal business name.",
                }
            },
        },
    },
    allowReplace=False,
)

Provisioning is never implicit in transform. Resource defaults and model
deployments remain administrator-owned. The service's DefaultsNotSet error
is surfaced rather than repaired by changing shared configuration.

REST requests under the hood

Action REST request
Analyze/submit POST {endpoint}/contentunderstanding/analyzers/{analyzerId}:analyze?api-version={version}
Poll GET {Operation-Location}
Create/replace analyzer PUT {endpoint}/contentunderstanding/analyzers/{analyzerId}?api-version={version}&allowReplace={bool}
Read analyzer GET {endpoint}/contentunderstanding/analyzers/{analyzerId}?api-version={version}

The analyze body uses inputs: [{url: ...}] or inputs: [{data: "<base64>"}],
with optional name, mimeType, and range. modelDeployments is a top-level
body property. stringEncoding and processingLocation are query parameters.
Analyzer creation polls its management operation before returning the definition.
POST/PUT calls are not automatically retried, redirects are disabled, and GET
polling has a bounded retry budget.

Persistence contract and limits

  • The journal is append-only. An accepted handle is committed before polling;
    each completed document or explicit PDF range gets its own result commit.
  • The writer is sequential and supports one writer per destination. batchSize
    bounds collected input rows, not the number of results per commit. A single
    document and response must still fit in memory.
  • Request hashes reject reuse of an ID with changed request content or options.
    Credential rotation is allowed. Changes behind an unchanged URL, analyzer
    name, or deployment name cannot be detected.
  • Submission 401/403/429 without a handle stops before recording that ID.
    Indeterminate submissions are recorded as Unknown and are not automatically
    resubmitted. Polling credential/transport/response errors retain saved handles.
    Missing or expired results become ResultUnavailable, allowing later IDs to
    proceed.
  • A crash between POST acceptance and the first journal commit can still cause
    duplicate submission. This is not an exactly-once external-service guarantee.
    Saved handles also remain subject to the service's 24-hour result retention.

Service experiments that informed the design

Experiments used only synthetic inputs and existing authorized resources.
No shared defaults or model deployments were changed.

Observation Design consequence
A Running response already contains an empty result; HTTP 200 can carry Failed Preserve service status separately from HTTP status. Only Succeeded is successful completion.
Usage and preview fields vary across analyzers and formats Keep the entire operation JSON instead of imposing a fixed extraction schema.
PDF range 2 retained original page 2 and reported one basic document page of usage Use explicit PDF range rows with independent durable IDs.
DOCX GA returned both page markers and table text as Markdown with documentPagesMinimal usage Support DOCX bytes without assuming a pages array or a PDF-shaped usage object.
DOCX range="2" still returned the whole document Document whole-DOCX durability. Do not promise DOCX page-level checkpoints.
Preview DOCX layout retained a metadata object Preserve unknown and preview fields without requiring wrapper changes.
Ranges 1-2 and 319-320 of a synthetic 320-page PDF succeeded in earlier experiments Expose opt-in ranges, while retaining service input/response limits and warning about changed extraction context.

How is this patch tested?

Remediation head: f3a7963c217677e9b6564cc6736495e0f1bcdce4.

  • 54 offline Content Understanding Scala tests passed with no skipped or canceled cases.
  • Root FuzzingTest passed all 10 active tests, including the four previously failing checks. One pre-existing getter/setter test remains ignored.
  • PipelineTestCoverageSuite passed, confirming CI still selects the separated suites.
  • Cognitive main/test compilation and scalastyle, core/cognitive code generation, and Python merging passed.
  • Four regenerated Python integration tests passed. The five live tests correctly skipped without explicit credentials.
  • Black 22.3.0 passed for all changed Python test files.
  • Exact-commit Fabric execution passed all five checked-in PDF/DOCX E2E tests with no skips, failures, or errors, plus the 320-page PDF/Delta recovery scenario.
  • Full Azure validation passed all 64 jobs on this head. One failed-jobs-only retry recovered two Azure certificate failures without code or configuration changes.

Four configuration/recovery regressions failed before the fixes and pass
afterward, including the writer's rejection of existing output/error column
names. Both table and path writers now accept those inputs, while transform
and transformSchema still reject overwrites before any service call.
A 3,000-row journal-plan probe with 1,000 document IDs confirmed that
only the two selected IDs' metadata reaches the driver. Spark already pushed
the original filter below the window; the reordered code produces an identical
optimized plan. The change makes that intent explicit, not a claimed speedup.

The reported Azure failure was a PR-specific registration defect, not a flaky
service call. Build 234599208 failed four core FuzzingTest checks because
ContentUnderstanding had no typed framework fuzzer. The exact-base master
build passed those four checks. This revision adds
ContentUnderstandingFuzzingSuite instead of exempting the new class.

Azure CI status

Build 234646332
tested PR merge 3dda7c401f3d9ca4f5e104abadd2f55ee193921e, whose parents are
base 8c7143875c843c649a817cf3e8ba9c7bee23689c and this head
f3a7963c217677e9b6564cc6736495e0f1bcdce4.

All four previously failing framework assertions passed in test run
1096850611. Azure also passed all 54 Content Understanding Scala tests, the
four Python integration tests, and the generated Python constructor test.
The five credential-gated live cases skipped in Azure and ran successfully
in the separate Fabric execution below.

Two of the 64 jobs initially failed outside Content Understanding:

Job Actual failure
UnitTests image Azure task initialization received a certificate for *.azureedge.net instead of msdata.visualstudio.com; no image tests started. Logs 915:39,69 and 918:15.
PythonTests deep-learning-hf All four tests passed, but test-result and coverage uploads failed TLS certificate validation. Logs 1595:1997, 1600:94 and 1621:21-22.

A single failed-jobs-only retry was requested on the same build at
2026-09-05 11:52 UTC and completed successfully at 12:21 UTC. The other 62 jobs
retained their passing first attempts. Image tests ran with six passes and two
pre-existing ignored cases; all four HF tests passed. Both jobs successfully
published their test results and coverage. All 64 jobs and all 65 Azure check
entries are now successful on the unchanged final source.

No source, pipeline, credentials, or TLS-validation settings were changed to
recover those infrastructure failures.

The line-by-line audit covered the REST protocol, authentication, persistence,
schema, generated Python, tests, and examples. All three review threads are
resolved. The final-head automated review has no active findings; the suppressed
suggestion about duplicating Spark read validation was
investigated and documented.
Maintainer review is still required; this PR has not been merged.

Exact-artifact Fabric evidence

The run used fabric-spark-cli and generated Python from this PR, not the
runtime-bundled SynapseML package. Runtime class-source checks verified the
loaded core and cognitive jar hashes.

  • Application: application_1788607300979_0001
  • Spark runtime: 3.5.5.5.4.20260807.1
  • Source commit: f3a7963c217677e9b6564cc6736495e0f1bcdce4
  • Both the scratch notebook and lakehouse were deleted after output capture.
Artifact SHA-256
Core jar ae46e713ae2335f676ad2a1c475428209811e1d485c6b8a2170f5f6a053a6633
Cognitive jar 4af886a2cfaf26e64f04bafd4edcc9bbe2b0814d88f30c6270b5c6371aeee65a
Generated Python and checked-in E2E suite archive 71afe0c016ca446c2511d7c23a1375befbb2efabc1d1d66519d060f84a7d2947

The five dedicated cases exercised GA PDF ranges, GA DOCX text/table extraction,
preview DOCX metadata, mixed DOCX/PDF table recovery, and submit-only path
resumption. The mixed case committed the DOCX result before a later PDF result
exceeded a deliberate response cap, then resumed the same operation handles.
The table retained exactly four journal rows. Both live writer cases also used
input columns whose names matched the stage's output/error settings.

The 320-page synthetic PDF scenario independently preserved an earlier result
after a later response exceeded its cap, resumed original pages 319-320 without
replacing either handle, and retained four Delta journal rows. Repeating a
completed path write retained two rows. Preview PDF metadata, invoice extraction
with explicit model mappings, and persistence of a definite service failure also
passed.

Content Understanding tests now have dedicated files:

File Coverage
ContentUnderstandingSuite.scala Public request API, REST protocol, status/errors, schema, parameters, copy/save/load, and analyzer management.
ContentUnderstandingWriterSuite.scala Table/path writes, request identity, IDs, submit-only mode, rejection and unavailable-result behavior.
ContentUnderstandingRecoverySuite.scala Oversized/chunked/gzip results, malformed responses, unknown acceptance, and recovery after polling errors.
ContentUnderstandingFileSystemSuite.scala Session-specific filesystem configuration.
ContentUnderstandingFuzzingSuite.scala Framework experiment, serialization, generated Python/R registration, and getter/setter coverage, using offline fixtures.
test_ContentUnderstanding.py Generated Python wrapper against a loopback REST fixture.
test_ContentUnderstandingE2E.py Separate opt-in Azure PDF/DOCX and durable-write cases.

The Scala suites remain in the existing document-service CI group. The live
suite skips when explicit credentials are absent. It creates only uniquely
named scratch output destinations and never provisions service resources.

Does this PR change any dependencies?

  • No.
  • Yes.

No dependency pins, workflows, release tooling, existing public JVM signatures,
or test exemptions were changed.

Does this PR add a new feature? If so, have you added samples on website?

  • No.
  • Yes.

The Content Understanding guide
includes Python/Scala signatures, REST mappings, Fabric credential handling,
PDF/DOCX guidance, analyzer configuration, recovery examples, and limits. It is
linked in the website sidebar.

Related Issues/PRs

No separate issue was supplied.

## Summary
Add a Scala-first Azure Content Understanding transformer and generated Python
API. Support current GA and opt-in preview requests, explicit custom analyzer
provisioning, and append-only operation journals in Spark tables and lakehouse
paths. Add offline service, persistence, and generated-wrapper coverage plus
Python/Scala usage documentation.

## Prompting Intent
The engineer requested an easy-to-configure Content Understanding integration
using current APIs that can run in Fabric. They also requested a persistence
layer that retains completed work when large-document processing fails, and
real-service experiments to guide the API and durability design.

## Linked Sources
- Analyze API: https://learn.microsoft.com/en-us/rest/api/contentunderstanding/content-analyzers/analyze?view=rest-contentunderstanding-2025-11-01
- Result API: https://learn.microsoft.com/en-us/rest/api/contentunderstanding/content-analyzers/get-result?view=rest-contentunderstanding-2025-11-01
- Analyzer provisioning: https://learn.microsoft.com/en-us/rest/api/contentunderstanding/content-analyzers/create-or-replace?view=rest-contentunderstanding-2025-11-01
- API releases: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/whats-new
- Model mappings: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/concepts/models-deployments
- Service limits: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/service-limits
- Result retention: https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/content-understanding/data-privacy
- Fabric credentials: https://learn.microsoft.com/en-us/fabric/data-engineering/notebookutils/notebookutils-credentials

## Rationale
Use 2025-11-01 by default and make 2026-06-01-preview explicit. Preserve the
complete operation JSON because live responses contain dynamic extraction
fields, preview metadata, and top-level usage, and Running responses can
already contain an empty result.

Commit each operation handle before polling and each result independently.
Keep uncertain submissions visible without automatically repeating billable
POSTs. Canonical request hashes prevent silently reusing IDs for changed
analysis requests. Driver-side single-writer orchestration trades throughput
for bounded input batches and durable per-operation commits. Explicit ranges
allow large-document progress without silently changing extraction context.

Do not terminalize definite authentication or throttling rejections; stop so a
later invocation can retry the same ID. Treat missing or expired result handles
as distinct terminal retrieval failures so later documents can still complete.
Keep HTTP 408/5xx submission outcomes uncertain rather than assuming that an
error response proves the service did not accept a billable request.

Use existing HTTP and Spark infrastructure without new dependencies. Require
an explicit endpoint and validate operation URLs before forwarding credentials.
Do not mutate resource defaults or change existing public JVM signatures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary
Leave response stream ownership with the enclosing HTTP response. Add large
chunked/gzip recovery coverage and a deterministic faulting-close regression.
Include the safe exception type in transport diagnostics.

## Prompting Intent
The engineer requested a Content Understanding integration that preserves
completed work for large-document jobs and runs in Fabric. During real-runtime
validation, a response-bound scenario produced a generic transport failure,
requiring investigation before publishing the feature.

## Linked Sources
- Content Understanding result API: https://learn.microsoft.com/en-us/rest/api/contentunderstanding/content-analyzers/get-result?view=rest-contentunderstanding-2025-11-01
- HttpClient response resource ownership: https://hc.apache.org/httpcomponents-client-4.5.x/current/tutorial/html/fundamentals.html

## Rationale
Closing an already-aborted entity stream in finally can replace the original
ResponseTooLarge exception with an IOException. The HTTP response already owns
and closes that stream, so remove the redundant close rather than suppressing
arbitrary cleanup errors. A faulting stream reproduces the exception masking;
public writer cases cover chunked and compressed responses and resumption.
Retain only the IOException class name, not potentially sensitive exception
messages, when distinguishing transport failures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings September 5, 2026 03:46
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

  • fix: Fix LightGBM crashes with empty partitions
  • feat: Make HTTP on Spark back-offs configurable
  • docs: Update Spark Serving usage
  • build: Add codecov support
  • perf: improve LightGBM memory usage
  • refactor: make python code generation rely on classes
  • style: Remove nulls from CNTKModel
  • test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

@ranadeepsingh

Copy link
Copy Markdown
Collaborator Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The durable writer currently computes latest(journal.read()) over the full journal before filtering to batch IDs, which can cause significant unnecessary window/shuffle work as the journal grows.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1 Medium severity

New issues introduced by this change (1)
Severity Finding
Medium severity cognitive/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​contentunderstanding/​ContentUnderstandingWriter.scala — In the writer loop, latest(journal.read()) computes a window over the entire journal and only…
What changed in this PR

Adds a new SynapseML Cognitive Services integration for Azure Content Understanding, providing a Scala-first SparkML transformer with generated Python wrappers plus a durable, resumable journal writer for lakehouse/table persistence. This expands the cognitive module’s document/AI service stages with a reliability-focused write/resume workflow and accompanying docs/tests.

Changes:

  • Introduces ContentUnderstanding transformer with request construction, validation, polling/submission modes, and analyzer provisioning APIs.
  • Adds ContentUnderstandingWriter and persistence helpers (writeToTable/writeToPath/readTable/readPath) implementing append-only journaling and resumption.
  • Adds comprehensive Scala + Python tests and a new docs page linked into the website sidebar.
File Description
website/​sidebars.js Adds the new Content Understanding guide to the docs sidebar navigation.
docs/​Explore Algorithms/​AI Services/​Content Understanding.md New end-to-end documentation for configuration, durability semantics, and Scala/Python usage.
cognitive/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​contentunderstanding/​ContentUnderstanding.scala New SparkML stage implementing request/poll logic, validation, and analyzer provisioning APIs.
cognitive/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​contentunderstanding/​ContentUnderstandingParams.scala Defines service parameters, defaults, and Python wrapper overrides for bytes + param transfer.
cognitive/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​contentunderstanding/​ContentUnderstandingPersistence.scala Exposes durable writer/read helpers on the stage and emits Python convenience methods.
cognitive/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​contentunderstanding/​ContentUnderstandingProtocol.scala Implements bounded HTTP exchange, retry-delay parsing, safe operation-location validation, and decoding.
cognitive/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​contentunderstanding/​ContentUnderstandingWriter.scala Driver-side single-writer journaling orchestration for submit/poll/append + resume behavior.
cognitive/​src/​test/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​form/​contentunderstanding/​ContentUnderstandingWriterSuite.scala Scala tests for journal durability/resumption, ID validation, and failure/rejection handling.
cognitive/​src/​test/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​form/​contentunderstanding/​ContentUnderstandingSuite.scala Scala tests for request correctness, schema validation, error preservation, polling semantics, and provisioning.
cognitive/​src/​test/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​form/​contentunderstanding/​ContentUnderstandingStub.scala HTTP stub infrastructure for deterministic offline testing of service behaviors and transport edge cases.
cognitive/​src/​test/​python/​synapsemltest/​services/​test_ContentUnderstanding.py Python integration tests validating generated wrapper behavior, persistence, and analyzer provisioning calls.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

## Summary
Register ContentUnderstanding with the framework's typed transformer fuzzers,
repairing the four failing core CI registration checks. Separate writer
recovery, filesystem, fuzzing, and live PDF/DOCX coverage into dedicated files.

Honor submit-only persistence, let poll mode use the version in its saved
operation URL, and resolve output filesystems from the Spark session. Express
the batch-ID journal filter before latest-state selection. Remove request data
from the Python test fixture's response headers. Document the exact public
signatures, raw REST calls, and the observed whole-document behavior of DOCX.

## Prompting Intent
The engineer asked to investigate failing tests in microsoft#2698,
make Content Understanding-specific unit and end-to-end failures identifiable,
review every changed line for necessity and bugs, validate PDFs and DOCX
against Azure, and make the PR's API and REST usage easy to understand.
No Azure DevOps work-item ID was supplied.

## Linked Sources
- Pull request and request context: microsoft#2698
- Failing Azure validation: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=234599208
- Exact-base comparison: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=234571994
- Test-header finding: microsoft#2698 (comment)
- Journal-filter finding: microsoft#2698 (comment)
- REST API: https://learn.microsoft.com/rest/api/contentunderstanding/content-analyzers/analyze?view=rest-contentunderstanding-2025-11-01
- Service limits: https://learn.microsoft.com/azure/ai-services/content-understanding/service-limits
- API release notes: https://learn.microsoft.com/azure/ai-services/content-understanding/whats-new

## Rationale
Ordinary TestBase suites did not satisfy the repository-wide experiment,
serialization, Python and R fuzzer registration contract. A typed fuzzer
repairs integration without suppressing checks or moving tests outside the
existing CI selector. Scoped loopback fixtures keep offline tests independent
of service credentials; real-service cases are explicitly opt-in.

Three before-fix regressions establish the recovery/configuration defects.
Submit mode now commits only accepted state, leaving result collection to a
later analyze call. Poll-only requests do not require columns unrelated to the
saved handle. Session-derived Hadoop configuration honors managed-runtime and
session-specific filesystem settings.

Live synthetic requests confirmed PDF range handling and DOCX text/table
extraction, but a DOCX page range still returned the whole file. The guide
therefore avoids promising DOCX page-level checkpoints. Existing public JVM
signatures, journal schema, response retention, request identity checks,
single-writer constraints and uncertain-submission protections remain intact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings September 5, 2026 10:31
@ranadeepsingh

Copy link
Copy Markdown
Collaborator Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The durable writer currently reuses transform-oriented schema validation and can unnecessarily reject valid inputs that already contain outputCol/errorCol.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1 Medium severity

New issues introduced by this change (1)
Severity Finding
Medium severity cognitive/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​contentunderstanding/​ContentUnderstandingWriter.scala — ContentUnderstandingWriter.validateInput calls analyzer.validateInputSchema(dataset.schema), which…
Issues resolved since last review (1)
Severity Finding
Medium severity cognitive/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​contentunderstanding/​ContentUnderstandingWriter.scala — In the writer loop, latest(journal.read()) computes a window over the entire journal and only… View resolved comment

## Summary
Allow durable table and path writers to consume input columns with the same
names as the stage's output and error columns. Keep overwrite protection in
the transform and transformSchema paths, which actually add those columns.

## Prompting Intent
Investigate failing tests in microsoft#2698, keep Content Understanding
tests in dedicated files, review every change, validate PDF and DOCX processing
with the Azure service, and explain the public API and raw REST contract.

## Linked Sources
- Pull request: microsoft#2698
- Current-head review: microsoft#2698 (comment)
- Guide: docs/Explore Algorithms/AI Services/Content Understanding.md

## Rationale
Both durable writers reuse request-schema validation but return a separate
operation journal rather than transformed input rows. Move the existing
overwrite guard to getInternalTransformer instead of dropping input columns,
adding a validation bypass, or changing public method signatures. Keep request
type, source, endpoint, and parameter checks shared. Add a before/after writer
regression plus explicit transform and transformSchema rejection coverage, and
exercise the same input-name case in the dedicated live PDF/DOCX writer tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings September 5, 2026 11:03
@ranadeepsingh

Copy link
Copy Markdown
Collaborator Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The PR introduces a substantial new public feature spanning Scala REST/protocol logic, durable persistence semantics, generated Python surface area, and multiple test suites, which warrants final human review and full CI validation.

Review tier: Lite
Findings: None

Issues resolved since last review (2)
Severity Finding
Medium severity cognitive/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​contentunderstanding/​ContentUnderstandingWriter.scala — ContentUnderstandingWriter.validateInput calls analyzer.validateInputSchema(dataset.schema), which… View resolved comment
Medium severity cognitive/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​services/​contentunderstanding/​ContentUnderstandingWriter.scala — In the writer loop, latest(journal.read()) computes a window over the entire journal and only… View resolved comment
Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/contentunderstanding/ContentUnderstandingWriter.scala:47

  • readTable/readPath currently accept null/blank destination names and (for readPath) any format string, even though writes explicitly restrict formats to delta/parquet. Adding the same input/format validation here would provide clearer, earlier errors and keep the public read API consistent with the write API.

@ranadeepsingh

Copy link
Copy Markdown
Collaborator Author

Reviewed the suppressed reader-validation suggestion on f3a7963. The read helpers intentionally delegate destination/provider resolution to Spark and then validate the exact Content Understanding journal schema. An eight-case probe through the generated public API confirmed that null/empty/blank destinations and unknown/null providers raise rather than return data; no Content Understanding endpoint or credentials were configured. Spark reports parser/path/provider errors, including its existing NullPointerException for a null table name or provider. Additional wrapper checks would improve those two messages, but are not needed to reject invalid reads or protect the journal. I kept Spark read-provider resolution rather than imposing the write-only delta/parquet restriction on reads. Writes remain restricted because they must provide supported append semantics. All active review threads are resolved; full current-head CI and the exact-artifact Fabric run are still in progress.

@ranadeepsingh

Copy link
Copy Markdown
Collaborator Author

Final-head Azure build 234646332 verified the repair: all four formerly failing framework assertions, all 54 Content Understanding Scala tests, four Python integration tests and the generated constructor passed. 62/64 jobs succeeded. The image job failed during Azure task initialization before tests started because msdata.visualstudio.com received a *.azureedge.net certificate; the HF tests passed but result/coverage uploads hit the same TLS problem. I requested one failed-jobs-only retry of the same build/source using the stage retry API, with forceRetryAllJobs=false. No TLS checks were bypassed and no source or pipeline settings changed. Exact-head Fabric PDF/DOCX and Delta recovery results are already in the PR description.

@ranadeepsingh

Copy link
Copy Markdown
Collaborator Author

The bounded infrastructure retry completed successfully. Build 234646332 is green on merge 3dda7c4, testing head f3a7963. All 64 Azure jobs and 65 Azure check entries succeeded. Only the two previously failed jobs reran: image executed six passing tests with two pre-existing ignored cases; HF executed four passing tests. Both published test results and coverage successfully. The Content Understanding regression, generated API, and exact-artifact Fabric PDF/DOCX/Delta evidence is in the updated description. No source or TLS-validation changes were needed for the retry. Maintainer review remains required; the PR is not merged.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.41894% with 41 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.68%. Comparing base (8c71438) to head (f3a7963).

Files with missing lines Patch % Lines
...ntunderstanding/ContentUnderstandingProtocol.scala 90.16% 24 Missing ⚠️
...es/contentunderstanding/ContentUnderstanding.scala 93.71% 11 Missing ⚠️
...tentunderstanding/ContentUnderstandingParams.scala 95.83% 4 Missing ⚠️
...nderstanding/ContentUnderstandingPersistence.scala 85.71% 1 Missing ⚠️
...tentunderstanding/ContentUnderstandingWriter.scala 99.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2698      +/-   ##
==========================================
+ Coverage   87.52%   87.68%   +0.16%     
==========================================
  Files         341      346       +5     
  Lines       21035    21658     +623     
  Branches     2232     2277      +45     
==========================================
+ Hits        18410    18990     +580     
- Misses       2625     2668      +43     
Files with missing lines Coverage Δ
...nderstanding/ContentUnderstandingPersistence.scala 85.71% <85.71%> (ø)
...tentunderstanding/ContentUnderstandingWriter.scala 99.00% <99.00%> (ø)
...tentunderstanding/ContentUnderstandingParams.scala 95.83% <95.83%> (ø)
...es/contentunderstanding/ContentUnderstanding.scala 93.71% <93.71%> (ø)
...ntunderstanding/ContentUnderstandingProtocol.scala 90.16% <90.16%> (ø)

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants