[FEATURE REQ] Expose stringEncoding on the convenience analyze overloads, Azure.AI.ContentUnderstanding (.NET, preview) #445
Unanswered
Downloading (Gabegi)
asked this question in
Product Feedback and Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Technical Feedback
The REST operation
/contentunderstanding/analyzers/{analyzerId}:analyzeaccepts astringEncodingquery parameter to select the unit for span offsets (utf8 bytes,utf16 code units, code points). In the .NET client this parameter is only reachable
through the protocol overload, which returns
Operation<BinaryData>. Theconvenience overload returns the typed
AnalyzeResultbut offers no way to setthe encoding, so callers get the service default.
Since C#
stringindexing uses utf16 code units, consumers of the typed API getoffsets on a different ruler and
Substring(offset, length)misindexes anycontent after the first non BMP character (emoji, CJK extensions, math symbols).
The mismatch is silent: nothing throws, spans just drift. This corrupts any
span based chunking or markdown slicing pipeline built on the typed models.
Desired Outcome
stringEncoding(or an equivalent enum) available on the typed analyze overloador its options type, ideally defaulting to utf16 for .NET so typed results index
correctly into C# strings.
Current Workaround
A custom
HttpPipelinePolicyregistered viaClientOptions.AddPolicy(..., HttpPipelinePosition.PerCall)that appendsstringEncoding=utf16to:analyzerequests. It works because both overloads share the pipeline, but it is fragile
and undiscoverable.
All reactions