Fix client-v2 Dynamic column BigDecimal type inference (truncation + overflow) - #2967
Open
polyglotAI-bot wants to merge 4 commits into
Open
Fix client-v2 Dynamic column BigDecimal type inference (truncation + overflow)#2967polyglotAI-bot wants to merge 4 commits into
polyglotAI-bot wants to merge 4 commits into
Conversation
valueToColumnForDynamicType inferred the Decimal width from BigDecimal.precision() alone and forced the scale to that width's maximum. Since DecimalN(S) is Decimal(P, S) with P fixed to the width precision (9/18/38/76) and 0 <= S <= P, forcing S = P left no room for an integer part and could not represent a scale larger than P. As a result a value whose scale exceeded the width maximum was silently truncated on write, and a value with an integer part (e.g. 19.99) overflowed and threw on insert. Size the width to hold both the integer digits and the value's scale, and keep the scale as wide as the width allows (maxScale - integerDigits) so the integer part still fits; throw a clear error when the value needs more than Decimal256 precision instead of truncating. Also write the actual column scale into the Dynamic type tag (it previously wrote the width maximum), so the tag matches the serialized value. Sub-1 values that already round-tripped are inferred to the identical type and are unchanged. Fixes: #2966
Client V2 CoverageCoverage Report
Class Coverage
|
JDBC V2 CoverageCoverage Report
Class Coverage
|
JDBC V1 CoverageCoverage Report
Class Coverage
|
Client V1 CoverageCoverage Report
Class Coverage
|
…decimal-scale-inference # Conflicts: # CHANGELOG.md
…decimal-scale-inference # Conflicts: # CHANGELOG.md
TriageCategory: Summary What this impacts
Concerns
Required reviewer action
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a1c993d. Configure here.
…c inference A numerically-zero BigDecimal whose scale/exponent implies more than the maximum 76 digits (e.g. 0E-77, precision 1 / scale 77) was rejected with a ClientException, even though zero rounds to zero at any scale with no data loss and such inserts round-tripped fine before this PR introduced the over-precision check. Restrict the rejection to non-zero values; map an over-wide zero to the widest Decimal256 band, capping the integer digits so the emitted scale stays within [0, maxScale]. Addresses a review comment on PR #2967.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #2966.
When a
BigDecimalis written into aDynamiccolumn, client-v2'sSerializerUtils.valueToColumnForDynamicType(Object)chose theDecimalwidth fromBigDecimal.precision()alone and then forced the scale to that width's maximum(
Decimal32→9,Decimal64→18,Decimal128→38,Decimal256→76). BecauseDecimalN(S)is
Decimal(P, S)withPfixed to the width precision and0 ≤ S ≤ P, forcingS = Pleaves no room for an integer part and cannot represent a scale larger than
P. So alow-precision value whose scale exceeded the width maximum was silently truncated on
write, and a value carrying an integer part (e.g. a price like
19.99) overflowed andthrew on insert. Only sub-1 values whose scale fit the width round-tripped.
The fix sizes the width to hold both the integer digits and the value's scale
(
requiredPrecision = integerDigits + valueScale), then keeps the scale as wide as thewidth allows without stealing room from the integer part (
scale = maxScale − integerDigits). A value that needs more thanDecimal256precision now fails loudly witha
ClientExceptioninstead of being truncated. A second, coupled site was also wrong: theDynamictype-tag writer emitteddataType.getMaxScale()for the tag's scale byte whilethe value itself is serialized at the column's actual scale — so the tag and data
disagreed whenever the scale wasn't the width maximum. It now writes the column's scale.
Changes
client-v2SerializerUtils.valueToColumnForDynamicType: infer theDecimalwidth fromintegerDigits + scale(notprecision()alone) and emitscale = maxScale − integerDigits; throwClientExceptionwhen the value exceedsDecimal256precision.client-v2SerializerUtilsDynamictype-tag writer: write the column's actual scaleinto the tag instead of the data type's maximum scale, so the tag matches the serialized
value.
CHANGELOG.md: bug-fix entry.Test
DataTypeTests.testDynamicColumnPreservesBigDecimalValue(integration,@DataProvider):inserts each
BigDecimalinto aDynamiccolumn (placed mid-row with a trailingfixed-width
Int32column) and asserts the value round-trips losslessly and the trailingcolumn stays intact (guards against a wrong width shifting following bytes). Covers the
previously-truncated cases, the previously-overflowing integer-part cases, negative scale
(
1E3), and aDecimal256value with an integer part. Fails onmain(2 truncationDataTypeTests.testDynamicColumnFractionalDecimalRepresentationUnchanged(integration):contrast case pinning that a sub-1 value (
0.5) still reads back as0.500000000(scale 9) — proving the fix does not change values that already round-tripped.
SerializerUtilsTest.testValueToColumnForDynamicTypeSizesDecimal(unit,@DataProvider):pins the exact inferred width and scale across all four widths, negative scale,
scale > precision, and the width boundaries.
SerializerUtilsTest.testValueToColumnForDynamicTypeRejectsOversizedDecimal(unit):asserts
ClientExceptionfor a value needing more thanDecimal256precision.Focused runs (devbox, ClickHouse 26.5):
DataTypeTests90/90,SerializerUtilsTest71/71,
InsertTests30/30 — all green; no existing tests modified.changes_checklist.md— “String, SQL, or serialized output changed”is stated below and pinned by a contrast test.
scale ≥ valueScalein every width (band selection guarantees
integerDigits + valueScale ≤ maxScale), so notruncation; unrepresentable values throw rather than round.
just local string equality — the type tag and serialized bytes are exercised through the
real insert/read path.
docs/features.mdcross-checked: noDynamic/Decimalinference behavior isdocumented there, so no features.md change is needed for this bug fix.
Pre-PR validation gate
Dynamic→ read back, CH 26.5)serializeData→case Dynamic), proven by an integration round-tripCompatibility impact
Non-breaking. Values that already round-tripped (sub-1
BigDecimals) are inferred to theidentical
Decimaltype and read back byte-for-byte as before. The change only affectsvalues that previously failed: over-scale values (were truncated) and values with an
integer part (threw on insert) now round-trip correctly. No public API change.