Optionally disable `validate_decimal_precision` check in `DecimalBuilder.append_value` for interop test #1767

viirya · 2022-05-31T02:19:41Z

Which issue does this PR close?

Closes #1766.

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

codecov-commenter · 2022-05-31T03:09:48Z

Codecov Report

Merging #1767 (5665071) into master (5cf06bf) will increase coverage by 0.17%.
The diff coverage is 97.61%.

@@            Coverage Diff             @@
##           master    #1767      +/-   ##
==========================================
+ Coverage   83.32%   83.49%   +0.17%     
==========================================
  Files         196      196              
  Lines       55961    55954       -7     
==========================================
+ Hits        46627    46719      +92     
+ Misses       9334     9235      -99

Impacted Files	Coverage Δ
arrow/src/ffi.rs	`86.97% <ø> (ø)`
arrow/src/ipc/writer.rs	`80.37% <ø> (ø)`
arrow/src/array/data.rs	`83.95% <96.42%> (+0.29%)`	⬆️
arrow/src/array/array_binary.rs	`93.22% <100.00%> (-0.09%)`	⬇️
arrow/src/array/builder.rs	`86.79% <100.00%> (-0.01%)`	⬇️
arrow/src/array/transform/mod.rs	`87.41% <100.00%> (+0.44%)`	⬆️
arrow/src/csv/reader.rs	`89.89% <100.00%> (ø)`
arrow/src/ipc/reader.rs	`90.84% <100.00%> (+2.11%)`	⬆️
parquet/src/util/cursor.rs	`63.86% <0.00%> (-13.45%)`	⬇️
parquet/src/util/io.rs	`88.33% <0.00%> (-1.67%)`	⬇️
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5cf06bf...5665071. Read the comment docs.

tustvold · 2022-05-31T08:17:58Z

I'm not sure how I feel about this, the promise of the builders is they won't let you construct an ArrayData that would fail validation, and can therefore elide this validation. I think this change violates that?

Tbh this feels like a bug in the C++ implementation... Perhaps we should raise a ticket to at the very least get context?

viirya · 2022-05-31T08:43:19Z

In C++, the similar check is done in validating ArrayData, not during appending values into the builder. I'm not sure which one is more correct under Arrow's context.

I'm trying to remove the check and (not committed yet) put the precision check into the validation of ArrayData (this is what C++ Arrow does).

In one way, as we don't do full validation when finishing the builder, it seems this check is necessary if it is the promise of the builder to always build a valid ArrayData. Then the check is needed and it seems C++ needs to add the check there. I will open a ticket there and try to get some feedbacks.

alamb

Getting Rust to follow the C++ implementation seems like a good idea to me. The ticket does a good job of explaining this.

cc @liukun4515 who originally wrote these checks, I think

alamb · 2022-05-31T11:50:55Z

arrow/src/array/array_binary.rs

@@ -1497,38 +1497,6 @@ mod tests {
        assert_eq!(16, decimal_array.value_length());
    }

-    #[test]
-    fn test_decimal_append_error_value() {


I recommend we change this test to show that it is ok to store these values in a decimal rather than removing it completely (aka change the test to validate there is no error).

Changed the test. As I move the precision check to ArrayData full validation, I add another test for that too.

alamb · 2022-05-31T11:54:35Z

In one way, as we don't do full validation when finishing the builder, it seems this check is necessary if it is the promise of the builder to always build a valid ArrayData. Then the check is needed and it seems C++ needs to add the check there. I will open a ticket there and try to get some feedbacks.

I think this is the right thing to do -- thank you @viirya

nevi-me

LGTM after @alamb's suggestion

viirya · 2022-05-31T17:16:40Z

Thanks @alamb @nevi-me @tustvold. I've opened a ticket https://issues.apache.org/jira/browse/ARROW-16696 to hear some feedback there.

viirya · 2022-05-31T17:22:45Z

arrow/src/array/data.rs

@@ -999,6 +999,27 @@ impl ArrayData {

    pub fn validate_dictionary_offset(&self) -> Result<()> {
        match &self.data_type {
+            DataType::Decimal(p, _) => {


C++ ArrayData full validation performs the precision check for decimal type. I think this is necessary to add even we don't remove validate_decimal_precision from DecimalBuilder.append_value.

I agree this code is necessary in validate 👍

viirya · 2022-05-31T17:55:41Z

arrow/src/array/array_binary.rs

@@ -1486,7 +1486,7 @@ mod tests {
            192, 219, 180, 17, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 36, 75, 238, 253,
            255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255,
        ];
-        let array_data = ArrayData::builder(DataType::Decimal(23, 6))
+        let array_data = ArrayData::builder(DataType::Decimal(38, 6))


Caught this invalid decimal value by decimal check in full validation. Increasing precision to pass it.

viirya · 2022-05-31T20:16:09Z

arrow/src/ipc/writer.rs

@@ -1130,6 +1131,7 @@ mod tests {
    }

    #[test]
+    #[cfg(not(feature = "force_validate"))]


I think these tests use the same 0.14.1 decimal file too.

alamb

Thank you for working on this @viirya

After thinking about it for a while, I think:

Adding validation checks to ArrayData::validate for decimal is ❤️
I am a little worried about removing the validation from DecimalBuilder -- I think one of the philosophies we are shooting for is if you use safe APIs you will create valid Arrow arrays.

What would you think about making validation optional for the builder?

Something like:

struct DecimalBuilder { 
 ...
  /// Should i128 values be validated for compatibility with scale and precision?
  /// defaults to true
  value_validation: true
}

impl DecomalBuilder {
  ...
  /// Disable validation
  pub unsafe disable_value_validation(mut self) -> Self {
    self.validate_values = false;
  }

  #[inline]
  pub fn append_value(&mut self, value: i128) -> Result<()> {
    let value = if sef.validate_values {
      validate_decimal_precision(value, self.precision)?
    } else {
      value
    }
   }
  ...
}

I think this would avoid having to add #[cfg(not(feature = "force_validate"))] and it would allow the interop test to bypass value validation only when needed?

alamb · 2022-06-02T10:50:46Z

arrow/src/array/data.rs

@@ -999,6 +999,27 @@ impl ArrayData {

    pub fn validate_dictionary_offset(&self) -> Result<()> {
        match &self.data_type {
+            DataType::Decimal(p, _) => {


I agree this code is necessary in validate 👍

alamb · 2022-06-02T10:56:23Z

arrow/src/array/data.rs

+                    };
+                    let as_array = raw_val.try_into();
+                    match as_array {
+                        Ok(v) if raw_val.len() == 16 => {


For some reason this code seems overly complicated -- I wonder if you could call i128::from_le_bytes directly on a slice like

for pos in 0..values_buffer.len() { let v = value_buffer.data[pos..pos+16]; let value = i128::from_le_bytes(v) }

or something 🤔

data is private. 😞

I simplified it as:

for pos in 0..values_buffer.len() { let raw_val = unsafe { std::slice::from_raw_parts( values_buffer.as_ptr().add(pos), 16_usize, ) }; let value = i128::from_le_bytes(raw_val.try_into().unwrap()); validate_decimal_precision(value, *p)?; }

viirya · 2022-06-02T16:04:49Z

I am a little worried about removing the validation from DecimalBuilder -- I think one of the philosophies we are shooting for is if you use safe APIs you will create valid Arrow arrays.

Hmm, seems at C++ implementation, the builder is unsafe API, in Rust parlance. This is based on the reply I got in the JIRA ticket. This seems the difference between Rust and C++ implementations.

viirya · 2022-06-02T16:14:31Z

What would you think about making validation optional for the builder?
I think this would avoid having to add #[cfg(not(feature = "force_validate"))] and it would allow the interop test to bypass value validation only when needed?

Generally it sounds good to me. But in the test, we are not using the builder directly. The builder is called by more higher API when reading JSON/Arrow files into Arrays. So seems we need more than adding an API like disable_value_validation to DecomalBuilder.

Let me try to play it around and see if an optional validation of the builder is possible to solve the interop test issue.

alamb · 2022-06-02T17:50:42Z

I wonder if this could be related to #1779 just filed by @tustvold 🤔

viirya · 2022-06-02T18:43:19Z

Hmm, I think it is not directly related. C++ also does this check but it does in full validation, not in the builder. Actually C++ Arrow avoids the full validation too in the interop to avoid similar issue.

viirya · 2022-06-03T05:59:44Z

I think this would avoid having to add #[cfg(not(feature = "force_validate"))]

Oh, BTW, adding #[cfg(not(feature = "force_validate"))] to several tests is for the added check in ArrayData::validate, not due to removal of the check from DecimalBuilder.

These tests build ArrayData with incompatible decimal values. As this adds the check when full validation, these tests will fail during that.

…t DecimalBuilder

viirya · 2022-06-03T06:19:03Z

Generally it sounds good to me. But in the test, we are not using the builder directly. The builder is called by more higher API when reading JSON/Arrow files into Arrays. So seems we need more than adding an API like disable_value_validation to DecomalBuilder.

I'm wrong. Fortunately, for interop test, we directly invoke DecimalBuilder to read JSON file to Array. So adding disable_value_validation call there can fix the interop issue. Verified it locally.

alamb

Thanks @viirya -- this looks wonderful. Thank you for sticking with it

alamb · 2022-06-03T10:24:58Z

arrow/src/array/data.rs

+
+    #[test]
+    #[cfg(not(feature = "force_validate"))]
+    fn test_decimal_full_validation() {


viirya · 2022-06-03T16:10:17Z

Thanks @alamb @tustvold @nevi-me

Remove precision check

6367da4

github-actions bot added the arrow Changes to the arrow crate label May 31, 2022

Remove test

e2ad586

Fix clippy

da44930

alamb approved these changes May 31, 2022

View reviewed changes

nevi-me approved these changes May 31, 2022

View reviewed changes

Move decimal precision check to ArrayData full validation and add test

3a221ce

viirya force-pushed the remove_precision_check branch from b179054 to 3a221ce Compare May 31, 2022 17:19

viirya commented May 31, 2022

View reviewed changes

Increase precision to pass existing test

b289f11

viirya commented May 31, 2022

View reviewed changes

viirya added 2 commits May 31, 2022 11:46

Fix clippy

2d8a69c

Fix tests

5665071

viirya commented May 31, 2022

View reviewed changes

alamb reviewed Jun 2, 2022

View reviewed changes

Add disable_value_validation to optionally disable value validation a…

d6eeba5

…t DecimalBuilder

viirya changed the title ~~Remove validate_decimal_precision check in DecimalBuilder.append_value~~ Optionally disable validate_decimal_precision check in DecimalBuilder.append_value for interop test Jun 3, 2022

viirya added 2 commits June 2, 2022 23:31

Simplify validation code

48798f0

Update test

7a6be90

alamb approved these changes Jun 3, 2022

View reviewed changes

arrow/src/array/data.rs

#[test]

#[cfg(not(feature = "force_validate"))]

fn test_decimal_full_validation() {

Copy link

Contributor

alamb Jun 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

alamb merged commit 29c49d8 into apache:master Jun 3, 2022

tustvold mentioned this pull request Jun 7, 2022

Unsound DecimalArray Validation in Unreleased arrow-rs #1813

Closed

asfimport mentioned this pull request Jun 3, 2022

[C++] Is Precision check necessary when appending values at decimal builder apache/arrow#32040

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optionally disable `validate_decimal_precision` check in `DecimalBuilder.append_value` for interop test #1767

Optionally disable `validate_decimal_precision` check in `DecimalBuilder.append_value` for interop test #1767

viirya commented May 31, 2022

codecov-commenter commented May 31, 2022 •

edited

Loading

tustvold commented May 31, 2022

viirya commented May 31, 2022

alamb left a comment

alamb May 31, 2022

viirya May 31, 2022

alamb commented May 31, 2022

nevi-me left a comment

viirya commented May 31, 2022

viirya May 31, 2022

alamb Jun 2, 2022

viirya May 31, 2022

viirya May 31, 2022

alamb left a comment

alamb Jun 2, 2022

alamb Jun 2, 2022

viirya Jun 3, 2022

viirya commented Jun 2, 2022

viirya commented Jun 2, 2022

alamb commented Jun 2, 2022

viirya commented Jun 2, 2022

viirya commented Jun 3, 2022

viirya commented Jun 3, 2022

alamb left a comment

alamb Jun 3, 2022

viirya commented Jun 3, 2022

Optionally disable validate_decimal_precision check in DecimalBuilder.append_value for interop test #1767

Optionally disable validate_decimal_precision check in DecimalBuilder.append_value for interop test #1767

Conversation

viirya commented May 31, 2022

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

codecov-commenter commented May 31, 2022 • edited Loading

Codecov Report

tustvold commented May 31, 2022

viirya commented May 31, 2022

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented May 31, 2022

nevi-me left a comment

Choose a reason for hiding this comment

viirya commented May 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya commented Jun 2, 2022

viirya commented Jun 2, 2022

alamb commented Jun 2, 2022

viirya commented Jun 2, 2022

viirya commented Jun 3, 2022

viirya commented Jun 3, 2022

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya commented Jun 3, 2022

Optionally disable `validate_decimal_precision` check in `DecimalBuilder.append_value` for interop test #1767

Optionally disable `validate_decimal_precision` check in `DecimalBuilder.append_value` for interop test #1767

codecov-commenter commented May 31, 2022 •

edited

Loading