Failure in CI: `Archery test With other arrows` - `binary_view Rust producing, .NET consuming`

[rluvaton]

I started to get unrelated failures in CI in Archery test With other arrows - binary_view Rust producing, .NET consuming

for example:

• feat: extract has_false and has_true from BooleanArray to BooleanBuffer and reuse for no nulls #9987

arrow .NET last release/push was done 2 weeks ago so it is probably coming from this repo

Failure Log

Integration: Test: Result
  ################# FAILURES #################
  FAILED TEST: binary_view Rust producing,  .NET consuming
  <class 'RuntimeError'>: Command failed: /__w/arrow-rs/arrow-rs/dotnet/artifacts/Apache.Arrow.IntegrationTest/Debug/net8.0/Apache.Arrow.IntegrationTest --mode validate -j /tmp/arrow-integration-xfpngs5h/generated_binary_view.json -a /tmp/tmpepjkh334/ae4404ef_generated_binary_view.json_as_file
  With output:
  --------------
  Unhandled exception. Xunit.Sdk.TrueException: Assert.True() Failure
  Expected: True
  Actual:   False
     at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 141
     at Xunit.Assert.True(Boolean condition) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 95
     at Apache.Arrow.Tests.ArrowReaderVerifier.ArrayComparer.CompareVariadicArrays[T](BinaryViewArray actualArray) in /__w/arrow-rs/arrow-rs/dotnet/test/Apache.Arrow.Tests/ArrowReaderVerifier.cs:line 365
     at Apache.Arrow.Tests.ArrowReaderVerifier.ArrayComparer.Visit(BinaryViewArray array) in /__w/arrow-rs/arrow-rs/dotnet/test/Apache.Arrow.Tests/ArrowReaderVerifier.cs:line 166
     at Apache.Arrow.BinaryViewArray.Accept(IArrowArrayVisitor visitor) in /__w/arrow-rs/arrow-rs/dotnet/src/Apache.Arrow/Arrays/BinaryViewArray.cs:line 258
     at Apache.Arrow.Tests.ArrowReaderVerifier.CompareArrays(IArrowArray expectedArray, IArrowArray actualArray, Boolean strictCompare) in /__w/arrow-rs/arrow-rs/dotnet/test/Apache.Arrow.Tests/ArrowReaderVerifier.cs:line 69
     at Apache.Arrow.Tests.ArrowReaderVerifier.CompareBatches(RecordBatch expectedBatch, RecordBatch actualBatch, Boolean strictCompare) in /__w/arrow-rs/arrow-rs/dotnet/test/Apache.Arrow.Tests/ArrowReaderVerifier.cs:line 63
     at Apache.Arrow.IntegrationTest.IntegrationCommand.Validate() in /__w/arrow-rs/arrow-rs/dotnet/test/Apache.Arrow.IntegrationTest/IntegrationCommand.cs:line 90
     at Apache.Arrow.IntegrationTest.IntegrationCommand.Execute() in /__w/arrow-rs/arrow-rs/dotnet/test/Apache.Arrow.IntegrationTest/IntegrationCommand.cs:line 54
     at Apache.Arrow.IntegrationTest.Program.Main(String[] args) in /__w/arrow-rs/arrow-rs/dotnet/test/Apache.Arrow.IntegrationTest/Program.cs:line 53
     at Apache.Arrow.IntegrationTest.Program.<Main>(String[] args)
  
  --------------
  
  
  FAILED TEST: binary_view Rust producing,  .NET consuming
  <class 'Xunit.Sdk.TrueException'>: Assert.True() Failure
  Expected: True
  Actual:   False
     at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 141
     at Xunit.Assert.True(Boolean condition) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 95
     at Apache.Arrow.Tests.ArrowReaderVerifier.ArrayComparer.CompareVariadicArrays[T](BinaryViewArray actualArray) in /__w/arrow-rs/arrow-rs/dotnet/test/Apache.Arrow.Tests/ArrowReaderVerifier.cs:line 365
     at Apache.Arrow.Tests.ArrowReaderVerifier.ArrayComparer.Visit(BinaryViewArray array) in /__w/arrow-rs/arrow-rs/dotnet/test/Apache.Arrow.Tests/ArrowReaderVerifier.cs:line 166
     at Apache.Arrow.BinaryViewArray.Accept(IArrowArrayVisitor visitor) in /__w/arrow-rs/arrow-rs/dotnet/src/Apache.Arrow/Arrays/BinaryViewArray.cs:line 258
     at Apache.Arrow.Tests.ArrowReaderVerifier.CompareArrays(IArrowArray expectedArray, IArrowArray actualArray, Boolean strictCompare) in /__w/arrow-rs/arrow-rs/dotnet/test/Apache.Arrow.Tests/ArrowReaderVerifier.cs:line 69
     at Apache.Arrow.Tests.ArrowReaderVerifier.CompareBatches(RecordBatch expectedBatch, RecordBatch actualBatch, Boolean strictCompare) in /__w/arrow-rs/arrow-rs/dotnet/test/Apache.Arrow.Tests/ArrowReaderVerifier.cs:line 63
     at InvokeStub_ArrowReaderVerifier.CompareBatches(Object, Span`1)
     at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
  
  2 failures, 319 skips

real	3m9.743s
user	10m55.535s
sys	1m10.432s
ERROR conda.cli.main_run:execute(148): `conda run ci/scripts/integration_arrow.sh /__w/arrow-rs/arrow-rs /build` failed. (See above for error)

[alamb]

I suspect this is related to a change in arrow:

• GH-49744: [Integration][Format] Remove unneeded skips for Rust and nanoarrow integration tests arrow#49910

[paleolimbot]

The issue seems to be that the views are not considered equal between what Rust is producing and what dotnet is reading:

https://github.com/apache/arrow-dotnet/blob/debff867ef4ac9a62cd5173e69397343b7da4335/test/Apache.Arrow.Tests/ArrowReaderVerifier.cs#L365

Debugging this kind of thing is a pain (I made a few PRs into the C# implementation when implementing nanoarrow's integration tests). Probably the first step would be to make that assert give better debug output from the dotnet side.

[CurtHagenlocher]

That assertion actually looks a little bogus to me. There's no specific requirement that the buffers be equal, only that the mapped parts of the buffers are equal.

EDIT: I've queued a build which comments that line out and I'm sure we'll see that it passes afterwards.

[CurtHagenlocher]

apache/arrow-dotnet#360

Do I need to publish a new version for this to be picked up everywhere or are the tests always against HEAD?

[kou]

Our integration tests always use HEAD.

[CurtHagenlocher]

Okay, the Rust integration tests are passing now that I've checked in a fix to arrow-dotnet so I'm going to close this.