Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix undefined behavior in FFI and enable MIRI checks on CI #323

Merged
merged 1 commit into from
May 23, 2021

Conversation

roee88
Copy link
Contributor

@roee88 roee88 commented May 18, 2021

Fix UB in FFI due to violation of aliasing rules

Which issue does this PR close?

Closes #322.

What changes are included in this PR?

  • Fix the issue by using raw pointers derived from the Box objects in private data rather than from moved boxes
  • Enable MIRI in CI for most tests in arrow crate
    test result: ok. 584 passed; 0 failed; 10 ignored; 0 measured; 102 filtered out

Specifically, the CI is changed to run miri test against the arrow crate (only) while skipping some tests:

  1. Individual tests that are either unsupported, fail, or get stuck are ignored using #[cfg_attr(miri, ignore)] with a code comment that indicates why it's disabled
  2. IO tests (csv, json, ipc) are skipped in CI due to slowness and/or failures.

The above was required because otherwise the CI would be stuck forever (e.g., in the test_can_cast_types test). This wasn't the case so far because the FFI issue failed the run. I suggest to track enabling more tests in #227.

Are there any user-facing changes?

No

- Fix UB due to aliasing
- Enable MIRI in CI for most tests in arrow crate

Signed-off-by: roee88 <roee88@gmail.com>
@codecov-commenter
Copy link

Codecov Report

Merging #323 (ab90087) into master (c863a2c) will increase coverage by 0.00%.
The diff coverage is 92.30%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #323   +/-   ##
=======================================
  Coverage   82.52%   82.52%           
=======================================
  Files         162      162           
  Lines       44007    44007           
=======================================
+ Hits        36316    36317    +1     
+ Misses       7691     7690    -1     
Impacted Files Coverage Δ
arrow/src/array/raw_pointer.rs 100.00% <ø> (ø)
arrow/src/compute/kernels/cast.rs 94.49% <ø> (ø)
arrow/src/compute/kernels/cast_utils.rs 92.50% <ø> (ø)
arrow/src/compute/kernels/length.rs 100.00% <ø> (ø)
arrow/src/util/bit_chunk_iterator.rs 96.55% <ø> (ø)
arrow/src/util/integration_util.rs 69.55% <ø> (ø)
arrow/src/ffi.rs 82.85% <92.30%> (+0.23%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c863a2c...ab90087. Read the comment docs.

Copy link
Member

@jorgecarleitao jorgecarleitao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks a lot, @roee88 !

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think this looks great -- it would be cool to file a follow on ticket to remove the remaining MIRI flagged issues but this seems like a great step forward to me

cargo miri test || true
# Currently only the arrow crate is tested with miri
# IO related tests and some unsupported tests are skipped
cargo miri test -p arrow -- --skip csv --skip ipc --skip json
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is epic. Nice work @roee88 !

@alamb alamb merged commit f316798 into apache:master May 23, 2021
@roee88 roee88 deleted the fix-miri branch May 23, 2021 10:18
@alamb alamb changed the title Fix undefined behavior in FFI Fix undefined behavior in FFI and enable MIRI checks on CI May 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Undefined behavior in FFI implementation
4 participants