Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-8506: [C++] Add tests to verify the encoded stream of RLE with bit_width > 8 #6976

Closed
wants to merge 1 commit into from

Conversation

kiszk
Copy link
Member

@kiszk kiszk commented Apr 18, 2020

This PR adds test cases to verify whether the encoded values are stored in a little-endian format when RLE with bit_width > 8 is applied. Parquet encoder/decoder handles the value as little-endian as follows:
https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/values/rle/RunLengthBitPackingHybridEncoder.java#L244
https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/values/rle/RunLengthBitPackingHybridDecoder.java#L87

There is no test of this type in the current test cases of Rle.

@github-actions
Copy link

@kiszk
Copy link
Member Author

kiszk commented Apr 18, 2020

retest this please

@kiszk
Copy link
Member Author

kiszk commented Apr 18, 2020

The failures may not be related to this PR.

       8: github.com/minio/minio@/cmd/fs-v1-helpers.go:320:cmd.fsCreateFile()
       7: github.com/minio/minio@/cmd/fs-v1.go:973:cmd.(*FSObjects).putObject()
       6: github.com/minio/minio@/cmd/fs-v1.go:887:cmd.(*FSObjects).PutObject()
       5: github.com/minio/minio@/cmd/config-common.go:62:cmd.saveConfig()
       4: github.com/minio/minio@/cmd/config-encrypted.go:291:cmd.migrateConfigPrefixToEncrypted()
       3: github.com/minio/minio@/cmd/config-encrypted.go:107:cmd.handleEncryptedConfigBackend()
       2: github.com/minio/minio@/cmd/server-main.go:194:cmd.initSafeMode()
       1: github.com/minio/minio@/cmd/server-main.go:447:cmd.serverMain()
/arrow/cpp/src/arrow/filesystem/s3fs_test.cc:731: Failure
Failed
'OutcomeToStatus(client_->CreateBucket(req))' failed with IOError: AWS Error [code 100]: Unable to parse ExceptionName: XMinioServerNotInitialized Message: Server not initialized, please try again. with address : 127.0.0.1
[  FAILED  ] TestS3FSGeneric.OpenInputFile (6041 ms)
[----------] 18 tests from TestS3FSGeneric (108737 ms total)

[----------] Global test environment tear-down
[==========] 39 tests from 4 test cases ran. (230555 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 38 tests, listed below:
[  FAILED  ] TestMinioServer.Connect
[  FAILED  ] TestS3FS.GetFileInfoRoot
[  FAILED  ] TestS3FS.GetFileInfoBucket
[  FAILED  ] TestS3FS.GetFileInfoObject
[  FAILED  ] TestS3FS.GetFileInfoSelector
[  FAILED  ] TestS3FS.GetFileInfoSelectorRecursive
[  FAILED  ] TestS3FS.CreateDir
[  FAILED  ] TestS3FS.DeleteFile
[  FAILED  ] TestS3FS.Move
[  FAILED  ] TestS3FS.OpenInputStream
[  FAILED  ] TestS3FS.OpenInputFile
[  FAILED  ] TestS3FS.OpenOutputStreamBackgroundWrites
[  FAILED  ] TestS3FS.OpenOutputStreamSyncWrites
[  FAILED  ] TestS3FS.OpenOutputStreamAbortBackgroundWrites
[  FAILED  ] TestS3FS.OpenOutputStreamAbortSyncWrites
[  FAILED  ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites
[  FAILED  ] TestS3FS.OpenOutputStreamDestructorSyncWrite
[  FAILED  ] TestS3FS.FileSystemFromUri
[  FAILED  ] TestS3FSGeneric.Empty
[  FAILED  ] TestS3FSGeneric.NormalizePath
[  FAILED  ] TestS3FSGeneric.CreateDir
[  FAILED  ] TestS3FSGeneric.DeleteDir
[  FAILED  ] TestS3FSGeneric.DeleteDirContents
[  FAILED  ] TestS3FSGeneric.DeleteFile
[  FAILED  ] TestS3FSGeneric.DeleteFiles
[  FAILED  ] TestS3FSGeneric.MoveFile
[  FAILED  ] TestS3FSGeneric.MoveDir
[  FAILED  ] TestS3FSGeneric.CopyFile
[  FAILED  ] TestS3FSGeneric.GetFileInfo
[  FAILED  ] TestS3FSGeneric.GetFileInfoVector
[  FAILED  ] TestS3FSGeneric.GetFileInfoSelector
[  FAILED  ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion
[  FAILED  ] TestS3FSGeneric.OpenOutputStream
[  FAILED  ] TestS3FSGeneric.OpenAppendStream
[  FAILED  ] TestS3FSGeneric.OpenInputStream
[  FAILED  ] TestS3FSGeneric.OpenInputFile

38 FAILED TESTS
/build/cpp/src/arrow/filesystem


99% tests passed, 1 tests failed out of 86
...
The following tests FAILED:
	 76 - arrow-s3fs-test (Failed)

Copy link
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, thank you. I've restarted the failing CI builds.

@pitrou
Copy link
Member

pitrou commented Apr 18, 2020

CI failures are unrelated, merging.

@pitrou pitrou closed this in c152ab6 Apr 18, 2020
kszucs pushed a commit that referenced this pull request Apr 20, 2020
…bit_width > 8

This PR adds test cases to verify whether the encoded values are stored in a little-endian format when RLE with bit_width > 8 is applied. Parquet encoder/decoder handles the value as little-endian as follows:
https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/values/rle/RunLengthBitPackingHybridEncoder.java#L244
https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/values/rle/RunLengthBitPackingHybridDecoder.java#L87

There is no test of this type in the current test cases of Rle.

Closes #6976 from kiszk/ARROW-8506

Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants