SPARK-4222 [CORE] use readFully in FixedLengthBinaryRecordReader

[industrial-sloth]

replaces the existing read() call with readFully().

[srowen]

Hm, but this also doesn't check how many bytes were actually read?

[industrial-sloth]

yep, but readFully will either block until the full number of bytes is available, or throw an error:
http://docs.oracle.com/javase/6/docs/api/java/io/DataInput.html#readFully(byte[])

[AmplabJenkins]

Can one of the admins verify this patch?

[industrial-sloth]

FWIW, we got bit by this over in the Thunder project, where we'd been using this class before Jeremy contributed it into Spark. The circumstance where it came up was doing big reads from S3, where the underlying buffer size was something like 16kb? I forget exactly... anyway, when the records didn't align with this buffer, badness happened.

Here's the equivalent change I made over there:
thunder-project/thunder@7ea31c8

The previous version on that patch shows my previous fix, which does check the number of bytes returned, before I realized that readFully() was a thing. Both that fix and the one here do resolve the original problem we encountered, though I'm having some trouble coming up with a clean unit test for it...

[mateiz]

Good catch, thanks! Can you check that this is the only version of read() in that code?

[mateiz]

BTW it would be good to open a JIRA issue for this on https://issues.apache.org/jira/browse/SPARK but unfortunately ASF JIRA seems to be down at the moment.

[mateiz]

Jenkins, this is ok to test

[SparkQA]

Test build #22904 has started for PR 3093 at commit a245c8a.

• This patch merges cleanly.

[SparkQA]

Test build #22904 has finished for PR 3093 at commit a245c8a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Params(
  • class RDDFunctions[T: ClassTag](self: RDD[T]) extends Serializable
  • class VectorUDT(UserDefinedType):
  • class NullType(PrimitiveType):
  • class UserDefinedType(DataType):
  • case class ScalaUdfBuilder[T: TypeTag](f: AnyRef)

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22904/
Test PASSed.

[industrial-sloth]

Thanks @mateiz - Oops, I actually did open https://issues.apache.org/jira/browse/SPARK-4222 for this, then (inexplicably...) left it off the commit message. Not sure offhand of how best to address that - would be happy to close this PR and open a new one with a proper message, let me know if that's how you'd like to proceed.

This was indeed the only use of read() in the FixedLengthBinary* files.

[srowen]

@industrial-sloth just change the PR title to "SPARK-4222 [CORE] ..." since that title will be the commit message when merged.

[industrial-sloth]

Thanks @srowen! Done.

[mateiz]

Cool, thanks. Will merge this soon.