[SPARK-24452][SQL][Core] Avoid possible overflow in int add or multiple #21481

kiszk · 2018-06-01T20:36:18Z

What changes were proposed in this pull request?

This PR fixes possible overflow in int add or multiply. In particular, their overflows in multiply are detected by Spotbugs

The following assignments may cause overflow in right hand side. As a result, the result may be negative.

long = int * int
long = int + int

To avoid this problem, this PR performs cast from int to long in right hand side.

How was this patch tested?

Existing UTs.

SparkQA · 2018-06-02T00:59:39Z

Test build #91404 has finished for PR 21481 at commit 324fd5c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2018-06-02T02:06:46Z

cc @ueshin @hvanhovell

kiszk · 2018-06-02T10:51:08Z

cc @cloud-fan

JoshRosen · 2018-06-02T17:11:29Z

Hey @kiszk, thanks for tracking this down. This change looks good to me.

I have a couple of questions, mostly aimed towards figuring out how we can categorically solve this problem:

What's the impact of this issue? Have you observed actual crashes or silent data corruption caused by this problem? I ask because it looks like this could be a plausible cause of a data corruption bug that I've been investigating.
Are these five files the only occurrence of this problem or are there potentially others? I've noticed that all of the files modified here are .java files: is that a coincidence or is that the result of running some Java linting tool? If the latter, is it possible that we have similar issues in other files which we haven't found yet?
Do you have any ideas for how we can categorically prevent this class of problem in the future? Are there compiler plugins or linters which can warn on these cases and turn them into compile-time errors? Or code-review checklists that we can employ to more easily spot these potential overflows? This is a hard problem, but I think it's worth brainstorming on categorical solutions since this isn't the first time we've hit this class of problem (but hopefully it can be the last!).

I think we should definitely backport this fix, at least to 2.3 and possibly earlier.

cloud-fan · 2018-06-03T04:07:09Z

+1 on @JoshRosen 's ideas.

I've already seen several overflow fixes from @kiszk , it will be good if we have some tools to check it, even we need to run the tool manually. One idea may be to force to use java 8's safe math functions: https://docs.oracle.com/javase/8/docs/api/java/lang/Math.html#addExact-int-int-

kiszk · 2018-06-03T10:00:37Z

Good questions.

For 2, at first I found one of these issues when I looked at a file. Then, I ran grep command with long .*=.*\* and long .*=.*\+ in .java file. Then, I picked them up manually. It looks labor-intensive.

For 3, here is my thought.
SpotBugs may be a good candidate to check it.
SpotBug is a successor of findBugs. When I ran FindBugs before, I found some problems regarding possible overflow and then made a PR that was integrated. On the other hand, these issues may not be detected at that time.

I will look at SpotBugs after my presentation at SAIS will be finished :)

kiszk · 2018-06-08T09:22:45Z

@JoshRosen @cloud-fan
Here is an update.

I have just apply findBugs to OffHeapColumnVector.java and UnsafeArrayData.java.
In OffHeapColumnVector.java, most of possible overflows are detected. But, not all.
In UnsafeArrayData.java, two possible overflows are detected. Line 86 and 456. I overlooked Line 86.

cloud-fan · 2018-06-08T16:50:44Z

is findBugs available for scala code as well?

kiszk · 2018-06-08T17:15:06Z

Since it is Java bytecode analysis, it is available for Scala code, too.
In my quick test, findBugs overlooked a possible overflow. On the other hand, findBugs found another redundant null check.

kiszk · 2018-06-10T19:17:37Z

Since I found an plug-in for maven, I will also include a patch to add findBugs/SpotBugs into maven in this PR.

JoshRosen · 2018-06-10T21:19:00Z

Let's merge this as-is and do the build improvements in a separate PR. That's important because we may want to backport the overflow fix to maintenance branches and may want to do so independent of the build changes.

kiszk · 2018-06-11T02:00:17Z

Thank you for your comment. I will create another PR for integrating findBugs/SpotBugs into maven.

cloud-fan · 2018-06-11T20:26:49Z

core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java

@@ -703,7 +703,7 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff
      // must be stored in the same memory page.
      // (8 byte key length) (key) (value) (8 byte pointer to next value)
      int uaoSize = UnsafeAlignedOffset.getUaoSize();
-      final long recordLength = (2 * uaoSize) + klen + vlen + 8;
+      final long recordLength = (2L * uaoSize) + (long)klen + (long)vlen + 8L;


do we need to cast everything to long? I think long + int + int + int is safe.

You are right. It was too conservative. (2L * uaoSize) + klen + vlen + 8 can generate LMUL or LADD as follows:

LDC 2 ILOAD 9 I2L LMUL ILOAD 4 I2L LADD ILOAD 8 I2L LADD LDC 8 LADD LSTORE 10

cloud-fan · 2018-06-11T20:30:19Z

...main/java/org/apache/spark/sql/catalyst/expressions/VariableLengthRowBasedKeyValueBatch.java

@@ -41,7 +41,7 @@
  @Override
  public UnsafeRow appendRow(Object kbase, long koff, int klen,
                             Object vbase, long voff, int vlen) {
-    final long recordLength = 8 + klen + vlen + 8;
+    final long recordLength = 8L + (long)klen + (long)vlen + 8L;


cloud-fan · 2018-06-11T20:33:06Z

@kiszk when you were running findBugs locally, did you find more overflow bugs that are not present in this PR? Let's put all discovered overflow bugs in this PR and have another PR to integrate findBugs with maven. Thanks!

SparkQA · 2018-06-12T17:43:35Z

Test build #91717 has finished for PR 21481 at commit 50a2585.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2018-06-15T16:13:52Z

@cloud-fan addressed all of the possible integer overflows detected by SpotBugs.

SparkQA · 2018-06-15T18:52:59Z

Test build #91922 has finished for PR 21481 at commit a87c417.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2018-06-15T20:49:52Z

thanks, merging to master/2.3!

This PR fixes possible overflow in int add or multiply. In particular, their overflows in multiply are detected by [Spotbugs](https://spotbugs.github.io/) The following assignments may cause overflow in right hand side. As a result, the result may be negative. ``` long = int * int long = int + int ``` To avoid this problem, this PR performs cast from int to long in right hand side. Existing UTs. Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Closes #21481 from kiszk/SPARK-24452. (cherry picked from commit 90da7dc) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

cloud-fan reviewed Jun 11, 2018

View reviewed changes

kiszk changed the title ~~[SPARK-24452][SQL][Core] Avoid possible overflow in int add or multiple~~ [WIP][SPARK-24452][SQL][Core] Avoid possible overflow in int add or multiple Jun 12, 2018

kiszk added 3 commits June 15, 2018 22:48

initial commit

7288e02

address review comments

f3a0ee9

fix possible interger overflow detected by Spottsbug

a87c417

kiszk force-pushed the SPARK-24452 branch from 50a2585 to a87c417 Compare June 15, 2018 14:26

kiszk changed the title ~~[WIP][SPARK-24452][SQL][Core] Avoid possible overflow in int add or multiple~~ [SPARK-24452][SQL][Core] Avoid possible overflow in int add or multiple Jun 15, 2018

asfgit closed this in 90da7dc Jun 15, 2018

kiszk mentioned this pull request Jun 15, 2018

[SPARK-24529][Build][test-maven] Add spotbugs into maven build process #21542

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-24452][SQL][Core] Avoid possible overflow in int add or multiple #21481

[SPARK-24452][SQL][Core] Avoid possible overflow in int add or multiple #21481

kiszk commented Jun 1, 2018 •

edited

Loading

SparkQA commented Jun 2, 2018

kiszk commented Jun 2, 2018

kiszk commented Jun 2, 2018

JoshRosen commented Jun 2, 2018

cloud-fan commented Jun 3, 2018

kiszk commented Jun 3, 2018

kiszk commented Jun 8, 2018

cloud-fan commented Jun 8, 2018

kiszk commented Jun 8, 2018

kiszk commented Jun 10, 2018

JoshRosen commented Jun 10, 2018

kiszk commented Jun 11, 2018

cloud-fan Jun 11, 2018

kiszk Jun 11, 2018

cloud-fan Jun 11, 2018

cloud-fan commented Jun 11, 2018 •

edited

Loading

SparkQA commented Jun 12, 2018

kiszk commented Jun 15, 2018

SparkQA commented Jun 15, 2018

cloud-fan commented Jun 15, 2018

[SPARK-24452][SQL][Core] Avoid possible overflow in int add or multiple #21481

[SPARK-24452][SQL][Core] Avoid possible overflow in int add or multiple #21481

Conversation

kiszk commented Jun 1, 2018 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Jun 2, 2018

kiszk commented Jun 2, 2018

kiszk commented Jun 2, 2018

JoshRosen commented Jun 2, 2018

cloud-fan commented Jun 3, 2018

kiszk commented Jun 3, 2018

kiszk commented Jun 8, 2018

cloud-fan commented Jun 8, 2018

kiszk commented Jun 8, 2018

kiszk commented Jun 10, 2018

JoshRosen commented Jun 10, 2018

kiszk commented Jun 11, 2018

cloud-fan Jun 11, 2018

Choose a reason for hiding this comment

kiszk Jun 11, 2018

Choose a reason for hiding this comment

cloud-fan Jun 11, 2018

Choose a reason for hiding this comment

cloud-fan commented Jun 11, 2018 • edited Loading

SparkQA commented Jun 12, 2018

kiszk commented Jun 15, 2018

SparkQA commented Jun 15, 2018

cloud-fan commented Jun 15, 2018

kiszk commented Jun 1, 2018 •

edited

Loading

cloud-fan commented Jun 11, 2018 •

edited

Loading