SPARK-7450 Use UNSAFE.getLong() to speed up BitSetMethods#anySet() #5897

tedyu · 2015-05-04T23:59:09Z

@JoshRosen
Please take a look

AmplabJenkins · 2015-05-05T00:02:10Z

Can one of the admins verify this patch?

JoshRosen · 2015-05-05T00:14:28Z

Jenkins, add to whitelist and test this please.

AmplabJenkins · 2015-05-05T00:17:11Z

Merged build triggered.

AmplabJenkins · 2015-05-05T00:17:20Z

Merged build started.

SparkQA · 2015-05-05T00:18:47Z

Test build #31800 has started for PR 5897 at commit 4ca0ef6.

SparkQA · 2015-05-05T00:20:47Z

Test build #31800 has finished for PR 5897 at commit 4ca0ef6.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-05T00:20:48Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-05T00:20:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31800/
Test FAILed.

rxin · 2015-05-05T00:47:31Z

unsafe/src/main/java/org/apache/spark/unsafe/bitset/BitSetMethods.java

@@ -71,7 +72,13 @@ public static boolean isSet(Object baseObject, long baseOffset, int index) {
   * Returns {@code true} if any bit is set.
   */
  public static boolean anySet(Object baseObject, long baseOffset, long bitSetWidthInBytes) {
-    for (int i = 0; i <= bitSetWidthInBytes; i++) {
+    long widthInLong = bitSetWidthInBytes / SIZE_OF_LONG;
+    for (long i = 0; i <= widthInLong; i++) {


actually we should use int instead of long for i. the reason is that JIT would inject a safepoint for loops over longs which prevents loop unrolling and incurs an extra check for every iteration

Addressed the above comment.

AmplabJenkins · 2015-05-05T01:07:11Z

Merged build triggered.

rxin · 2015-05-05T01:07:12Z

unsafe/src/main/java/org/apache/spark/unsafe/bitset/BitSetMethods.java

@@ -71,7 +72,13 @@ public static boolean isSet(Object baseObject, long baseOffset, int index) {
   * Returns {@code true} if any bit is set.
   */
  public static boolean anySet(Object baseObject, long baseOffset, long bitSetWidthInBytes) {
-    for (int i = 0; i <= bitSetWidthInBytes; i++) {
+    long widthInLong = bitSetWidthInBytes / SIZE_OF_LONG;


width needs to be an int too :)

AmplabJenkins · 2015-05-05T01:07:16Z

Merged build started.

SparkQA · 2015-05-05T01:08:44Z

Test build #31808 has started for PR 5897 at commit 63ee050.

SparkQA · 2015-05-05T01:10:46Z

Test build #31808 has finished for PR 5897 at commit 63ee050.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-05T01:10:47Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-05T01:10:47Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31808/
Test FAILed.

JoshRosen · 2015-05-05T01:11:21Z

unsafe/src/main/java/org/apache/spark/unsafe/bitset/BitSetMethods.java

+        return true;
+      }
+    }
+    for (int i = (int)(SIZE_OF_LONG * widthInLong); i < bitSetWidthInBytes; i++) {


It looks like this block now contains two for loops. A bad merge conflict resolution, perhaps?

I don't think so.
The second loop is for the remaining bytes :-)

Ah, sorry for overlooking that. I guess I was thinking of optimizing for the case where the bitset width was a multiple of the word size.

That case is covered by the first loop.

do we need the 2nd loop? I think the assumption is that all bitsets are word-aligned. We should definitely document that though.

tedyu · 2015-05-05T01:11:51Z

From failed test:
bq. [error] oro#oro;2.0.8!oro.jar origin location must be absolute: file:/home/jenkins/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar

Pretty sure the above is not caused by my change

AmplabJenkins · 2015-05-05T01:12:10Z

Merged build triggered.

AmplabJenkins · 2015-05-05T01:12:15Z

Merged build started.

SparkQA · 2015-05-05T01:12:36Z

Test build #31809 has started for PR 5897 at commit 093b7a4.

SparkQA · 2015-05-05T02:24:41Z

Test build #31809 has finished for PR 5897 at commit 093b7a4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ShuffleHandle(val shuffleId: Int) extends Serializable
- * class SomethingNotSerializable
- logDebug(s" + cloning the object $obj of class $
- abstract class Evaluator extends Params
- abstract class PipelineStage extends Params with Logging
- class BinaryClassificationEvaluator extends Evaluator with HasRawPredictionCol with HasLabelCol
- trait LDAOptimizer
- class EMLDAOptimizer extends LDAOptimizer
- class OnlineLDAOptimizer extends LDAOptimizer
- class SaslEncryption
- static class EncryptedMessage extends AbstractReferenceCounted implements FileRegion
- class SaslRpcHandler extends RpcHandler
- public class SaslServerBootstrap implements TransportServerBootstrap
- public class SparkSaslClient implements SaslEncryptionBackend
- public class SparkSaslServer implements SaslEncryptionBackend
- public class ByteArrayWritableChannel implements WritableByteChannel
- class ParamGridBuilder(object):
- abstract class Dialect
- class DialectException(msg: String, cause: Throwable) extends Exception(msg, cause)
- case class HiveDatabase(
- abstract class TableType
- case class HiveStorageDescriptor(
- case class HivePartition(
- case class HiveColumn(name: String, hiveType: String, comment: String)
- case class HiveTable(
- trait ClientInterface
- class ClientWrapper(
- class IsolatedClientLoader(
- protected trait ReflectionMagic
- protected implicit class InstanceMagic(a: Any)
- protected implicit class StaticMagic(c: Class[_])

AmplabJenkins · 2015-05-05T02:24:46Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-05T02:24:47Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31809/
Test FAILed.

tedyu · 2015-05-05T02:34:13Z

From https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31809/testReport/junit/org.apache.spark.deploy/SparkSubmitSuite/includes_jars_passed_in_through___jars/ :

sbt.ForkMain$ForkError: The code passed to failAfter did not complete within 60 seconds.
at org.scalatest.concurrent.Timeouts$$anonfun$failAfter$1.apply(Timeouts.scala:249)
at org.scalatest.concurrent.Timeouts$$anonfun$failAfter$1.apply(Timeouts.scala:249)

I think the above was not related to my change.

bq. This patch adds the following public classes (experimental)

This change doesn't add any public class.

rxin · 2015-05-05T05:18:15Z

Jenkins, retest this please.

SparkQA · 2015-05-06T21:51:05Z

Test build #32026 has finished for PR 5897 at commit 1719c5b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Model(Transformer):
- class PipelineModel(Model):
- class CrossValidator(Estimator):
- class CrossValidatorModel(Model):
- class JavaModel(Model, JavaTransformer):

AmplabJenkins · 2015-05-06T21:51:10Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-06T21:51:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32026/
Test FAILed.

tedyu · 2015-05-06T22:01:01Z

[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala:769: not found: type HiveCompatibilitySuite
[error] extends HiveCompatibilitySuite with BeforeAndAfter {
[error] ^
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala:828: value testCases is not a member of org.scalatest.BeforeAndAfter
[error] override def testCases: Seq[(String, File)] = super.testCases.filter {
[error] ^

I don't think the above error was related to my change.
BitSetSuite passed:

[info] Test org.apache.spark.unsafe.bitset.BitSetSuite.basicOps started
[info] Test org.apache.spark.unsafe.bitset.BitSetSuite.traversal started
[info] Test run finished: 0 failed, 0 ignored, 2 total, 0.012s

SparkQA · 2015-05-06T22:03:10Z

Test build #771 has started for PR 5897 at commit 1719c5b.

JoshRosen · 2015-05-06T22:04:16Z

unsafe/src/main/java/org/apache/spark/unsafe/bitset/BitSet.java

+   * @return whether any bit in the BitSet is set
+   */
+  public boolean anySet() {
+    return BitSetMethods.anySet(baseObject, baseOffset, numWords*BitSetMethods.WORD_SIZE);


As I mentioned in an earlier comment, we can assume that BitSetMethods.anySet will be called with a size that's word aligned, so for consistency with nextSetBit I think we should change anySet to accept a size that's measured in numbers of words. This would let us avoid the mod calculations or multiplications by word size.

AmplabJenkins · 2015-05-06T22:17:10Z

Merged build triggered.

AmplabJenkins · 2015-05-06T22:17:16Z

Merged build started.

SparkQA · 2015-05-06T22:19:02Z

Test build #32037 has started for PR 5897 at commit 473bf9d.

SparkQA · 2015-05-06T23:51:55Z

Test build #771 has finished for PR 5897 at commit 1719c5b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-05-07T00:01:59Z

Test build #32037 has finished for PR 5897 at commit 473bf9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-07T00:02:03Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-07T00:02:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32037/
Test PASSed.

andrewor14 · 2015-05-07T19:36:57Z

@tedyu also please file a JIRA that tracks this improvement and put it in the title. See how other PRs are opened.

tedyu · 2015-05-07T20:53:25Z

SPARK-7450 has been filed

JoshRosen · 2015-05-07T23:51:55Z

This looks fine; I have two minor nits because I'm really pedantic, but I'll just deal with it on merge.

JoshRosen · 2015-05-07T23:52:05Z

unsafe/src/main/java/org/apache/spark/unsafe/bitset/BitSetMethods.java

@@ -28,7 +28,7 @@
 */
 public final class BitSetMethods {

-  private static final long WORD_SIZE = 8;
+  static final long WORD_SIZE = 8;


Could remain private.

tedyu · 2015-05-07T23:53:55Z

bq. I'm really pedantic

I like that :-)

Author: tedyu <yuzhihong@gmail.com> Closes #5897 from tedyu/master and squashes the following commits: 473bf9d [tedyu] Address Josh's review comments 1719c5b [tedyu] Correct upper bound in for loop b51dcaf [tedyu] Add unit test in BitSetSuite for BitSet#anySet() 83f9f87 [tedyu] Merge branch 'master' of github.com:apache/spark 817e3f9 [tedyu] Replace constant 8 with SIZE_OF_LONG 75a467b [tedyu] Correct offset for UNSAFE.getLong() 855374b [tedyu] Remove second loop since bitSetWidthInBytes is WORD aligned 093b7a4 [tedyu] Use UNSAFE.getLong() to speed up BitSetMethods#anySet() 63ee050 [tedyu] Use UNSAFE.getLong() to speed up BitSetMethods#anySet() 4ca0ef6 [tedyu] Use UNSAFE.getLong() to speed up BitSetMethods#anySet() 3e9b691 [tedyu] Use UNSAFE.getLong() to speed up BitSetMethods#anySet() (cherry picked from commit 88063c6) Signed-off-by: Josh Rosen <joshrosen@databricks.com>

JoshRosen · 2015-05-07T23:58:21Z

Merged to master and branch-1.4 (1.4.0). Thanks!

Author: tedyu <yuzhihong@gmail.com> Closes apache#5897 from tedyu/master and squashes the following commits: 473bf9d [tedyu] Address Josh's review comments 1719c5b [tedyu] Correct upper bound in for loop b51dcaf [tedyu] Add unit test in BitSetSuite for BitSet#anySet() 83f9f87 [tedyu] Merge branch 'master' of github.com:apache/spark 817e3f9 [tedyu] Replace constant 8 with SIZE_OF_LONG 75a467b [tedyu] Correct offset for UNSAFE.getLong() 855374b [tedyu] Remove second loop since bitSetWidthInBytes is WORD aligned 093b7a4 [tedyu] Use UNSAFE.getLong() to speed up BitSetMethods#anySet() 63ee050 [tedyu] Use UNSAFE.getLong() to speed up BitSetMethods#anySet() 4ca0ef6 [tedyu] Use UNSAFE.getLong() to speed up BitSetMethods#anySet() 3e9b691 [tedyu] Use UNSAFE.getLong() to speed up BitSetMethods#anySet()

Use UNSAFE.getLong() to speed up BitSetMethods#anySet()

3e9b691

Use UNSAFE.getLong() to speed up BitSetMethods#anySet()

4ca0ef6

rxin reviewed May 5, 2015
View reviewed changes

Use UNSAFE.getLong() to speed up BitSetMethods#anySet()

63ee050

rxin reviewed May 5, 2015
View reviewed changes

Use UNSAFE.getLong() to speed up BitSetMethods#anySet()

093b7a4

JoshRosen reviewed May 5, 2015
View reviewed changes

JoshRosen reviewed May 6, 2015
View reviewed changes

Address Josh's review comments

473bf9d

tedyu changed the title ~~Use UNSAFE.getLong() to speed up BitSetMethods#anySet()~~ SPARK-7450 Use UNSAFE.getLong() to speed up BitSetMethods#anySet() May 7, 2015

JoshRosen reviewed May 7, 2015
View reviewed changes

asfgit closed this in 88063c6 May 7, 2015

SPARK-7450 Use UNSAFE.getLong() to speed up BitSetMethods#anySet() #5897

SPARK-7450 Use UNSAFE.getLong() to speed up BitSetMethods#anySet() #5897

Conversation

tedyu commented May 4, 2015

AmplabJenkins commented May 5, 2015

JoshRosen commented May 5, 2015

AmplabJenkins commented May 5, 2015

AmplabJenkins commented May 5, 2015

SparkQA commented May 5, 2015

SparkQA commented May 5, 2015

AmplabJenkins commented May 5, 2015

AmplabJenkins commented May 5, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented May 5, 2015

Choose a reason for hiding this comment

AmplabJenkins commented May 5, 2015

SparkQA commented May 5, 2015

SparkQA commented May 5, 2015

AmplabJenkins commented May 5, 2015

AmplabJenkins commented May 5, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tedyu commented May 5, 2015

AmplabJenkins commented May 5, 2015

AmplabJenkins commented May 5, 2015

SparkQA commented May 5, 2015

SparkQA commented May 5, 2015

AmplabJenkins commented May 5, 2015

AmplabJenkins commented May 5, 2015

tedyu commented May 5, 2015

rxin commented May 5, 2015

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

tedyu commented May 6, 2015

SparkQA commented May 6, 2015

Choose a reason for hiding this comment

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

SparkQA commented May 6, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

andrewor14 commented May 7, 2015

tedyu commented May 7, 2015

JoshRosen commented May 7, 2015

Choose a reason for hiding this comment

tedyu commented May 7, 2015

JoshRosen commented May 7, 2015