Skip to content

Commit

Permalink
Sort MemSQL partitions by ordinal in MemSQLRDD getPartitions()
Browse files Browse the repository at this point in the history
Summary:
As of MemSQL 6, "SHOW PARTITIONS ON `${dbName}`" no longer guarantees that the result set is in sorted order by ordinal, so users were hitting error messages like `requirement failed: partitions(0).partition == 21, but it should equal 0 ` (https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L2 )

This diff causes MemSQLRDD getPartitions() to sort the partition array by index before returning it.

Test Plan: sbt test

Reviewers: carl, jdkaplan

Reviewed By: jdkaplan

Subscribers: engineering-list

JIRA Issues: AP-1693

Differential Revision: https://grizzly.memsql.com/D24036
  • Loading branch information
Lucy Yu committed Jan 18, 2018
1 parent 7e876e6 commit 56abf86
Show file tree
Hide file tree
Showing 4 changed files with 7 additions and 4 deletions.
5 changes: 4 additions & 1 deletion CHANGELOG
@@ -1,4 +1,7 @@
2017-8-31 Version 2.0.4
2018-01-18 Version 2.0.5
* Explicitly sort MemSQLRDD partitions due to MemSQL 6.0 no longer returning partitions in sorted order by ordinal.

2017-08-31 Version 2.0.4
* Switch threads in LoadDataStrategy so that the parent thread reads from the RDD and the new thread writes
to MemSQL so that Spark has access to the thread-local variables it expects

Expand Down
2 changes: 1 addition & 1 deletion README.md
Expand Up @@ -21,7 +21,7 @@ Installation
Inside a project definition you can depend on the MemSQL Connector using sbt:

```
libraryDependencies += "com.memsql" %% "memsql-connector" % "2.0.4"
libraryDependencies += "com.memsql" %% "memsql-connector" % "2.0.5"
```

or Maven:
Expand Down
2 changes: 1 addition & 1 deletion build.sbt
Expand Up @@ -10,7 +10,7 @@ lazy val testScalastyle = taskKey[Unit]("testScalastyle")

lazy val commonSettings = Seq(
organization := "com.memsql",
version := "2.0.4",
version := "2.0.5",
scalaVersion := "2.11.8",
assemblyScalastyle := org.scalastyle.sbt.ScalastylePlugin.scalastyle.in(Compile).toTask("").value,
assembly <<= assembly dependsOn assemblyScalastyle,
Expand Down
Expand Up @@ -80,7 +80,7 @@ case class MemSQLRDD[T: ClassTag](@transient sc: SparkContext,

new MemSQLRDDPartition(ordinal, connInfo, Some(partitionQuery))
})
.toArray
.toArray.sortBy(_.index).asInstanceOf[Array[Partition]]
}
case None => {
getSinglePartition
Expand Down

0 comments on commit 56abf86

Please sign in to comment.