[SPARK-9451] [SQL] Support entries larger than default page size in BytesToBytesMap & integrate with ShuffleMemoryManager #7762

JoshRosen · 2015-07-30T00:19:52Z

This patch adds support for entries larger than the default page size in BytesToBytesMap. These large rows are handled by allocating special overflow pages to hold individual entries.

In addition, this patch integrates BytesToBytesMap with the ShuffleMemoryManager:

Move BytesToBytesMap from unsafe to core so that it can import ShuffleMemoryManager.
Before allocating new data pages, ask the ShuffleMemoryManager to reserve the memory:
- putNewKey() now returns a boolean to indicate whether the insert succeeded or failed due to a lack of memory. The caller can use this value to respond to the memory pressure (e.g. by spilling).
UnsafeFixedWidthAggregationMap. getAggregationBuffer() now returns null to signal failure due to a lack of memory.
Updated all uses of these classes to handle these error conditions.
Added new tests for allocating large records and for allocations which fail due to memory pressure.
Extended the afterAll() test teardown methods to detect ShuffleMemoryManager leaks.

SparkQA · 2015-07-30T02:13:37Z

Test build #38926 has finished for PR 7762 at commit c801788.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-31T01:07:01Z

Test build #39118 has finished for PR 7762 at commit 2cd3570.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class UnixTimestamp(timeExp: Expression, format: Expression)
- case class FromUnixTime(sec: Expression, format: Expression)
- case class TimeAdd(start: Expression, interval: Expression)
- case class TimeSub(start: Expression, interval: Expression)
- case class AddMonths(startDate: Expression, numMonths: Expression)
- case class MonthsBetween(date1: Expression, date2: Expression)
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

JoshRosen · 2015-07-31T01:42:22Z

/cc @rxin for review. Added a very basic test; can add more focused unit tests later.

SparkQA · 2015-07-31T03:54:17Z

Test build #39132 has finished for PR 7762 at commit bea1152.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2015-07-31T04:53:23Z

unsafe/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java

+      if (useOverflowPage) {
+        // The record is larger than the page size, so allocate a special overflow page just to hold
+        // that record.
+        MemoryBlock overflowPage = memoryManager.allocatePage(requiredSize + 8);


don't we need to check whether we can get this page or not? it's possible we are running out, isn't it?

BytesToBytesMap does not currently integrate with the ShuffleMemoryManager since it doesn't support spilling. It probably should, though, if only to ensure that its memory consumption is counted towards the task's overall memory allowance. Currently, any failures to allocate here will manifest as OOMs.

I think that part of the problem is that BytesToBytesMap itself doesn't really know how to handle failure to allocate memory since it can't spill on its own. Intuitively, I guess that the right approach is to make sure that allocation failures do not leave the map in an inconsistent state then to propagate the failure to the caller so that they can determine what to do (fallback, spill the map, sort, etc).

Unfortunately, we're going to have to move BytesToBytesMap out of unsafe and into core so it can integrate with ShuffleMemoryManager.

rxin · 2015-07-31T04:55:21Z

The approach lgtm.

SparkQA · 2015-07-31T20:16:24Z

Test build #39258 has finished for PR 7762 at commit ec4484c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…nager can be integrated

SparkQA · 2015-07-31T23:03:08Z

Test build #39290 has finished for PR 7762 at commit 82fc657.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-07-31T23:03:43Z

sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala

        valuesBuffer, PlatformDependent.BYTE_ARRAY_OFFSET, valuesSize)
+      if (!putSuceeded) {
+        throw new IOException("Could not allocate memory to grow BytesToBytesMap")


This can never get hit; I only added this check for completeness.

rxin · 2015-07-31T23:41:43Z

This currently fails compilation because of some missing imports.

SparkQA · 2015-08-01T00:46:59Z

Test build #39287 has finished for PR 7762 at commit 31a525a.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

[SPARK-9451] [SQL] Support entries larger than default page size in BytesToBytesMap

SparkQA · 2015-08-01T01:04:46Z

Test build #39289 has finished for PR 7762 at commit 34ab943.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

rxin · 2015-08-01T01:05:17Z

LGTM.

SparkQA · 2015-08-01T02:06:16Z

Test build #39302 has finished for PR 7762 at commit ae7bc56.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-08-01T02:17:27Z

Test failure is unrelated, so I'm going to merge this into master.

…ernalSorter This patch extends UnsafeExternalSorter to support records larger than the page size. The basic strategy is the same as in #7762: store large records in their own overflow pages. Author: Josh Rosen <joshrosen@databricks.com> Closes #7891 from JoshRosen/large-records-in-sql-sorter and squashes the following commits: 967580b [Josh Rosen] Merge remote-tracking branch 'origin/master' into large-records-in-sql-sorter 948c344 [Josh Rosen] Add large records tests for KV sorter. 3c17288 [Josh Rosen] Combine memory and disk cleanup into general cleanupResources() method 380f217 [Josh Rosen] Merge remote-tracking branch 'origin/master' into large-records-in-sql-sorter 27eafa0 [Josh Rosen] Fix page size in PackedRecordPointerSuite a49baef [Josh Rosen] Address initial round of review comments 3edb931 [Josh Rosen] Remove accidentally-committed debug statements. 2b164e2 [Josh Rosen] Support large records in UnsafeExternalSorter. (cherry picked from commit ab8ee1a) Signed-off-by: Reynold Xin <rxin@databricks.com>

…ernalSorter This patch extends UnsafeExternalSorter to support records larger than the page size. The basic strategy is the same as in #7762: store large records in their own overflow pages. Author: Josh Rosen <joshrosen@databricks.com> Closes #7891 from JoshRosen/large-records-in-sql-sorter and squashes the following commits: 967580b [Josh Rosen] Merge remote-tracking branch 'origin/master' into large-records-in-sql-sorter 948c344 [Josh Rosen] Add large records tests for KV sorter. 3c17288 [Josh Rosen] Combine memory and disk cleanup into general cleanupResources() method 380f217 [Josh Rosen] Merge remote-tracking branch 'origin/master' into large-records-in-sql-sorter 27eafa0 [Josh Rosen] Fix page size in PackedRecordPointerSuite a49baef [Josh Rosen] Address initial round of review comments 3edb931 [Josh Rosen] Remove accidentally-committed debug statements. 2b164e2 [Josh Rosen] Support large records in UnsafeExternalSorter.

Basic support for large rows in BytesToBytesMap.

07ff9ef

JoshRosen force-pushed the large-rows branch from c801788 to 07ff9ef Compare July 30, 2015 22:55

Remove accidental duplicated code

2cd3570

JoshRosen changed the title ~~[SPARK-9451] [SQL] [WIP] Support records larger than default page size in BytesToBytesMap~~ [SPARK-9451] [SQL] [WIP] Support entries larger than default page size in BytesToBytesMap Jul 30, 2015

Add basic test.

bea1152

JoshRosen changed the title ~~[SPARK-9451] [SQL] [WIP] Support entries larger than default page size in BytesToBytesMap~~ [SPARK-9451] [SQL]\ Support entries larger than default page size in BytesToBytesMap Jul 31, 2015

JoshRosen changed the title ~~[SPARK-9451] [SQL]\ Support entries larger than default page size in BytesToBytesMap~~ [SPARK-9451] [SQL] Support entries larger than default page size in BytesToBytesMap Jul 31, 2015

rxin reviewed Jul 31, 2015
View reviewed changes

JoshRosen added 2 commits July 31, 2015 12:49

Rename size to numElements

642ed69

Move BytesToBytesMap from unsafe package to core.

ec4484c

JoshRosen added 4 commits July 31, 2015 13:28

Move code to sql/core and spark/core packages so that ShuffleMemoryMa…

626b33c

…nager can be integrated

Integrate BytesToBytesMap with ShuffleMemoryManager.

31a525a

Remove semi

34ab943

Merge remote-tracking branch 'origin/master' into large-rows

82fc657

JoshRosen reviewed Jul 31, 2015
View reviewed changes

Fix compilation

ae7bc56

JoshRosen changed the title ~~[SPARK-9451] [SQL] Support entries larger than default page size in BytesToBytesMap~~ [SPARK-9451] [SQL] Support entries larger than default page size in BytesToBytesMap & integrate with ShuffleMemoryManager Aug 1, 2015

rxin added a commit to rxin/spark that referenced this pull request Aug 1, 2015

Merge pull request apache#7762 from JoshRosen/large-rows

cc6141c

[SPARK-9451] [SQL] Support entries larger than default page size in BytesToBytesMap

asfgit closed this in 8cb415a Aug 1, 2015

JoshRosen mentioned this pull request Aug 3, 2015

[SPARK-9452] [SQL] Support records larger than page size in UnsafeExternalSorter #7891

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-9451] [SQL] Support entries larger than default page size in BytesToBytesMap & integrate with ShuffleMemoryManager #7762

[SPARK-9451] [SQL] Support entries larger than default page size in BytesToBytesMap & integrate with ShuffleMemoryManager #7762

JoshRosen commented Jul 30, 2015

SparkQA commented Jul 30, 2015

SparkQA commented Jul 31, 2015

JoshRosen commented Jul 31, 2015

SparkQA commented Jul 31, 2015

rxin Jul 31, 2015

JoshRosen Jul 31, 2015

JoshRosen Jul 31, 2015

rxin commented Jul 31, 2015

SparkQA commented Jul 31, 2015

SparkQA commented Jul 31, 2015

JoshRosen Jul 31, 2015

rxin commented Jul 31, 2015

SparkQA commented Aug 1, 2015

SparkQA commented Aug 1, 2015

rxin commented Aug 1, 2015

SparkQA commented Aug 1, 2015

JoshRosen commented Aug 1, 2015

[SPARK-9451] [SQL] Support entries larger than default page size in BytesToBytesMap & integrate with ShuffleMemoryManager #7762

[SPARK-9451] [SQL] Support entries larger than default page size in BytesToBytesMap & integrate with ShuffleMemoryManager #7762

Conversation

JoshRosen commented Jul 30, 2015

SparkQA commented Jul 30, 2015

SparkQA commented Jul 31, 2015

JoshRosen commented Jul 31, 2015

SparkQA commented Jul 31, 2015

rxin Jul 31, 2015

Choose a reason for hiding this comment

JoshRosen Jul 31, 2015

Choose a reason for hiding this comment

JoshRosen Jul 31, 2015

Choose a reason for hiding this comment

rxin commented Jul 31, 2015

SparkQA commented Jul 31, 2015

SparkQA commented Jul 31, 2015

JoshRosen Jul 31, 2015

Choose a reason for hiding this comment

rxin commented Jul 31, 2015

SparkQA commented Aug 1, 2015

SparkQA commented Aug 1, 2015

rxin commented Aug 1, 2015

SparkQA commented Aug 1, 2015

JoshRosen commented Aug 1, 2015