[SPARK-19520][streaming] Do not encrypt data written to the WAL. #16862

vanzin · 2017-02-09T00:55:00Z

Spark's I/O encryption uses an ephemeral key for each driver instance.
So driver B cannot decrypt data written by driver A since it doesn't
have the correct key.

The write ahead log is used for recovery, thus needs to be readable by
a different driver. So it cannot be encrypted by Spark's I/O encryption
code.

The BlockManager APIs used by the WAL code to write the data automatically
encrypt data, so changes are needed so that callers can to opt out of
encryption.

Aside from that, the "putBytes" API in the BlockManager does not do
encryption, so a separate situation arised where the WAL would write
unencrypted data to the BM and, when those blocks were read, decryption
would fail. So the WAL code needs to ask the BM to encrypt that data
when encryption is enabled; this code is not optimal since it results
in a (temporary) second copy of the data block in memory, but should be
OK for now until a more performant solution is added. The non-encryption
case should not be affected.

Tested with new unit tests, and by running streaming apps that do
recovery using the WAL data with I/O encryption turned on.

Spark's I/O encryption uses an ephemeral key for each driver instance. So driver B cannot decrypt data written by driver A since it doesn't have the correct key. The write ahead log is used for recovery, thus needs to be readable by a different driver. So it cannot be encrypted by Spark's I/O encryption code. The BlockManager APIs used by the WAL code to write the data automatically encrypt data, so changes are needed so that callers can to opt out of encryption. Aside from that, the "putBytes" API in the BlockManager does not do encryption, so a separate situation arised where the WAL would write unencrypted data to the BM and, when those blocks were read, decryption would fail. So the WAL code needs to ask the BM to encrypt that data when encryption is enabled; this code is not optimal since it results in a (temporary) second copy of the data block in memory, but should be OK for now until a more performant solution is added. The non-encryption case should not be affected. Tested with new unit tests, and by running streaming apps that do recovery using the WAL data with I/O encryption turned on.

vanzin · 2017-02-09T00:55:17Z

@tdas @zsxwing

SparkQA · 2017-02-09T03:07:45Z

Test build #72614 has finished for PR 16862 at commit bdbe267.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2017-02-09T17:55:51Z

Kafka test error (hopefully unrelated?). Retest this please

SparkQA · 2017-02-09T20:46:09Z

Test build #72654 has finished for PR 16862 at commit bdbe267.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

squito · 2017-02-09T21:21:16Z

core/src/main/scala/org/apache/spark/storage/BlockManager.scala

@@ -758,9 +761,33 @@ private[spark] class BlockManager(
      blockId: BlockId,
      bytes: ChunkedByteBuffer,
      level: StorageLevel,
-      tellMaster: Boolean = true): Boolean = {
+      tellMaster: Boolean = true,
+      encrypt: Boolean = false): Boolean = {


I think its worth documenting this param. At first I was going to suggest that it should be called allowEncryption like the other one, but I realize its more complicated than that. Maybe something like

If true, the given bytes should be encrypted before they are stored. Note that in most cases, the given bytes will already be encrypted if encryption is on. An important exception to this is with the streaming WAL. Since the WAL does not support encryption, those bytes are generated un-encrypted. But we still encrypt those bytes before storing in the block manager.

Maybe too wordy but I think its worth documenting.

squito

one minor suggestion for a doc, otherwise lgtm

zsxwing · 2017-02-09T23:31:45Z

streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala

@@ -158,13 +158,16 @@ class WriteAheadLogBackedBlockRDD[T: ClassTag](
      logInfo(s"Read partition data of $this from write ahead log, record handle " +
        partition.walRecordHandle)
      if (storeInBlockManager) {
-        blockManager.putBytes(blockId, new ChunkedByteBuffer(dataRead.duplicate()), storageLevel)
+        blockManager.putBytes(blockId, new ChunkedByteBuffer(dataRead.duplicate()), storageLevel,


Why encrypt should be true here? In the following codes, it just reads the block using maybeEncrypted = false.

This is explained in the summary.

The code that uses maybeEncrypted = false is deserializing data read from the WAL. This code is adding the block to the block manager, which later is read with getBlockFromBlockManager which calls blockManager.get which does decryption automatically.

zsxwing · 2017-02-09T23:58:18Z

Looks good.

SparkQA · 2017-02-10T03:57:54Z

Test build #72667 has finished for PR 16862 at commit 42d5d10.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2017-02-13T22:19:06Z

We can remove the putBytes workaround when I (or someone else) get around to fixing SPARK-19556.

For now, I'm merging this to master and 2.1.

Spark's I/O encryption uses an ephemeral key for each driver instance. So driver B cannot decrypt data written by driver A since it doesn't have the correct key. The write ahead log is used for recovery, thus needs to be readable by a different driver. So it cannot be encrypted by Spark's I/O encryption code. The BlockManager APIs used by the WAL code to write the data automatically encrypt data, so changes are needed so that callers can to opt out of encryption. Aside from that, the "putBytes" API in the BlockManager does not do encryption, so a separate situation arised where the WAL would write unencrypted data to the BM and, when those blocks were read, decryption would fail. So the WAL code needs to ask the BM to encrypt that data when encryption is enabled; this code is not optimal since it results in a (temporary) second copy of the data block in memory, but should be OK for now until a more performant solution is added. The non-encryption case should not be affected. Tested with new unit tests, and by running streaming apps that do recovery using the WAL data with I/O encryption turned on. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #16862 from vanzin/SPARK-19520. (cherry picked from commit 0169360) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

Spark's I/O encryption uses an ephemeral key for each driver instance. So driver B cannot decrypt data written by driver A since it doesn't have the correct key. The write ahead log is used for recovery, thus needs to be readable by a different driver. So it cannot be encrypted by Spark's I/O encryption code. The BlockManager APIs used by the WAL code to write the data automatically encrypt data, so changes are needed so that callers can to opt out of encryption. Aside from that, the "putBytes" API in the BlockManager does not do encryption, so a separate situation arised where the WAL would write unencrypted data to the BM and, when those blocks were read, decryption would fail. So the WAL code needs to ask the BM to encrypt that data when encryption is enabled; this code is not optimal since it results in a (temporary) second copy of the data block in memory, but should be OK for now until a more performant solution is added. The non-encryption case should not be affected. Tested with new unit tests, and by running streaming apps that do recovery using the WAL data with I/O encryption turned on. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes apache#16862 from vanzin/SPARK-19520.

squito reviewed Feb 9, 2017

View reviewed changes

zsxwing reviewed Feb 9, 2017

View reviewed changes

Add comment about "encrypt" parameter.

42d5d10

asfgit closed this in 0169360 Feb 13, 2017

vanzin deleted the SPARK-19520 branch February 15, 2017 00:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-19520][streaming] Do not encrypt data written to the WAL. #16862

[SPARK-19520][streaming] Do not encrypt data written to the WAL. #16862

vanzin commented Feb 9, 2017

vanzin commented Feb 9, 2017

SparkQA commented Feb 9, 2017

vanzin commented Feb 9, 2017

SparkQA commented Feb 9, 2017

squito Feb 9, 2017

zsxwing Feb 9, 2017

squito left a comment

zsxwing Feb 9, 2017

vanzin Feb 9, 2017

zsxwing Feb 9, 2017

zsxwing commented Feb 9, 2017

SparkQA commented Feb 10, 2017

vanzin commented Feb 13, 2017

[SPARK-19520][streaming] Do not encrypt data written to the WAL. #16862

[SPARK-19520][streaming] Do not encrypt data written to the WAL. #16862

Conversation

vanzin commented Feb 9, 2017

vanzin commented Feb 9, 2017

SparkQA commented Feb 9, 2017

vanzin commented Feb 9, 2017

SparkQA commented Feb 9, 2017

squito Feb 9, 2017

Choose a reason for hiding this comment

zsxwing Feb 9, 2017

Choose a reason for hiding this comment

squito left a comment

Choose a reason for hiding this comment

zsxwing Feb 9, 2017

Choose a reason for hiding this comment

vanzin Feb 9, 2017

Choose a reason for hiding this comment

zsxwing Feb 9, 2017

Choose a reason for hiding this comment

zsxwing commented Feb 9, 2017

SparkQA commented Feb 10, 2017

vanzin commented Feb 13, 2017