[WIP] Secure Dictionary Server Implementation #1152

sounakr · 2017-07-10T04:34:05Z

Secure Dictionary Implementation Along with Non Secure.

asfgit · 2017-07-10T04:34:07Z

Can one of the admins verify this patch?

asfgit · 2017-07-10T04:34:07Z

Can one of the admins verify this patch?

CarbonDataQA · 2017-07-10T04:37:41Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2983/

CarbonDataQA · 2017-07-10T04:48:43Z

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/394/

sounakr · 2017-07-10T11:18:14Z

retest this please

CarbonDataQA · 2017-07-10T11:21:40Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2991/

CarbonDataQA · 2017-07-10T11:28:46Z

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/402/

sounakr · 2017-07-10T11:31:51Z

retest this please

CarbonDataQA · 2017-07-10T11:33:41Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2992/

CarbonDataQA · 2017-07-10T11:43:46Z

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/403/

sounakr · 2017-07-10T13:19:04Z

retest this please

CarbonDataQA · 2017-07-10T13:23:13Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/406/

CarbonDataQA · 2017-07-10T13:26:25Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2995/

CarbonDataQA · 2017-07-10T13:42:24Z

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/408/

CarbonDataQA · 2017-07-10T13:48:46Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2997/

CarbonDataQA · 2017-07-10T15:45:32Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/417/

CarbonDataQA · 2017-07-10T16:04:30Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3006/

CarbonDataQA · 2017-07-11T04:13:05Z

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/424/

CarbonDataQA · 2017-07-11T04:14:29Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3012/

gvramana · 2017-07-12T06:04:18Z

...main/java/org/apache/carbondata/core/dictionary/client/NonSecureDictionaryClientHandler.java

@@ -57,7 +57,7 @@ public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception
    try {
      ByteBuf data = (ByteBuf) msg;
      DictionaryMessage key = new DictionaryMessage();
-      key.readData(data);
+      key.readNonSecureData(data);


function can be named readskiplength

gvramana · 2017-07-12T06:04:45Z

...src/main/java/org/apache/carbondata/core/dictionary/generator/ServerDictionaryGenerator.java

@@ -26,6 +26,7 @@
 import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
 import org.apache.carbondata.core.metadata.schema.table.column.CarbonDimension;

+


remove extra line

gvramana · 2017-07-12T06:05:59Z

core/src/main/java/org/apache/carbondata/core/dictionary/generator/key/DictionaryMessage.java

@@ -50,7 +50,32 @@
   */
  private DictionaryMessageType type;

-  public void readData(ByteBuf byteBuf) {
+  public void readNonSecureData(ByteBuf byteBuf) {
+    byteBuf.resetReaderIndex();


remove this

gvramana · 2017-07-12T06:08:04Z

core/src/main/java/org/apache/carbondata/core/dictionary/server/DictionaryServer.java

-    // Configure the server.
-    bindToPort(port);
-  }
+  public void startServer(SparkConf conf, String host, int port);


This interface should not contain SparkConf and hostname

gvramana · 2017-07-12T06:08:50Z

core/src/main/java/org/apache/carbondata/core/dictionary/server/DictionaryServer.java

-      i++;
-    }
-  }
+  public void bindToPort(SparkConf orgConf, String host, int port);


only port is sufficient

gvramana · 2017-07-12T06:11:13Z

core/src/main/java/org/apache/carbondata/core/dictionary/server/DictionaryServer.java

-    key.setType(DictionaryMessageType.WRITE_DICTIONARY);
-    dictionaryServerHandler.processMessage(key);
-  }
+  public String getSecretKey();


This is common interface remove getSecretKey

gvramana · 2017-07-12T06:12:32Z

core/src/main/java/org/apache/carbondata/core/dictionary/server/NonSecureDictionaryServer.java

+    nonSecureDictionaryServerHandler.processMessage(key);
+  }
+
+  @Override public String findLocalIpAddress() {


move all the common function to abstract class

All cannot be moved as some of the class belongs to Spark2 and some in Core.

gvramana · 2017-07-12T06:13:40Z

...main/java/org/apache/carbondata/core/dictionary/server/NonSecureDictionaryServerHandler.java

@@ -76,8 +75,7 @@ public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception
   * @param ctx
   * @param cause
   */
-  @Override
-  public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
+  @Override public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {


revert this change

gvramana · 2017-07-12T06:14:49Z

core/src/main/java/org/apache/carbondata/core/dictionary/service/DictionaryOnePassService.java

+
+package org.apache.carbondata.core.dictionary.service;
+
+public class DictionaryOnePassService {


Add class headers explaining purpose

gvramana · 2017-07-12T06:15:11Z

...n/java/org/apache/carbondata/core/dictionary/service/NonSecureDictionaryServiceProvider.java

+    this.port = port;
+  }
+
+  //  @Override public DictionaryServer getDictionaryServer() {


use dictionary server provider to get instance

Sending the server instance in provider is not needed.

gvramana · 2017-07-12T06:18:24Z

...ation/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala

@@ -59,7 +59,7 @@ class SerializableConfiguration(@transient var value: Configuration) extends Ser
  @transient
  private val LOGGER = LogServiceFactory.getLogService(this.getClass.getCanonicalName)

-  private def writeObject(out: ObjectOutputStream): Unit =
+  private def writeObject(out: ObjectOutputStream): Unit = {


revert this change

gvramana · 2017-07-12T07:11:10Z

...c/main/java/org/apache/carbondata/spark/dictionary/client/SecureDictionaryClientHandler.java

+    DictionaryMessage dictionaryMessage;
+    ByteBuffer resp = null;
+    try {
+


clean buffers and data in finally block

Handles. If response is Null the data will be handled.

gvramana · 2017-07-12T07:14:35Z

...c/main/java/org/apache/carbondata/spark/dictionary/client/SecureDictionaryClientHandler.java

+      ByteBuf buffer = ByteBufAllocator.DEFAULT.heapBuffer();
+      key.writeData(buffer);
+      resp = client.sendRpcSync(buffer.nioBuffer(), 100000);
+    } catch (Exception e) {


resp failure should be handled

aLready Handled

gvramana · 2017-07-12T07:19:45Z

...c/main/java/org/apache/carbondata/spark/dictionary/client/SecureDictionaryClientHandler.java

+    return new OneForOneStreamManager();
+  }
+
+  private static class DictionaryChannelFutureListener implements ChannelFutureListener {


remove this listener

gvramana · 2017-07-12T07:23:04Z

...ark2/src/main/java/org/apache/carbondata/spark/dictionary/server/SecureDictionaryServer.java

+   */
+  @Override
+  public void startServer(SparkConf conf, String host, int port) {
+    secureDictionaryServerHandler = new SecureDictionaryServerHandler();


this configurations should be converted to TransportConf configurations

Secure Dictionary Implementation Along with Non Secure.

sounakr · 2017-07-12T14:41:07Z

retest this please

CarbonDataQA · 2017-07-12T14:41:19Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3047/

CarbonDataQA · 2017-07-12T14:44:48Z

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/459/

CarbonDataQA · 2017-07-12T14:53:21Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3048/

CarbonDataQA · 2017-07-12T14:54:48Z

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/460/

CarbonDataQA · 2017-08-02T08:45:44Z

Can one of the admins verify this patch?

CarbonDataQA · 2017-09-13T00:33:47Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/37/

ravipesala · 2017-10-23T09:51:09Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1215/

CarbonDataQA · 2017-10-23T15:17:11Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/584/

sounakr force-pushed the Secure_Dictionry branch from d0aca3a to fa7d4b0 Compare July 10, 2017 13:07

sounakr force-pushed the Secure_Dictionry branch from fa7d4b0 to f3693f3 Compare July 10, 2017 15:42

sounakr force-pushed the Secure_Dictionry branch from f3693f3 to cf365df Compare July 11, 2017 03:56

sounakr changed the title ~~[WIP] Secure Dictionary Server Implementation~~ [CARBONDATA-1288] Secure Dictionary Server Implementation Jul 11, 2017

gvramana reviewed Jul 12, 2017

View reviewed changes

Secure Dictionary Server Implementation

28aa235

Secure Dictionary Implementation Along with Non Secure.

sounakr force-pushed the Secure_Dictionry branch from cf365df to 28aa235 Compare July 12, 2017 14:29

sounakr changed the title ~~[CARBONDATA-1288] Secure Dictionary Server Implementation~~ [WIP] Secure Dictionary Server Implementation Aug 2, 2017

sounakr closed this Dec 14, 2017

		@@ -26,6 +26,7 @@
		import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
		import org.apache.carbondata.core.metadata.schema.table.column.CarbonDimension;


		package org.apache.carbondata.core.dictionary.service;

		public class DictionaryOnePassService {

[WIP] Secure Dictionary Server Implementation #1152

[WIP] Secure Dictionary Server Implementation #1152

Conversation

sounakr commented Jul 10, 2017

asfgit commented Jul 10, 2017

asfgit commented Jul 10, 2017

CarbonDataQA commented Jul 10, 2017

CarbonDataQA commented Jul 10, 2017

sounakr commented Jul 10, 2017

CarbonDataQA commented Jul 10, 2017

CarbonDataQA commented Jul 10, 2017

sounakr commented Jul 10, 2017

CarbonDataQA commented Jul 10, 2017

CarbonDataQA commented Jul 10, 2017

sounakr commented Jul 10, 2017

CarbonDataQA commented Jul 10, 2017

CarbonDataQA commented Jul 10, 2017

CarbonDataQA commented Jul 10, 2017

CarbonDataQA commented Jul 10, 2017

CarbonDataQA commented Jul 10, 2017

CarbonDataQA commented Jul 10, 2017

CarbonDataQA commented Jul 11, 2017

CarbonDataQA commented Jul 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gvramana Jul 12, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gvramana Jul 12, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sounakr commented Jul 12, 2017

CarbonDataQA commented Jul 12, 2017

CarbonDataQA commented Jul 12, 2017

CarbonDataQA commented Jul 12, 2017

CarbonDataQA commented Jul 12, 2017

CarbonDataQA commented Aug 2, 2017

CarbonDataQA commented Sep 13, 2017

ravipesala commented Oct 23, 2017

CarbonDataQA commented Oct 23, 2017

gvramana Jul 12, 2017 •

edited

gvramana Jul 12, 2017 •

edited