[BUG]: Hyperspace CreateIndex failed #580

pherbel · 2020-07-06T11:18:12Z

pherbel
Jul 6, 2020

Describe the bug
I try to test Hyperspace index with our dataset (CSV) and it seems failed during the creation.

To Reproduce

Steps to reproduce the behavior:

Create new .Net Spark Console app
Add .Net spark extension nuget package Hyperspace
Create Session
Add data frame
Create index

            var hs = new Hyperspace(spark);
            hs.CreateIndex(dataFrame, new IndexConfig("CarIndex", new string[] { "Car" }));
            hs.CreateIndex(dataFrame, new IndexConfig("TimeIndex", new string[] { "Tim" }));

            hs.Indexes().Show();

Expected behavior
New index created and app will show in the log.

Desktop (please complete the following information):

OS: Win 10 2004
Spark Version: 2.4.6
.Net Spark Version: 0.12.1

Additional context
Log
20/07/06 13:06:53 ERROR DotnetBackendHandler: Failed to execute 'createIndex' on 'com.microsoft.hyperspace.Hyperspace' with args=([Type=org.apache.spark.sql.Dataset, Value: [Car: int, Tim: int ... 16 more fields]], [Type=com.microsoft.hyperspace.index.IndexConfig, Value: [indexName: TimeIndex; indexedColumns: tim; includedColumns: ]])
[2020-07-06T11:06:53.2501359Z] [PHERBEL-DESKTOP] [Error] [JvmBridge] JVM method execution failed: Nonstatic method 'createIndex' failed for class '13' when called with 2 arguments ([Index=1, Type=DataFrame, Value=Microsoft.Spark.Sql.DataFrame], [Index=2, Type=IndexConfig, Value=[indexName: TimeIndex; indexedColumns: tim; includedColumns: ]], )
[2020-07-06T11:06:53.2501973Z] [PHERBEL-DESKTOP] [Error] [JvmBridge] com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
Serialization trace:
classes (sun.misc.Launcher$AppClassLoader)
classloader (java.security.ProtectionDomain)
context (java.security.AccessControlContext)
acc (org.apache.spark.util.MutableURLClassLoader)
classLoader (org.apache.hadoop.conf.Configuration)
conf (org.apache.hadoop.io.compress.BZip2Codec)
codecs (org.apache.hadoop.io.compress.CompressionCodecFactory)
codecFactory (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat)
fileFormat (com.microsoft.hyperspace.index.serde.package$HadoopFsRelationWrapper)
relation (com.microsoft.hyperspace.index.serde.package$LogicalRelationWrapper)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:361)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:302)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:629)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:86)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:113)
at com.esotericsoftware.kryo.serializers.DefaultSerializers$TreeMapSerializer.write(DefaultSerializers.java:684)
at com.esotericsoftware.kryo.serializers.DefaultSerializers$TreeMapSerializer.write(DefaultSerializers.java:680)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
at com.microsoft.hyperspace.index.serde.KryoSerDeUtils$.serialize(KryoSerDeUtils.scala:41)
at com.microsoft.hyperspace.index.serde.LogicalPlanSerDeUtils$.serialize(LogicalPlanSerDeUtils.scala:46)
at com.microsoft.hyperspace.actions.CreateActionBase.getIndexLogEntry(CreateActionBase.scala:61)
at com.microsoft.hyperspace.actions.CreateAction.logEntry$lzycompute(CreateAction.scala:36)
at com.microsoft.hyperspace.actions.CreateAction.logEntry(CreateAction.scala:35)
at com.microsoft.hyperspace.actions.Action$class.begin(Action.scala:49)
at com.microsoft.hyperspace.actions.Action$class.run(Action.scala:86)
at com.microsoft.hyperspace.actions.CreateAction.run(CreateAction.scala:27)
at com.microsoft.hyperspace.index.IndexCollectionManager.create(IndexCollectionManager.scala:42)
at com.microsoft.hyperspace.index.CachingIndexCollectionManager.create(CachingIndexCollectionManager.scala:77)
at com.microsoft.hyperspace.Hyperspace.createIndex(Hyperspace.scala:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:145)
at org.apache.spark.api.dotnet.DotnetBackendHandler.handleBackendRequest(DotnetBackendHandler.scala:85)
at org.apache.spark.api.dotnet.DotnetBackendHandler.channelRead0(DotnetBackendHandler.scala:28)
at org.apache.spark.api.dotnet.DotnetBackendHandler.channelRead0(DotnetBackendHandler.scala:23)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Caused by: java.util.ConcurrentModificationException
at java.util.Vector$Itr.checkForComodification(Unknown Source)
at java.util.Vector$Itr.next(Unknown Source)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:99)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
... 81 more

[2020-07-06T11:06:53.2638537Z] [PHERBEL-DESKTOP] [Exception] [JvmBridge] JVM method execution failed: Nonstatic method 'createIndex' failed for class '13' when called with 2 arguments ([Index=1, Type=DataFrame, Value=Microsoft.Spark.Sql.DataFrame], [Index=2, Type=IndexConfig, Value=[indexName: TimeIndex; indexedColumns: tim; includedColumns: ]], )
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
Unhandled exception. System.Exception: JVM method execution failed: Nonstatic method 'createIndex' failed for class '13' when called with 2 arguments ([Index=1, Type=DataFrame, Value=Microsoft.Spark.Sql.DataFrame], [Index=2, Type=IndexConfig, Value=[indexName: TimeIndex; indexedColumns: tim; includedColumns: ]], )
---> Microsoft.Spark.JvmException: com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
Serialization trace:
classes (sun.misc.Launcher$AppClassLoader)
classloader (java.security.ProtectionDomain)
context (java.security.AccessControlContext)
acc (org.apache.spark.util.MutableURLClassLoader)
classLoader (org.apache.hadoop.conf.Configuration)
conf (org.apache.hadoop.io.compress.BZip2Codec)
codecs (org.apache.hadoop.io.compress.CompressionCodecFactory)
codecFactory (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat)
fileFormat (com.microsoft.hyperspace.index.serde.package$HadoopFsRelationWrapper)
relation (com.microsoft.hyperspace.index.serde.package$LogicalRelationWrapper)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:361)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:302)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:629)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:86)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:113)
at com.esotericsoftware.kryo.serializers.DefaultSerializers$TreeMapSerializer.write(DefaultSerializers.java:684)
at com.esotericsoftware.kryo.serializers.DefaultSerializers$TreeMapSerializer.write(DefaultSerializers.java:680)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
at com.microsoft.hyperspace.index.serde.KryoSerDeUtils$.serialize(KryoSerDeUtils.scala:41)
at com.microsoft.hyperspace.index.serde.LogicalPlanSerDeUtils$.serialize(LogicalPlanSerDeUtils.scala:46)
at com.microsoft.hyperspace.actions.CreateActionBase.getIndexLogEntry(CreateActionBase.scala:61)
at com.microsoft.hyperspace.actions.CreateAction.logEntry$lzycompute(CreateAction.scala:36)
at com.microsoft.hyperspace.actions.CreateAction.logEntry(CreateAction.scala:35)
at com.microsoft.hyperspace.actions.Action$class.begin(Action.scala:49)
at com.microsoft.hyperspace.actions.Action$class.run(Action.scala:86)
at com.microsoft.hyperspace.actions.CreateAction.run(CreateAction.scala:27)
at com.microsoft.hyperspace.index.IndexCollectionManager.create(IndexCollectionManager.scala:42)
at com.microsoft.hyperspace.index.CachingIndexCollectionManager.create(CachingIndexCollectionManager.scala:77)
at com.microsoft.hyperspace.Hyperspace.createIndex(Hyperspace.scala:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:145)
at org.apache.spark.api.dotnet.DotnetBackendHandler.handleBackendRequest(DotnetBackendHandler.scala:85)
at org.apache.spark.api.dotnet.DotnetBackendHandler.channelRead0(DotnetBackendHandler.scala:28)
at org.apache.spark.api.dotnet.DotnetBackendHandler.channelRead0(DotnetBackendHandler.scala:23)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Caused by: java.util.ConcurrentModificationException
at java.util.Vector$Itr.checkForComodification(Unknown Source)
at java.util.Vector$Itr.next(Unknown Source)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:99)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
... 81 more

--- End of inner exception stack trace ---
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object arg0, Object arg1)
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference objectId, String methodName, Object arg0, Object arg1)
at Microsoft.Spark.Interop.Ipc.JvmObjectReference.Invoke(String methodName, Object arg0, Object arg1)
at Microsoft.Spark.Extensions.Hyperspace.Hyperspace.CreateIndex(DataFrame df, IndexConfig indexConfig)
at SparkConsoleAppTest.Program.Main(String[] args) in C:\Repos\src\SparkConsoleAppTest\Program.cs:line 37

suhsteve · 2020-07-06T16:45:06Z

suhsteve
Jul 6, 2020
Maintainer

@AFFogarty can you look into this ?

0 replies

imback82 · 2020-07-06T17:29:27Z

imback82
Jul 6, 2020

I see the following:

at com.microsoft.hyperspace.index.serde.KryoSerDeUtils$.serialize(KryoSerDeUtils.scala:41)
at com.microsoft.hyperspace.index.serde.LogicalPlanSerDeUtils$.serialize(LogicalPlanSerDeUtils.scala:46)
at com.microsoft.hyperspace.actions.CreateActionBase.getIndexLogEntry(CreateActionBase.scala:61)
at com.microsoft.hyperspace.actions.CreateAction.logEntry$lzycompute(CreateAction.scala:36)
at com.microsoft.hyperspace.actions.CreateAction.logEntry(CreateAction.scala:35)
at com.microsoft.hyperspace.actions.Action$class.begin(Action.scala:49)
at com.microsoft.hyperspace.actions.Action$class.run(Action.scala:86)
at com.microsoft.hyperspace.actions.CreateAction.run(CreateAction.scala:27)
at com.microsoft.hyperspace.index.IndexCollectionManager.create(IndexCollectionManager.scala:42)
at com.microsoft.hyperspace.index.CachingIndexCollectionManager.create(CachingIndexCollectionManager.scala:77)
at com.microsoft.hyperspace.Hyperspace.createIndex(Hyperspace.scala:41)

@pherbel Can you share how you are creating dataFrame?

cc: @apoorvedave1 / @pirz Can you try to repro with a CSV file format?

0 replies

apoorvedave1 · 2020-07-06T19:06:23Z

apoorvedave1
Jul 6, 2020

Thanks, @imback82
I can't repro the issue with this scala code. This was done locally and on scala.

object Test extends App {
  val spark: SparkSession = SparkSession
    .builder()
    .master(s"local[*]")
    .getOrCreate()
  import spark.implicits._
  val df = spark.range(1,5).toDF("id")
  df.write.mode("overwrite").option("header", true).csv("csvtest")


  val df2 = spark.read.option("header", true).csv("csvtest")
  val hs = new Hyperspace(spark)
  hs.createIndex(df2, new IndexConfig("ic1", Seq("id"), Seq()))      // worked fine
}

0 replies

pherbel · 2020-07-07T07:23:33Z

pherbel
Jul 7, 2020
Author

@imback82

Here is the other part of the source code.

            // Create a Spark session
            SparkSession spark = SparkSession
                .Builder()
                .AppName("CSVMessage_Processing")
                .GetOrCreate()
                .EnableHyperspace();


            // Create initial DataFrame
            DataFrame dataFrame = spark
                .Read()
                .Option("header", true)
                .Option("sep", "|")
                .Schema("Car INT, Tim INT,Msg INT, Lat INT, Lon INT, Dir INT,Spd INT," +
                        "Sta INT, Sns INT, Par INT, An0 INT, An1 INT, An2 INT, An3 INT," +
                        "An4 INT, An5 INT, An6 INT, An7 INT")
                .Csv(@"C:\Repos\sampledata\datalake\year=2020\month=06\day=22\hour=11\*\*.txt");

CSV format:

Car|Tim|Msg|Lat|Lon|Dir|Spd|Sta|Sns|Par|An0|An1|An2|An3|An4|An5|An6|An7|
4562344|1592823595|2|47393500|17375500|240|6|1|67108865|0|431|615|0|590|1023|1023|1592823596|0|

0 replies

pherbel · 2020-07-07T07:41:30Z

pherbel
Jul 7, 2020
Author

Also I don't know exactly that it could be a problem or not, but I installed the jar files manually for local development.

So, I downloaded the jar files (Delta, Hyperspace) and installed into the "jars" directory in my local spark install.
It seems to start working, but got the error.

0 replies

apoorvedave1 · 2020-07-07T16:50:22Z

apoorvedave1
Jul 7, 2020

Thanks @pherbel, I could repro the issue. I will update with my findings soon.

0 replies

apoorvedave1 · 2020-07-07T18:42:36Z

apoorvedave1
Jul 7, 2020

Thanks @pherbel, did you materialize the data before creating the index? I could repro the issue only if I call df.show() before hs.createIndex(). If I don't call df.show(), it works fine and creates index without issue.

We are working to fix this problem. In the mean time, could you avoid calling df.show() or any other form of materializing the df before creating index?

Here's my repro

object Test extends App {
  val spark: SparkSession = ...
  
  val df = spark.read.csv("csv/*/*")
  df.show()                            // commenting this line eliminates the issue
  val hs = new Hyperspace(spark)
  hs.createIndex(df, new IndexConfig("ic2", Seq("_c0"), Seq()))
}

0 replies

apoorvedave1 · 2020-07-07T18:47:08Z

apoorvedave1
Jul 7, 2020

Explanation: In case of csv, on materializing the df before creating index, it creates some internal objects which don't support serialization (specifically CompressionCodecFactory)

PR with fix: microsoft/hyperspace#82

0 replies

imback82 · 2020-07-08T04:15:42Z

imback82
Jul 8, 2020

@pherbel This issue has been fixed and SNAPSHOT artifacts have been published. And you can consume it as follows:

spark-submit --packages com.microsoft.hyperspace:hyperspace-core_2.11:0.2.0-SNAPSHOT --repositories https://oss.sonatype.org/content/repositories/snapshots/  <other args>

Please let us know if this fixes your issue. Thanks!

0 replies

pherbel · 2020-07-08T05:47:24Z

pherbel
Jul 8, 2020
Author

@apoorvedave1 Thanks for the quick work on it. This is great.
I don't call dataframe.Show() before the index creation.

@imback82 Unfortunately, I couldn't get the packages with this method. Does this need authentication?
Can I access the jar directly somehow?

0 replies

imback82 · 2020-07-08T05:58:14Z

imback82
Jul 8, 2020

@pherbel Can you get to this? https://oss.sonatype.org/content/repositories/snapshots/com/microsoft/hyperspace/hyperspace-core_2.11/0.2.0-SNAPSHOT/

0 replies

pherbel · 2020-07-08T06:56:24Z

pherbel
Jul 8, 2020
Author

@imback82 Thanks. I could download it.

Now It seems works! Great!

I really appreciate your quick work on it. Thanks for everybody.

Now I can start to test the index.

0 replies

pherbel · 2020-07-08T07:05:44Z

pherbel
Jul 8, 2020
Author

After the next version will be out. I will check and we can close this.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG]: Hyperspace CreateIndex failed #580

{{title}}

Replies: 13 comments

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[BUG]: Hyperspace CreateIndex failed #580

Replies: 13 comments

suhsteve Jul 6, 2020 Maintainer

pherbel Jul 7, 2020 Author

pherbel Jul 7, 2020 Author

pherbel Jul 8, 2020 Author

pherbel Jul 8, 2020 Author

pherbel Jul 8, 2020 Author

suhsteve
Jul 6, 2020
Maintainer

pherbel
Jul 7, 2020
Author

pherbel
Jul 7, 2020
Author

pherbel
Jul 8, 2020
Author

pherbel
Jul 8, 2020
Author

pherbel
Jul 8, 2020
Author