Distinct with JSONL would cause JVM core dump

[Yicong-Huang]

The code dump happens after some execution with Distinct operator + JSONL. This could be related to null fields produced by JSONL operator (with a nullable source data), and it has some bugs with LinkedHashSet that used internally in Distinct Op.

Some error does happen like this:

[05-06-2021 03:20:44.035] [WARN ] [WorkerActorVirtualIdentity(Layer(3,Distinct-operator-8be11d7b-89d1-4a95-b2dc-48f97a10abf2,main)[0]).logWarning(40)] - Receiver class scala/collection/mutable/HashSet must be the current class or a subtype of interface scala/collection/mutable/HashTable
scala.collection.mutable.HashTable.findOrAddEntry$(HashTable.scala:167)
scala.collection.mutable.LinkedHashSet.findOrAddEntry(LinkedHashSet.scala:44)
scala.collection.mutable.LinkedHashSet.add(LinkedHashSet.scala:68)
edu.uci.ics.texera.workflow.operators.distinct.DistinctOpExec.processTexeraTuple(DistinctOpExec.scala:18)
edu.uci.ics.texera.workflow.common.operators.OperatorExecutor.processTuple(OperatorExecutor.scala:14)
edu.uci.ics.texera.workflow.common.operators.OperatorExecutor.processTuple$(OperatorExecutor.scala:10)
edu.uci.ics.texera.workflow.operators.distinct.DistinctOpExec.processTuple(DistinctOpExec.scala:10)
edu.uci.ics.amber.engine.architecture.worker.DataProcessor.processInputTuple(DataProcessor.scala:95)
edu.uci.ics.amber.engine.architecture.worker.DataProcessor.handleInputTuple(DataProcessor.scala:190)
edu.uci.ics.amber.engine.architecture.worker.DataProcessor.edu$uci$ics$amber$engine$architecture$worker$DataProcessor$$runDPThreadMainLogic(DataProcessor.scala:144)
edu.uci.ics.amber.engine.architecture.worker.DataProcessor$$anon$1.run(DataProcessor.scala:42)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748) 

[Yicong-Huang]

Apparently, the original implementation used LinkedHashSet in scala@2.12.8 which has some bug to cause JVM core dump. The issue is possibly related to scala/bug#8774. I have tried to update the scala version to 2.12.9 ~ 2.12.10 but it does not fix the bug. In scala@2.13.0 there is a new HashMap/HashSet implementation that could potentially fix the issue, however, our code does not compatible with anything more than 2.12.10.

Alternatively, I have used HashSet instead of LinkedHashSet to fix the issue. The original reason to use LinkedHashSet is to maintain the input stream order. Since we are now moving onto hash partition anyway, such input order is no longer needed to be maintained.