Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] test_from_json_map failed #6359

Closed
jlowe opened this issue Aug 18, 2022 · 2 comments
Closed

[BUG] test_from_json_map failed #6359

jlowe opened this issue Aug 18, 2022 · 2 comments
Assignees
@jlowe
Labels
bug Something isn't working cudf_dependency An issue or PR with this label depends on a new feature in cudf P0 Must have for release

Comments

@jlowe
Copy link
Member

jlowe commented Aug 18, 2022

test_from_json_map failed in a nightly test run:

def test_from_json_map():
        # The test here is working around some inconsistencies in how the keys are parsed for maps
        # on the GPU the keys are dense, but on the CPU they are sparse
        json_string_gen = StringGen("{\"a\": \"[0-9]{0,5}\"(, \"b\": \"[A-Z]{0,5}\")?}")
>       assert_gpu_and_cpu_are_equal_collect(
            lambda spark : unary_op_df(spark, json_string_gen)\
                    .selectExpr("from_json(a, \"MAP<STRING,STRING>\") as parsed")\
                    .selectExpr("parsed[\"a\"] as pa", "parsed[\"b\"] as pb"),
            conf={"spark.rapids.sql.expression.JsonToStructs": "true"})

../../src/main/python/json_test.py:367: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../src/main/python/asserts.py:508: in assert_gpu_and_cpu_are_equal_collect
    _assert_gpu_and_cpu_are_equal(func, 'COLLECT', conf=conf, is_cpu_first=is_cpu_first)
../../src/main/python/asserts.py:428: in _assert_gpu_and_cpu_are_equal
    run_on_gpu()
../../src/main/python/asserts.py:422: in run_on_gpu
    from_gpu = with_gpu_session(bring_back, conf=conf)
../../src/main/python/spark_session.py:132: in with_gpu_session
    return with_spark_session(func, conf=copy)
../../src/main/python/spark_session.py:99: in with_spark_session
    ret = func(_spark)
../../src/main/python/asserts.py:201: in <lambda>
    bring_back = lambda spark: limit_func(spark).collect()
/home/jenkins/agent/workspace/rapids_it-AQE-dev-github/jars/spark-3.1.2-bin-hadoop3.2/python/lib/pyspark.zip/pyspark/sql/dataframe.py:677: in collect
    sock_info = self._jdf.collectToPython()
/home/jenkins/agent/workspace/rapids_it-AQE-dev-github/jars/spark-3.1.2-bin-hadoop3.2/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py:1304: in __call__
    return_value = get_return_value(
/home/jenkins/agent/workspace/rapids_it-AQE-dev-github/jars/spark-3.1.2-bin-hadoop3.2/python/lib/pyspark.zip/pyspark/sql/utils.py:111: in deco
    return f(*a, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

answer = 'xro328854'
gateway_client = <py4j.java_gateway.GatewayClient object at 0x7f913cf454f0>
target_id = 'o328853', name = 'collectToPython'

    def get_return_value(answer, gateway_client, target_id=None, name=None):
        """Converts an answer received from the Java gateway into a Python object.
    
        For example, string representation of integers are converted to Python
        integer, string representation of objects are converted to JavaObject
        instances, etc.
    
        :param answer: the string returned by the Java gateway
        :param gateway_client: the gateway client used to communicate with the Java
            Gateway. Only necessary if the answer is a reference (e.g., object,
            list, map)
        :param target_id: the name of the object from which the answer comes from
            (e.g., *object1* in `object1.hello()`). Optional.
        :param name: the name of the member from which the answer comes from
            (e.g., *hello* in `object1.hello()`). Optional.
        """
        if is_error(answer)[0]:
            if len(answer) > 1:
                type = answer[1]
                value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
                if answer[1] == REFERENCE_TYPE:
>                   raise Py4JJavaError(
                        "An error occurred while calling {0}{1}{2}.\n".
                        format(target_id, ".", name), value)
E                   py4j.protocol.Py4JJavaError: An error occurred while calling o328853.collectToPython.
E                   : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3933.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3933.0 (TID 4648) (10.233.89.240 executor 0): java.lang.IllegalArgumentException: Creating an empty list column of nested types is not currently supported
E                   	at ai.rapids.cudf.ColumnVector.makeList(ColumnVector.java:455)
E                   	at org.apache.spark.sql.rapids.GpuJsonToStructs.$anonfun$makeMap$5(GpuJsonToStructs.scala:92)
E                   	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:46)
E                   	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:44)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.withResource(GpuExpressions.scala:186)
E                   	at org.apache.spark.sql.rapids.GpuJsonToStructs.$anonfun$makeMap$3(GpuJsonToStructs.scala:91)
E                   	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:46)
E                   	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:44)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.withResource(GpuExpressions.scala:186)
E                   	at org.apache.spark.sql.rapids.GpuJsonToStructs.makeMap(GpuJsonToStructs.scala:87)
E                   	at org.apache.spark.sql.rapids.GpuJsonToStructs.$anonfun$doColumnar$6(GpuJsonToStructs.scala:141)
E                   	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:46)
E                   	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:44)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.withResource(GpuExpressions.scala:186)
E                   	at org.apache.spark.sql.rapids.GpuJsonToStructs.$anonfun$doColumnar$1(GpuJsonToStructs.scala:140)
E                   	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:28)
E                   	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:26)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.withResource(GpuExpressions.scala:186)
E                   	at org.apache.spark.sql.rapids.GpuJsonToStructs.doColumnar(GpuJsonToStructs.scala:105)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.doItColumnar(GpuExpressions.scala:192)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.$anonfun$columnarEval$1(GpuExpressions.scala:203)
E                   	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:28)
E                   	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:26)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.withResource(GpuExpressions.scala:186)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.columnarEval(GpuExpressions.scala:202)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$ReallyAGpuExpression.columnarEval(implicits.scala:34)
E                   	at com.nvidia.spark.rapids.GpuBinaryExpression.columnarEval(GpuExpressions.scala:252)
E                   	at com.nvidia.spark.rapids.GpuBinaryExpression.columnarEval$(GpuExpressions.scala:251)
E                   	at org.apache.spark.sql.rapids.GpuGetMapValue.columnarEval(complexTypeExtractors.scala:190)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$ReallyAGpuExpression.columnarEval(implicits.scala:34)
E                   	at com.nvidia.spark.rapids.GpuAlias.columnarEval(namedExpressions.scala:109)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$ReallyAGpuExpression.columnarEval(implicits.scala:34)
E                   	at com.nvidia.spark.rapids.GpuExpressionsUtils$.columnarEvalToColumn(GpuExpressions.scala:93)
E                   	at com.nvidia.spark.rapids.GpuProjectExec$.projectSingle(basicPhysicalOperators.scala:102)
E                   	at com.nvidia.spark.rapids.GpuProjectExec$.$anonfun$project$1(basicPhysicalOperators.scala:109)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$MapsSafely.$anonfun$safeMap$1(implicits.scala:216)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$MapsSafely.$anonfun$safeMap$1$adapted(implicits.scala:213)
E                   	at scala.collection.immutable.List.foreach(List.scala:392)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$MapsSafely.safeMap(implicits.scala:213)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$AutoCloseableProducingSeq.safeMap(implicits.scala:248)
E                   	at com.nvidia.spark.rapids.GpuProjectExec$.project(basicPhysicalOperators.scala:109)
E                   	at com.nvidia.spark.rapids.GpuProjectExec$.projectAndClose(basicPhysicalOperators.scala:73)
E                   	at com.nvidia.spark.rapids.GpuProjectExec.$anonfun$doExecuteColumnar$1(basicPhysicalOperators.scala:149)
E                   	at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
E                   	at com.nvidia.spark.rapids.ColumnarToRowIterator.$anonfun$fetchNextBatch$2(GpuColumnarToRowExec.scala:228)
E                   	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:28)
E                   	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:26)
E                   	at com.nvidia.spark.rapids.ColumnarToRowIterator.withResource(GpuColumnarToRowExec.scala:174)
E                   	at com.nvidia.spark.rapids.ColumnarToRowIterator.fetchNextBatch(GpuColumnarToRowExec.scala:225)
E                   	at com.nvidia.spark.rapids.ColumnarToRowIterator.loadNextBatch(GpuColumnarToRowExec.scala:202)
E                   	at com.nvidia.spark.rapids.ColumnarToRowIterator.hasNext(GpuColumnarToRowExec.scala:242)
E                   	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
E                   	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:345)
E                   	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
E                   	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
E                   	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
E                   	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
E                   	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
E                   	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
E                   	at org.apache.spark.scheduler.Task.run(Task.scala:131)
E                   	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
E                   	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
E                   	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
E                   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
E                   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
E                   	at java.lang.Thread.run(Thread.java:750)
E                   
E                   Driver stacktrace:
E                   	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2258)
E                   	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2207)
E                   	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2206)
E                   	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
E                   	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
E                   	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
E                   	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2206)
E                   	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1079)
E                   	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1079)
E                   	at scala.Option.foreach(Option.scala:407)
E                   	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1079)
E                   	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2445)
E                   	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2387)
E                   	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2376)
E                   	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
E                   	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:868)
E                   	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2196)
E                   	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2217)
E                   	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2236)
E                   	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2261)
E                   	at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030)
E                   	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
E                   	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
E                   	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
E                   	at org.apache.spark.rdd.RDD.collect(RDD.scala:1029)
E                   	at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:390)
E                   	at org.apache.spark.sql.Dataset.$anonfun$collectToPython$1(Dataset.scala:3519)
E                   	at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
E                   	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
E                   	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
E                   	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
E                   	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
E                   	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
E                   	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3685)
E                   	at org.apache.spark.sql.Dataset.collectToPython(Dataset.scala:3516)
E                   	at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
E                   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
E                   	at java.lang.reflect.Method.invoke(Method.java:498)
E                   	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
E                   	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
E                   	at py4j.Gateway.invoke(Gateway.java:282)
E                   	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
E                   	at py4j.commands.CallCommand.execute(CallCommand.java:79)
E                   	at py4j.GatewayConnection.run(GatewayConnection.java:238)
E                   	at java.lang.Thread.run(Thread.java:750)
E                   Caused by: java.lang.IllegalArgumentException: Creating an empty list column of nested types is not currently supported
E                   	at ai.rapids.cudf.ColumnVector.makeList(ColumnVector.java:455)
E                   	at org.apache.spark.sql.rapids.GpuJsonToStructs.$anonfun$makeMap$5(GpuJsonToStructs.scala:92)
E                   	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:46)
E                   	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:44)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.withResource(GpuExpressions.scala:186)
E                   	at org.apache.spark.sql.rapids.GpuJsonToStructs.$anonfun$makeMap$3(GpuJsonToStructs.scala:91)
E                   	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:46)
E                   	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:44)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.withResource(GpuExpressions.scala:186)
E                   	at org.apache.spark.sql.rapids.GpuJsonToStructs.makeMap(GpuJsonToStructs.scala:87)
E                   	at org.apache.spark.sql.rapids.GpuJsonToStructs.$anonfun$doColumnar$6(GpuJsonToStructs.scala:141)
E                   	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:46)
E                   	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:44)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.withResource(GpuExpressions.scala:186)
E                   	at org.apache.spark.sql.rapids.GpuJsonToStructs.$anonfun$doColumnar$1(GpuJsonToStructs.scala:140)
E                   	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:28)
E                   	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:26)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.withResource(GpuExpressions.scala:186)
E                   	at org.apache.spark.sql.rapids.GpuJsonToStructs.doColumnar(GpuJsonToStructs.scala:105)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.doItColumnar(GpuExpressions.scala:192)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.$anonfun$columnarEval$1(GpuExpressions.scala:203)
E                   	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:28)
E                   	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:26)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.withResource(GpuExpressions.scala:186)
E                   	at com.nvidia.spark.rapids.GpuUnaryExpression.columnarEval(GpuExpressions.scala:202)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$ReallyAGpuExpression.columnarEval(implicits.scala:34)
E                   	at com.nvidia.spark.rapids.GpuBinaryExpression.columnarEval(GpuExpressions.scala:252)
E                   	at com.nvidia.spark.rapids.GpuBinaryExpression.columnarEval$(GpuExpressions.scala:251)
E                   	at org.apache.spark.sql.rapids.GpuGetMapValue.columnarEval(complexTypeExtractors.scala:190)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$ReallyAGpuExpression.columnarEval(implicits.scala:34)
E                   	at com.nvidia.spark.rapids.GpuAlias.columnarEval(namedExpressions.scala:109)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$ReallyAGpuExpression.columnarEval(implicits.scala:34)
E                   	at com.nvidia.spark.rapids.GpuExpressionsUtils$.columnarEvalToColumn(GpuExpressions.scala:93)
E                   	at com.nvidia.spark.rapids.GpuProjectExec$.projectSingle(basicPhysicalOperators.scala:102)
E                   	at com.nvidia.spark.rapids.GpuProjectExec$.$anonfun$project$1(basicPhysicalOperators.scala:109)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$MapsSafely.$anonfun$safeMap$1(implicits.scala:216)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$MapsSafely.$anonfun$safeMap$1$adapted(implicits.scala:213)
E                   	at scala.collection.immutable.List.foreach(List.scala:392)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$MapsSafely.safeMap(implicits.scala:213)
E                   	at com.nvidia.spark.rapids.RapidsPluginImplicits$AutoCloseableProducingSeq.safeMap(implicits.scala:248)
E                   	at com.nvidia.spark.rapids.GpuProjectExec$.project(basicPhysicalOperators.scala:109)
E                   	at com.nvidia.spark.rapids.GpuProjectExec$.projectAndClose(basicPhysicalOperators.scala:73)
E                   	at com.nvidia.spark.rapids.GpuProjectExec.$anonfun$doExecuteColumnar$1(basicPhysicalOperators.scala:149)
E                   	at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
E                   	at com.nvidia.spark.rapids.ColumnarToRowIterator.$anonfun$fetchNextBatch$2(GpuColumnarToRowExec.scala:228)
E                   	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:28)
E                   	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:26)
E                   	at com.nvidia.spark.rapids.ColumnarToRowIterator.withResource(GpuColumnarToRowExec.scala:174)
E                   	at com.nvidia.spark.rapids.ColumnarToRowIterator.fetchNextBatch(GpuColumnarToRowExec.scala:225)
E                   	at com.nvidia.spark.rapids.ColumnarToRowIterator.loadNextBatch(GpuColumnarToRowExec.scala:202)
E                   	at com.nvidia.spark.rapids.ColumnarToRowIterator.hasNext(GpuColumnarToRowExec.scala:242)
E                   	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
E                   	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:345)
E                   	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
E                   	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
E                   	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
E                   	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
E                   	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
E                   	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
E                   	at org.apache.spark.scheduler.Task.run(Task.scala:131)
E                   	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
E                   	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
E                   	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
E                   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
E                   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
E                   	... 1 more
@jlowe jlowe added bug Something isn't working ? - Needs Triage Need team to review and classify P0 Must have for release labels Aug 18, 2022
@jlowe jlowe self-assigned this Aug 18, 2022
@jlowe
Copy link
Member Author

jlowe commented Aug 18, 2022

This was caused by rapidsai/cudf#11364 and will be fixed by rapidsai/cudf#11566.

@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Aug 18, 2022
@jlowe
Copy link
Member Author

jlowe commented Aug 19, 2022

cudf fix has been merged and integrated into the published spark-rapids-jni snapshot. Test is now passing.

@jlowe jlowe closed this as completed Aug 19, 2022
@jlowe jlowe added the cudf_dependency An issue or PR with this label depends on a new feature in cudf label Aug 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jlowe jlowe

Labels
bug Something isn't working cudf_dependency An issue or PR with this label depends on a new feature in cudf P0 Must have for release
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jlowe @sameerz