Skip to content
Permalink
Browse files

[SPARK-23963][SQL] Properly handle large number of columns in query o…

…n text-based Hive table

## What changes were proposed in this pull request?

TableReader would get disproportionately slower as the number of columns in the query increased.

I fixed the way TableReader was looking up metadata for each column in the row. Previously, it had been looking up this data in linked lists, accessing each linked list by an index (column number). Now it looks up this data in arrays, where indexing by column number works better.

## How was this patch tested?

Manual testing
All sbt unit tests
python sql tests

Author: Bruce Robbins <bersprockets@gmail.com>

Closes #21043 from bersprockets/tabreadfix.
  • Loading branch information...
bersprockets authored and gatorsmile committed Apr 13, 2018
1 parent a1c56b6 commit 5bcb7bdccf967ff5ad3d8c76f4ad8c9c4031e7c2
Showing with 1 addition and 1 deletion.
  1. +1 −1 sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
@@ -381,7 +381,7 @@ private[hive] object HadoopTableReader extends HiveInspectors with Logging {

val (fieldRefs, fieldOrdinals) = nonPartitionKeyAttrs.map { case (attr, ordinal) =>
soi.getStructFieldRef(attr.name) -> ordinal
}.unzip
}.toArray.unzip

/**
* Builds specific unwrappers ahead of time according to object inspector

0 comments on commit 5bcb7bd

Please sign in to comment.
You can’t perform that action at this time.