[SPARK-12907][SQL] Use bitvectos to represent null fields for reducing memory footprint

[kiszk]

Use BitSet for OnHeapColumnVector and BitSetMethod for OffHeapColumnVector

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49691/
Test FAILed.

[nongli]

This was explicitly not done. See this benchmark:
https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchBenchmark.scala#L314

Memory footprint is not the biggest concern for this component. It's not clear to me this is a better approach.

Why should we do this?

[kiszk]

I see the OnHeap case. I overlooked this.
May I add the similar benchmark for OffHeap for information? Or, do you have the performance data for off-heap, too?

I thought that to reduce memory foodprint would be good in general. Do you want to focus on performance to write data by eliminating overhead to use bit vector?

[kiszk]

According to

spark/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnarBatch.java

Line 38 in 3708d13

* is designed to maximize CPU efficiency and not storage footprint. Since it is expected that

, this structure is designed for performance rather than memory footprint.
Close this PR for now