[sql] fix DEFAULT_INITIAL_BUFFER_SIZE

[scwf]

No description provided.

[AmplabJenkins]

Can one of the admins verify this patch?

[rxin]

While this is definitely a bug fix, it is also pretty dangerous. This means a 100 column table would use 1G of buffer per task, and with 32 cores, that's 32G of ram just for the buffer. @marmbrus @liancheng

[marmbrus]

Yeah, lets wait to fix this until we fix the other issues....

[scwf]

i am voluntary to fix this issue, can you assign this issue for me？@rxin @marmbrus

[liancheng]

@marmbrus has already filed SPARK-2650 to track this issue and assigned to me. The 104 here is definitely a stupid typo (my bad, sorry...). But the more important missing piece here is that we haven't implemented the logic to calculate a proper initial buffer size like what Shark does. Shark initializes the buffer size by estimating the size of the target column based on DFS block size and ColumnType.defaultSize (see here and here).

[scwf]

ok.thanks!