[SPARK-42359][SQL] Support row skipping when reading CSV files#39907

Closed

ted-jenks wants to merge 36 commits intoapache:masterfrom

ted-jenks:SPARK-42359

Commits on Feb 6, 2023

add skiplines
ted jenks
committed
reformat and parameterize tests
ted jenks
committed

Commits on Feb 7, 2023

Commits on Feb 8, 2023

explicitely order blank line removal and comments
ted-jenks
committed
renme func
ted-jenks
committed
revert respect skipline in schema inference
ted-jenks
committed
Revert "revert respect skipline in schema inference"
ted-jenks
committed
Revert "renme func"
ted-jenks
committed
Revert "explicitely order blank line removal and comments"
ted-jenks
committed
return to comments and blanks first
ted-jenks
committed
add test for comments and blanks in dataset
ted-jenks
committed
make expectation for new test right
ted-jenks
committed
change test name
ted-jenks
committed

Commits on Feb 9, 2023

fix header assert
ted-jenks
committed
rework CSVExprUtils
ted-jenks
committed
update CSV Utils
ted-jenks
committed
Revert "change test name"
ted-jenks
committed
Revert "Revert "change test name""
ted-jenks
committed
Revert "update CSV Utils"
ted-jenks
committed
Revert "rework CSVExprUtils"
ted-jenks
committed
set order to blank, skip, comment
ted-jenks
committed
revert order to original
ted-jenks
committed
update docs
ted-jenks
committed
remove filter comment and empty
ted-jenks
committed

Commits on Feb 10, 2023

don't infer scheam
ted-jenks
committed
cast int to string
ted-jenks
committed
use iterator instead
ted-jenks
committed

Commits on Feb 13, 2023

fix compile
ted-jenks
committed
set type corectly
ted-jenks
committed
fix none header case
ted-jenks
committed

Commits on Feb 14, 2023

stringify all
ted-jenks
committed

Commits on Feb 16, 2023

remove sip with index
ted-jenks
committed

Commits on Feb 17, 2023

use an iter
ted-jenks
committed

Commits on Mar 29, 2023

comments
ted-jenks
committed