[HUDI-1790] Added SqlSource to fetch data from any partitions for backfill use case

[vingov]

Tips

• Thank you very much for contributing to Apache Hudi.
• Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request

This pull request adds a new source to delta streamer, to perform snapshot queries mainly used for backfilling historical partitions.

Brief change log

• Added a new SqlSource to delta streamer to handle backfills for any specific date range snapshot queries.

Verify this pull request

This change added tests and can be verified as follows:

• Added TestSqlSource to verify the change.
• Manually verified the change by running a job locally.

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[codecov-commenter]

Codecov Report

Merging #2896 (9ac435c) into master (afbafe7) will decrease coverage by 46.91%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master   #2896       +/-   ##
============================================
- Coverage     55.35%   8.43%   -46.92%     
+ Complexity     4025      62     -3963     
============================================
  Files           520      70      -450     
  Lines         25291    2880    -22411     
  Branches       2872     359     -2513     
============================================
- Hits          13999     243    -13756     
+ Misses         9905    2616     -7289     
+ Partials       1387      21     -1366     

Flag	Coverage Δ
hudicli	?
hudiclient	?
hudicommon	?
hudiflink	?
hudihadoopmr	?
hudisparkdatasource	?
hudisync	6.79% <ø> (-44.66%)	⬇️
huditimelineservice	?
hudiutilities	9.09% <0.00%> (-61.97%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...a/org/apache/hudi/utilities/sources/SqlSource.java	0.00% <0.00%> (ø)
...va/org/apache/hudi/utilities/IdentitySplitter.java	0.00% <0.00%> (-100.00%)	⬇️
...va/org/apache/hudi/utilities/schema/SchemaSet.java	0.00% <0.00%> (-100.00%)	⬇️
...a/org/apache/hudi/utilities/sources/RowSource.java	0.00% <0.00%> (-100.00%)	⬇️
.../org/apache/hudi/utilities/sources/AvroSource.java	0.00% <0.00%> (-100.00%)	⬇️
.../org/apache/hudi/utilities/sources/JsonSource.java	0.00% <0.00%> (-100.00%)	⬇️
...rg/apache/hudi/utilities/sources/CsvDFSSource.java	0.00% <0.00%> (-100.00%)	⬇️
...g/apache/hudi/utilities/sources/JsonDFSSource.java	0.00% <0.00%> (-100.00%)	⬇️
...apache/hudi/utilities/sources/JsonKafkaSource.java	0.00% <0.00%> (-100.00%)	⬇️
...pache/hudi/utilities/sources/ParquetDFSSource.java	0.00% <0.00%> (-100.00%)	⬇️
... and 498 more

[vingov]

Hi @n3nash, Can you please review this PR since you have more context on this feature?

[nsivabalan]

minor. Try to see if we can reuse the variable declared in source code rather than hardcoding the config key here. we could avoid any typos.

[vingov]

The variable declared in the source is private, hence hardcoded it here, but to avoid typos made it as a final variable and used it everywhere.

[nsivabalan]

@vingov : hey vinoth. Did you get a chance to check out my feedback. We can merge this in once addressed.

[vingov]

@nsivabalan - I've addressed all the review comments, please review it again, thanks!

[vingov]

The test failures are not related to this change, its on hudi-client/hudi-spark-client modules.

[nsivabalan]

awesome. LGTM. Have added the test flakiness to tracking ticket: https://issues.apache.org/jira/browse/HUDI-1989
Have re-triggered CI. lets see. if it succeeds, will merge this in.

[nsivabalan]

Thanks for your contribution :)