[HUDI-1211] clean up spark session for each test of FunctionalTestHar…

[kwondw]

…ness

Tips

• Thank you very much for contributing to Apache Hudi.
• Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request

To fix the issue of HUDI-1211,

Brief change log

• close spark session after each test in FunctionalTestHarness.java

Verify this pull request

• This change is only for tests

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[codecov-io]

Codecov Report

Merging #2697 (3c88186) into master (161d530) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##             master   #2697   +/-   ##
========================================
  Coverage      9.55%   9.55%           
  Complexity       48      48           
========================================
  Files            53      53           
  Lines          1958    1958           
  Branches        233     233           
========================================
  Hits            187     187           
  Misses         1758    1758           
  Partials         13      13           

Flag	Coverage Δ	Complexity Δ
hudiutilities	9.55% <ø> (ø)	0.00 <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[vinothchandar]

@kwondw this is done intentionally to speed up tests.

cc @xushiyan can you please comment on this ?

[xushiyan]

I would like to understand if the linked issue was reproducible and solved by this change.

The original intention was to share spark session for test cases in each module that has functional test suite configured. And all test cases tied to the functional test suite are ran sequentially so the shared spark session should be okay. In the maven profile functional-tests, we configured these to ensure one new JVM process to execute all tests in one Maven module.

<forkCount>1</forkCount>
<reuseForks>true</reuseForks>

I'm not against this change, though it looks like will be doing many more spark init, just want to understand what exactly the issue was and how this fixed it. Thanks.

[nsivabalan]

fyi, we did merge a hot fix couple of days back on similar lines. locally, both funcational and Hudi-utilities tests ran parallel and we ran into some issues.

[vinothchandar]

@nsivabalan are you saying this pr is already subsumed by other fixes on master? Can you please clarify what your suggestion for this pr is

[nsivabalan]

@kwondw : am in line with @xushiyan. Don't think we need to close and init resources for each test. If you see any issues by one time instantiation and tear down per class, let us know. we can consider the fix. If not, would prefer to leave it that way as this might increase our (test execution/)running time as well.

[vinothchandar]

Closing due to inactivity