Generate flink-conf.yaml file automatically to set optimum conf values

[chandrashekar-s]

Description of what I changed

Fixes #823

• Enabled a feature flag to automate the generation of flink-conf.yaml based on the numThreads in application.yaml file and the number of cores available in the machine.
• If the flag is disabled, then the flink-conf.yaml directed by the env. var FLINK_CONF_DIR will be used. If var is empty, then default conf will be used.
• JAVA_OPTS can be set to override the default JVM heap and other parameter values.
• The application will fail to launch if the JVM Off Heap memory is insufficient for the Flink cluster to launch.
• Fixes the regression introduced by this change where the maxWorkers was defaulted to 1 and in turn the num of shards for parquet files was also 1.

E2E test

Tested Full Run and Incremental Run by enabling and disabling the flags. Also tested for different numThread configurations.

TESTED:

Load tested the application on a 48 core machine. Given below are the results

Run #	Run Type	# of Patient Records	Time in secs (Before Change)	Time in secs (After Change)	Remarks
1	Full	9K	400	358	Full run timing remains the same before and after the changes
2	Full (Repeat)	9K	360	256
3	Incremental	9K Upfront + 100 additional	140	30	The improvement in time after change is because of fixing the regression (increased num of shards from 1 to N)
4	Incremental	9K Upfront + 500 additional	180	32

After fixing the regression the time taken for incremental run has significantly reduced

Checklist: I completed these to help reviewers :)

• I have read and will follow the review process.

• I am familiar with Google Style Guides for the language I have coded in.

  No? Please take some time and review Java and Python style guides.

• My IDE is configured to follow the Google code styles.

  No? Unsure? -> configure your IDE.

• I have added tests to cover my changes. (If you refactored existing code that was well tested you do not have to add tests)

• I ran mvn clean package right before creating this pull request and added all formatting changes to my commit.

• All new and existing tests passed.

• My pull request is based on the latest changes of the master branch.

  No? Unsure? -> execute command git pull --rebase upstream master

[chandrashekar-s]

@bashir2 The build is failing currently, I am looking into it.

[chandrashekar-s]

@bashir2 The PR is ready for review. Since changes were made to create target jars compatible with jdk 17, the docker images had to be updated for streaming module.

[bashir2]

@bashir2 The PR is ready for review. Since changes were made to create target jars compatible with jdk 17, the docker images had to be updated for streaming module.

Thanks @chandrashekar-s for the updates. I'll review this by tomorrow.

[bashir2]

Thanks @chandrashekar-s for this change.

[chandrashekar-s]

@bashir2 Thanks for reviewing the changes. I have addressed/responded to the comments. Can you please have a look. Also, for few cases where unit test cases could not be written, I feel we need few more tests for e2e or performance to avoid regressions.

[bashir2]

Thanks @chandrashekar-s the remaining comments are all questions or minor suggestions. Please feel free to merge after addressing them.

[chandrashekar-s]

Thanks @bashir2 for the reviewing the changes. I have addressed some of the comments in the latest commit. Also I have created these 2 issues #893 and #891 for Validating Flink in non-local mode and Investigating/fixing the reshuffle operations for writing to parquet files respectively.

[chandrashekar-s]

The performance results have been attached in the PR description. No noticeable changes for the Full run, but for Incremental run the timing has improved after fixing the number of Shards to be N (Flink parallelism)