Evaluate feasibility of scheduling hoodie pipelines w/o need for external workflow managers

[vinothchandar]

Per http://spark.apache.org/docs/latest/job-scheduling.html , Spark already can do schedule tasks internally (we do this at scale already). and Spark has APIs to request and relinquish executors.

https://spark.apache.org/docs/2.0.2/api/java/org/apache/spark/SparkContext.html#requestExecutors(int)

Even for batch pipelines, we would like hoodie pipelines to be run efficiently, like Spark Streaming, except we give up containers when not in use..

Blocker : #123

[SemanticBeeng]

Do we want to consider what this means more exactly from perspective of HudiIO? (integration with Apache Beam, how Spark session is managed for long running jobs, what workflow manager means, etc)

From https://issues.apache.org/jira/browse/HUDI-70

"connection management"

See https://github.com/apache/beam/search?q=ProcessContext&unscoped_q=ProcessContext
Can we use something like that to hold the session ?

[vinothchandar]

@SemanticBeeng this is orthogonal to connection management I believe.. This ticket was around figuring out how to deploy hudi in a long running mode.. Some aspects like givining up containers on dynamic allocation etc could still be useful. let me post this into HUDI-70

[vinothchandar]

Once we approach beam, we can consider ProcessContext. for now, I think SparkStreamingContext or equivalent would do something similar I beleieve