[CHIP-9] fetcher refactor + support derivations in chaining

[hzding621]

Summary

Refactor fetch join calls and support derivations in chaining.

• Refactor fetchJoin into multiple sub-functions: fetchBaseJoin, fetchDerivations, fetchModelTransforms and instrumentAndLog
• Update the call of fetchJoin from JoinSourceRunner into calling each sub-functions separately
  • for derivations, instead of calling fetchDerivations, use spark session to apply derivations
  • the new sequence is:
    • 1st mapPartitions: invoke fetchBaseJoin
    • run derivations using the main session
    • 2nd mapPartitions: invoke fetchModelTransforms and instrumentAndLog
• Also, support UDFs loading (via setups) in JoinSourceRunner

Why / Goal

Currently, if a join contains derivations, it cannot be used as a joinSource for chaining, because in the chaining streaming job (JoinSourceRunner), spark executor will call fetcher.fetchJoin which invokes its own local spark session, and Spark doesn't like this - results in Caused by: org.apache.spark.SparkException: Failed to get broadcast_1_piece0 of broadcast_1

Before model transforms chaining this is fine, because the same derivations can be expressed using GroupBy's query. But with model transforms chaining, running derivations as part of the join is a mandatory step to prepare model inputs. This is to be done together with UDF usage in derivations.

Test Plan

• Added Unit Tests
• Covered by existing CI
• Integration tested

Checklist

• Documentation update

Reviewers

[nikhil-zlai]

should we discuss the direction a bit more as a larger group - before we go deep?

Lets also ask users if they prefer join.inferenceSpec or a standalone model object in the API.