SAMZA-2501 : Optimizing startpoint manager to not make successive bootstrapMessage calls to coordinator-store

[rmatharu-zz]

Symptom: Currently the startpoint manager queries metastore for task-ssps and for each ssp queries coordinator-store, issuing a re-read on the coordinator store. This causes increased AM startup time and depending on the number of input SSPs to a job (thousands in case of regex), YARN may timeout the AM.
Cause: Above.
Fix: This change reduces to number of metastore reads to 2; one for startpoints keyed by ssp and the other for startpoints keyed by ssp+taskname.
API changes: None
Upgrade Instructions: None

[mynameborat]

With the current approach, the cost of fanOut is improved although it does introduce additional memory overhead for the single fetches for tools & other use cases.

I was thinking along the lines of making changes to only readStartpoint(..) in fanOut to readStartpointFromMetadataMap(...) or similar and fetch the raw metadata once in the method and pass the metadata map to read methods. We will need to refactor out the conversion of value bytes to startpoint so that it can be reused in the above read methods.

1. It still gets us the improvement and is slightly better than the current version since we make only one .all() call and additionally, keeping the metadata map as raw <string, byte[]> just keeps the option open to reuse the map for data that belong to the same namespace but not startpoint.
2. Single read paths are unaffected and can still use the old code path minimizing the churn.

Also, can you add a config to disable startpoint fanout? It will be either ways useful for jobs that don't really need startpoint in the first place.

[rmatharu-zz]

+1, simplified and addressed both comments.

[Sanil15]

If this config is called job.startpoint.fanout.enabled, isn't it better to be a part of JobConfig than ClusterManagerConfig?

[Sanil15]

Once you decide where it fits, since this is user-facing, please add docs to doc markdown. Thanks!

[mynameborat]

Cool! Thanks for scoping the config to the feature level 👍
Feel free to merge once travis passes