Infer Number of Available Executors on Different Environments

[moritzmeister]

If we want to support Maggy on arbitrary Spark clusters, we need a reliable way of inferring the number of executors that are available on the Spark cluster.

Additionally, as a fallback the user should be able to specify a lower number of executors in case it's a shared cluster.

We have to look at the following properties:
spark.dynamicAllocation.maxExecutors
spark.executor.instances for static

Find equivalents on Databricks.