error when runing spark on travis

[miguelgfierro]

with

def spark(app_name="App", url="local[*]", memory="10G", cores="16"):
    """Start Spark if not started
    Args:
        app_name (str): Set name of the application
        url (str): URL for spark master.
        memory (str): Size of memory for spark driver and executor.  
        cores (str): Number of executor cores
    Returns:
        obj: Spark context.
    Examples: 
        >>> spark = spark()
        >>> spark is not None
        True
    """
    return (
        SparkSession.builder.appName(app_name)
        .master(url)
        .config("spark.driver.memory", memory)
        .config("spark.executor.cores", cores)
        .config("spark.executor.memory", memory)
        .config("spark.memory.fraction", "0.9")
        .config("spark.memory.stageFraction", "0.3")
        .config("spark.executor.instances", 1)
        .config("spark.executor.heartbeatInterval", "36000s")
        .config("spark.network.timeout", "10000000s")
        .config("spark.driver.maxResultSize", "50g")
        .getOrCreate()
    )

In travis I get:

=================================== FAILURES ===================================
________________ [doctest] python.pyspark_base.spark_conf.spark ________________
005 Start Spark if not started
006     Args:
007         app_name (str): Set name of the application
008         url (str): URL for spark master.
009         memory (str): Size of memory for spark driver.  
010     Returns:
011         obj: Spark context.
012     Examples: 
013         >>> spark = spark()
UNEXPECTED EXCEPTION: TypeError("'SparkSession' object is not callable",)
Traceback (most recent call last):
  File "/home/travis/miniconda/envs/codebase/lib/python3.6/doctest.py", line 1330, in __run
    compileflags, 1), test.globs)
  File "<doctest python.pyspark_base.spark_conf.spark[0]>", line 1, in <module>
TypeError: 'SparkSession' object is not callable

However, in a machine configured with spark it works.

Related to #292

[miguelgfierro]

explore:
https://github.com/massmutual/sample-pyspark-application/blob/master/.travis.yml

https://github.com/holdenk/spark-testing-base/blob/master/.travis.yml

https://github.com/databricks/tensorframes/blob/master/.travis.yml

[miguelgfierro]

Same error in spark 2.3.1 and 2.3.2

[miguelgfierro]

return SparkSession.builder.getOrCreate() not working in a functio, but when I do >>> spark = SparkSession.builder.getOrCreate() in the tests, it works

[miguelgfierro]

wierd error, it looks it is related to travis, skip it in the code