PySpark long instructions style

[vmgustavo]

Describe the style change
It is frequent to use long instructions for PySpark chained with the dot operator. The official Apache Spark tutorials on how to use PySpark suggest using \ to break the statement. The current style splits the statement simply to avoid crossing the char columns limit and doesn't consider readability. By splitting the statement at the dot operators this could be better.

Examples in the current Black style

spark = (
    SparkSession.builder.appName("Python Spark SQL basic example")
    .config("spark.some.config.option", "some-value")
    .getOrCreate()
)

data.repartition("column").write.partitionBy("column").mode("ignore").option(
    "compress", "snappy"
).parquet(output_path)

Desired style
Should follow the Apache Spark code style

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL basic example") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()

Or maybe if not using \ to split the lines

spark = (
    SparkSession
    .builder
    .appName("Python Spark SQL basic example")
    .config("spark.some.config.option", "some-value")
    .getOrCreate()
)

Additional context

• This could be applied to code using Pandas as well
• This is similar to Split long call chains joined by binary operators on the operator #1781

[JelleZijlstra]

We will not use the style with backslashes; that's against Black general style principles.

I'm most interested in changing the output for your second example, with data.repartition. That does look ugly right now, and I'd welcome suggestions for how to make it better.

[vmgustavo]

By simpling wrapping the instructions with parenthesis the style is already great:

(data.repartition("column").write.partitionBy("column").mode("ignore").option(
    "compress", "snappy"
).parquet(output_path))

(
    data.repartition("column")
    .write.partitionBy("column")
    .mode("ignore")
    .option("compress", "snappy")
    .parquet(output_path)
)

The bad formmating only happens when there is no parenthesis. Would it be possible to "force add" parenthesis?

[felix-hilden]

@vmgustavo We have an open issue about improvements to fluent interfaces: #571

[felix-hilden]

And I think the case Jelle was interested in is also covered by that issue, so I'll close this as a duplicate.