Best Practices guide for creation of good GeoParquet files (focused on distribution)

[cholmes]

Attempt to pull together recommendations / best practices as discussed in #251.

More work needed, feedback / help is very welcome. Likely more to discuss to get the recommendations right, but wanted to put up something for people to react to.

[paleolimbot]

Feel free to take out the comments (those were more for me writing this or for a future blog post).

@jiayuasu Is this about right?

Suggested change

	### Sedona
	### Sedona
	```python
	import glob
	from sedona.spark import SedonaContext, GridType
	from sedona.utils.structured_adapter import StructuredAdapter
	from sedona.sql.st_functions import ST_GeoHash
	# Configuring this line to do the right thing can be tricky
	# https://sedona.apache.org/latest/setup/install-python/?h=python#prepare-sedona-spark-jar
	config = (
	SedonaContext.builder()
	.config("spark.executor.memory", "6G")
	.config("spark.driver.memory", "6G")
	.getOrCreate()
	)
	sedona = SedonaContext.create(config)
	# Read from GeoParquet or some other datasource + do any spatial ops/transformations
	# using Sedona pyspark or SQL
	df = sedona.read.format("geoparquet").load(
	"/Users/dewey/gh/geoarrow-data/microsoft-buildings/files/microsoft-buildings_point_geo.parquet"
	)
	# Create the partitioning. KDBTREE provides a nice balance providing
	# tight (but well-separated) partitions with approximately equal numbers of
	# features in each file. Note that num_partitions is only a suggestion
	# (actual value may differ)
	rdd = StructuredAdapter.toSpatialRdd(df, "geometry")
	rdd.analyze()
	# We call the WithoutDuplicates() variant to ensure that we don't introduce
	# duplicate features (i.e., each feature is assigned a single partition instead of
	# each feature being assigned to every partition it intersects). For points the
	# behaviour of spatialPartitioning() and spatialPartitioningWithoutDuplicates()
	# is identical.
	rdd.spatialPartitioningWithoutDuplicates(GridType.KDBTREE, num_partitions=8)
	# Get the grids for this partitioning (you can reuse this partitioning
	# by passing it to some other spatialPartitioningWithoutDuplicates() to
	# ensure a different write has identical partition extents)
	rdd.getPartitioner().getGrids()
	df_partitioned = StructuredAdapter.toSpatialPartitionedDf(rdd, sedona)
	# Optional: sort within partitions for tighter rowgroup bounding boxes within files
	df_partitioned = (
	df_partitioned.withColumn("geohash", ST_GeoHash(df_partitioned.geometry, 12))
	.sortWithinPartitions("geohash")
	.drop("geohash")
	)
	# Write in parallel directly from each executor node. This scales nicely to
	# (much) bigger-than-memory data, particularly if done with a configured cluster
	# (e.g., Databricks, Glue, Wherobots).
	# There are several options for geoparquet writing:
	# https://sedona.apache.org/latest/tutorial/files/geoparquet-sedona-spark/
	df_partitioned.write.format("geoparquet").mode("overwrite").save(
	"buildings_partitioned"
	)
	# The output files have funny names because Spark writes them this way
	files = glob.glob("buildings_partitioned/*.parquet")
	len(files)

[jiayuasu]

Yes. LGTM!