Repository navigation
Releases: timrobertson100/map-spark-sql
Releases · timrobertson100/map-spark-sql
Release list
Initial working version
This code works with a couple of minor issues and runs in 1.2hrs for Mercator only when executed with the same resources as the production nightly build:
spark2-submit --class org.gbif.demo.SparkMapApp --master yarn --driver-memory 4G --conf spark.dynamicAllocation.enabled=false --executor-memory 48G --num-executors 100 --executor-cores 6 --conf spark.yarn.executor.memoryOverhead=4096 --conf spark.sql.shuffle.partitions=1200 --conf spark.network.timeout=600s spark-map-sql-1.0.0-2.3.0.jar
Archived early experiments
This contains various code some of which is very scrappy, which will now be removed.
This release serves as a checkpoint in case it needs to be revisited.

