Skip to content

Releases: timrobertson100/map-spark-sql

Initial working version

Pre-release

Choose a tag to compare

This code works with a couple of minor issues and runs in 1.2hrs for Mercator only when executed with the same resources as the production nightly build:

spark2-submit --class org.gbif.demo.SparkMapApp --master yarn --driver-memory 4G --conf spark.dynamicAllocation.enabled=false --executor-memory 48G --num-executors 100 --executor-cores 6 --conf spark.yarn.executor.memoryOverhead=4096 --conf spark.sql.shuffle.partitions=1200 --conf spark.network.timeout=600s spark-map-sql-1.0.0-2.3.0.jar

SparkMapTest-Spark-Jobs

GBIF-Occurrence-Maps-—-Toolbox

Archived early experiments

Pre-release

Choose a tag to compare

@timrobertson100 timrobertson100 released this 06 Oct 08:01

This contains various code some of which is very scrappy, which will now be removed.
This release serves as a checkpoint in case it needs to be revisited.