Skip to content

Initial working version

Pre-release
Pre-release

Choose a tag to compare

@timrobertson100 timrobertson100 released this 06 Oct 08:57
· 16 commits to main since this release

This code works with a couple of minor issues and runs in 1.2hrs for Mercator only when executed with the same resources as the production nightly build:

spark2-submit --class org.gbif.demo.SparkMapApp --master yarn --driver-memory 4G --conf spark.dynamicAllocation.enabled=false --executor-memory 48G --num-executors 100 --executor-cores 6 --conf spark.yarn.executor.memoryOverhead=4096 --conf spark.sql.shuffle.partitions=1200 --conf spark.network.timeout=600s spark-map-sql-1.0.0-2.3.0.jar

SparkMapTest-Spark-Jobs

GBIF-Occurrence-Maps-—-Toolbox