Permalink
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
199 lines (133 sloc) 6.82 KB

Example Archive

This is a collection of code snippets that represent common GeoTrellis tasks.

Get a Layer from a GeoTiff on S3

Motivation: You have a potentially large GeoTiff on S3 that you'd like to form a proper keyed GeoTrellis layer out of. While you're at it, you'd like to change the projection of the imagery too.

New as of GeoTrellis 1.1, S3GeoTiffRDD provides a powerful set of functions for fetching Tiles from the cloud. Here's an example that encompasses your desired work-flow:

def layerFromS3(bucket: String, key: String)(implicit sc: SparkContext): (Int, TileLayerRDD[SpatialKey]) = {
 /* Extract Tiles efficiently from a remote GeoTiff. */
 val ungridded: RDD[(ProjectedExtent, Tile)] =
   S3GeoTiffRDD.spatial(bucket, key)

 /* Say it was in WebMercator and we want it in ConusAlbers. */
 val reprojected: RDD[(ProjectedExtent, Tile)] =
   ungridded.reproject(ConusAlbers)

 /* Our layer will need metadata. Luckily, this can be derived mostly for free. */
 val (zoom, meta): (Int, TileLayerMetadata[SpatialKey]) =
   TileLayerMetadata.fromRDD(reprojected, ZoomedLayoutScheme(ConusAlbers))

 /* Recut our Tiles to form a proper gridded "layer". */
 val gridded: RDD[(SpatialKey, Tile)] = reprojected.tileToLayout(meta)

 (zoom, ContextRDD(gridded, meta))
}

Want to pyramid up to zoom-level 0 while you're at it?

def pyramid(bucket: String, key: String)(implicit sc: SparkContext): Stream[(Int, TileLayerRDD[SpatialKey])] = {
 val (zoom, layer) = layerFromS3(bucket, key)

 Pyramid.levelStream(layer, ZoomedLayoutScheme(ConusAlbers), zoom)
}

Get the Extent of a Tile

Motivation: Given a pair (SpatialKey, Tile), you'd like to know the geographic area on the earth that they cover (the Extent). This Extent has many uses within GeoTrellis.

This solution assumes you have a LayoutDefinition on hand. If you're working with a TileLayerRDD, as many GeoTrellis operations do, then grabbing one is just:

val layer: TileLayerRDD[SpatialKey] = ...  /* Output of previous work */

val layout: LayoutDefinition = layer.metadata.layout

Now to find the Extent of the key/tile pair:

val (key, tile): (SpatialKey, Tile) = ...

val extent1: Extent = key.extent(layout)        /* As of GeoTrellis 1.2 */
val extent2: Extent = layout.mapTransform(key)  /* For versions < 1.2 */

Create a Layer from a single Tile

Motivation: You've gotten a single Tile as output from some GeoTrellis function. Now you'd like to promote it as-is as a GeoTrellis layer, perhaps for easy output.

First, it is assumed that you know the projection (CRS) that your Tile is in, and that you've calculated its Extent. If so, you can construct a ProjectedRaster:

val tile: Tile = ...
val crs: CRS = ...
val extent: Extent = ...

val raster: ProjectedRaster[Tile] = ProjectedRaster(Raster(tile, extent), crs)

This is the minimum amount of information required to construct a Layer. A function that does that could look like this:

import geotrellis.raster._
import geotrellis.spark._
import geotrellis.spark.SpatialKey._  /* To get a `Boundable` instance for `SpatialKey` */
import geotrellis.spark.tiling._
import geotrellis.vector._
import org.apache.spark._
import org.apache.spark.rdd._

/** Convert an in-memory `Tile` into a GeoTrellis layer. */
def toLayer(pr: ProjectedRaster[Tile])(implicit sc: SparkContext): TileLayerRDD[SpatialKey] = {

  val layer: RDD[(ProjectedExtent, Tile)] =
    sc.parallelize(List((ProjectedExtent(pr.raster.extent, pr.crs), pr.raster.tile)))

  val scheme: LayoutScheme = ZoomedLayoutScheme(pr.crs)

  /* The metadata, plus the zoom level corresponding to the size of the Tile.
   * We don't need the zoom level here, but it deserves a note.
   */
  val meta: (Int, TileLayerMetadata[SpatialKey]) = layer.collectMetadata(scheme)

  ContextRDD(layer.tileToLayout[SpatialKey](meta._2), meta._2)
}

Work with S3 using a custom S3Client configuration

Motivation: You would like to work with assets on S3, but you want to use an S3 client (or clients) with a configuration (various configurations) different from the default client configuration.

This can be accomplished by sub-classing S3AttributeStore and/or S3ValueReader, perhaps anonymously.

import geotrellis.spark.io.s3._
import com.amazonaws.services.s3.{AmazonS3Client=>AWSAmazonS3Client}

val aws: AWSAmazonS3Client = ??? /* Special configuration happens here */
val specialS3client = new AmazonS3Client(aws)

val attributeStore = new S3AttributeStore("my-bucket", "my-prefix") {
   override def s3Client = specialS3Client
}

val valueReader = new S3ValueReader(attributeStore) {
   override def s3Client = specialS3Client
}

Saving the Tiles of a Layer as GeoTiffs to S3

Motivation: You would like to save the Tiles of your layer as GeoTiffs to a S3.

If the size of your Tiles are fine and you're to save them:

import geotrellis.raster.io.geotiff._
import geotrellis.spark.io.s3._

val rdd: TileLayerRDD[SpatialKey] = ???

// Convert the values of the layer to SinglebandGeoTiffs
val geoTiffRDD: RDD[(K, SinglebandGeoTiff)] = rdd.toGeoTiffs()

// Convert the GeoTiffs to Array[Byte]]
val byteRDD: RDD[(K, Array[Byte])] = geoTiffRDD.mapValues { _.toByteArray }

// In order to save files to S3, we need a function that converts the
// Keys of the layer to URIs of their associated values.
val keyToURI = (k: SpatialKey) => s"s3://path/to/geotiffs/${k.col}_${k.row}.tif"

byteRDD.saveToS3(keyToURI)

If you'd like the size of the Tiles in the layer to be a different size before saving:

import geotrellis.raster.io.geotiff._
import geotrellis.spark.io.s3._
import geotrellis.spark.regrid._

val rdd: TileLayerRDD[SpatialKey] = ???

// Regrid the Tiles so that they are 512x512
val regridedRDD: TileLayerRDD[SpatialKey] = rdd.regrid(512, 512)

// Convert the values of the layer to SinglebandGeoTiffs
val geoTiffRDD: RDD[(K, SinglebandGeoTiff)] = regridedRDD.toGeoTiffs()

// Convert the GeoTiffs to Array[Byte]]
val byteRDD: RDD[(K, Array[Byte])] = geoTiffRDD.mapValues { _.toByteArray }

// In order to save files to S3, we need a function that converts the
// Keys of the layer to URIs of their associated values.
val keyToURI = (k: SpatialKey) => s"s3://path/to/geotiffs/${k.col}_${k.row}.tif"

byteRDD.saveToS3(keyToURI)