Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added save action to ETL #1764

Merged
merged 5 commits into from Nov 7, 2016

Conversation

@lossyrob
Copy link
Member

commented Nov 1, 2016

This feature allows us to include client code in the save process for ETL. Lets us do things like:

    EtlConf(args) foreach { conf =>
      val etl = Etl(conf)

      val sourceTiles = etl.load[ProjectedExtent, Tile]
      val (zoom, tiled) = etl.tile[ProjectedExtent, Tile, SpatialKey](sourceTiles)

      etl.save[SpatialKey, Tile](LayerId(etl.input.name, zoom), tiled, { (attributeStore, layerWriter, layerId, rdd) =>
        // Save off histogram computed from zoom 9, store in zoom 0's attributes.
        if(layerId.zoom == 9) {
          val histogram = rdd.histogram(512)
          attributeStore.write(
            layerId.copy(zoom = 0),
            "histogram",
            histogram: Histogram[Double])
        }
      })
    }

which is otherwise very difficult to do in the ETL context.

Also in this PR

Changing the layerExists functionality on AttributeStore to check specifically for the metadata attribute, so that you can store attributes in layer IDs without that layer being considered as existing.

@lossyrob lossyrob force-pushed the lossyrob:feature/etl-output-hook branch from acac3ed to b751e2f Nov 1, 2016

@lossyrob lossyrob added this to the 1.0 milestone Nov 1, 2016

@lossyrob lossyrob changed the title Added post save hook to ETL Added save action to ETL Nov 1, 2016

* @param rdd Tiled raster RDD with TileLayerMetadata
* @param id Layout ID to b
* @param rdd Tiled raster RDD with TileLayerMetadata
* @param postSave Function to allow saving additional attributes or layers per layer saved.

This comment has been minimized.

Copy link
@echeipesh

echeipesh Nov 1, 2016

Contributor

wrong comment name

@lossyrob

This comment has been minimized.

Copy link
Member Author

commented Nov 3, 2016

@echeipesh your comment was addressed. GTG?

Made requested change

@echeipesh echeipesh merged commit 6d49a7a into locationtech:master Nov 7, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@lossyrob lossyrob deleted the lossyrob:feature/etl-output-hook branch Nov 7, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
3 participants
You can’t perform that action at this time.