Create GeoMesa suproject #1621
Create GeoMesa suproject #1621
Conversation
pomadchin
commented
Sep 5, 2016
•
pomadchin
commented
- layer reader
- layer writer
- tests coverage
- GeoMesa 1.2.7
| * http://www.opensource.org/licenses/apache2.0.php. | ||
| *************************************************************************/ | ||
|
|
||
| package org.locationtech.geomesa.jobs.mapreduce |
There was a problem hiding this comment.
should be removed after moving to geomesa 1.2.7 locationtech/geomesa#1077
There was a problem hiding this comment.
Can you please create an issue for this?
| if(temporal) sftb.add(whenField, classOf[java.util.Date]) | ||
| val sft = sftb.buildFeatureType | ||
| if(temporal) sft.getUserData.put(Constants.SF_PROPERTY_START_TIME, whenField) // when field is date | ||
| sft.getUserData.put("geomesa.mixed.geometries", java.lang.Boolean.TRUE) // allow GeoMesa to index points and extents together |
There was a problem hiding this comment.
Looks like these tags are specific to Geomesa, so the object should be in geotrellis.geomsea package.
| if(temporal) sft.getUserData.put(Constants.SF_PROPERTY_START_TIME, whenField) // when field is date | ||
| sft.getUserData.put(SimpleFeatureTypes.MIXED_GEOMETRIES, java.lang.Boolean.TRUE) // allow GeoMesa to index points and extents together | ||
| sft.getUserData.put(Hints.USE_PROVIDED_FID, java.lang.Boolean.FALSE) // generate feature ids | ||
| sft |
There was a problem hiding this comment.
I think this might be missing some things, e.g. visibility and periodicity:
There was a problem hiding this comment.
Then again we don't really have those concepts in GeoTrellis layers, so this might be a good thing to make a note/issue out of and add once we put those concepts into GeoTrellis (perhaps through sfcurve usage?)
| import org.opengis.feature.simple.SimpleFeature | ||
|
|
||
| trait FeatureToGeoMesaSimpleFeatureMethods[G <: Geometry, T] extends MethodExtensions[Feature[G, T]] { | ||
| def toSimpleFeature(featureName: String, featureId: Option[String] = Some(null), crs: Option[CRS] = None)(implicit transmute: T => Seq[(String, Any)]): SimpleFeature = |
There was a problem hiding this comment.
doesn't Some(null) kind of break the Option contract?
There was a problem hiding this comment.
Weird, is this PR diff just not updating properly?
There was a problem hiding this comment.
._. looks like these comments attached to a certain sha; anyway it's perfect that this pr was not merged in (due to 8f962f3)
There was a problem hiding this comment.
Oh. Yeah you can comment commits rather than prs, might of been what happened here.
There was a problem hiding this comment.
Looks like it's just the new review functionality for github - it no longer removes comments when a new change comes in that addresses the issue. Which makes it harder to see if issues are addressed, which is pretty crappy of GitHub.
|
|
||
| val sft = sftb.buildFeatureType | ||
| if(data.map(_._1).contains(whenField)) sft.getUserData.put(Constants.SF_PROPERTY_START_TIME, whenField) // when field is date | ||
| sft.getUserData.put(SimpleFeatureTypes.MIXED_GEOMETRIES, java.lang.Boolean.TRUE) // allow GeoMesa to index points and extents together |
There was a problem hiding this comment.
FYI you actually don't need this (although it won't hurt) - it's only if you specify the geometry binding as Geometry, which you don't do in your match above.
| val sft = sftb.buildFeatureType | ||
| if(data.map(_._1).contains(whenField)) sft.getUserData.put(Constants.SF_PROPERTY_START_TIME, whenField) // when field is date | ||
| sft.getUserData.put(SimpleFeatureTypes.MIXED_GEOMETRIES, java.lang.Boolean.TRUE) // allow GeoMesa to index points and extents together | ||
| sft.getUserData.put(Hints.USE_PROVIDED_FID, java.lang.Boolean.FALSE) // generate feature ids |
There was a problem hiding this comment.
This won't get used actually - you have to set (or not set) the hint on a per-feature basis. If you don't explicitly set the hint then generating IDs is the default behavior.
| } | ||
| data.foreach({ case (key, value) => sfb.add(value) }) | ||
|
|
||
| sfb.buildFeature(featureId.orNull) |
There was a problem hiding this comment.
as an optimization, you might want to cache the simple feature types - it's probably going to be slow to generate it for each simple feature you create.
| numPartitions: Option[Int] = None | ||
| ): RDD[SimpleFeature] = { | ||
| val dataStore = instance.accumuloDataStore | ||
| dataStore.createSchema(simpleFeatureType) |
There was a problem hiding this comment.
this is a no-op if the schema already exists - but if the schema doesn't exist then there's not going to be any data in it... maybe that is ok here.
There was a problem hiding this comment.
added an existence check
| try { | ||
| sf.foreach { rawFeature => | ||
| val newFeature = featureWriter.next() | ||
| attrNames.foreach(an => newFeature.setAttribute(an, rawFeature.getAttribute(an))) |
There was a problem hiding this comment.
as a minor optimization, you could do
(0 until sft.getAttributeCount).foreach(i => newFeature.setAttribute(i, rawFeature.getAttribute(i))
lookup by index is always faster than lookup by attribute name
| val sft = sftb.buildFeatureType | ||
| if(temporal) sft.getUserData.put(Constants.SF_PROPERTY_START_TIME, whenField) // when field is date | ||
| sft.getUserData.put(SimpleFeatureTypes.MIXED_GEOMETRIES, java.lang.Boolean.TRUE) // allow GeoMesa to index points and extents together | ||
| sft.getUserData.put(Hints.USE_PROVIDED_FID, java.lang.Boolean.FALSE) // generate feature ids |
There was a problem hiding this comment.
same as noted above - these 2 options won't really have any effect here.
There was a problem hiding this comment.
lines removed and created an issue #1654
|
Thanks for the review @elahrvivaz. @pomadchin are all the review comments addressed? |
|
@lossyrob yes, walked through all comments |
|
Looks like there's some optimizations and other comments that still need to be addressed...need to either make the changes or reply to comments about why the suggested change shouldn't happen. |
…ection kryo serializer
