-
Notifications
You must be signed in to change notification settings - Fork 484
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Geospatial search #8239
Merged
kcondon
merged 38 commits into
IQSS:develop
from
GlobalDataverseCommunityConsortium:GDCC/geosearch
Nov 30, 2022
Merged
Geospatial search #8239
Changes from all commits
Commits
Show all changes
38 commits
Select commit
Hold shift + click to select a range
f74e0c2
add bonding box indexing
qqmyers f3db9ee
Merge remote-tracking branch 'IQSS/develop' into
qqmyers 7b15877
Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch
qqmyers d9eed2f
Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch
qqmyers 93a3cb6
Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch
qqmyers 84e9614
Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch
qqmyers e765a8d
Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch
qqmyers d5430e4
Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch
qqmyers 4ec3d0f
Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch
qqmyers 6da3b2a
Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch
qqmyers 3b8c859
Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch
qqmyers 8ce67a3
Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch
qqmyers 1007a6b
Update conf/solr/8.11.1/schema.xml
qqmyers b903e2a
release note and addition to search doc
qqmyers d79ba87
Merge branch 'GDCC/geosearch' of https://github.com/GlobalDataverseCo…
qqmyers c210821
add space
qqmyers 8c11b7d
Merge branch 'develop' into GDCC/geosearch #8239
pdurbin de079fd
Merge branch 'develop' into GDCC/geosearch #8239
pdurbin 0b0c3b9
add multivalued in schema
qqmyers 23a6d58
Merge branch 'GDCC/geosearch' of https://github.com/GlobalDataverseCo…
qqmyers cf9b4ae
case matters
qqmyers f542925
north < south latitude is an error
qqmyers 4f9434e
fix another non-physical box
qqmyers 1db095f
handle multiples - make bbox a single surrounding box
qqmyers 202438a
typo
qqmyers 8c048a7
wrong scope
qqmyers d48c29a
Merge branch 'develop' into GDCC/geosearch #8239
pdurbin c7c16d4
add geo_point and geo_radius #8239
pdurbin 3d647f4
move hard coded strings to SearchFields class #8239
pdurbin 7272de6
add geospatial search test #8239
pdurbin efa8e9c
Merge branch 'develop' into GDCC/geosearch #8239
pdurbin e5187b2
Avoid DatasetCreate exception with only one coordinate #8239
pdurbin 28215fb
rename solr_srpt to geolocation and solr_bboxtype to boundingBox #8239
pdurbin ff32672
add error checking for geo_point and geo_radius #8239
pdurbin 6e7499e
update docs and release note (supported via API) #8239
pdurbin b5383f4
Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch
qqmyers 364e347
test invalid lat/long (too large) #8239
pdurbin 9d8332d
Merge branch 'develop' into GDCC/geosearch #8239
pdurbin File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
Support for indexing the "Geographic Bounding Box" fields ("West Longitude", "East Longitude", "North Latitude", and "South Latitude") from the Geospatial metadata block has been added. | ||
|
||
Geospatial search is supported but only via API using two new parameters: `geo_point` and `geo_radius`. | ||
|
||
A Solr schema update is required. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,6 +6,7 @@ | |
import edu.harvard.iq.dataverse.DataFileTag; | ||
import edu.harvard.iq.dataverse.Dataset; | ||
import edu.harvard.iq.dataverse.DatasetField; | ||
import edu.harvard.iq.dataverse.DatasetFieldCompoundValue; | ||
import edu.harvard.iq.dataverse.DatasetFieldConstant; | ||
import edu.harvard.iq.dataverse.DatasetFieldServiceBean; | ||
import edu.harvard.iq.dataverse.DatasetFieldType; | ||
|
@@ -37,6 +38,7 @@ | |
import java.io.IOException; | ||
import java.io.InputStream; | ||
import java.sql.Timestamp; | ||
import java.text.NumberFormat; | ||
import java.text.SimpleDateFormat; | ||
import java.time.LocalDate; | ||
import java.util.ArrayList; | ||
|
@@ -947,6 +949,70 @@ public SolrInputDocuments toSolrDocs(IndexableDataset indexableDataset, Set<Long | |
} | ||
} | ||
} | ||
|
||
//ToDo - define a geom/bbox type solr field and find those instead of just this one | ||
if(dsfType.getName().equals(DatasetFieldConstant.geographicBoundingBox)) { | ||
String minWestLon=null; | ||
String maxEastLon=null; | ||
String maxNorthLat=null; | ||
String minSouthLat=null; | ||
for (DatasetFieldCompoundValue compoundValue : dsf.getDatasetFieldCompoundValues()) { | ||
String westLon=null; | ||
String eastLon=null; | ||
String northLat=null; | ||
String southLat=null; | ||
for(DatasetField childDsf: compoundValue.getChildDatasetFields()) { | ||
switch (childDsf.getDatasetFieldType().getName()) { | ||
case DatasetFieldConstant.westLongitude: | ||
westLon = childDsf.getRawValue(); | ||
break; | ||
case DatasetFieldConstant.eastLongitude: | ||
eastLon = childDsf.getRawValue(); | ||
break; | ||
case DatasetFieldConstant.northLatitude: | ||
northLat = childDsf.getRawValue(); | ||
break; | ||
case DatasetFieldConstant.southLatitude: | ||
southLat = childDsf.getRawValue(); | ||
break; | ||
} | ||
} | ||
if ((eastLon != null || westLon != null) && (northLat != null || southLat != null)) { | ||
// we have a point or a box, so proceed | ||
if (eastLon == null) { | ||
eastLon = westLon; | ||
} else if (westLon == null) { | ||
westLon = eastLon; | ||
} | ||
if (northLat == null) { | ||
northLat = southLat; | ||
} else if (southLat == null) { | ||
southLat = northLat; | ||
} | ||
//Find the overall bounding box that includes all bounding boxes | ||
if(minWestLon==null || Float.parseFloat(minWestLon) > Float.parseFloat(westLon)) { | ||
minWestLon=westLon; | ||
} | ||
if(maxEastLon==null || Float.parseFloat(maxEastLon) < Float.parseFloat(eastLon)) { | ||
maxEastLon=eastLon; | ||
} | ||
if(minSouthLat==null || Float.parseFloat(minSouthLat) > Float.parseFloat(southLat)) { | ||
minSouthLat=southLat; | ||
} | ||
if(maxNorthLat==null || Float.parseFloat(maxNorthLat) < Float.parseFloat(northLat)) { | ||
maxNorthLat=northLat; | ||
} | ||
//W, E, N, S | ||
solrInputDocument.addField(SearchFields.GEOLOCATION, "ENVELOPE(" + westLon + "," + eastLon + "," + northLat + "," + southLat + ")"); | ||
} | ||
} | ||
//Only one bbox per dataset | ||
//W, E, N, S | ||
if ((minWestLon != null || maxEastLon != null) && (maxNorthLat != null || minSouthLat != null)) { | ||
solrInputDocument.addField(SearchFields.BOUNDING_BOX, "ENVELOPE(" + minWestLon + "," + maxEastLon + "," + maxNorthLat + "," + minSouthLat + ")"); | ||
} | ||
|
||
} | ||
} | ||
|
||
for(String metadataBlockName : metadataBlocksWithValue) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I rewatched Jim's recent "Experimenting with Geospatial indexing" talk ( https://osf.io/84pnw ) and it reminded me that I asked for an test to be added to SearchIT. I still want this and I'm happy to try to add it, if it make sense for me to jump in the code. 😄 |
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want to catch a nonnumeric exception here and give more feedback?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, getGeoPoint and getGeoRadius will already throw exceptions with feedback about non-numeric values.
For example, if the user does this:
They'll get this error:
That said, I may not be following the use cases, the concerns. I'm happy to add more tests for these.