New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OGR failing on VISIT_859 shapefile #61

Closed
MattReimer opened this Issue Jun 20, 2017 · 7 comments

Comments

Projects
None yet
2 participants
@MattReimer
Collaborator

MattReimer commented Jun 20, 2017

Visit 859

From @KellyMWhitehead :

I cannot for the life of me figure out why this shapefile is having problems loading features using the 'shapefile_loader.py" in champ metrics.

It keeps bombing out on line 4380 of ogr.py when it trys to convert the geometry to json.

On the surface, it looks just like a normal breaklines shapefile (x,y,z). This was taken from a Bridge Creek visit, of which all surveys from 2012-2013 appear to be affected. It could that non-standard workflow/equipment could have produced data in a slightly different format, but it seems very odd that this kind of effect would carry over from gdb to shapefile.

Anyway, if you are able, could you please try loading this and let me know if you are getting the same type of error?

@MattReimer MattReimer added the bug label Jun 20, 2017

@MattReimer MattReimer self-assigned this Jun 20, 2017

@MattReimer

This comment has been minimized.

Show comment
Hide comment
@MattReimer

MattReimer Jun 20, 2017

Collaborator

When OGR loads this shapefile here is what it sees for the first feature:

{
  "geometry": {
    "type": "Unknown",
    "coordinates": null
  },
  "type": "Feature",
  "properties": {
    "Shape_Leng": 21.9605148484,
    "TARGET_FID": 0,
    "DESCRIPTIO": "lw",
    "LineType": "SOFT",
    "Join_Count": 19
  },
  "id": 0
}

When I load it in GIS I see the type is set to Line (WKB type: "LineStringZM") and I wonder if this could be related

Collaborator

MattReimer commented Jun 20, 2017

When OGR loads this shapefile here is what it sees for the first feature:

{
  "geometry": {
    "type": "Unknown",
    "coordinates": null
  },
  "type": "Feature",
  "properties": {
    "Shape_Leng": 21.9605148484,
    "TARGET_FID": 0,
    "DESCRIPTIO": "lw",
    "LineType": "SOFT",
    "Join_Count": 19
  },
  "id": 0
}

When I load it in GIS I see the type is set to Line (WKB type: "LineStringZM") and I wonder if this could be related

@MattReimer

This comment has been minimized.

Show comment
Hide comment
@MattReimer

MattReimer Jun 20, 2017

Collaborator

More investigation:

Notice the "coordinates": null above. That's REALLY weird.

OGR's GetGeomType() shows me 3002 --> ogr.wkbLineStringZM so that looks fine

it's feat.ExportToJson() that shows me "type": "unknown". Very weird

I did this for all the features and I get:

{"geometry": {"type": "Unknown", "coordinates": null}, "type": "Feature", "properties": {"Shape_Leng": 21.9605148484, "TARGET_FID": 0, "DESCRIPTIO": "lw", "LineType": "SOFT", "Join_Count": 19}, "id": 0}
{"geometry": {"type": "Unknown", "coordinates": null}, "type": "Feature", "properties": {"Shape_Leng": 20.7922476024, "TARGET_FID": 1, "DESCRIPTIO": "wg", "LineType": "SOFT", "Join_Count": 16}, "id": 1}
{"geometry": {"type": "Unknown", "coordinates": null}, "type": "Feature", "properties": {"Shape_Leng": 59.4708222009, "TARGET_FID": 2, "DESCRIPTIO": "wg", "LineType": "SOFT", "Join_Count": 39}, "id": 2}
{"geometry": {"type": "Unknown", "coordinates": null}, "type": "Feature", "properties": {"Shape_Leng": 34.6729095871, "TARGET_FID": 3, "DESCRIPTIO": "wg", "LineType": "SOFT", "Join_Count": 25}, "id": 3}
{"geometry": {"type": "Unknown", "coordinates": null}, "type": "Feature", "properties": {"Shape_Leng": 101.719932978, "TARGET_FID": 4, "DESCRIPTIO": "to", "LineType": "HARD", "Join_Count": 47}, "id": 4}
..... etc

Sidenote: How can something have Shape_Leng > 0 when there are no points?!?

but when I do ExportToWkt() I get:

LINESTRING (717732.7373 4948354.8319 588.4162,717731.9184 4948355.9094 588.377,717731.5187 4948356.8366 588.4856,717730.9576 4948357.6243 588.6037,717730.0095 4948359.5566 588.3758,717728.6046 4948360.9441 588.3897,717726.7332 4948363.1009 588.4402,717725.4148 4948364.709 588.3485,717723.8071 4948366.4298 588.2651,717721.6738 4948368.3347 588.2319,717720.1761 4948369.9768 588.3397,717718.2568 4948371.7471 588.1031,717717.7232 4948372.8208 587.9782)
Collaborator

MattReimer commented Jun 20, 2017

More investigation:

Notice the "coordinates": null above. That's REALLY weird.

OGR's GetGeomType() shows me 3002 --> ogr.wkbLineStringZM so that looks fine

it's feat.ExportToJson() that shows me "type": "unknown". Very weird

I did this for all the features and I get:

{"geometry": {"type": "Unknown", "coordinates": null}, "type": "Feature", "properties": {"Shape_Leng": 21.9605148484, "TARGET_FID": 0, "DESCRIPTIO": "lw", "LineType": "SOFT", "Join_Count": 19}, "id": 0}
{"geometry": {"type": "Unknown", "coordinates": null}, "type": "Feature", "properties": {"Shape_Leng": 20.7922476024, "TARGET_FID": 1, "DESCRIPTIO": "wg", "LineType": "SOFT", "Join_Count": 16}, "id": 1}
{"geometry": {"type": "Unknown", "coordinates": null}, "type": "Feature", "properties": {"Shape_Leng": 59.4708222009, "TARGET_FID": 2, "DESCRIPTIO": "wg", "LineType": "SOFT", "Join_Count": 39}, "id": 2}
{"geometry": {"type": "Unknown", "coordinates": null}, "type": "Feature", "properties": {"Shape_Leng": 34.6729095871, "TARGET_FID": 3, "DESCRIPTIO": "wg", "LineType": "SOFT", "Join_Count": 25}, "id": 3}
{"geometry": {"type": "Unknown", "coordinates": null}, "type": "Feature", "properties": {"Shape_Leng": 101.719932978, "TARGET_FID": 4, "DESCRIPTIO": "to", "LineType": "HARD", "Join_Count": 47}, "id": 4}
..... etc

Sidenote: How can something have Shape_Leng > 0 when there are no points?!?

but when I do ExportToWkt() I get:

LINESTRING (717732.7373 4948354.8319 588.4162,717731.9184 4948355.9094 588.377,717731.5187 4948356.8366 588.4856,717730.9576 4948357.6243 588.6037,717730.0095 4948359.5566 588.3758,717728.6046 4948360.9441 588.3897,717726.7332 4948363.1009 588.4402,717725.4148 4948364.709 588.3485,717723.8071 4948366.4298 588.2651,717721.6738 4948368.3347 588.2319,717720.1761 4948369.9768 588.3397,717718.2568 4948371.7471 588.1031,717717.7232 4948372.8208 587.9782)
@MattReimer

This comment has been minimized.

Show comment
Hide comment
@MattReimer

MattReimer Jun 20, 2017

Collaborator

Followup:

I filed a bug at OSGEO

It's my first one so we'll see how that goes:

https://trac.osgeo.org/gdal/ticket/6935

Collaborator

MattReimer commented Jun 20, 2017

Followup:

I filed a bug at OSGEO

It's my first one so we'll see how that goes:

https://trac.osgeo.org/gdal/ticket/6935

@MattReimer

This comment has been minimized.

Show comment
Hide comment
@MattReimer

MattReimer Jun 20, 2017

Collaborator

More followup:

https://trac.osgeo.org/gdal/changeset/39216

It's been fixed as of Jun 20, 2017 1:43:20 PM

but we won't be able to consume the fix since I refuse to recompile a new version of trunk gdal on any of our systems.

Collaborator

MattReimer commented Jun 20, 2017

More followup:

https://trac.osgeo.org/gdal/changeset/39216

It's been fixed as of Jun 20, 2017 1:43:20 PM

but we won't be able to consume the fix since I refuse to recompile a new version of trunk gdal on any of our systems.

@KellyMWhitehead

This comment has been minimized.

Show comment
Hide comment
@KellyMWhitehead

KellyMWhitehead Jun 21, 2017

Collaborator

@MattReimer Please verify the code updates here: https://gist.github.com/KellyMWhitehead/e4369d9a72b2c6287a8ed78397ff117e

Two issues: 1) Breaklines were still causing an error when trying to load geom as json (so unable to even test if type was unknown in your fix). 2) line 104, I was having issues with some features that did not have a geometry (rare situation). Any advice on when to use 'None' types in theses situations would be appreciated.

Its very crude, but seems to work for me. Feel free to change or improve.

Collaborator

KellyMWhitehead commented Jun 21, 2017

@MattReimer Please verify the code updates here: https://gist.github.com/KellyMWhitehead/e4369d9a72b2c6287a8ed78397ff117e

Two issues: 1) Breaklines were still causing an error when trying to load geom as json (so unable to even test if type was unknown in your fix). 2) line 104, I was having issues with some features that did not have a geometry (rare situation). Any advice on when to use 'None' types in theses situations would be appreciated.

Its very crude, but seems to work for me. Feel free to change or improve.

@MattReimer

This comment has been minimized.

Show comment
Hide comment
@MattReimer

MattReimer Jun 21, 2017

Collaborator

@KellyMWhitehead Ok. I think it's better to do this than to hack it up like I did. Feel free to make the change and don't just comment out my code. Really get rid of it!

Collaborator

MattReimer commented Jun 21, 2017

@KellyMWhitehead Ok. I think it's better to do this than to hack it up like I did. Feel free to make the change and don't just comment out my code. Really get rid of it!

@MattReimer

This comment has been minimized.

Show comment
Hide comment
@MattReimer

MattReimer Oct 16, 2017

Collaborator

Hey @KellyMWhitehead I put in some try/catching blocks so that validation can close. Can you just quickly eyeball a2de2ed and make sure I didn't make a mess?

Collaborator

MattReimer commented Oct 16, 2017

Hey @KellyMWhitehead I put in some try/catching blocks so that validation can close. Can you just quickly eyeball a2de2ed and make sure I didn't make a mess?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment