Merge in esri2geo #7

Merged
merged 24 commits into from Jun 20, 2013

Projects

None yet

2 participants

@calvinmetcalf

This merges in the work I've done on esri2geo you may want to not do a strait merge as this overwrites most of your stuff and instead use this as a template but the improvements include:

  • Handles null geometries, lines with only one distinct point, polygons with less then 3 distinct points, donuts holes.
  • Handles dates
  • Has 2 other tools one for merging multiple files into one (i.e. if you have point and line data related to something) and one for doing a batch import.
  • Automatically converts to wgs84.

I actually started mine a while ago but the use case fell through and I never finished the corner cases, until your tool inspired me to finish it.

Calvin Metcalf added some commits Apr 3, 2013
Calvin Metcalf added git ignore a0713a9
Calvin Metcalf merge in esri2open c3a693f
Calvin Metcalf updated readme 42cad8b
Calvin Metcalf added a few more todos fa1d2d9
@feomike

these are stellar! nice work. i am going to play w/ these, and merge them. what i might do, is a test, get all the right code in place, merge the pull to make the project clean, then overwite the resulting merge w/ hand edited code. i am likely just to do this in phases over time. hope you don't mind if i keep this open for a while. let me know what you think.

@calvinmetcalf

That is probably a wise choice, in my attempt to make it as modular as possible the control flow got somewhat complex, but it's paying off adding a new format in is pretty easy, I did some work on sqlite support on another branch.

The three scripts are their own files because as far as I can tell that's the only way you can do it it in arcgis.

@feomike

i am also a big believer in the kiss principle. right now the structure is pretty simple and 'works as advertised' (attrib - James Fee). would love to try to stay to that principle. not saying what you have doesn't, just going to move slow.

Calvin Metcalf updated gitignore 2ba5deb
@calvinmetcalf

most of the complexity really came from me trying to make it modular, without actually using modules, I need to test this when I get to a windows computer tomorrow, but this should make it easier to take bits on their own.

@calvinmetcalf

I reorganized it so that all of the file specific code is in it's own file, this makes it it even more modular and should make it easier to add more file types or geometry options. I finished the last bit on a computer without arcMap so no testing of that until monday.

Calvin Metcalf added some commits Apr 5, 2013
@calvinmetcalf

I'm just going to close this and we can deal with specific features to improve in their own issues

@feomike

sorry, been slammed, and just haven't gotten to it. you sure about the close?

@calvinmetcalf calvinmetcalf reopened this Apr 10, 2013
@calvinmetcalf

I did some radical changes so that even if we were to merge we'd probably want a new pull request

@calvinmetcalf

specifically I was working on sqlite and reorganized everything so that I could add in more, realized I made everything far to complicated and reorganized them again to simplify things, I was more afraid that you were getting an alert every time I pushed a commit.

Calvin Metcalf added some commits Apr 10, 2013
@calvinmetcalf

I'm going on vacation next week so I'm not going to be able to work on it so here's a status report:

  • I've tried to strike a balance between simple and well organized e.g. having one file is much simpler as you don't have to worry about importing stuff, but breaking large self contained chunks of code into files can make it much easier to deal with, e.g. all the geometry parsing stuff is in a single file, not mixed in with the code for writing to the file. The balance is between a million files and 1 file so the code to close things up is not in it's own file, but the code to prepare the file is.
  • The structure is set up to have three phases:

    1. prepare: open up the file and do the initial stuff like set up the columns for csv or write the {"type":"FeatureCollection","features":[ for geojson
    2. write the rows from the input feature class into the file
    3. close the file up.

    the idea behind this is to let multiple feature classes that couldn't be merged in ESRI (e.g. routes and stations) be merged into one output file by opening it once, writing multiple files in and then closing it up, so things related to geometry are in step two.

  • Details related to particular file types are also separated so that adding a new one can be straight forward, this allowed me to add sqlite and almost get it working.
  • Geometry is also separated meaning new ones can be added like WKT (working) and WKB (maybe the issue preventing sqlite from working)
@calvinmetcalf

sqlite works

@feomike

sorry for leaving this for so long. i am taking a look at it. i got git out of synch with the merge i just accepted, so might need to do some jujitsu here. thanks for your patience

Calvin Metcalf merge in master 6d1101a
@calvinmetcalf

no problem, the toolbox is going to be the main problem because the other file you have I don't have, I can do a merge on my end which should make it easier for you.

@calvinmetcalf

I went back and looked and the json meta data standards apply to catalogs not to the individual data pieces.

@feomike

thats right. my logic here is that making data available as part of a web ecology, is remarkably helpful. huge lists of download files, WMS or other assets are moderately helpful, but when JSON (geoJSON) is posted as a URL/I, then the data is naturally part of the web, w/o any additional overhead, eg. software/hardware (arguable for sure but..). i'll be able to do the toolbox thing. again massive apologies for the delay.

@calvinmetcalf

yeah at the moment trying to get ESRI stuff out onto the web is not easy, most of my work flows historically had a "save to shapefile, open with quantum gis" step in them.

@calvinmetcalf

*cough\*

@feomike

its on me. am sorry. sigh. hopefully this week.

@calvinmetcalf

no worries, and I have been dog fooding my branch and my geojson repo was made with this with no problems

@feomike

so i finally just did a quick couple of tests. your branch works great. i don't think i need to make any edits at all. you ok w/ the current readme etc? anything else before i accept?

@calvinmetcalf

nothing on my end looks good.

@feomike

k; great. for some reason i thought i needed to do something to the .tbx, but you got that nailed. am checking one last thing. nice work. (and nice write up in the atlantic)

@calvinmetcalf

thanks

@feomike feomike merged commit 2ea1b53 into project-open-data:master Jun 20, 2013
@feomike

Calvin, congrats. nice work!

@calvinmetcalf calvinmetcalf deleted the calvinmetcalf:merge-geo branch Jun 20, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment