Add GeoJSON support #876

dracos · 2021-01-24T18:54:40Z

This PR hopefully adds GeoJSON support, in much the same way as Shapefiles.

I had to add another save-sheet command to the canvas so that you could save from it, that didn't appear to work on Shapefiles previously.

https://github.com/martinjc/UK-GeoJSON/blob/master/json/administrative/wal/lad.json opened with vd -f geojson lad.json and selecting the English name column as a key column gives:

And pressing g. with the English name column highlighted gives:

Hope that's useful.

saulpw

Hi @dracos, thanks for putting this together! Overall it looks good, just a few changes and questions.

saulpw · 2021-01-24T23:00:13Z

dev/formats.jsonl

@@ -18,7 +18,7 @@
 {"filetype": "ttf", "filetype_url": "#ttf", "aliases": "otf", "requirements": "fonttools", "format": "TrueType Font", "VisiData loader": "yes", "VisiData saver": "", "version_added": "1.1", "created": "1991", "creator": "Apple", "description": "", "open format": "yes", "nestable": "", "plottable": "plottable", "format_url": "https://en.wikipedia.org/wiki/TrueType"}
 {"filetype": "dot", "filetype_url": "#pcap", "aliases": "", "requirements": "", "format": "Graphviz diagram", "VisiData loader": " ", "VisiData saver": "from pcap", "version_added": "1.2", "created": "1991", "creator": "", "description": "output for pcap only", "open format": "yes", "nestable": "", "plottable": "", "format_url": "https://graphviz.gitlab.io/_pages/doc/info/lang.html"}
 {"filetype": "dta", "aliases": "", "requirements": "pandas", "format": "Stata", "VisiData loader": "yes", "VisiData saver": "", "version_added": "1.2", "created": "1985", "creator": "StataCorp", "description": "", "open format": "no", "nestable": "", "plottable": "", "format_url": "https://www.loc.gov/preservation/digital/formats/fdd/fdd000471.shtml"}
-{"filetype": "geojson", "filetype_url": "#shp", "aliases": "", "requirements": "", "format": "Geographic JSON", "VisiData loader": " ", "VisiData saver": "from shp", "version_added": "1.2", "created": "2008", "creator": "", "description": "derivative of JSON", "open format": "yes", "nestable": "", "plottable": "plottable", "format_url": "http://geojson.org/"}
+{"filetype": "geojson", "filetype_url": "#shp", "aliases": "", "requirements": "", "format": "Geographic JSON", "VisiData loader": "yes", "VisiData saver": "", "version_added": "1.2", "created": "2008", "creator": "", "description": "derivative of JSON", "open format": "yes", "nestable": "", "plottable": "plottable", "format_url": "http://geojson.org/"}


Thanks for remembering this! Let's also change the version_added to 2.2 (which is the next release).

saulpw · 2021-01-24T23:27:46Z

visidata/loaders/geojson.py

+                        self.colnames[prop] = c
+                        self.addColumn(c)
+
+            yield from Progress(features)


Let's yield feature in the first pass instead of doing two passes over features (and move the Progress into that for loop).

saulpw · 2021-01-24T23:43:31Z

visidata/loaders/geojson.py

+            for feature in features:
+                for prop in feature.get('properties', {}).keys():
+                    if prop not in self.colnames:
+                        c = Column(name=prop, getter=getter_factory(prop))


You should be able to use ItemColumn with a nested name: "properties."+prop.

AttrColumn handles nested properties, but these are normal dicts, and ItemColumn does not handle that.

You are right! It really seems like it should, sorry to give you the runaround. This again tempts me to make it nestable, even at the cost of having it not work with dict keys that have . in them. But that's not for tonight, so we can keep the getter_factory for now.

saulpw · 2021-01-25T05:34:58Z

visidata/loaders/geojson.py

+        self.reset()
+
+        for row in Progress(self.sourceRows):
+            k = tuple(col.getValue(row) for col in self.source.keyCols)


Use self.source.rowkey(row).

I'll update the shp loader as well, presumably?

Yes, please! Thanks for helping to keep it clean.

saulpw · 2021-01-25T05:42:01Z

visidata/loaders/geojson.py

+    if (isinstance(vs, ShapeMap)):
+        return save_shp_geojson(vd, p, vs)
+
+    isinstance(vs, Canvas) or vd.fail("must save GeoJSON from map sheet")


This only works for GeoJSONMap (not all Canvas). With the change in decorators we can remove this.

saulpw · 2021-01-25T05:47:04Z

visidata/loaders/shp.py

@@ -71,8 +73,7 @@ def reload(self):

        self.refresh()

-@VisiData.api
-def save_geojson(vd, p, vs):
+def save_shp_geojson(vd, p, vs):


This should actually be save_geojson with @ShapeMap.api decorator.

saulpw · 2021-01-25T05:48:36Z

visidata/loaders/shp.py

@@ -83,7 +84,7 @@ def save_geojson(vd, p, vs):
                'coordinates': [[x, y] for x, y in coords],
            },
            'properties': {
-                col.name: col.getTypedValue(row) for col in vs.source.visibleCols
+                col.name: col.getTypedValue(row) for col in vs.source.keyCols


This is the one change I'm not sure of...don't we want all the visibleCols as saved properties? Does some other GeoJSON tool balk at unneeded columns?

Not sure what I thought here, I guess I confused key with visible. Will revert

saulpw · 2021-01-25T05:50:33Z

visidata/loaders/geojson.py

+
+@VisiData.api
+def save_geojson(vd, p, vs):
+    if (isinstance(vs, ShapeMap)):


This isn't needed if we use the @GeoJSONMap.api decorator (instead of @VisiData.api`). Then we'd also change the original save_geojson (noted below).

I made this change (and similar shp one), but I now get "no function to save as type geojson" on both maps.

Hm, that should have worked. Unfortunately I'm out of steam for tonight. I'll take a look at this tomorrow and we'll hopefully get this merged just in time for 2.2!

@dracos Please update your visidata fork, it should work now with this change! 5849b88

Thanks! Yes, that seems to have worked.

saulpw · 2021-01-25T05:58:26Z

visidata/loaders/geojson.py

+            fp.write(chunk)
+
+GeoJSONSheet.addCommand('.', 'plot-row', 'vd.push(GeoJSONMap(name+"_map", sheet, sourceRows=[cursorRow], textCol=cursorCol, source=sheet))', 'plot geospatial vector in current row')
+GeoJSONSheet.addCommand('g.', 'plot-rows', 'vd.push(GeoJSONMap(name+"_map", sheet, sourceRows=rows, textCol=cursorCol, source=sheet))', 'plot all geospatial vectors in current sheet')


What do you think about changing g. to be plot-selected and have them use someSelectedRows?

Hmm, g. to me implies global-plot. I'd say if anything . on its own should plot current row if no selection, or selected rows if some selected.

I see what you mean. To be honest, plotting only the current row has never been very useful, so I wonder if . itself should be someSelectedRows, and g. should always be "all rows". That wouldn't be consistent with most of the TableSheet g bindings though. Thoughts for another PR 🧷

saulpw · 2021-01-25T06:01:22Z

visidata/loaders/geojson.py

+
+GeoJSONSheet.addCommand('.', 'plot-row', 'vd.push(GeoJSONMap(name+"_map", sheet, sourceRows=[cursorRow], textCol=cursorCol, source=sheet))', 'plot geospatial vector in current row')
+GeoJSONSheet.addCommand('g.', 'plot-rows', 'vd.push(GeoJSONMap(name+"_map", sheet, sourceRows=rows, textCol=cursorCol, source=sheet))', 'plot all geospatial vectors in current sheet')
+InvertedCanvas.addCommand('^S', 'save-sheet', 'vd.saveSheets(inputPath("save to: ", value=getDefaultSaveName(sheet)), sheet, confirm_overwrite=options.confirm_overwrite)', 'save current sheet to filename in format determined by extension (default .tsv)')


Maybe ultimately this should go on BaseSheet, but for now I think this should be on GeoJSONSheet specifically and not all InvertedCanvas.

GeoJSONSheet already has a save-sheet on it, do you mean GeoJSONMap?

This adds loading and saving support for GeoJSON files, and fixes the Shapefile GeoJSON saving.

saulpw

Thanks again @dracos!

aborruso · 2021-01-26T15:37:53Z

@dracos thank you very much!

How view a geojson in vd?

anjakefala · 2021-01-26T17:35:15Z

@aborruso Update your installed VisiData to the most recent develop! There will be a release sometime this week. =)

aborruso · 2021-01-26T17:39:05Z

I know it and I have already updated it.

When I open a geojson I see this. Do I must use only point layer?

Thank you

dracos · 2021-01-26T17:55:56Z

How did you open it, and what is its filename? Its filename needs to end with .geojson or you open it with -f geojson as in my description in the PR body.

aborruso · 2021-01-26T18:11:01Z

Hi @dracos thank you again. I have opened it using vd aree.geojson.

And it's the same if I use -f geojson.

I have saul.pw/VisiData v2.2dev

anjakefala · 2021-01-26T18:12:12Z

@aborruso I think you are likely referencing an earlier commit of VisiData 2.2dev in your environment. =(

aborruso · 2021-01-26T18:14:23Z

@anjakefala I'm installing it using pip3 install --no-warn-script-location git+https://github.com/saulpw/visidata.git@develop

anjakefala · 2021-01-26T18:16:16Z

@aborruso I would try to completely uninstall all versions of VisiData (pip3 uninstall visidata), and use a search to hunt for any stray files in your computer, and then install. If you are still having trouble after doing that, I will take a look!

aborruso · 2021-01-26T18:18:03Z

I have it, I have added --upgrade

Now that I have a table, how to render the geoms?

Thank you

aborruso · 2021-01-26T18:18:32Z

Using g ., it's great!!!

dracos · 2021-01-26T18:18:51Z

As per the instructions I gave in the PR body, (. for a single row, or) g. for all rows (though see discussion above about how that may change in future). Glad to hear it.

anjakefala · 2021-01-26T18:19:43Z

Yay @aborruso 💜!! I am so glad.

dracos force-pushed the add-geojson-support branch from d1ecebb to 13d3b98 Compare January 24, 2021 20:40

anjakefala changed the base branch from stable to develop January 24, 2021 20:51

saulpw requested changes Jan 25, 2021

View reviewed changes

dracos requested a review from saulpw January 25, 2021 08:17

Add GeoJSON support.

7cb0ac6

This adds loading and saving support for GeoJSON files, and fixes the Shapefile GeoJSON saving.

dracos force-pushed the add-geojson-support branch from 3683dbe to 7cb0ac6 Compare January 26, 2021 08:34

saulpw approved these changes Jan 26, 2021

View reviewed changes

saulpw merged commit b201946 into saulpw:develop Jan 26, 2021

anjakefala mentioned this pull request Jan 27, 2021

[geojson] save in GeoJSON format from any sheet with lat/long columns #879

Closed

Add GeoJSON support #876

Add GeoJSON support #876

Conversation

dracos commented Jan 24, 2021 • edited

saulpw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saulpw left a comment

Choose a reason for hiding this comment

aborruso commented Jan 26, 2021

anjakefala commented Jan 26, 2021

aborruso commented Jan 26, 2021

dracos commented Jan 26, 2021 • edited

aborruso commented Jan 26, 2021

anjakefala commented Jan 26, 2021

aborruso commented Jan 26, 2021

anjakefala commented Jan 26, 2021 • edited

aborruso commented Jan 26, 2021

aborruso commented Jan 26, 2021

dracos commented Jan 26, 2021 • edited

anjakefala commented Jan 26, 2021

dracos commented Jan 24, 2021 •

edited

dracos commented Jan 26, 2021 •

edited

anjakefala commented Jan 26, 2021 •

edited

dracos commented Jan 26, 2021 •

edited