Add support for vector fields (such as wind fields, ocean currents) via GeoJSON, CSV, etc.

[rsignell-usgs]

We often want to display fields of arrows to represent winds or ocean currents at each observed or model grid location. For example, this is a snapshot in time of predicted tidal currents from a triangular mesh ocean model:

(See the full Jupyter notebook in case you are interested in how this was generated)

We could obviously serve this up as WMS and consume them in TerriaJS that way, but it seems also useful to think about displaying the fields of arrows as time-dependent fields of vector data.

As you can see in the snapshot above, I tried generating a file "test.json" in python and dropped that into TerriaJS, which results in only the basic styling as point data:

Would a better vehicle for delivering these fields of time-dependent arrows be CZML?

I see this python package to generate CZML, but I didn't see anything about arrows...

Any advice would be greatly appreciated!

[rsignell-usgs]

With such (ahem) beautiful graphics embedded in this issue, I'm surprised that I didn't get a nibble. 🎣

Is this because:
A. too hard a problem, likely will not answer anytime soon
B. too busy right now, will try to answer when coming up for air
C. @rsignell-usgs is on blacklist for asking too many questions 😿

[kring]

Hey @rsignell-usgs, you're definitely not on the blacklist; we really appreciate your ideas and participation, even though we're not entirely able to keep up! :) This issue prompted at least one "wow such a cool issue!" on our internal slack, for what it's worth.

To try to answer your question: for a modest number of points like you've shown here (say, less than 10k), I think CZML will work well. I believe CZML lets you specify that there should be an arrowhead on a polyline, but that may not be the best approach here. I'd try creating an "arrow" image (e.g. a PNG), and then use that as the image for a CZML billboard for each point. You can specify the rotation and the scale of the billboard to create something pretty close to your first screenshot above. You should be able to generate the CZML fairly easily with the Python package you mentioned, but me know if you run into trouble.

[kring]

By the way, it's also easy in CZML to specify that the rotation and scale of each billboard is a function of time.

[ocefpaf]

I believe CZML lets you specify that there should be an arrowhead on a polyline, but that may not be the best approach here

I agree. I thought about this problem a lot in the past few days and polylines are not appealing.

I'd try creating an "arrow" image (e.g. a PNG), and then use that as the image for a CZML billboard for each point.

Interesting. The mplleaflet that @rsignell-usgs showed above is a similar solution, but each point has a html feature in the JSON holding its own SVG arrow. Having 1 image and setting the rotation sounds cleaner though. However, reading a "feature-with-an-icon-image" seems more robust in the long run. Image showing arrows of equal lengths but color-coded based on the speed? I am unfamiliar with CZML and I am sure it can be done, but embedding the SVG is "free" for matplotlib users.

Again, I know nothing about CZML, but is there an interest to add an option to read the icon from a HTML/SVG feature in the JSON?

[rsignell-usgs]

I asked @mramato via e-mail about rendering a bunch (98K) of polygons in Cesium, and because I think it's relevant to this issue as well, I'm going to quote it here:

Matt said:
"I took a look at the sample file you provided, and the only issue with it is the overall number of polygons (there are over 98,000 of them in the file). This runs into 2 similar, but not identical problems.

1. Rendering a polygon in WebGL (whether it's a 3D or 2D view) is actually much more complicated than simple Canvas based renderers and it requires more CPU and additional memory to compute. Unfortunately, JavaScript makes this process slower than desktop apps and the browser also artificially limits how much memory an application can use.
2. Browsers in general are very bad a brute-forcing data. 98k polygons is a lot for a single geojson file (as far as the web is concerned), and even non-WebGL solutions I tried seem to choke on loading your file. Cesium has room for optimization in this area and is something we will be looking into in the future. I'm hoping newer browser standards, like ES6's Map object, will allow us to streamline memory usage.

Going to CZML won't help in this particular issue, but the good news is we are actively working on new open standards to make this type of visualization a breeze; specifically, 3D Tiles, which you can read about here: http://cesiumjs.org/2015/08/10/Introducing-3D-Tiles/. There are some cool demos on that link that already show us loading over 1.1 million 3D buildings into Cesium. 3D vector tiles are next on our to-do list and will let you load millions of polygons into Cesium without breaking a sweat. We hope to eventually provide tools for easily converting geojson and other formats into 3D tiles for use in client apps. So while I don't have a simple solution to your problem right now, this should become a non-issue within the next few months."

[rsignell-usgs]

Another way to solve this problem might be to generate vector tiles on the fly in the browser.

As described in this awesome blog post, the geojson-vt will give us the capability to quickly render large geoJSON, regardless of type (e.g. triangular meshes or vector arrows).

Here's geojson-vt in action using Mapbox GL JS,
dynamically loading a 100Mb US zip codes GeoJSON with 5.4 million points

[kring]

I'm a huge fan of vector tiling. We'll see a solid solution in Cesium based on the new 3D Tiles spec before too long. I've often thought about teaching Cesium / TerriaJS how to render MapBox vector tiles (even if it's by rasterizing them on the client, initially) but haven't ever gotten around to doing it.

Cutting up vector tiles on the client is an interesting idea, too, though it's really only applicable in that relatively narrow band where your data is too big to render brute force, and yet small enough that sending the whole thing to the client and processing it there is a reasonable thing to do.

For your original problem, though, I'm not sure anything this sophisticated is necessary. If you can get away with a scaled/rotated icon for each point, rendering ~100k points should work just fine (at least in Cesium. The Leaflet fallback in TerriaJS might not fare so well).

[ocefpaf]

@kring would it be possible to add leaflet's divIcon in TerriaJS? If so then we could easily create GeoJSONs with a SVG icon exported from matplotlib.

Here is an example using another library that wraps leaflet in Python:

https://ocefpaf.github.io/python4oceanographers/blog/2015/11/16/folium_quiver/

[stevage]

Ok now that #1089 has been opened, can we close this?

[rsignell-usgs]

I would vote no, since only after closing #1089 will we know if this actually works effectively to solve this issue. But I'm not hard over, so if you want to close, that's fine.

[stevage]

Ok, now that I understand this a bit better, a couple of comments:

#1089 is kind of a hacky way of supporting customer markers in GeoJSON files. In Leaflet, a "divIcon" is a way of embedding custom HTML at a point, and is being used here as a way of jamming the contents of an SVG file at the point, as expressed by an html property. It would be cleaner to indicate explicitly that this is an image, like markerImage or something. Unfortunately SimpleStyle doesn't support custom markers (only names of maki icons). So, if we're going to support custom markers in GeoJSON, let's rework this slightly.

Alternatively, we could explicitly support vector fields. That is, we could support either GeoJSON or CSV (more compact) with attributes like "direction" and "magnitude", and automatically render arrows. This would probably be easier on the provider as they wouldn't need to go through the hacky process of generating SVGs of arrows in all the different directions and sizes.

I'd probably prefer the second approach, but it's not really an either-or.

Finally, as a short term solution, WMS is a pretty good way to go.

[rsignell-usgs]

The need for plotting lots of vectors is raising it's ugly head again. I'm definitely in favor of explicitly supporting vector fields, supporting GeoJSON or CSV with eastward/northward components u,v in addition to speed, direction, since direction often leads to errors. Is it direction toward or from, relative to true North or math convention, radians or degrees?

[kring]

Updated the title to reflect that this issue is primarily about adding explicit support for vector fields. #1089 describes the simpler case of attaching particular marker images to points.