-
Notifications
You must be signed in to change notification settings - Fork 126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to build other API output formats? #195
Comments
@timwis It is certainly possilble but we are planning some refactoring of the exporter code so its sort of a minefield. Right now all the data exports are pumped through ogr2ogr on the command line. If you look at these lines you'll see the formats: https://github.com/Esri/koop/blob/master/lib/Exporter.js#L13-L20 They match a given mime type in a url (http://...ckan/:host/:id.{kml,csv,zip,geojson}) We need to change this though. The koop exporting code should be an easily extendable ecosystem that that supports converting geojson to any format - so long a format converter is included (not sure that make sense.) Long story short: its a bit of work that we are planning on doing but havent got to yet. |
Thanks @chelm but I think this is a bit different than export format -- it's closer to the featureservice output format in that it's a queryable API. I'm talking about enabling querystring params to filter the results, potentially geometry queries, etc. In my opinion there are a lot of possibilities with this. Further exploration pointed me to |
@timwis Oh im sorry! I jumped to conclusion. Cool so I am totally new to Odata. Will look at it. Koop does pass the where and geometry querystring params down to the DB. So this should be doable. For really large data its crucial that queries happen at the DB level. So im interested in module that converts Odata queries into SQL or similar. Will look. |
@timwis I feel like something like this would be useful to have here: https://github.com/koopjs/koop-pgcache/blob/master/index.js#L324 where the options param is actually already the entire query string from the request. It could be parsed into a query. |
This looks useful https://github.com/auth0/node-odata-parser It could also make sense to add this at the cache wrapper level in the |
@timwis great idea BTW |
Thanks! I appreciate the prompt responses. I didn't realize the output format and the query operations were separate like that. I see that the query operations even affect the GeoJSON output format: That lexer library looks like a good start, though I imagine applying the resulting object to a postgres database call is still a significant step. I saw some other libs out there for odata too though, so hopefully they'll get us the rest of the way. |
@timwis there have only been a few new "output API" built - I think by adding more like ODATA or SODA it should illuminate what the common patterns are to then make the development of these very straightforward. Personally I think it should be decoupled output vs. input - so you can write an ODATA output that is deployed with the GitHub/AGOL/CKAN inputs and choose if you want PGCache or ESCache. A bit like pluggable modules in the three layer stack. |
Agreed, that sounds ideal |
Got a working proof of concept. Just finishing up at Code for Philly so I will reply in the next day or two with what I learnt and ideas on a more thorough implementation, but wanted to show you in the meantime: A koop plugin that does 2 things: parse the querystring (the API) and output XML (it's not technically OData compliant but proof of concept): And I had to modify one of the providers to use it (which I have some ideas about): |
@timwis Very cool. I have some ideas on this too, and your branch looks like a great start. I may make a few PRs to your fork if thats cool. Nice work |
Thanks @chelm! Here are some thoughts I had along the way: Output format vs API Modular / plugins if( ckan.koop[req.params.api] !== undefined ) {
ckan.koop[req.params.api]._beforeFind(req, res);
} (The plugin would modify Query language req.query.$select.split(',').forEach(function(field) {
dbQuery.select(field);
}); Let me know if this is off base - part of me thinks that it should hit the API logic before it hits the provider, but at a certain point I'm probably overthinking it. EDIT: I'm now noticing that the |
Hey guys, any thoughts on direction here? Didn't want to go too far down the path I described above without feedback from you as you're far more familiar with the tool than I am. |
My recommendation is the following:
If you do this right from the base controller/base model, every provider should be able to get this functionality "for free" just by virtue of being part of Koop. @ngoldman what do you think? |
I'm not sure this is going to work for our use-case. I updated my comment above because we don't actually need to generate something specific to Feature Services. It's just a standard where clause. At the end of the day it makes the most sense to generate the query at the cache level. Think about the difference between Elasticsearch and SQL syntax for example. There's also the issue that this doesn't knex.js doesn't seem to support queries for data stored as JSON (which is specific to koop-pgcache). |
Okay @dmfenton, having reread your comment, it looks like you're recommending making the ODATA format part of the core koop code. It seemed like @ajturner and I were talking about making output modular instead. In theory, that could mean koop core just provides GeoJSON interaction (perhaps with a basic Here's my understanding of how it currently works versus the modular output idea: Current path for queries
Proposed path for queries
|
That's one way to go. The other would be to take the path of writing a plugin. You still would end up modifying the core koop code to enable that plugin. https://github.com/koopjs/koop/blob/master/index.js#L104-L106 Let me parse through the other part of your question later today. |
I think the idea of moving towards an architecture of modular inputs and outputs with koop's core acting as the standard interface for querying GeoJSON makes sense. This would take time but it's the right direction. Right now the outputs are fixed to GeoJSON, FeatureService, KML, Shapefile, and CSV, but there's no reason other than time and work hours to make those components rather than the sole defacto outputs. @timwis I think we can use oData as a test case for developing a reasonable API that would fit this scenario. Keep in mind this would be long term in terms of the roadmap (at least a major version bump), as we need to continue supporting the current architecture in production. I'll do some research and try to come up with a reasonable path. |
I think something worth discussing is more distinct terminology. "provider" is generic and applies to both Source API and Client API. a few areas worth investigating:
|
If we're talking about redefining terminology, architecture, and API conventions, why not just call them inputs and outputs? Then you have an ecosystem like: Inputs
Caches
Outputs
Core would take a request, use input module to perform ETL and stream data to cache, then use output module to stream response filtered as necessary. So:
If data was already cached, skip steps 2 through 4. As an aside, in my opinion the only piece handling raw requests and responses should be the server. ETL, cache management, GeoJSON manipulation, and querying should have nothing to do with web requests. There's no need to pass those objects around. |
(I see the need for an ORM or something along those lines to deal with the filtering at the core level) |
@ngoldman What do you mean "server" in this quote?
The term "server" is not mentioned in list of terms :) |
Chelm is being cheeky but he's got a point, I'm not sure I understand what you mean. Also, one tricky thing about decoupling inputs and outputs totally is that the logic for creating cache keys is currently inside providers and is not standardized. |
@chelm I mean the express middleware piece of koop. Right now requests and responses are handed over to providers. I think that's an unnecessary coupling. @dmfenton As for the logic for creating cache keys being inside of providers... ideally it shouldn't be there, and should be standardized :) We're discussing a big set of tasks here but I think it's worth talking about. I know there are real hurdles to implementing this in the near-term, but it's something we can put on the roadmap. |
This is straight forward to address as part of this architecture discussion. In fact it should be another part of Koop-core or a concrete Koop-cache-provider that encapsulates any concepts of cache and cache-keys. Agreed with @ngoldman server means Koop core. Or call it Koop Kernel 🌽 |
I use "always" loosely of course. Everything can change and I think this is a great discussion to be having. |
If we modify the base model and base controller (in the right way), we should not have to touch providers at all. |
Doesn't supporting oData mean defining new routes (which are all defined by each individual provider for now)? |
@ngoldman isnt supporting odata just adding querystring params? Are there really new routes? |
Good point. Maybe the providers need to inherit routes. |
Thanks for all the replies, guys! A few thoughts:
Check out my comment above - IMO there are 3 entities: "source type/provider," "output format," and "interface/API." I don't have much opinion on what they end up getting called of course, but I'm just trying to point out that "output" doesn't cover both. For example, the FeatureService "interface" can be "output" in Esri JSON, KML, HTML, GeoJSON, etc. OData has a There are some implications of that on your list above - for instance, you may want to have the interface process the query first into some kind of common syntax or ORM object. (That way you can let the database handle the filtering when possible) @dmfenton: the plugin route you described is how I implemented the As regards a short-term solution, I'd point out that while OData is the topic at hand, the next one up is the SODA2 interface (already started), and after that perhaps the CKAN API interface (unless we think of a cooler one). So hard references to OData in core might not hold us over very long. For what it's worth, I'm happy to keep working on this and even submit pull requests in line with where you want it to go. I'm really excited about the possibilities this could create, but I don't want to go too far down any of those paths without clarity from you guys on the direction. So I appreciate the conversation and am happy to participate! |
Isn't the SODA2 interface covered in Koop-Socrata? Is that spec used elsewhere? |
Koop-Socrata enables you to connect to SODA2 APIs as a source/provider, but you have to interact with them using the FeatureService interface. It doesn't provide the ability to A SODA2 interface module would allow you to interact with any source/provider using the SODA2 query params. So you could have your data in CKAN or Esri and query it with the SODA2 API, just as we've been discussing with OData. |
@timwis I'm starting to see your point about interfaces vs. outputs... I was lumping exports (downloads) with interfaces into the output category but it may make sense to make those distinct. |
Pulled a SODA2 parser together |
I made a pull request based on the proposed path for queries above (koopjs/koop-provider-ckan#9). It doesn't solve (a) the fact that we'd have to do this for every provider, or (b) the fact that we're translating to FeatureServer query language before translating to a database query, but it seems like the best option before the more heavy changes discussed above. |
Here's a basic, working version of a SODA2 interface for koop, working the same way as the OData interface: https://github.com/timwis/koop-soda |
The above interface, koop-soda, works with very basic conditionals, ie. |
@timwis I'd rather not force everything to get translated twice before interacting with the cache, and there's no need to make FeatureService queries the defacto format. However coming up with a query standard to translate other APIs into is not trivial and would take some time (would need to cover queries from many different interfaces). If you're not in a rush we can figure that out, but I don't want to hold up your project :) |
In the meantime, I've just finished the first prototype of soda-postgres which basically translates the SODA-specific functions (like within_box) to their Postgres equivalents, outputs json in the right format, etc. Demo links at the bottom of the readme to see it in action. Interesting to think about for this discussion. |
@timwis demo links to IP server that isn't up or available public |
Sorry about that @ajturner, should be up now. It's really hard to come by a free postgres db with postgis enabled. Open Shift is the only one that offers it and I spent hours debugging their ambiguous logs before finally deciding to just do a $5 digitalocean droplet. Crossing my fingers that "forever" keeps it running like supervisor would. |
This discussion is well covered in the roadmap. |
That's awesome! Just read the roadmap - extremely well articulated - can't wait! Let me know if there's anything I can do to help |
Will do. Next steps for us are to write out some documentation for the V3 spec. Then we'll prototype something. There's definitely space in there for you to write a package that supports something other than geoservices. More to come in the following months. |
Hey folks! Just saw the splash image in the readme suggests this was implemented! :-O Is that the case? |
Yep! Check out koopjs/koop-output-geoservices and koopjs/koop-output-wfs for example |
Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.7.1 to 2.8.9. - [Release notes](https://github.com/npm/hosted-git-info/releases) - [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md) - [Commits](npm/hosted-git-info@v2.7.1...v2.8.9) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Hey guys, how would you recommend I go about building API output formats other than FeatureServer? For instance, how would I output a GeoJSON file in the OData spec? I see clearly how I'd add a provider, but this is on the other side of the fence and it's not quite clear, and I've perused the codebase.
EDIT: Looks like we'd need to add this to every provider as a method in the controller, eh? Any other way come to mind? I figured each provider just got it into a standard format (ie. GeoJSON) and then there were "output formats" that transformed the GeoJSON into the various output formats like FeatureService.
The text was updated successfully, but these errors were encountered: