REST API Documentation #1646
Comments
|
easiest option is to look at the controller section of the source code and checkout the Api directories (same structure is used for both the plugins and core). https://github.com/opnsense/core/tree/master/src/opnsense/mvc/app/controllers/OPNsense Api enabled pages are easy to find, all url's start with /ui/ for the user interface part, /api/ for the rest part. |
|
@AdSchellevis This is not super awesome. Can we at least have an endpoint that gives a list of registered API endpoints. That would be start instead of going on a wild goose chase through the code. |
|
not planned either, if you would like to work on documentation on the subject, just let us know. |
|
a fast variant: |
|
|
fichtner
added
feature
|
timeout due to inactivity |
|
If its not documented, it doesn't exist. Have you considered using a framework like swagger for api documentation? https://swagger.io/tools/ |
|
Sure, but we can only work on a limited number of things and from an open source perspective API docs are not high value targets and nobody else so far helped fund them. Cheers, |
|
@fichtner they also would be always incomplete and maybe wrong because the plugins change frequently. |
|
From a OSS perspective public API points do not matter as much? :) You mean, thats how to guarantee you cannot build and extern ecosystem of integrations since that does not matter? Its not only that the endpoints are entirely missing, its basically the payloads to if you want to run CRUD on any models. Since phalconphp has been wrapped but there are no docs for that either. I wonder how many people which are interested in integrating your API for anything they offer or have as OSS, can reverse engineer on how to create entity X on an endpoint. And the example given i guess is the most simple one can expect, but it wont help a lot in real case scenarios. Thats a huge gap here, pitty. @fichtner how should anybody help out with this realistically, when you need to understand the API structure prior that, including the payloads, what and how Thats not going to happen because there is probably no foundation for that either - closing this ticket has not helped in addition i guess |
|
This is just ridiculous... if you offer an API, tell users how to use it. Argumenting with "either you're a nerd and can reverse engineer our code or the feature won't be usable for you at all" is the most hilarious answer I've read in a while. That's the same like "we won't tell you how all the functions of a car work unless you can decipher it's electronic blueprints". I was considering opnsense as being a great alternative for pfsense in the way of automatic deployment and configuration, but now seeing where the project goes, it's really not... |
|
I am going to slowly contribute to the documention of the API as it's not actually that had to figure out if you open the network tools in Firefox
…________________________________
From: Julian Zielke <notifications@github.com>
Sent: Friday, April 27, 2018 10:16:10 AM
To: opnsense/core
Cc: Subscribed
Subject: Re: [opnsense/core] REST API Documentation (#1646)
This is just ridiculous... if you offer an API, tell users how to use it. Argumenting with "either you're a nerd and can reverse engineer or code or the feature won't be usable for you at all" is the most hilarious answer I've read in a while. That's the same like "we won't tell you how all the functions of a car work unless you can decipher it's electronic blueprints".
I was considering opnsense as being a great alternative for pfsense in the way of automatic deployment and configuration, but now seeing where the project goes, it's really not...
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<#1646 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ADddQnMRLiWYy9Sl7sLOnrnLoGLST_WEks5tsuHagaJpZM4Nb6DN>.
|
|
Also this is a good starting point as I used it to easily figure out the IDs/IPS API API docs<https://docs.opnsense.org/development/how-tos/api.html>
…________________________________
From: Jay Townsend <townsend891@hotmail.com>
Sent: Friday, April 27, 2018 10:19:52 AM
To: opnsense/core; opnsense/core
Cc: Subscribed
Subject: Re: [opnsense/core] REST API Documentation (#1646)
I am going to slowly contribute to the documention of the API as it's not actually that had to figure out if you open the network tools in Firefox
________________________________
From: Julian Zielke <notifications@github.com>
Sent: Friday, April 27, 2018 10:16:10 AM
To: opnsense/core
Cc: Subscribed
Subject: Re: [opnsense/core] REST API Documentation (#1646)
This is just ridiculous... if you offer an API, tell users how to use it. Argumenting with "either you're a nerd and can reverse engineer or code or the feature won't be usable for you at all" is the most hilarious answer I've read in a while. That's the same like "we won't tell you how all the functions of a car work unless you can decipher it's electronic blueprints".
I was considering opnsense as being a great alternative for pfsense in the way of automatic deployment and configuration, but now seeing where the project goes, it's really not...
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<#1646 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ADddQnMRLiWYy9Sl7sLOnrnLoGLST_WEks5tsuHagaJpZM4Nb6DN>.
|
No it's not. You are imposing your need over the needs and time constraints of an open source project. The only solutions are offering time, guidance or work. Ranting, contrary to popular belief, will not fix it. :) |
|
@fichtner It's not about my needs but about how the argument of bad documentation is twisted and turned. Being not a primary goal on this project is a reason everyone would understand, but
clearly tells me that some of you guys either love security by obscurity or just like programs without man or --help functionality. Also, publishing the so far available api would lead to quicker bugfixes and suggestions by the community. I was making my point here and the way the peoples questions were answered left me no other way than calling this behavior "ridiculous". This wasn't meant to be personally but somehow the answers were kinda (pf)senseless and even a bit ignorant. If you call this "ranting" I can't help you. |
I really don't know if you're trolling or not. We've written elaborate man pages and docs and if you don't see it that's perfectly fine, but it lessens your argument by an order of magnitude: Picking the example where we lack documentation after having ourselves stated that is true is a perfectly fine way of beating a dead horse. You could have said you would like to see this progress, maybe even asked what's needed or if this would be reopened. Some people have experience with API doc generation, great, let's get that on the table. Truth be told, you are right about everything and I don't disagree. Though you can see how the choice of doing none of the above lessens your entry and we are not afraid of calling this out, because at the end of the day when someone asks others to do work or lessen the work of others by telling how bad or lacking it is there is not much to be gained on either side. With all due respect, |
|
@jzielke-nli have you actually read all the documentation we've created over the years??? The whole concept is explained from how to build apps to how to build an entire system including lots of manual pages for the tools that we use.... As with with everything in the world.... there is always room for improvement, something like slate to document endpoints would be great to have, but takes time and maintenance, which we currently have no funding for. If people want to use our endpoints, their usually pretty easy to find using either the available inline PHPdocs or inspect the requests the browser is doing... Historically they haven't changed very much for the core system as we tend to keep them backwards compatible. Contributions are always more then welcome, as long as they lead to actionable results. Best, |
|
Ok guys let me get this straight: I have no time nor the intention to troll anyone here. I've been using pfsense for high productive firewall clusters running for many clients throughout a couple of years now, using complex routings and even written my own provisioning mechanism for virtual IPs. So I think I know the product quite well. So far for the question on reading the docs. Now when I stumbled upon your fork, which i really appreciate - and (again) I totally understand that feature requests aren't changing you priority on development in a matter of seconds - I see this API as a feature list on your website. And yes, I've looked up the api documentation as well. But - and that's the only thing which I was complaining about here - when people ask for help, the argument I quoted above just doesn't make sense, nor does it help the community in any way. Stating that the API is still an undocumented (or at least unfinished) feature on your website as well as here at the beginning of the discussion would have lead to the understanding of all the people here I guess. But come on @AdSchellevis you can't really belief that documenting the api endpoints is a security expose, it's the complete opposite. And that's the only point I was complaining about. I don't complain about your priorities nor about your efforts on different topics, but please, just be honest when a feature isn't really there rather than putting it on your website when in the end, users get stuck because the feature doesn't fully exist - at least not for ppl who're able to read and understand all your code which exists for this functionality so far. So hopefully I could explain my concerns and the API will be completed one day for everyone to be easy to use. |
|
huh, I never stated that it's a security expose, nor do I think so. |
|
Yes, thank you. I apologise for being as blunt as I have been.
So that's part of @AdSchellevis answer that seems to cause a bit of a stir. Maybe he can explain a bit more in detail himself, it's easy to ask. :) |
|
Right, thats the point. On the question
The response "not be a good idea for security reasons" kinda seemed very odd. |
|
Getting a list of available API endpoints with the required parameters is currently not really possible because the information may not exist and it depends on the the installed plugins as all OPNsense specific plugins do work with the API. You will always find the same kind of API calls for CRUD actions. If you need to automate something, you can open the page, open the network tools of Firefox, right click the request ans select copy as curl. After that you can just past it into your favourite editor and remove the unnecessary headers like the user agent, the cookies etc. and add the authentication ( |
|
Neither is the payload similar, nor the output or the responses. The reasons that there is no REST documentation is, that it is not even REST, and neither a framework has been used to do that. You will not be able to use swagger or anything else. What we have in opnsense is a self made, HTTP "api" and obviously its very hard to have docs for that - since you have to write and maintain them manually. This will always be a pain point. |
|
@jzielke-nli I see what you mean, not intended that way. Leaking endpoints to the outside world is usually not a good idea for security reasons, same reason why you don't want your attacker to know which webserver version you're running on a lot of public facing servers. Being able to collect help from the machine itself with the usual --help option obviously isn't a security concern (our configd system for example can also output it's known commands). My point was (and is) that providing a list of what your installation can provide in terms of services is easy, you can already do that, using grep for example. Giving context and making it useful is the hard part and requires more work then we can process at the moment. I would love to see proper endpoint documentation using something like slate (https://github.com/lord/slate) as long as the maintenance process to keep it useful is also in place. Eventually we might end up with something in addition to what we have now, endpoints are easy to identify so if anyone want to use them, their easy to find, documentation about how to use them would be valuable and would ideally cover both the core system as the plugins. Feel free to discuss further, I'm not going to, I see more comments flowing in while typing about what OPNsense is or isn't baring statements that are either incomplete or false. |
|
@EugenMayer can you explain? In all plugins, you can find an API call for get/set if you see some grid on it where you can do CRUD actions, you find a search API call, one for get, one for set, one for delete, … - and it is very similar everywhere. I don't want to say you are wrong about REST because deleting should be DELETE and not POST and so on but there is a call to delete stuff. Maybe @AdSchellevis can explain this design decision - it was not mine. As far as I know, also rails/rails has some fallback to support (legacy) browsers, which cannot handle other methods than GET/POST. In that case, a filed Which data you get, depends on the Model in the background. The fields do always look the same but it depends on the field. |
|
Neither there is anything like a "resource", nor VERBs are used at all, rather any "verb" is done using some kind of custom "subresource" like api/myplugin/deleteXXX or api/myplugin/deleteYYY or createYY..update... and so on. This is complete handmade, there is no consense and neither a pattern nor it matches anything you find out there. This means, you litterally have to reverse-engineer by reading the code, because even the payload is by no means a standard, nor you can guess it, nor its an simple annotation you read on the method or whatever. It And this all is not about some estethical, "RFC would be nice" kind of thing - it has directy implication on the way and effort you need to integrate it even *after you have reverse engineered it the endpoints. Creating something like https://github.com/EugenMayer/opnsense-cli was just as bad is it gets. I had to break with every single pattern i have prio implemented by following the existing plugins (check the git history, i entirely rewritten it in terms of REST/payload/response code after writing the plugin initially)... like As i have written above an API can encourage people to build third party integrations, to integrate with OPNsense, to "spread the word" and to broaden its audience. Or it does not :) |
|
@AdSchellevis Thanks for your detailed explanation. As for the comments on the APi design itself I won't jump in since I don't know much about it's implementation from the codebase side of view. |
|
I found this thread just now and I too have been met (struggling) with the above for a coupla months. However, I still think the API is a great addition to the project and one of the main bullet-points that guided me to using OPNsense. From what I read above, we need to start out with an inventory list of which api calls are currently available. I've quite easily scraped this info from the source code (php and volt) files. Then we need to make sense of that list. So far I have identified: crud items, plain settings (get/set only), services, packages, diagnostics and firmware calls. There's specifics for each category, but api calls within the same area tend to follow some similar patterns. I'll look into compiling a more comprehensive list... @L1ghtn1ng I can probably help you out and give some pointers about where the start. Once I spent some time going over the api calls it started to 'click' and the api actually made a lot of sense. Can it be improved? Sure! But for now let's roll with what we have and make incremental improvements towards an even better product. Honestly, I had also looked at documenting the REST api and had all the frustations @EugenMayer has. I'll gladly help out where I can. I can't gauge the effort needed to implement something like OpenAPI Specification and if it's even feasible within the OPNsense framework. |
|
OK, so I played around with the opnsense-sphinx-docker and got some start on the API list documentation. Now, what is the best format to represent all of this data? I'm hoping to get the page(s) auto-generated in bulk since I have a lot of the data already. Do you just want a plain big list? Maybe sorted by category or a separate page for each plug-in? Maybe there's also a need to provide some guidelines on how to best implement new API calls in the future. This could limit the differences between developers and ease the consumption of the API by 3rd parties. I also glanced at the MVC framework and it seems to me that it's possible for a plugin to adhere to the As for now, I have settled for a json lookup table in my project. That way I don't need to hardcode the exact API call in my code anymore, I can just reference the object & action and this gets translated into the actual api url. So I can have consistancy in the code and not be dependent on a strict convention for API calls. |
|
I have created PR opnsense/docs#24 with a proposal for the API reference. |
The docs for the REST api do not show any end points or how to get them. The only docs are here: https://docs.opnsense.org/development/how-tos/api.html. The example script gives 2 end points /core/firmware/status and /core/firmware/update but nothing further is documented that I can find. Is this all that is currently supported, or is the documentation missing?
The text was updated successfully, but these errors were encountered: