Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REST API Scanning #15087

Closed
andresriancho opened this issue Feb 1, 2017 · 16 comments

Comments

Projects
None yet
2 participants
@andresriancho
Copy link
Owner

commented Feb 1, 2017

User story

As a user I would like to be able to scan REST APIs which might or might not be consumed by a single page application frontend.

Problem being solved

No scanner is able to identify the URLs and parameters which are exposed by a REST API because, unlike with web services and WSDL, there is no "index" or "web service definition". In order to scan REST APIs w3af needs the help of "the documentation" which we can parse from the popular swagger.io

Conditions of satisfaction

Note: updated conditions of satisfaction and TODO list is here

  • w3af can parse the Top3 REST API documentation frameworks. This should be implemented using document parser subclasses. The document parsers should:
  • w3af should be able to extract the REST API information in two different scenarios:
  • User points the target to the REST API documentation URL
  • User points the target to api.target.com. The scanner should try to identify the REST API documentation in the most common URLs. This requires a new crawling plugin to be enabled.
  • The user should be able to specify an authentication method (header, cookie, token, etc.). This is already implemented in the framework, but we need to give the user better information on how this integrates with the REST API scanning. For example, if we detect the REST API documentation, try to consume a service and it returns 401, we should create a new information object in the knowledge base letting the user know that we were unable to consume the REST API and that authentication should be configured following steps in documentation (provide link).
  • Documentation is written to explain users how to use these features
  • There are unittests for all features
  • Handle security definitions part of the open api spec. Idea: Let the user know he needs to set a specific header if he hasn't yet.
  • Make sure that spec.api_url doesn't point to a domain outside our target
  • externalDocs URLs should be included in the get_refs
  • Make sure REST API call parameters are properly filled with a value, just like with HTML forms
  • Handle download of additional json files https://github.com/Yelp/bravado/blob/18561ca3ca744af389918353742486cdc622affc/bravado/client.py#L115
  • Handle array objects which have complex types inside (see comment near if parameter_type == 'array':)
  • When enabling the REST API scanning "support" we should also enable FileNameMutant via fuzz_url_filenames in order to fuzz the 42 in /pets/dogs/42
  • Just in case... do NOT scan calls with DELETE method. Warn the user about w3af ignoring these.
  • Support parameter.location == 'path' (example: /person/{:id}) when creating the fuzzable request
  • Support parameter.location == 'header' (example: api-key: ...) when creating the fuzzable request
  • Silence this warning or handle it? w3af/local/lib/python2.7/site-packages/bravado_core/spec_flattening.py:192: Warning: Un-referenced models cannot be un-flattened if spec_definitions is not present category=Warning,
@andresriancho

This comment has been minimized.

Copy link
Owner Author

commented Aug 28, 2017

@mackinra

This comment has been minimized.

Copy link

commented Feb 23, 2018

Was this ever completed? If not, how far along would you say it is?

@andresriancho

This comment has been minimized.

Copy link
Owner Author

commented Feb 26, 2018

Well, I found that generating mutants based on the REST API specification was complicated (7f8694f) because of all the existing data types, references, and the way they can be nested / combined.

Coding that will take considerable time.

Also, I was coding this as part of an agreement I have with a project sponsor; and they decreased the priority of this task in favor of other more critical bug fixes.

I believe that the project sponsor will want this completed in ~2 months.

Taking that into account, I would say that once I come back to this issue it will take ~20 hours for me to complete it.

@mackinra

This comment has been minimized.

Copy link

commented Mar 6, 2018

@andresriancho So beyond generating mutants (which I assume only refers to varying the input data to API requests?), how useful is this branch for testing REST APIs? I take it the swagger piece is complete?

@andresriancho

This comment has been minimized.

Copy link
Owner Author

commented Mar 6, 2018

The swagger / open API parser is not complete.

I'll be working on this tomorrow, hopefully finishing it by the end of the week!

@andresriancho

This comment has been minimized.

Copy link
Owner Author

commented Mar 9, 2018

@andresriancho

This comment has been minimized.

Copy link
Owner Author

commented Mar 11, 2018

The REST API scanning feature is still a work in progress.

I've put several hours into this (~14) and I'm still fighting to finish the most important part: supporting all of the different parameter types which can be documented by open api.

The code for this feature is at:

https://github.com/andresriancho/w3af/tree/feature/rest-api-scanning/w3af/core/data/parsers/doc/open_api

As you can see in the JSON files which are inside this directory and the unit tests in this file there are many complexities in the open api format.

I would say that I'm 80% done with the support for the different primitive types, models, and the ways in which they can be combined / nested.

After finishing the tests in test_specification.py I'll have to write the much simpler code at https://github.com/andresriancho/w3af/blob/feature/rest-api-scanning/w3af/core/data/parsers/doc/open_api/requests.py , which translates the parameters I got from https://github.com/andresriancho/w3af/blob/feature/rest-api-scanning/w3af/core/data/parsers/doc/open_api/specification.py into fuzzable requests which are ready to send to the framework.

This task is complex!

@mackinra

This comment has been minimized.

Copy link

commented Mar 11, 2018

Cool stuff, will check it out. By the way, will you be supporting both JSON and YAML swagger files, or just JSON initially?

@andresriancho

This comment has been minimized.

Copy link
Owner Author

commented Mar 12, 2018

Bravado (the library we use for parsing the open api specification) can parse both json and yaml files, so we support them both.

@andresriancho

This comment has been minimized.

Copy link
Owner Author

commented Mar 12, 2018

Organizing my thoughts a little bit here, in order to be able to keep on working on this on the next sprint:

  • Need to complete and unittest the specification / parameter part
  • Add support for https://swagger.io/docs/specification/data-models/oneof-anyof-allof-not/
  • Silence or handle this warning: bravado_core/spec_flattening.py:192: Warning: Un-referenced models cannot be un-flattened if spec_definitions is not present category=Warning,
  • Change the open_api.py parser to ignore DELETE calls
  • Need to work on translating operations (with the fill attribute set) generated by specification / parameter into form and form parameter instances. This means that the requests.py code might need to be renamed / redone. Unittest in detail, there are many cases: parameters in path, query string, headers, body (with different encodings).
  • Create a crawl plugin, open_api that will perform requests to the most common REST API documentation URLs. If one is found it should send them to the framework using get_api_calls and then creating the fuzzable requests, in a very similar way than web_spider.
  • The crawl plugin should also let the user can configure:
    • Headers to be sent in API requests (in case api key is sent in headers)
    • Query string parameters to be sent in API requests (in case api key is sent in QS)
    • These should be added to the fuzzable requests returned by the parser before sending them to the core
  • When enabling the crawl.open_api plugin is enabled we should forcefully enable FileNameMutant via fuzz_url_filenames in order to fuzz the 42 in /pets/dogs/42
  • Add documentation to the main docs to explain how everything works

Nice to haves:

  • If crawl.open_api is enabled and there are no credentials auth.open_api and at least one of the GET requests found in the specification yields 401 when consumed, then we should warn the user that he needs to configure authentication
  • Extend the open_api.py parser to extract externalDocs URLs from the open api documents and include them in the result of get_refs
  • If the open api parser finds a specification which contains at least one operation and the crawl.open_api / auth.open_api plugins are not enabled, then we need to let the user know that he should enable them in order to increase the scan coverage

Time required to do all this: 16h.

@andresriancho

This comment has been minimized.

Copy link
Owner Author

commented Mar 12, 2018

giphy

@andresriancho

This comment has been minimized.

Copy link
Owner Author

commented Apr 5, 2018

File names to be queried by crawl.open_api:

  • swagger.json
  • openapi.json
  • openapi.yaml

They should be queried in all application directories.

Also, I should query those filenames in:

  • /api/
  • /api/v2/
  • /api/v1/
  • /api/v2.0/
  • /api/v2.1/
  • /api/v1.0/
  • /api/v1.1/
  • /api/2.0/
  • /api/2.1/
  • /api/1.0/
  • /api/1.1/

andresriancho pushed a commit that referenced this issue Apr 6, 2018

Andres Riancho
@andresriancho

This comment has been minimized.

Copy link
Owner Author

commented Apr 6, 2018

@mackinra this feature is complete and ready to use in the develop branch, documentation here:

http://docs.w3af.org/en/develop/scan-rest-apis.html

@andresriancho

This comment has been minimized.

Copy link
Owner Author

commented Apr 6, 2018

@mackinra

This comment has been minimized.

Copy link

commented Apr 9, 2018

Cool! Will give it a try. I take it I can manually provide the swagger file too (i.e., it doesn't have to be served by the API server)?

@andresriancho

This comment has been minimized.

Copy link
Owner Author

commented Apr 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.