Asynchronous RESTful API to interact with sqlmap engine

[bdamele]

Design and develop an asynchronous RESTful API to interact with sqlmap engine. This is useful to use/call sqlmap from custom scripts, web interface, third-party tools or similar as opposed to use it from command line or wrap it as in a call similar to os.popen('sqlmap...').

This API will replace the XML-RPC service (#287).

[bdamele]

With regards to the potential technologies to use, I was looking into two micro web framework for the RESTful web server: bottle and flask and json Python built-in library for the JSON part.

Feedback on Python web server and JSON framework/libraries is welcome!

[bdamele]

Some references:

• JSON tutorial
• Python micro web frameworks comparison slide-deck - bottle (MIT) wins.

[bdamele]

First related commit, 6e31e87

[bdamele]

As of 4c4cb85 all features present in the XML-RPC interface have been ported to the RESTful API.

[andresriancho]

I totally support the idea of a sqlmap API (talked with stamparm about that when we met). It's a shame you guys decided against xmlrpc since it seemed the best option from my side; but I can imagine the reasons to move towards RESTful.

Question: Isn't the whole idea of an API to get the results in a programatic-friendly (if that even exists) way? I mean, what's the benefit that a user gets from calling the RESTful API (other than sqlmap being remote) if the output to a call is:

{"output": "[15:57:29] [INFO] GET parameter 'artist' is 'MySQL > 5.0.11 AND...

On the other side, I liked these answers from the API:

{"taskid": "d35abc1537ca591729d1aa5f639811ee"}
{"success": true}

[bdamele]

Question: Isn't the whole idea of an API to get the results in a programatic-friendly (if that even exists) way? I mean, what's the benefit that a user gets from calling the RESTful API (other than sqlmap being remote) if the output to a call is:

{"output": "[15:57:29] [INFO] GET parameter 'artist' is 'MySQL > 5.0.11 AND...

This is a work in progress feature. Currently, it no longer return the whole console output. Now the /scan/:taskid/output returns the query's output retrieved chunk whereas /scan/:taskid/log returns the log messages' chunks.

[bdamele]

As of 9766f60, the method /scan/:taskid/log returns a json-friendly output:

curl -s -H Content-Type: application/json -X POST -d '{url: http://debiandev/sqlmap/mysql/get_int.php?id=1, tech: TU, verbose: 2}' http://127.0.0.1:8775/scan/2e62d7515f55d1fdabf8431ee1c3e1a0/start
{
    "success": true
}

curl -s http://127.0.0.1:8775/scan/2e62d7515f55d1fdabf8431ee1c3e1a0/log
{
    "log": [
        {
            "text": "cleaning up configuration parameters", 
            "id": 0, 
            "levelname": "DEBUG"
        }
    ]
}
curl -s http://127.0.0.1:8775/scan/2e62d7515f55d1fdabf8431ee1c3e1a0/log
{
    "log": [
        {
            "text": "cleaning up configuration parameters", 
            "id": 0, 
            "levelname": "DEBUG"
        }, 
        {
            "text": "setting the HTTP timeout", 
            "id": 1, 
            "levelname": "DEBUG"
        }, 
        {
            "text": "setting the HTTP method to GET", 
            "id": 2, 
            "levelname": "DEBUG"
        }, 
[...]

[andresriancho]

Hi guys, did this achieve a stable state? Would you recommend me to use it in w3af?

[bdamele]

@andresriancho, not yet, working on it lately..

[bdamele]

As of edd6699, the RESTful API is usable. However, there's work to be done to allow the API to get the partial output (real-time enumeration): at the moment, the API has knowledge of the enumerated data only at the end of the enumeration of the requested switch.

[bdamele]

We can consider this to be closed now: the API is usable, although improvements can be done and will be documented in a separate ticket.

[riramar]

Hi All,

Sorry if this is a stupid question but I didn't find a documentation about how to run sqlmap testing a restful interface which accept only JSON. Could you help me?

Thanks!
Ricardo

[stamparm]

@riramar

There is currently none :). I'll just give you a quick intro.

At server side:

$ python sqlmapapi.py -s -H 0.0.0.0
[19:42:00] [INFO] Running REST-JSON API server at '0.0.0.0:8775'..
[19:42:00] [INFO] Admin ID: cfdd0c84a8ebbccf40a97fe6eaaeac9d
[19:42:00] [DEBUG] IPC database: /tmp/sqlmapipc-QUdQ7m
[19:42:00] [DEBUG] REST-JSON API server connected to IPC database

At client side:

$ curl http://127.0.0.1:8775/task/new
{
    "taskid": "4be40bb5e98a03c2", 
    "success": true
}
$ curl -H "Content-Type: application/json" -X POST -d '{"url": "http://testphp.vulnweb.com/artists.php?artist=1"}' http://127.0.0.1:8775/scan/4be40bb5e98a03c2/start
{
    "engineid": 3068, 
    "success": true
}
$ curl http://127.0.0.1:8775/scan/4be40bb5e98a03c2/data
{
    "data": [], 
    "success": true, 
    "error": []
}
$ curl http://127.0.0.1:8775/scan/4be40bb5e98a03c2/log
{
    "log": [
        {
            "message": "testing connection to the target URL", 
            "level": "INFO", 
            "time": "19:44:23"
        }, 
        {
            "message": "testing if the target URL is stable. This can take a couple of seconds", 
            "level": "INFO", 
            "time": "19:44:24"
        }, 
        {
            "message": "target URL is stable", 
            "level": "INFO", 
            "time": "19:44:25"
        }, 
        {
            "message": "testing if GET parameter 'artist' is dynamic", 
            "level": "INFO", 
            "time": "19:44:25"
        }, 
        {
            "message": "confirming that GET parameter 'artist' is dynamic", 
            "level": "INFO", 
            "time": "19:44:25"
        }, 
        {
            "message": "GET parameter 'artist' is dynamic", 
            "level": "INFO", 
            "time": "19:44:26"
        }, 
        {
            "message": "heuristic (basic) test shows that GET parameter 'artist' might be injectable (possible DBMS: 'MySQL')", 
            "level": "INFO", 
            "time": "19:44:26"
        }, 
        {
            "message": "testing for SQL injection on GET parameter 'artist'", 
            "level": "INFO", 
            "time": "19:44:26"
        }, 
        {
            "message": "testing 'AND boolean-based blind - WHERE or HAVING clause'", 
            "level": "INFO", 
            "time": "19:44:26"
        }, 
        {
            "message": "GET parameter 'artist' seems to be 'AND boolean-based blind - WHERE or HAVING clause' injectable ", 
            "level": "INFO", 
            "time": "19:44:27"
        }, 
        {
            "message": "testing 'MySQL >= 5.0 AND error-based - WHERE or HAVING clause'", 
            "level": "INFO", 
            "time": "19:44:27"
        }, 
        {
            "message": "testing 'MySQL >= 5.1 AND error-based - WHERE or HAVING clause (EXTRACTVALUE)'", 
            "level": "INFO", 
            "time": "19:44:27"
        }, 
        {
            "message": "testing 'MySQL >= 5.1 AND error-based - WHERE or HAVING clause (UPDATEXML)'", 
            "level": "INFO", 
            "time": "19:44:28"
        }, 
        {
            "message": "testing 'MySQL >= 4.1 AND error-based - WHERE or HAVING clause'", 
            "level": "INFO", 
            "time": "19:44:28"
        }, 
        {
            "message": "testing 'MySQL >= 5.0 OR error-based - WHERE or HAVING clause'", 
            "level": "INFO", 
            "time": "19:44:28"
        }, 
        {
            "message": "testing 'MySQL >= 5.1 OR error-based - WHERE or HAVING clause (EXTRACTVALUE)'", 
            "level": "INFO", 
            "time": "19:44:29"
        }, 
        {
            "message": "testing 'MySQL >= 5.1 OR error-based - WHERE or HAVING clause (UPDATEXML)'", 
            "level": "INFO", 
            "time": "19:44:29"
        }, 
        {
            "message": "testing 'MySQL >= 4.1 OR error-based - WHERE or HAVING clause'", 
            "level": "INFO", 
            "time": "19:44:29"
        }, 
        {
            "message": "testing 'MySQL OR error-based - WHERE or HAVING clause'", 
            "level": "INFO", 
            "time": "19:44:29"
        }, 
        {
            "message": "testing 'MySQL >= 5.0 error-based - Parameter replace'", 
            "level": "INFO", 
            "time": "19:44:30"
        }, 
        {
            "message": "testing 'MySQL >= 5.1 error-based - Parameter replace (EXTRACTVALUE)'", 
            "level": "INFO", 
            "time": "19:44:30"
        }, 
        {
            "message": "testing 'MySQL >= 5.1 error-based - Parameter replace (UPDATEXML)'", 
            "level": "INFO", 
            "time": "19:44:30"
        }, 
        {
            "message": "testing 'MySQL inline queries'", 
            "level": "INFO", 
            "time": "19:44:30"
        }, 
        {
            "message": "testing 'MySQL > 5.0.11 stacked queries'", 
            "level": "INFO", 
            "time": "19:44:31"
        }, 
        {
            "message": "testing 'MySQL < 5.0.12 stacked queries (heavy query)'", 
            "level": "INFO", 
            "time": "19:44:31"
        }, 
        {
            "message": "testing 'MySQL > 5.0.11 AND time-based blind'", 
            "level": "INFO", 
            "time": "19:44:31"
        }, 
        {
            "message": "GET parameter 'artist' seems to be 'MySQL > 5.0.11 AND time-based blind' injectable ", 
            "level": "INFO", 
            "time": "19:44:42"
        }, 
        {
            "message": "testing 'MySQL UNION query (NULL) - 1 to 20 columns'", 
            "level": "INFO", 
            "time": "19:44:42"
        }, 
        {
            "message": "automatically extending ranges for UNION query injection technique tests as there is at least one other (potential) technique found", 
            "level": "INFO", 
            "time": "19:44:42"
        }, 
        {
            "message": "ORDER BY technique seems to be usable. This should reduce the time needed to find the right number of query columns. Automatically extending the range for current UNION query injection technique test", 
            "level": "INFO", 
            "time": "19:44:42"
        }, 
        {
            "message": "target URL appears to have 3 columns in query", 
            "level": "INFO", 
            "time": "19:44:43"
        }, 
        {
            "message": "GET parameter 'artist' is 'MySQL UNION query (NULL) - 1 to 20 columns' injectable", 
            "level": "INFO", 
            "time": "19:44:44"
        }, 
        {
            "message": "the back-end DBMS is MySQL", 
            "level": "INFO", 
            "time": "19:44:45"
        }
    ], 
    "success": true
}

[riramar]

Hi @stamparm ,

First of all thanks so much for your quick replay.
I'm not sure if I got the whole idea of sqlmapapi. Let me try to describe my "environment" to you so maybe you can help me. I'm not allowed to give specific details since this is a closed project.
Nowadays we have a RESTful server (let's say https://www.xptorest.com.br) exposed acting as a Back-End that it will be integrate with some customers (let's say https://www.mycustomer.com.br) which will act as a Front-End.
The Front-End will be like a GUI for the final user and it will do some RESTful requests to our Back-End. On the Back-End we are using MySQL as a database.
I have all the specification of our RESTful, most of them accept POST HTTP method and specific JSON as a paylod.
I'm using another commercial tool in order to scan the Back-End against some attacks including JSON injection and SQL Injection but I'm not happy with the results (too many false-positives). This tool accept the RESTful specifications in a specific format as a parameters for the scan.
Reading your answer I was thinking that sqlmapapi.py will act like a bridge between curl malicious requests and my Back-End translating the curl request to RESTful requests but for me it doesn't make sense and I'll explain why. In your case you use testphp.vulnweb.com insted of www.xptorest.com.br (my case).
Since you didn't specify any RESTful request to sqlmapapi.py it'll impossible to create a valid request to the Back-End.
So I'm not a native English speaker but I hope that you understand my doubt and help to fin the solution.

Thanks!
Ricardo Iramar

[stamparm]

Since you didn't specify any RESTful request to sqlmapapi.py it'll impossible to create a valid request to the Back-End. - I've given you an example with VALID RESTful requests to sqlmapapi.py.

All those examples I've given in At client side: part are RESTful requests.

This tool accept the RESTful specifications in a specific format as a parameters for the scan. - every tool has it's own specification. Our specification says that you have to provide options like you would do with sqlmap.conf.

Reading your answer I was thinking that sqlmapapi.py will act like a bridge between curl malicious requests and my Back-End translating the curl request to RESTful requests but for me it doesn't make sense and I'll explain why - you are wrong here. Curl is just used for sending RESTful requests toward the server itself (sqlmapapi.py).

I would suggest you to research a bit what RESTful means and please try to understand what does command lines do in example(s) I've given to you. I am really not sure how you could conclude Reading your answer I was thinking that sqlmapapi.py will act like a bridge between curl malicious requests if you know what RESTful means.

[riramar]

Hi @stamparm

As i said I'm not a native speaker and sometimes I have difficult to express myself in English. I'm really sorry about that.
Maybe I didn't write exactly what I wanted with this phrase. Really sorry for that. :(

Reading your answer I was thinking that sqlmapapi.py will act like a bridge between curl malicious requests...

Sometimes I think I'm been rude with people when I try to talk in English. Maybe I need to improve my English more than my knowledge in RESTful interface. For my own good I'll do both, study RESTful and English. Maybe after that I'll be able to understand better your explanation how sqlmapapi.py works.

Thanks!
Ricardo Iramar

[stamparm]

Just one thing that maybe left in the air: sqlmapapi.py will act like a bridge between curl malicious requests <- not true. sqlmapapi.py is a RESTful interface toward sqlmap instance running on server side, not a curl bridge.

[riramar]

OK! Thanks! I'll try to do some tests here.

[riramar]

Hi @stamparm

Now I see the big misunderstood that I made. I was confusing about the purpose of sqlmapapi. :(
I thought that sqlmapapi was a tool that load a configuration file (eg. wadl) and create some sql injection tests (eg. valid RESTful requests) for sqlmap.
Reading all the thread from the beginning again and again I understood the real purpose of the sqlmapapi that's different from I thought.
Really sorry for this big mistake and thanks again for your help. You and @inquisb are doing a great job.

Ricardo Iramar

[daremon]

Hi @stamparm

Is there a way to check if sqlmap has finished? Neither /data nor /log seem to offer this unless I missed something.

[stamparm]

@daremon yes, there is.

http://127.0.0.1:8775/scan/<taskid>/status

See example in attached picture:

After the task finishes/terminates sqlmap API status call manifests like this:

[daremon]

Perfect, thank you!