Merge pull request #244 from xmendez/deve

Deve

Makefile

@@ -1,7 +1,9 @@
 .PHONY: docs
-test:
+tox:
 	pip install tox
 	tox --recreate
+test:
+	pytest -v -s tests/
 flake8:
 	black --check src tests
 	flake8 src tests

docs/user/advanced.rst

@@ -466,6 +466,7 @@ value|replace('what', 'with')    value|r('what', 'with') Returns value replacing
 value|unique()                   value|u()               Returns True if a value is unique.
 value|startswith('value')        value|sw('value')       Returns true if the value string starts with param
 value|gregex('expression')       value|gre('exp')        Returns first regex group that matches in value
+value|diff(expression)                                   Returns diff comparison between value and expression
 ================================ ======================= =============================================
 
 * When a FuzzResult is available, you could perform runtime introspection of the objects using the following symbols
@@ -482,7 +483,7 @@ lines        l              Wfuzz's result HTTP response lines
 words        w              Wfuzz's result HTTP response words
 md5                         Wfuzz's result HTTP response md5 hash
 history      r              Wfuzz's result associated FuzzRequest object
-plugins                     Wfuzz's results associated plugins result in the form of {'plugin id': ['result']}
+plugins                     Wfuzz's plugins scan results 
 ============ ============== =============================================
 
 FuzzRequest object's attribute (you need to use the r. prefix) such as:
@@ -609,7 +610,7 @@ The payload to filter, specified by the -z switch must precede --slice command l
 
 The specified expression must return a boolean value, an example, using the unique operator is shown below::
 
-    $ wfuzz-cli.py -z list --zD one-two-one-one --slice "FUZZ|u()" http://localhost:9000/FUZZ
+    $ wfuzz -z list --zD one-two-one-one --slice "FUZZ|u()" http://localhost:9000/FUZZ
 
     ********************************************************
     * Wfuzz 2.2 - The Web Fuzzer                           *
@@ -633,6 +634,37 @@ The specified expression must return a boolean value, an example, using the uniq
 It is worth noting that, the type of payload dictates the available language symbols. For example, a dictionary payload such as in the example
 above does not have a full FuzzResult object context and therefore object fields cannot be used.
 
+When slicing a FuzzResult payload, you are accessing the FuzzResult directly, therefore given a previous session such as::
+
+    $ wfuzz -z range --zD 0-0 -u http://www.google.com/FUZZ --oF /tmp/test1
+    ...
+    000000001:   404        11 L     72 W       1558 Ch     "0"                                                                                                                                                 
+    ...
+
+this can be used to filter the payload::
+
+    $ wfpayload -z wfuzzp --zD /tmp/test1 --slice "c=404"
+    ...
+    000000001:   404        11 L     72 W       1558 Ch     "0"                                                                                                                                                 
+    ...
+
+    $ wfpayload -z wfuzzp --zD /tmp/test1 --slice "c!=404"
+    ...
+    wfuzz.py:168: UserWarning:Fatal exception: Empty dictionary! Please check payload or filter.
+    ...
+
+In fact, in this situation, FUZZ refers to the previous result (if any)::
+
+    $ wfuzz -z wfuzzp --zD /tmp/test1 -u FUZZ --oF /tmp/test2
+    ...
+    000000001:   404        11 L     72 W       1558 Ch     "http://www.google.com/0"                                                                                                                           
+    ...
+
+    $ wfpayload -z wfuzzp --zD /tmp/test2 --efield r.headers.response.date --efield FUZZ[r.headers.response.date]
+    ...
+    000000001:   404        11 L     72 W       1558 Ch     "http://www.google.com/0 | Mon, 02 Nov 2020 19:29:03 GMT | Mon, 02 Nov 2020 19:27:27 GMT"                                                           
+    ...
+
 Re-writing a payload
 """""""
 
@@ -740,6 +772,12 @@ The above command will generate HTTP requests such as the following::
 
 You can filter the payload using the filter grammar as described before.
 
+Reutilising previous results
+--------------------------------------
+
+Plugins results contain a treasure trove of data. Wfuzz payloads and object introspection (explained in the filter grammar section) exposes a Python object interface to plugins results.
+This allows you to perform semi-automatic tests based on plugins results or compile a set of results to be used in another tool.
+
 Request mangling
 ^^^^^^^^^
 

docs/user/basicusage.rst

@@ -252,7 +252,7 @@ For example, to show results in JSON format use the following command::
 
     $ wfuzz -o json -w wordlist/general/common.txt http://testphp.vulnweb.com/FUZZ
 
-When using the default output you can also select additional FuzzResult's fields to show, using --efield, together with the payload description::
+When using the default or raw output you can also select additional FuzzResult's fields to show, using --efield, together with the payload description::
 
     $ wfuzz -z range --zD 0-1 -u http://testphp.vulnweb.com/artists.php?artist=FUZZ --efield r
     ...
@@ -262,7 +262,7 @@ When using the default output you can also select additional FuzzResult's fields
                                                         Host: testphp.vulnweb.com
     ...
 
-The above is useful, for example, to debug what exact HTTP request Wfuzz sent to the remote Web server.
+The above command is useful, for example, to debug what exact HTTP request Wfuzz sent to the remote Web server.
 
 To completely replace the default payload output you can use --field instead::
 
@@ -279,4 +279,14 @@ To completely replace the default payload output you can use --field instead::
     000000001:   200        104 L    364 W    4735 Ch     "0 | http://testphp.vulnweb.com/artists.php?artist=0 | 4735"                                                                                        
     ...
 
+The field printer can be used with a --efield or --field expression to list only the specified filter expressions without a header or footer::
+
+
+    $ wfuzz -z list --zD https://www.airbnb.com/ --script=links --script-args=links.regex=.*js$,links.enqueue=False -u FUZZ -o field --field plugins.links.link | head -n3
+    https://a0.muscache.com/airbnb/static/packages/4e8d-d5c346ee.js
+    https://a0.muscache.com/airbnb/static/packages/7afc-ac814a17.js
+    https://a0.muscache.com/airbnb/static/packages/7642-dcf4f8dc.js
+
+The above command is useful, for example, to pipe wfuzz into other tools or perform console scripts.
+
 --efield and --field are in fact filter expressions. Check the filter language section in the advance usage document for the available fields and operators.

docs/user/installation.rst

@@ -26,7 +26,7 @@ You can either clone the public repository::
 
     $ git clone git://github.com/xmendez/wfuzz.git
 
-Or download last `release <https://github.com/xmendez/wfuzz/releases/latest>_`.
+Or download last `release <https://github.com/xmendez/wfuzz/releases/latest>`_.
 
 Once you have a copy of the source, you can embed it in your own Python
 package, or install it into your site-packages easily::
@@ -91,6 +91,16 @@ If you get errors such as::
     
 Run brew update && brew upgrade
 
+If you get an error such as::
+
+    ImportError: pycurl: libcurl link-time ssl backends (secure-transport, openssl) do not include compile-time ssl backend (none/other)
+
+That might indicate that pycurl was reinstalled and not linked to the SSL correctly. Uninstall pycurl as follows::
+
+    $ pip uninstall pycurl
+
+and re-install pycurl starting from step 4 above.
+
 Pycurl on Windows
 -----------------
 

requirements.txt

@@ -6,7 +6,6 @@
 #
 attrs==20.1.0             # via pytest
 chardet==3.0.4            # via wfuzz (setup.py)
-future==0.18.2            # via wfuzz (setup.py)
 iniconfig==1.0.1          # via pytest
 more-itertools==8.5.0     # via pytest
 packaging==20.4           # via pytest

setup.py

@@ -32,7 +32,6 @@
     'pycurl',
     'pyparsing<2.4.2;python_version<="3.4"',
     'pyparsing>=2.4*;python_version>="3.5"',
-    'future',
     'six',
     'configparser;python_version<"3.5"',
     'chardet',

src/wfuzz/__init__.py

@@ -1,5 +1,5 @@
 __title__ = "wfuzz"
-__version__ = "3.0.3"
+__version__ = "3.1.0"
 __build__ = 0x023000
 __author__ = "Xavier Mendez"
 __license__ = "GPL 2.0"

src/wfuzz/core.py

@@ -30,11 +30,12 @@ def __init__(self, options):
         # genReq ---> seed_queue -> [slice_queue] -> http_queue/dryrun -> [round_robin -> plugins_queue] * N
         # -> [recursive_queue -> routing_queue] -> [filter_queue] -> [save_queue] -> [printer_queue] ---> results
 
+        self.options = options
         self.qmanager = QueueManager(options)
         self.results_queue = MyPriorityQueue()
 
         if options["allvars"]:
-            self.qmanager.add("allvars_queue", AllVarQ(options))
+            self.qmanager.add("seed_queue", AllVarQ(options))
         else:
             self.qmanager.add("seed_queue", SeedQ(options))
 
@@ -56,8 +57,12 @@ def __init__(self, options):
         if options.get("script"):
             self.qmanager.add("plugins_queue", JobQ(options))
 
-        if options.get("script") or options.get("rlevel") > 0:
+        if options.get("rlevel") > 0:
             self.qmanager.add("recursive_queue", RecursiveQ(options))
+
+        if (options.get("script") or options.get("rlevel") > 0) and options.get(
+            "transport"
+        ) == "http":
             rq = RoutingQ(
                 options,
                 {
@@ -115,7 +120,7 @@ def __next__(self):
     def stats(self):
         return dict(
             list(self.qmanager.get_stats().items())
-            + list(self.qmanager["transport_queue"].job_stats().items())
+            + list(self.qmanager["transport_queue"].http_pool.job_stats().items())
             + list(self.options.stats.get_stats().items())
         )
 

src/wfuzz/dictionaries.py

@@ -1,6 +1,6 @@
 from .exception import FuzzExceptNoPluginError, FuzzExceptBadOptions
 from .facade import Facade
-from .filters.ppfilter import FuzzResFilterSlice
+from .filters.ppfilter import FuzzResFilterSlice, FuzzResFilter
 from .fuzzobjects import FuzzWord, FuzzWordType
 
 
@@ -119,7 +119,8 @@ def next_word(self):
 
 class SliceIt(BaseDictionary):
     def __init__(self, payload, slicestr):
-        self.ffilter = FuzzResFilterSlice(filter_string=slicestr)
+        self.ffilter = FuzzResFilter(filter_string=slicestr)
+        self.ffilter_slice = FuzzResFilterSlice(filter_string=slicestr)
         self.payload = payload
 
     def count(self):
@@ -128,10 +129,18 @@ def count(self):
     def get_type(self):
         return self.payload.get_type()
 
+    def _get_filtered_value(self, item):
+        if item.type == FuzzWordType.FUZZRES:
+            filter_ret = self.ffilter.is_visible(item.content)
+        else:
+            filter_ret = self.ffilter_slice.is_visible(item.content)
+
+        return filter_ret
+
     def next_word(self):
         # can be refactored using the walrus operator in python 3.8
         item = next(self.payload)
-        filter_ret = self.ffilter.is_visible(item.content)
+        filter_ret = self._get_filtered_value(item)
 
         if not isinstance(filter_ret, bool) and item.type == FuzzWordType.FUZZRES:
             raise FuzzExceptBadOptions(
@@ -140,7 +149,7 @@ def next_word(self):
 
         while isinstance(filter_ret, bool) and not filter_ret:
             item = next(self.payload)
-            filter_ret = self.ffilter.is_visible(item.content)
+            filter_ret = self._get_filtered_value(item)
 
         if not isinstance(filter_ret, bool):
             return FuzzWord(filter_ret, item.type)

src/wfuzz/externals/reqresp/Response.py

@@ -146,7 +146,7 @@ def parseResponse(self, rawheader, rawbody=None, type="curl"):
         tp = TextParser()
         tp.setSource("string", rawheader)
 
-        tp.readUntil(r"(HTTP\S*) ([0-9]+)")
+        tp.readUntil(r"(HTTP/[0-9.]+) ([0-9]+)")
         while True:
             while True:
                 try:
@@ -162,7 +162,7 @@ def parseResponse(self, rawheader, rawbody=None, type="curl"):
                 if self.code != "100":
                     break
                 else:
-                    tp.readUntil(r"(HTTP\S*) ([0-9]+)")
+                    tp.readUntil(r"(HTTP/[0-9.]+) ([0-9]+)")
 
             self.code = int(self.code)
 
@@ -176,7 +176,7 @@ def parseResponse(self, rawheader, rawbody=None, type="curl"):
             # curl sometimes sends two headers when using follow, 302 and the final header
             # also when using proxies
             tp.readLine()
-            if not tp.search(r"(HTTP\S*) ([0-9]+)"):
+            if not tp.search(r"(HTTP/[0-9.]+) ([0-9]+)"):
                 break
             else:
                 self._headers = []

src/wfuzz/externals/reqresp/TextParser.py

@@ -130,9 +130,8 @@ def readLine(self):
             if self.oldindex >= 0:
                 self.newindex = self.string.find("\n", self.oldindex, len(self.string))
                 if self.newindex == -1:
-                    self.lastFull_line = self.string[self.oldindex : len(self.string)]
-                else:
-                    self.lastFull_line = self.string[self.oldindex : self.newindex + 1]
+                    self.newindex = len(self.string) - 1
+                self.lastFull_line = self.string[self.oldindex : self.newindex + 1]
 
                 self.oldindex = self.newindex + 1
             else:

src/wfuzz/facade.py

@@ -8,8 +8,6 @@
 
 import os
 
-# python2 and 3: metaclass
-from future.utils import with_metaclass
 
 ERROR_CODE = -1
 BASELINE_CODE = -2
@@ -64,8 +62,7 @@ def get_plugin(self, identifier):
             )
 
 
-# python2 and 3: class Facade(metaclass=utils.Singleton):
-class Facade(with_metaclass(Singleton, object)):
+class Facade(metaclass=Singleton):
     def __init__(self):
 
         self.__plugins = dict(

src/wfuzz/factories/fuzzresfactory.py

@@ -27,6 +27,7 @@ class FuzzResultDictioBuilder:
     def __call__(self, options, dictio_item):
         res = copy.deepcopy(options["compiled_seed"])
         res.item_type = FuzzType.RESULT
+        res.discarded = False
         res.payload_man.update_from_dictio(dictio_item)
         res.update_from_options(options)
 
@@ -69,6 +70,7 @@ class FuzzResultAllVarBuilder:
     def __call__(self, options, var_name, payload):
         fuzzres = copy.deepcopy(options["compiled_seed"])
         fuzzres.item_type = FuzzType.RESULT
+        fuzzres.discarded = False
         fuzzres.payload_man = payman_factory.create("empty_payloadman", payload)
         fuzzres.payload_man.update_from_dictio([payload])
         fuzzres.history.wf_allvars_set = {var_name: payload.content}
@@ -97,6 +99,7 @@ def __call__(self, seed):
             new_seed.rlevel_desc += " - "
         new_seed.rlevel_desc += seed.payload_man.description()
         new_seed.item_type = FuzzType.SEED
+        new_seed.discarded = False
         new_seed.payload_man = payman_factory.create(
             "payloadman_from_request", new_seed.history
         )
@@ -113,6 +116,7 @@ def __call__(self, seed, url):
             fr.rlevel_desc += " - "
         fr.rlevel_desc += seed.payload_man.description()
         fr.item_type = FuzzType.BACKFEED
+        fr.discarded = False
         fr.is_baseline = False
 
         fr.payload_man = payman_factory.create(

src/wfuzz/factories/plugin_factory.py

@@ -12,6 +12,7 @@ def __init__(self):
                 "plugin_from_recursion": PluginRecursiveBuilder(),
                 "plugin_from_error": PluginErrorBuilder(),
                 "plugin_from_finding": PluginFindingBuilder(),
+                "plugin_from_summary": PluginFindingSummaryBuilder(),
             },
         )
 
@@ -38,12 +39,29 @@ def __call__(self, name, exception):
 
 
 class PluginFindingBuilder:
-    def __call__(self, name, message):
+    def __call__(self, name, itype, message, data, severity):
         plugin = FuzzPlugin()
         plugin.source = name
         plugin.issue = message
+        plugin.itype = itype
+        plugin.data = data
         plugin._exception = None
         plugin._seed = None
+        plugin.severity = severity
+
+        return plugin
+
+
+class PluginFindingSummaryBuilder:
+    def __call__(self, message):
+        plugin = FuzzPlugin()
+        plugin.source = FuzzPlugin.OUTPUT_SOURCE
+        plugin.itype = FuzzPlugin.SUMMARY_ITYPE
+        plugin.severity = FuzzPlugin.NONE
+        plugin._exception = None
+        plugin.data = None
+        plugin._seed = None
+        plugin.issue = message
 
         return plugin