Updated support for Web modules and analysis techniques

[tasos-r7]

This request contains core classes for the new Web-related functionality (along with a few updates to existing code) such as:

Anemone

Has been hacked to support modular path extractors and dirbusting.

Path extractors (/lib/anemone/extractors)

Single-responsibility components which when passed an Anemone::Page return an array of paths.

Dirbuster (/lib/anemone/extractors/dirbuster.rb /lib/anemone/extractors/dirbuster/directories)

Implemented as a path extractor, feeds the crawler a bunch of directory names.

Auxiliary::Web (/lib/msf/core/auxiliary/web.rb)

Mixin which provides a bunch of helpers to the web modules for:

• logging vulns
• fingerprinting
• detection of server-side resources
• HTTP requests (Web::HTTP)

Auxiliary::Web::HTTP (/msf3/lib/msf/core/auxiliary/web/http.rb)

Provides a more sensible interface for the task at hand, much more high-level than Rex::Proto::Http::Client, which it wraps into a pretty bow.
Takes into account query params and merges them with explicit get params, handles redirects, can detect custom 404 pages, etc.
Generally makes life much easier...

Auxiliary::Web::Target (/msf3/lib/msf/core/auxiliary/web/target.rb)

Holds target info like Mdm::Service, Mdm::WebSite, along with a bunch of helper attributes (like vhost, host, port, etc.).

Most interesting attributes are:

• #forms -- Array of form elements (Auxiliary::Web::Form)
• #paths -- Array of path elements (Auxiliary::Web::Path)
• #auditable -- Merge of the above two.

Auxiliary::Web::Form (/msf3/lib/msf/core/auxiliary/web/form.rb)

Represents a form element and holds its method, action, inputs and the original Mdm::WebForm model.
Also provides a bunch of domain relevant helpers like query parsing, encoding, permutation generation and more.

Inherits from Auxiliary::Web::Fuzzable.

Auxiliary::Web::Path (/msf3/lib/msf/core/auxiliary/web/path.rb)

Same deal as Auxiliary::Web::Form apart from a few differences in their guts since even though their
API needs to be identical their internal representation differs wildly.

Inherits from Auxiliary::Web::Fuzzable.

Auxiliary::Web::Fuzzable (/msf3/lib/msf/core/auxiliary/web/fuzzable.rb)

Doesn't do much, provides a way to #submit the elements and #fuzz them using their permutations.
Also provides comparison and #dup methods.

The cool stuff is that it dynamically loads and includes analysis modules (Ruby modules, not framework components) under its namespace.

Auxiliary::Web::Analysis (/msf3/lib/msf/core/auxiliary/web/analysis/)

Namespace under which all analysis techniques reside.

Auxiliary::Web::Analysis::Taint (/pro/msf3/lib/msf/core/auxiliary/web/analysis/taint.rb)

Provides Taint analysis support to Fuzzzable elements, basically submits their permutations and uses the fuzzer's (web module's) #find_proof to determine whether or not a response matches a vulnerability signature.

Auxiliary::Web::Analysis::Differential (/msf3/lib/msf/core/auxiliary/web/analysis/differential.rb)

Provides Differential analysis using fault/boolean injection pairs to determine whether the back-end logic can be manipulated and uses text refinement (Rex::Text.refine) to remove irrelevant/out-of-context noise data in order to make response comparisons possible.

Auxiliary::Web::Analysis::Timing (/pro/msf3/lib/msf/core/auxiliary/web/analysis/timing.rb)

Provides timing-attack/response-time analysis. Injects delay increments and ensures responsiveness between injections to make sure that the response times are manipulatable and not just dumb luck or a dead server.

Changes to core libs

Added:

• Rex::Text.to_words -- Splits a String to an Array of words.
• Rex::Text.refine -- Removes noise from 2 Strings and return a refined String version.

[jlee-r7]

Should one of these be o_words - s_words ? Looks like you're just getting two copies of each thing in s_words that isn't in o_words

[tasos-r7]

Yes, you're right, will push the fix shortly.