Skip to content

Result Sharing

Jump to bottom
Dustin Born edited this page Apr 10, 2020 · 8 revisions

Introduction

One feature of the AVAIN framework is that modules can share results which are automatically aggregated by the AVAIN core. This decreases the overall scanning / analysis work that modules have to do, as prior results can just be reused. It also allows for the creation of more sophisticated modules. As of now, result sharing only works for so called (AVAIN) shared results, which each have a predefined format and their own result processor class that is responsible for the aggregation and other tasks. New result types require new result processors. The format of all shared results is JSON or a Python dict, respectively. Apart from shared results, the idea exists to add simple result sharing without any aggregation etc. in the future.

This wiki page describes the format of the three currently available shared result types: scan results, webserver map results and vuln score results. How modules can actually share these results is described on the Creating a New Module wiki page.

Scan Result

Scan results store general information about the services and OS of assessed systems. As such, an example scan result looks like the following (as JSON):

{
   "192.168.178.36": {
      "os": [
         {
            "name": "Apple OS X 10.10.X",
            "cpes": [
               "cpe:/o:apple:mac_os_x:10.10"
            ],
            "accuracy": "97"
         }
      ],
      "ip": {
         "addr": "192.168.178.36",
         "type": "ipv4"
      },
      "tcp": {
         "80": [
            {
               "portid": "80",
               "protocol": "tcp",
               "cpes": [
                  "cpe:/a:apache:http_server:2.4.33"
               ],
               "name": "Apache httpd 2.4.33",
               "service": "http"
            }
         ]
      }
   },
   "trust": 4
}

For brevity, only one host is shown above. All hosts are indexed by their (IPv4) network address. Their content is another object that can contain several more fields. AVAIN requires the os, tcp and udp fields to abide by a certain format:

  • The os node is a list that contains one or more OS information nodes. If an OS information node has an entry called cpes, it has to be a list of CPEs for the detected / presumed OS. name has to be the OSs name as string.
  • The tcp and udp entries are each dissected into objects for every detected port, hence their indexing by port numbers.
  • Every port node contains one or more port information nodes stored within a list, similar to the "os" node. For port information nodes the same rules apply for the name and cpes entry as for OS information nodes. Furthermore, if the service field exists, it has to be a string describing the service, e.g, "ssh" or "http".

Additionally, a scan result can contain one or more "trust" fields that can reside on any one of the different hierarchy levels. This fields symbolizes how much a module trusts its results, i.e. how accurate and valuable it perceives its results to be. It is basically a quality of data indicator. The value of the "trust" field can be any integer or floating point number. If a scan result does not contain a "trust" field, it is assigned the configured default value by the scan result processor. Apart from the aforementioned restrictions, scan results can contain other custom information, which is not specially aggregated like the constrained information is.

Aggregation

The following gives a rough idea about how scan results are aggregated:

  • Non overlapping port nodes and OS nodes of several scan results are simply united because they contain no overlapping information.
  • Port nodes from different scan results that overlap, i.e. that have the same portid, are first grouped together by their CPE and service name. Then the groups are aggregated, by choosing the most specific information available, i.e. CPE cpe:/o:apple:mac_os_x:10.12.4 would be chosen over cpe:/o:apple:mac_os_x:10.12. A similiar concept is used for service name aggregation if no CPEs are available. The trust of the aggregated port node is either the maximum available value if per configuration scan_trust_aggr_scheme = TRUST_MAX or a newly computed one based on the original ones if scan_trust_aggr_scheme = TRUST_AGGR. Next, we have only dissimilar port nodes for the current portid remaining. These are aggregated depending on the scan_result_aggr_scheme config parameter to get the final result:
    • SINGLE: the single result with the highest trust rating is chosen
    • MULTIPLE: the results are kept without further processing
    • FILTER: similar products are filtered out, i.e. out of macOS 10.12 and macOS 10.13, only the one with the highest trust rating is chosen
  • OS nodes are aggregated in the same way as port nodes
  • Other attributes apart from CPE, name and service info that show more miscellaneous information are simply united and not specially aggregated

Result files

The result files can be found in the subdirectory scan_result_aggregation/ relative to the base of AVAIN's output directory. The result files consist of the final scan result and the intermediate results for the two aggregation steps.

Webserver Map Result

Webserver map results store information about the structure of web servers. Structure refers to available paths and their return codes, GET / POST parameters and cookies. An example webserver map result looks like the following:

{
   "192.168.42.101": {
      "80": {
         "example.lan": {
            "200": {
               "/login.php": {
                  "GET": [
                     "lang"
                  ],
                  "POST": [
                     "username",
                     "password",
                     "login"
                  ],
                  "cookies": [
                     "PHPSESSID"
                  ],
                  "instances": [
                     {
                        "GET": {
                           "lang": "en"
                        },
                        "POST": {
                           "username": "",
                           "passowrd": "",
                           "login": "Submit"
                        },
                        "cookies": {
                           "PHPSESSID": "rs9651vpai7amd3hme8118d22o"
                        }
                     }
                  ]
               },
               "/index.php": {}
            },
            "301": {
               "/admin/": {
                  "misc_info": "redirect to /login.php"
               }
            },
            "403": {
               "/secret/": {}
            }
         }
      }
   }
}

As you can see, at the top level the webserver map result is indexed by IP addresses. A level lower, the data is indexed by port numbers. Therefore, this result shows data for a webserver at the network endpoint 192.168.56.101:80. The data on the third level is indexed by domain / host names to be able to deal with virtual hosts. Let's call such a node on the third level a host node. A host node is indexed by HTTP response codes. Every response code node is indexed by paths on the web server. Paths must start with a leading "/" and be absolute. Similarly to before, let's call such a node a path node. A path node can contain 5 different keys, which are first GET, POST and cookies to store a list of GET / POST parameters and cookie names that are used on the current path. Second, a path node can contain an instances key under which is stored a list of instances of the path including their parameter keys and values. This helps AVAIN to reconstruct actually viable combinations of the path's parameters. In the above example, we see within the instances node of the path /login.php an instance of a login attempt with an empty username and password and a session id. The fifth accepted key for a path node is misc_info which stores other interesting information like the target of a redirect (see above the path /admin/).

Aggregation

Two webserver maps are mostly aggregated by simply uniting them as there is not really any data that could contradict each other. Duplicates at any level as well as empty instances are removed.

Result files

The result files of this type can be found in the subdirectory webserver_map_aggregation/ starting from the base of AVAIN's output directory. In this subdirectory are two files that show respectively:

  1. The aggregated webserver map as JSON file
  2. A text file containing a simple overview of discovered URLs, ordered by their status code.

Vuln Score Result

The purpose of vuln score results is to store quantified vulnerability information for every assessed host. The idea is to end up with a simple score that can easily be interpreted. The format for vulnerability score results is fairly simple. An example result is:

{
    "192.168.0.24": 6.8,
    "192.168.0.42": 7.9,
    "192.168.0.64": 9.8,
    "192.168.0.78": 8.2,
    "192.168.0.101": 7.4
}

It is as simple as listing every host with its vulnerability / security score. The score has to be a float between 0 and 10. Apart from that there are no restrictions. The semantic meaning behind the score should be somewhat aligned to the CVSSv3.1 framework by considering the attack vector of the discovered vulnerability of a host.

Aggregation

The basis is that we have different vulnerability scores per host coming from different modules. In the aggregation, for one host the highest available vulnerability score returned by any module is chosen as its final vulnerability score. Thereafter, what remains is essentially one list of (host, vuln score) pairs. As the final output of AVAIN is a score indicating the (in)security or vulnerability level of the entire assessed network, these host scores need to be aggregated. This is done via a weighted arithmetic mean that tries to have high vulnerability scores be more impactful to the security score of the network, while not simply choosing the highest one. This is done to reflect the fact that while an insecure host in the network is a big threat, it does not necessarily make the whole network as insecure as that one host.

Result files

The result files can be found in the subdirectory vuln_score_aggregation/ relative to the base of AVAIN's output directory. There are three files in this subdirectory that show respectively:

  1. An overview of what module assigned which score to what host
  2. Every host with its aggregated vulnerability score
  3. The aggregated vulnerability score of the assessed network, i.e. the final result
Clone this wiki locally