Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update `retina` imports to better handle parsing #11954

Merged
merged 5 commits into from Jun 7, 2019

Conversation

Projects
None yet
4 participants
@jmartin-r7
Copy link
Contributor

commented Jun 6, 2019

When parsing Retina XML exports REXML lib in ruby is injecting with space not found inside tags.

  • Adjust for whitespace issue in a backwards compatible way.
  • Add context tag parsing to support adding service details when supplied in the XML.
    • Looks about right: <context>TCP:443 ([redacted]), SHA256[=][redacted], Serial[=][redacted]</context>
  • Add detection for Retina files when Xml Declaration is present a beginning of the file.
    • Indeed, the file starts with <?xml version="1.0" encoding="utf-8"?> now

Verification

List the steps needed to make sure this thing works

  • Start msfconsole
  • db_import <PATH_TO_RETINA_XML>
  • Verify a sample file imports hosts.

@jmartin-r7 jmartin-r7 force-pushed the jmartin-r7:adjust-retina-import branch from e843a37 to f646a97 Jun 7, 2019

Show resolved Hide resolved lib/rex/parser/retina_xml.rb Outdated

@wvu-r7 wvu-r7 added bug library labels Jun 7, 2019

@wvu-r7 wvu-r7 self-assigned this Jun 7, 2019

@wvu-r7

This comment has been minimized.

Copy link
Contributor

commented Jun 7, 2019

I've changed your PR description to have checkboxes so I can verify functionality as I review.

wvu-r7 added some commits Jun 7, 2019

Remove warning
<context> provides service info now:

<context>TCP:443 ([redacted]), SHA256[=][redacted], Serial[=][redacted]</context>
Add nil check
Just in case.
@wvu-r7

This comment has been minimized.

Copy link
Contributor

commented Jun 7, 2019

New/fixed behavior:

msf5 > db_import ~/Downloads/[redacted]
[*] Importing 'Retina XML' data
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Successfully imported /Users/wvu/Downloads/[redacted]
msf5 >
@wvu-r7

This comment has been minimized.

Copy link
Contributor

commented Jun 7, 2019

Independent fixing/testing to reproduce the issue:

msf5 > db_import ~/Downloads/[redacted]
[*] Importing 'IP360 XML v3' data
[-] Failed to import /Users/wvu/Downloads/[redacted]: The nCircle IP360 ASPL file is not present.
    Download ASPL from nCircle VNE | Administer | Support | Resources, unzip it, and import it first
msf5 > git stash pop
[*] exec: git stash pop

On branch master
Your branch is behind 'upstream/master' by 6 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   lib/msf/core/db_manager/import.rb
	modified:   lib/rex/parser/retina_xml.rb

no changes added to commit (use "git add" and/or "git commit -a")
Dropped refs/stash@{0} (d5c6149be15975f4ff64a59a6f9c754eaf9aa5b6)
msf5 > git diff
[*] exec: git diff

diff --git a/lib/msf/core/db_manager/import.rb b/lib/msf/core/db_manager/import.rb
index 9ae2724fd5..252febd480 100644
--- a/lib/msf/core/db_manager/import.rb
+++ b/lib/msf/core/db_manager/import.rb
@@ -407,6 +407,9 @@ module Msf::DBManager::Import
         when /ReportInfo/
           @import_filedata[:type] = "Foundstone"
           return :foundstone_xml
+        when /scanJob/
+          @import_filedata[:type] = "Retina XML"
+          return :retina_xml
         when /ScanGroup/
           @import_filedata[:type] = "Acunetix"
           return :acunetix_xml
diff --git a/lib/rex/parser/retina_xml.rb b/lib/rex/parser/retina_xml.rb
index 16517be6d3..c1abe4c017 100644
--- a/lib/rex/parser/retina_xml.rb
+++ b/lib/rex/parser/retina_xml.rb
@@ -27,6 +27,8 @@ class RetinaXMLStreamParser
   end

   def text(str)
+    return if str.blank?
+
     case @state
     when :in_ip
       @host["address"] = str
@@ -35,7 +37,7 @@ class RetinaXMLStreamParser
     when :in_netbiosname
       @host["netbios"] = str
     when :in_mac
-      @host["mac"] = str
+      @host["mac"] = str.split.first
     when :in_os
       @host["os"] = str
     when :in_rthid
msf5 > reload_lib -a
[*] Reloading /rapid7/metasploit-framework/lib/msf/core/db_manager/import.rb
[*] Reloading /rapid7/metasploit-framework/lib/rex/parser/retina_xml.rb
msf5 > db_import ~/Downloads/[redacted]
[*] Importing 'Retina XML' data
[-]
[-] Warning: The Retina XML format does not associate vulnerabilities with the
[-] specific service on which they were found.
[-] This makes it impossible to correlate exploits to discovered vulnerabilities
[-] in a reliable fashion.
[-]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Importing host [redacted]
[*] Successfully imported /Users/wvu/Downloads/[redacted]
[-] Please note that there was one warning
msf5 > vulns -p 445

Vulnerabilities
===============

Timestamp  Host  Name  References
---------  ----  ----  ----------

msf5 >

Good job on the service info. I've removed the warning thusly.

@wvu-r7 wvu-r7 merged commit dd101a0 into rapid7:master Jun 7, 2019

3 checks passed

Metasploit Automation - Sanity Test Execution Successfully completed all tests.
Details
Metasploit Automation - Test Execution Successfully completed all tests.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

wvu-r7 added a commit that referenced this pull request Jun 7, 2019

msjenkins-r7 added a commit that referenced this pull request Jun 7, 2019

@wvu-r7

This comment has been minimized.

Copy link
Contributor

commented Jun 7, 2019

Release Notes

The Retina XML importer now uses the current XML format. It also now cross-references services to vulnerabilities.

@pbarry-r7

This comment has been minimized.

Copy link
Contributor

commented Jun 7, 2019

Thanks, @jmartin-r7 and @wvu-r7! 👏

@jmartin-r7 jmartin-r7 deleted the jmartin-r7:adjust-retina-import branch Jun 7, 2019

@tdoan-r7 tdoan-r7 added the rn-fix label Jun 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.