Skip to content

lotus_domino_hashes scanner speedup and fixes #1042

Closed
wants to merge 2 commits into from

3 participants

@bpoole
bpoole commented Nov 8, 2012
  • Record the FullName of the account and not the ShortName. The
    ShortName is not always allowed to be used for logging in to servers
    and is disabled by default post-Domino 6
    (http://www-01.ibm.com/support/docview.wss?uid=swg21218067).

    NOTE: One issue that has not been cleanly resolved here is that the
    FullName parameter is actually a list of names. Currently I do not
    process the FullName after retrieving it and store the list of
    usernames as the username. Thus an entry for the user 'Joe
    Schmoe/US/COMPANY; Joe Schmoe' is created. This isn't particularly
    clean or correct IMO but I'm not sure I like either of the other two
    options I can think of either:

    1) Split on the ; and store the first entry. Downside: loses an
    alternate account name that may be useful information.
    2) Split on the ; and store each entry as a separate credential.
    Downside: Many duplicates in the credential database. As the names
    are not necessarily easy to uniquify this could make reporting on
    the credentials more painful.

  • Support retrieving entries via Readviewentries in larger batches than by default. Also actually process the Readviewentries response as a list of account IDs rather than just grabbing the first account ID each time. Together these two changes provide a significant speedup on large sites as it halves the total number of HTTP requests needed. I set the batch size at what I believe is a reasonable number (10000). Documentation I have seen says the largest possible Count value is 65535 but I left it a bit lower to not put too much load in a single request. 10k accounts is probably about a 5MB XML response.
  • Support restarting hash retrieval from a particular index in Readviewentries. This is very useful if you have say pulled 6000 of 12000 password hashes when the server times out a connection.
  • Make exception handling consistent throughout and handle all
    connection errors. If we fail partway through a site report the index
    we were requesting so that the user can set the index to resume from.

    NOTE: I'm new to ruby and I went with blanket rescues here
    because I couldn't find an authoritative list of exceptions that
    might result in the HTTP connection failing. As I went with
    catch-alls I made the begin/rescue blocks as small as possible
    around the send_request_raw calls but I'm open to better ways
    to do it. It is critical that failed HTTP connections be handled
    so that we know what account index we were on though. In
    dumping large databases the previous version would silently
    stop on a network error fooling the user into thinking the dump
    had completed successfully. We need to be able to reliably
    report there was a failure and where the failure occurred so
    resuming dumps is possible.

  • Support basic and form authentication using the same settings. Use the HTTP response code to figure out which route to go. Previously if you set NOTES_USER/NOTES_PASS these creds would not be used if the Domino server was configured to require basic auth. I thought this confusing so changed it around.
  • Fix prints to report the correct protocol when using SSL.
  • Check report_service's return value is valid before using.
@bpoole bpoole Speedup code and fix lots of random issues.
- Record the FullName of the account and not the ShortName. The
  ShortName is not always allowed to be used for logging in to servers
  and is disabled by default post-Domino 6
  (http://www-01.ibm.com/support/docview.wss?uid=swg21218067).

  NOTE: One issue that has not been cleanly resolved here is that the
  FullName parameter is actually a list of names. Currently I do not
  process the FullName after retrieving it and store the list of
  usernames as the username. Thus an entry for the user 'Joe
  Schmoe/US/COMPANY; Joe Schmoe' is created. This isn't particularly
  clean or correct IMO but I'm not sure I like either of the other two
  options I can think of either:

  1) Split on the ; and store the first entry. Downside: loses an
     alternate account name that may be useful informational.
  2) Split on the ; and store each entry as a separate credential.
     Downside: Many duplicates in the credential database. As the names
     are not necessarily easy to uniquify this could make reporting on
     the credentials more painful.
- Support retrieving entries via Readviewentries in larger batches than
  by default. Also actually process the Readviewentries response as a
  list of account IDs rather than just grabbing the first account ID
  each time. Together these two changes provide a significant speedup on
  large sites as it halves the total number of HTTP requests needed.
- Support restarting hash retrieval from a particular index in
  Readviewentries. This is very useful if you have say pulled 6000 of
  8000 password hashes when the server times out a connection.
- Make exception handling consistent throughout and handle all
  connection errors. If we fail partway through a site report the index
  we were requesting so that the user can set the index to resume from.
- Support basic and form authentication using the same settings. Use the
  HTTP response code to figure out which route to go.
- Fix prints to report the correct protocol when using SSL.
- Check report_service's return before using.
0ef8a80
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
@@ -41,110 +43,167 @@ def run_host(ip)
pass = datastore['NOTES_PASS'].to_s
$uri = datastore['URI'].to_s
- if (user.length == 0 and pass.length == 0)
- print_status("http://#{vhost}:#{rport} - Lotus Domino - Trying dump password hashes without credentials")
+ proto = ssl ? "https" : "http"
+ $site = "#{proto}://#{vhost}:#{rport}"
@brandonprry
brandonprry added a note Nov 9, 2012

Global variables are bad

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
@@ -41,110 +43,167 @@ def run_host(ip)
pass = datastore['NOTES_PASS'].to_s
$uri = datastore['URI'].to_s
@brandonprry
brandonprry added a note Nov 9, 2012

I know you didn't change this part but wtf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
- cookie = "DomAuthSessId=#{$1}"
- elsif res.headers['Set-Cookie'] and res.headers['Set-Cookie'].match(/LtpaToken=(.*);(.*)/i)
- cookie = "LtpaToken=#{$1}"
- else
- print_error("http://#{vhost}:#{rport} - Lotus Domino - Unrecognized 302 response")
- return :abort
- end
- print_good("http://#{vhost}:#{rport} - Lotus Domino - SUCCESSFUL authentication for '#{user}'")
- print_status("http://#{vhost}:#{rport} - Lotus Domino - Getting password hashes")
- get_views(cookie,$uri)
-
- elsif (res and res.body.to_s =~ /names.nsf\?Login/)
- print_error("http://#{vhost}:#{rport} - Lotus Domino - Authentication error: failed to login as '#{user}'")
- return :abort
-
+ if (res and res.code == 302 )
@brandonprry
brandonprry added a note Nov 9, 2012

if res and res.code == 302

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
end
- else
- print_status("http://#{vhost}:#{rport} - Lotus Domino - Trying dump password hashes with given credentials")
- do_login(user, pass)
+ if (res and res.body.to_s =~ /\<viewentries/)
@brandonprry
brandonprry added a note Nov 9, 2012

if res and res.body =~ /<viewentries/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
- end
+ elsif (res and res.body.to_s =~ /names.nsf\?Login/)
@brandonprry
brandonprry added a note Nov 9, 2012

elsif res and res.body =~ /names.nsf\?Login/

Don't think body needs .to_s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
+ elsif (res and res.code == 401)
@brandonprry
brandonprry added a note Nov 9, 2012

elsif res and res.code == 401

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
+ elsif (res and res.code == 401)
+ if (user.length == 0 and pass.length == 0)
@brandonprry
brandonprry added a note Nov 9, 2012

if user.length == 0 and pass.length == 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
- rescue ::Rex::ConnectionRefused, ::Rex::HostUnreachable, ::Rex::ConnectionTimeout
- rescue ::Timeout::Error, ::Errno::EPIPE
+ elsif (res and res.body.to_s =~ /names.nsf\?Login/)
@brandonprry
brandonprry added a note Nov 9, 2012

elsif res and res.body =~ /names.nsf\?Login/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
[
OptString.new('NOTES_USER', [false, 'The username to authenticate as', '']),
OptString.new('NOTES_PASS', [false, 'The password for the specified username' ]),
OptString.new('URI', [false, 'Define the path to the names.nsf file', '/names.nsf']),
+ OptString.new('START_INDEX', [false, 'Index to start at', '1']),
@brandonprry
brandonprry added a note Nov 9, 2012

OptInt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
[
OptString.new('NOTES_USER', [false, 'The username to authenticate as', '']),
OptString.new('NOTES_PASS', [false, 'The password for the specified username' ]),
OptString.new('URI', [false, 'Define the path to the names.nsf file', '/names.nsf']),
+ OptString.new('START_INDEX', [false, 'Index to start at', '1']),
+ OptString.new('STEP_SIZE', [false, 'Number of records to enumerate at once', '10000']),
@brandonprry
brandonprry added a note Nov 9, 2012

OptInt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
end
def get_views(cookie,uri)
+ start = datastore['START_INDEX'].to_i
@brandonprry
brandonprry added a note Nov 9, 2012

should be an optint, no casting should be needed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
end
def get_views(cookie,uri)
+ start = datastore['START_INDEX'].to_i
+ step = datastore['STEP_SIZE'].to_i
@brandonprry
brandonprry added a note Nov 9, 2012

should be an optint, no casting should be needed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
begin
res = send_request_raw({
'method' => 'GET',
'uri' => "#{uri}\/$defaultview?Readviewentries",
'cookie' => cookie,
}, 25)
- if (res and res.body)
- max = res.body.scan(/siblings=\"(.*)\"/)[0].join
+ rescue
+ print_error("#{$site} - Lotus Domino - Request to enumerate entries failed")
+ return
+ end
+
+ if (not res or not res.body)
@brandonprry
brandonprry added a note Nov 9, 2012

if not res or not res.body

I actually prefer if !res or !res.body, but whatev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
+ (start .. max.to_i).step(step) {|i|
+ begin
+ res = send_request_raw({
+ 'method' => 'GET',
+ 'uri' => "#{uri}\/$defaultview?Readviewentries&Start=#{i}&Count=#{step}",
+ 'cookie' => cookie,
@brandonprry
brandonprry added a note Nov 9, 2012

trailing comma

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
end
- rescue ::Rex::ConnectionRefused, ::Rex::HostUnreachable, ::Rex::ConnectionTimeout
- rescue ::Timeout::Error, ::Errno::EPIPE
- end
+ if (not res or not res.body)
@brandonprry
brandonprry added a note Nov 9, 2012

if not res or not res.body

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
end
- rescue ::Rex::ConnectionRefused, ::Rex::HostUnreachable, ::Rex::ConnectionTimeout
- rescue ::Timeout::Error, ::Errno::EPIPE
- end
+ if (not res or not res.body)
+ print_error("#{$site} - Lotus Domino - Request for batch of users starting at #{i} failed, stopping dump. Use START_INDEX to resume the dump")
+ return
+ end
+
+ current = i
+ res.body.scan(/unid="([^\s]+)"/){ |viewId|
@brandonprry
brandonprry added a note Nov 9, 2012

res.body.scan do |view_id|

end

no {}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
- rescue ::Rex::ConnectionRefused, ::Rex::HostUnreachable, ::Rex::ConnectionTimeout
- rescue ::Timeout::Error, ::Errno::EPIPE
- end
+ if (not res or not res.body)
+ print_error("#{$site} - Lotus Domino - Request for batch of users starting at #{i} failed, stopping dump. Use START_INDEX to resume the dump")
+ return
+ end
+
+ current = i
+ res.body.scan(/unid="([^\s]+)"/){ |viewId|
+ current = current + 1
+ if dump_hashes(viewId[0],cookie,uri) == -1
+ print_error("#{$site} - Lotus Domino - Request for entry #{current} failed, stopping dump. Use START_INDEX to resume the dump")
+ return
+ end
+ }
@brandonprry
brandonprry added a note Nov 9, 2012

read above

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
- if (res and res.body)
- short_name = res.body.scan(/<INPUT NAME=\"ShortName\" TYPE=(?:.*) VALUE=\"([^\s]+)"/i).join
- user_mail = res.body.scan(/<INPUT NAME=\"InternetAddress\" TYPE=(?:.*) VALUE=\"([^\s]+)"/i).join
- pass_hash = res.body.scan(/<INPUT NAME=\"dspHTTPPassword\" TYPE=(?:.*) VALUE=\"([^\s]+)"/i).join
+ if (not res or not res.body)
@brandonprry
brandonprry added a note Nov 9, 2012

if res or res.body

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
- if user_mail.to_s.strip.empty?
- user_mail = 'NULL'
- end
+ if full_name.to_s.strip.empty?
@brandonprry
brandonprry added a note Nov 9, 2012

already a string no? to_s is redundant

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
- if pass_hash.to_s.strip.empty?
- pass_hash = 'NULL'
- end
+ if user_mail.to_s.strip.empty?
@brandonprry
brandonprry added a note Nov 9, 2012

same as above

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
- )
- report_auth_info(
- :host => rhost,
- :port => rport,
- :sname => (ssl ? "https" : "http"),
- :user => short_name,
- :pass => pass_hash,
- :ptype => "domino_hash",
- :source_id => domino_svc.id,
- :source_type => "service",
- :proof => "WEBAPP=\"Lotus Domino\", USER_MAIL=#{user_mail}, HASH=#{pass_hash}, VHOST=#{vhost}",
- :active => true
- )
- end
- end
+ if pass_hash.to_s.strip.empty?
@brandonprry
brandonprry added a note Nov 9, 2012

same as above

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
- report_auth_info(
- :host => rhost,
- :port => rport,
- :sname => (ssl ? "https" : "http"),
- :user => short_name,
- :pass => pass_hash,
- :ptype => "domino_hash",
- :source_id => domino_svc.id,
- :source_type => "service",
- :proof => "WEBAPP=\"Lotus Domino\", USER_MAIL=#{user_mail}, HASH=#{pass_hash}, VHOST=#{vhost}",
- :active => true
- )
- end
- end
+ if pass_hash.to_s.strip.empty?
+ pass_hash = 'NULL'
@brandonprry
brandonprry added a note Nov 9, 2012

uh, pass_hash = nil?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brandonprry brandonprry commented on an outdated diff Nov 9, 2012
modules/auxiliary/scanner/lotus/lotus_domino_hashes.rb
- rescue ::Rex::ConnectionRefused, ::Rex::HostUnreachable, ::Rex::ConnectionTimeout
- rescue ::Timeout::Error, ::Errno::EPIPE
+ print_good("#{$site} - Lotus Domino - Account Found: #{full_name}, #{user_mail}, #{pass_hash}")
+
+ if pass_hash != 'NULL'
@brandonprry
brandonprry added a note Nov 9, 2012

see above

@brandonprry
brandonprry added a note Nov 9, 2012

if pass_hash is nil, you can simply evaluate pass_hash

if pass_hash

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@bpoole bpoole Address brandonprry's initial feedback.
Also added a few more small fixes for things I noticed.
e91831d
@bpoole
bpoole commented Nov 9, 2012

Thanks for the initial pass. I believe I addressed it all, along with a few other little things I noticed.

@todb-r7
todb-r7 commented Mar 18, 2013

Hi @bpoole and @brandonprry -- the pull request is now conflicting, so that will need to be resolved. If one of you could address that and then reopen, that would be swell.

It's a small conflict, but I'm not sure which parts are intended to be preserved for this module, honestly.

You should be able to reproduce the conflict yourself by attempting to merge this branch against the master rapid7 branch.

For tips on resolving merge conflicts, see the Git SCM book, here:

http://git-scm.com/book/en/Git-Branching-Basic-Branching-and-Merging

You should be able to reproduce the conflict yourself by trying to merge this branch with your local copy of the rapid7/metasploit-framework/master branch.

@todb-r7 todb-r7 closed this Mar 18, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.