Freeze on RPC Bind #19

bao7uo · 2017-04-06T10:13:55Z

When executing check or display commands, after autodiscover, "Binding to RPC" causes the ruler process to go to 99% CPU usage and nothing happens after waiting for a long time. This is on the Linux 64 troopers release.

On the slightly older (but most recently available) Windows release the following error occurs:
[x] An error occurred setting up RPC.
[x] Couldn't setup RPC channel - illegal base64 data at input byte 5

staaldraad · 2017-04-06T11:43:56Z

Hi! Thanks for the report.

I think the issue is either authentication failure against the rpc end-point (which would cause the [x] Couldn't setup RPC channel - illegal base64 data at input byte 5 error message in older clients)
I just tested by giving incorrect creds for and I also got the hanging.

The fact that it gets to the RPC indicates that auth was successful against autodiscover though. So It's weird that there is this is failing. It could be that the RPC proxy is disabled.

I'll try push add a check to see that RPC has authenticated correctly or if there is an error coming back from the channel setup. I'm struggling with channel synchronisation. In the meantime, could you try with --verbose and try authenticate with the URL displayed using the browser?

staaldraad · 2017-04-06T12:38:57Z

If you could try the update I pushed into dev it may give some more insight into what is breaking :)

bao7uo · 2017-04-06T13:22:54Z

I can't build dev... but is my first time to download a specific branch from git (rather than just default), so maybe I am doing something wrong?

Command:
git clone https://github.com/sensepost/ruler.git -b dev && cd ruler && go build
Relevant output:

./ruler.go:397: too many arguments in call to mapi.QueryRows
./ruler.go:410: undefined: utils.Clear in utils.Clear.Printf
./ruler.go:413: undefined: mapi.STAT
./ruler.go:417: too many arguments in call to mapi.QueryRows
./ruler.go:424: undefined: utils.Clear in utils.Clear.Printf
./ruler.go:446: too many arguments in call to mapi.QueryRows
./ruler.go:457: undefined: mapi.STAT
./ruler.go:462: too many arguments in call to mapi.QueryRows

staaldraad · 2017-04-06T13:44:30Z

Seems like some of the address book stuff I'm working on didn't get synced to the dev branch.

I've pushed a pre-built binary to releases: https://github.com/sensepost/ruler/releases/tag/v2.0.18

Thanks for helping troubleshoot this 👍

bao7uo · 2017-04-06T13:57:56Z

Thanks here is the new message.
https://hastebin.com/apazigavom.cs
I don't have access to port 6001 on the target, but I get a 200 response for that whole RPC url in my browser. As I understand it, I don't need access to port 6001 as long as the web server hosting the rpcproxy.dll has access to it?

staaldraad · 2017-04-06T14:04:01Z

Interesting, it all looks right. Can you try leaving off "--domain" ?
If you don't mind pinging me on etienne 'at' sensepost.com - we could troubleshoot that way, it might be faster / easier

bao7uo · 2017-04-06T14:07:41Z

Further update:

using --domain DOMAIN --username USER gives the error shown in the link above after a few seconds.
using --username USER@DOMAIN gives this immediately:

[+] Binding to RPC
[*] Invalid HTTP response: [HTTP/1.1 401 Unauthorized]
ERROR: 2017/04/06 15:06:27 mapi: a transport layer error occurred. An error occurred setting up RPC. 
Invalid HTTP response: [HTTP/1.1 401 Unauthorized]

using --username DOMAIN\user causes autodiscovery to fail with 401 so it doesn't even get to Binding to RPC

bao7uo · 2017-04-06T14:08:21Z

sorry, just seen your message to ping you.. will do...

staaldraad · 2017-04-06T14:42:30Z

Error located! When BASIC auth was being used on the rpcproxy, I disabled RPC_C_AUTHN_LEVEL_PKT_PRIVACY and RPC_C_AUTHN_WINNT. Meaning the RPC messages were incorrect (which points to another possible bug). Turning encryption back on it should now work.

Fixes in dev branch and in prelease: https://github.com/sensepost/ruler/releases/tag/v2.0.18

funoverip · 2017-11-28T10:50:20Z

Hi,
I'm sorry to inform you that I have the same issue (99% cpu / freeze on RPC Bind).

root@debian:~/go/src/github.com/sensepost/ruler# ./ruler  --username myuser --email myuser@mydomain.tld --verbose   check
Password:
[+] Retrieving MAPI/HTTP info
[*] Autodiscover step 0 - URL: https://autodiscover.mydomain.tld/autodiscover/autodiscover.xml
[*] RPC URL set: https://mydomain.tld/rpc/rpcproxy.dll?b4d581b4-feb1-493e-8712-77f60c41d787@mydomain.tld:6001
[*] Setting up channels
[+] Binding to RPC
^C

I did build the last source code from the master branch (2.1.9). Is there any dev branch I can try ?
I would love to help you troubleshooting this !

Cheers!

staaldraad · 2017-11-28T12:07:38Z

Hi, thanks for reporting this.

These are notoriously difficult to debug. Could you try run with --debug, this should give some more information regarding the state of the RPC connection.

funoverip · 2017-11-28T13:04:07Z

Yeah, sorry for the news.

Using "--debug" doesn't provide any additional information unfortunately.

Tcpdump doesn't show any traffic during the freeze. From time to time, I see some 0 length TCP packets (could be some keep-alive ..).

I think having tried many combinations including --hash / --noencrypt / --basic (access denied) / --rpc / ...

PS: Next week I will be in UK for the Sensepost BlackHat training. If you're part of the trainers, it might be easier to troubleshot.

staaldraad · 2017-11-28T13:32:38Z

Ok, what I usually do is to use http://github.com/staaldraad/tcpprox to proxy the traffic and see what the issue might be. But seems like the connection is going through but Ruler isn't parsing the response packets correctly. One option might be to try build 2.1.7 or earlier. I changed the RPC parsing logic around then and maybe the newer logic breaks in this instance.

I'm no longer with SensePost, but hopefully Vlad or Evangelos can help out if we haven't resolved this before then.

P.S the training is awesome, hope you enjoy it!

funoverip · 2017-11-28T14:05:38Z

No luck with 2.1.6 nor 2.1.7.

Not sure to use tcpprox correctly. Could you confirm this syntax ?:
"./tcpprox -s -l 127.0.0.1 -p 8080 -r a.b.c.d:443"
and just adding "--proxy http://127.0.0.1:8080" to ruler.

tcpprox says:

[*] Listening...
[*] Accepted from: 127.0.0.1:39356
[*][0] Connected to server: a.b.c.d:443

but 'ruler' fails to perform the autodiscovery now. Funny to see that tcpprox itself is at 99% of CPU !

Thanks.

staaldraad · 2017-11-28T14:12:11Z

with tcpprox it acts as a transparent proxy, so what you'll need to do is
./tcpprox -s -l 127.0.0.1 -p 8080 -r 1.1.1.1:443
where 1.1.1.1 is the IP address of the target end-point. And then you add a /etc/hosts entry changing target.domain.com to 127.0.0.1.

So you have tried with --domain blah ? shouldn't be an issue but it is the only thing I can think of. Alternatively it could be the port on this https://mydomain.tld/rpc/rpcproxy.dll?b4d581b4-feb1-493e-8712-77f60c41d787@mydomain.tld:6001 since Ruler is hard-coded to use 6001. You could use the config file (https://github.com/sensepost/ruler/wiki/Getting-Started#the-config-file) to change that port to 6002, 6004, 6005. Which might work.

funoverip · 2017-11-29T15:05:18Z

Some more information..

Still no luck (or time) to retry tcpprox.

I have:

tested ports 6001 to 6005 : no changes.
with and without --domain : no differences
set some PrintLn() and can say it enters this func:
https://github.com/sensepost/ruler/blob/master/rpc-http/rpctransport.go#L294
but and this condition is always false:
https://github.com/sensepost/ruler/blob/master/rpc-http/rpctransport.go#L560

funoverip · 2017-12-01T13:16:40Z

Does it help to see this debug output ?

Printf in RPCRead():

//RPCRead function takes a call ID and searches for the response in
//our list of received responses. Blocks until it finds a response
func RPCRead(callID int) (RPCResponse, error) {
        c := make(chan RPCResponse, 1)

        go func() {
                stop := false
                utils.Info.Println("RPCRead() entering")
                for stop != true {
                        utils.Info.Println("RPCRead() Loop level1 - begin")
                        for k, v := range responses {

                                utils.Info.Println("RPCRead() Loop level2 - begin")
                                //if the PFCFlags is set to 1, this is a fragmented packet. wait to update it first
                                fmt.Printf("v.Header.CallID: %d, uint32(callID): %d, v.Header.PFCFlags: 0b%b\n", v.Header.CallID, uint32(callID), v.Header.PFCFlags )
                                if v.Header.CallID == uint32(callID) && v.Header.PFCFlags != 1 {
                                        utils.Info.Println("RPCRead() if() - begin")
                                        responses = append(responses[:k], responses[k+1:]...)
                                        stop = true
                                        c <- v
                                        break
                                }
                        }
                }
        }()
....

Output :

[*] Setting up channels
[+] Binding to RPC
[+] RPCRead() entering
[+] RPCRead() Loop level1 - begin
[+] RPCRead() Loop level2 - begin
v.Header.CallID: 1634620704, uint32(callID): 0, v.Header.PFCFlags: 0b1010000
[+] RPCRead() Loop level2 - begin
v.Header.CallID: 1684095520, uint32(callID): 0, v.Header.PFCFlags: 0b1010000
[+] RPCRead() Loop level2 - begin
v.Header.CallID: 1344293965, uint32(callID): 0, v.Header.PFCFlags: 0b1001111
[+] RPCRead() Loop level1 - begin
[+] RPCRead() Loop level2 - begin
v.Header.CallID: 1634620704, uint32(callID): 0, v.Header.PFCFlags: 0b1010000
[+] RPCRead() Loop level2 - begin
v.Header.CallID: 1684095520, uint32(callID): 0, v.Header.PFCFlags: 0b1010000
[+] RPCRead() Loop level2 - begin
v.Header.CallID: 1344293965, uint32(callID): 0, v.Header.PFCFlags: 0b1001111
[+] RPCRead() Loop level1 - begin
[+] RPCRead() Loop level2 - begin
... repeat ...

staaldraad · 2017-12-01T13:39:25Z

hey!

Strange that the callID isn't incrementing. Could you try with && v.Header.PFCFlags != 1 removed?

otherwise if you could check out ruler the version around this commit: 89b36c6

The subsequent commit is where I brought in the "fragmentation" detection. That could be the breaker.

Alternatively 2.1.1 or there about should also have the older RPC parsing logic and would hopefully work differently.

funoverip · 2017-12-01T14:32:20Z

Mmmm, no luck with versions 2.1.0, 2.1.1or 2.1.8, or by removing && v.Header.PFCFlags != 1

Regarding v.Header.CallID, could you confirm that you expect so big values ? Since it loops over these 3 values, does it means that they are bogus messages ?

staaldraad · 2017-12-01T14:38:46Z

Those are definitely bogus values, the expected is sequential starting at 1 😢

You could add some output to the "scan" method, and we could see what the packets look like:
https://github.com/sensepost/ruler/blob/master/rpc-http/rpctransport.go#L597

fmt.Printf("%x\n",data)

Should help give some insight, I know what to expect :)

Thanks for the thorough work on debugging this!

funoverip · 2017-12-01T14:42:42Z

With pleasure :)
Before posting this , could you confirm there is no sensitive data inside ? ;)

staaldraad · 2017-12-01T14:49:29Z

hmmm, there shouldn't be (you might spot NTLM auth header in the HTTP data, but you could sanitize that) - The RPC data would consist of the server side response to the NTLM hand-shake, but nothing more.

staaldraad · 2017-12-01T14:51:14Z

if you use fmt.Printf("%s\n", hex.Dump(data)) you will be able to see the clear-text and the hex data. maybe this helps prevent sensitive data making a way in

funoverip · 2017-12-01T15:07:00Z

I think you will hate me ...

The "Setting up channel" ends with a 401 (auth issue), I have 3 headers WWW-Authenticate in the response (basic/NTLM/negociate)

and the RPCBind ends with a 400 error claiming "Bad Request - Invalid Verb"

:-/

funoverip · 2017-12-01T15:20:24Z

Ok, it works !
Here is the problem: I've done most of my test using --hash parameter. And it fails all the times.

No if I type the password instead AND provide --domain (as you suggested), it works !

Sorry for the wasted time. Maybe a check on the HTTP response code would be good for stupid peoples like me :-/

staaldraad · 2017-12-01T15:23:51Z

hahaha! brilliant!

And this hasn't been a waste of time :) that freeze used to happen if I missed the HTTP response code. I thought I'd fixed the parser to handle it, seems like there is still an edge case that I've missed 👎 At least it's easier to fix than RPC issues!!

P.S never write your own HTTP parser...

Thanks again for a great report and all the effort! Enjoy the Black-ops training and say hi to Vlad and the guys for me

funoverip · 2017-12-01T15:32:10Z

Thanks for your time ;) I will double check my NTLM hash and will do more tests on that side. Cheers

…

On 1 December 2017 16:23:54 CET, Etienne Stalmans ***@***.***> wrote: hahaha! brilliant! And this hasn't been a waste of time :) that freeze used to happen if I missed the HTTP response code. I thought I'd fixed the parser to handle it, seems like there is still an edge case that I've missed 👎 At least it's easier to fix than RPC issues!! P.S never write your own HTTP parser... Thanks again for a great report and all the effort! Enjoy the Black-ops training and say hi to Vlad and the guys for me -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #19 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

funoverip · 2017-12-01T15:53:22Z

Just a last note, I feel stupid but the --domain parameter confused me a bit, since it doesn't have the same effect as in autodiscover mode. Maybe renaming --domain to --fqdn (or something else) in autodiscover mode would prevent this (?)

staaldraad · 2017-12-01T16:32:48Z

you are absolutely right, it is confusing 😕 I like the idea of --fqdn for autodiscover/brute-forcing.

staaldraad pushed a commit that referenced this issue Apr 6, 2017

fix for #19. go back to 'always encrypt' for RPC requests

dcaf64a

staaldraad added the bug label Apr 6, 2017

staaldraad self-assigned this Apr 6, 2017

staaldraad mentioned this issue Apr 6, 2017

Merge updates and fixes #20

Merged

staaldraad closed this as completed in 31a6680 Apr 6, 2017

zz124lkjkchvba mentioned this issue Apr 7, 2017

Same RPC issue as #19 despite latest build #23

Closed

staaldraad mentioned this issue May 11, 2017

Improve RPC/HTTP #27

Closed

LeonMessi mentioned this issue Aug 23, 2018

Freeze on RPC Bind like #19 #89

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Freeze on RPC Bind #19

Freeze on RPC Bind #19

bao7uo commented Apr 6, 2017 •

edited

staaldraad commented Apr 6, 2017

staaldraad commented Apr 6, 2017

bao7uo commented Apr 6, 2017

staaldraad commented Apr 6, 2017

bao7uo commented Apr 6, 2017

staaldraad commented Apr 6, 2017

bao7uo commented Apr 6, 2017

bao7uo commented Apr 6, 2017

staaldraad commented Apr 6, 2017

funoverip commented Nov 28, 2017

staaldraad commented Nov 28, 2017

funoverip commented Nov 28, 2017

staaldraad commented Nov 28, 2017

funoverip commented Nov 28, 2017 •

edited

staaldraad commented Nov 28, 2017

funoverip commented Nov 29, 2017

funoverip commented Dec 1, 2017

staaldraad commented Dec 1, 2017

funoverip commented Dec 1, 2017

staaldraad commented Dec 1, 2017

funoverip commented Dec 1, 2017

staaldraad commented Dec 1, 2017

staaldraad commented Dec 1, 2017

funoverip commented Dec 1, 2017 •

edited

funoverip commented Dec 1, 2017

staaldraad commented Dec 1, 2017

funoverip commented Dec 1, 2017 via email

funoverip commented Dec 1, 2017

staaldraad commented Dec 1, 2017

Freeze on RPC Bind #19

Freeze on RPC Bind #19

Comments

bao7uo commented Apr 6, 2017 • edited

staaldraad commented Apr 6, 2017

staaldraad commented Apr 6, 2017

bao7uo commented Apr 6, 2017

staaldraad commented Apr 6, 2017

bao7uo commented Apr 6, 2017

staaldraad commented Apr 6, 2017

bao7uo commented Apr 6, 2017

bao7uo commented Apr 6, 2017

staaldraad commented Apr 6, 2017

funoverip commented Nov 28, 2017

staaldraad commented Nov 28, 2017

funoverip commented Nov 28, 2017

staaldraad commented Nov 28, 2017

funoverip commented Nov 28, 2017 • edited

staaldraad commented Nov 28, 2017

funoverip commented Nov 29, 2017

funoverip commented Dec 1, 2017

staaldraad commented Dec 1, 2017

funoverip commented Dec 1, 2017

staaldraad commented Dec 1, 2017

funoverip commented Dec 1, 2017

staaldraad commented Dec 1, 2017

staaldraad commented Dec 1, 2017

funoverip commented Dec 1, 2017 • edited

funoverip commented Dec 1, 2017

staaldraad commented Dec 1, 2017

funoverip commented Dec 1, 2017 via email

funoverip commented Dec 1, 2017

staaldraad commented Dec 1, 2017

bao7uo commented Apr 6, 2017 •

edited

funoverip commented Nov 28, 2017 •

edited

funoverip commented Dec 1, 2017 •

edited