Remove OpenSSL from Windows Meterp, packet header changes, and TLV packet encryption

[OJ]

Associated Metasploit Payloads PR: rapid7/metasploit-payloads#211
Associated Mettle PR: < coming shortly I promise, I'm not yet finished >

Those payloads will need to be built and tested alongside this if anything is going to work.

This PR supercedes the previous MSF side PR: #8570, which has now been closed.

Description

This all began after some of the powers that be (and me too!) decided to remove the OpenSSL library from Windows Meterpreter. As a result of that discussion, @busterb went ahead and did the initial OpenSSL removal work in rapid7/metasploit-payloads#205. As shown in the PR comments, various community members expressed a perfectly valid concern about now having encryption on reverse_tcp transports. While we initially felt that the lack of encryption in the short term was something we were happy to live, we instead took it on ourselves to go ahead and satiate this need before changing anything.

This is another large set of changes, and comes with a large PR description. You'll have to fortive me. I'll do my best to break up the boring bits with amusing memes, gifs and useful tidbits of information.

High-level rationale goes like this:

• OpenSSL was huge, and was only used in two scenarios (the Python extension, and for reverse_tcp transports).
• The only other transport with transport-level encryption (reverse_https) makes use of CSP/Schannel, and so doesn't rely on the OpenSSL lib that comes with metsrv.
• Removal of OpenSSL would reduce the size of the metsrv DLL by a whopping 80%! A saving that shouldn't be sniffed at.
• After removing the library, the intention would be to re-add some form of encryption using CSP/Schannel as well.

It turned out that we had a few other needs that were in some way related, and given that we don't like to constantly break binary compatibility with old payloads, it was a good idea to be "in for a penny, in for a pound" and break everything at once so we didn't have to do it again any time soon. The primary need was to modify the packet header so that it included the Session GUID that has been added recently. This would mean that we would have the ability to associate a packet with a session handler, regardless of the transport that the packet came from, and for us to handle the routing of this packet to the relevant session handler that contains the per-session encryption key. This opens up a world of fun things that we can do down the track (more to come on that in future PRs, but trust me, it's going to be funzies!).

Given that we were going to make a point of implementing encryption for reverse_tcp, it made more sense to enable encryption at the TLV packet level, rather than binding it to the transport. As a result this work contains not only the removal of OpenSSL, but addition of a bunch of code that enables per-session encryption using encryption tha tis specific to the session, with key sharing via RSA. More on this in a moment. XORing the packet with a per-packet XOR key is still needed to help obfuscate the packet headers, and to make it so that the few packets that might fly in plain view are still obscured.

For various reasons (including not updating all the Meterpreters, and needing to have "raw" packets to begin with), the packet needed to be modified to indicate whether or not the packet contained encrypted data. Combine that with the session GUID requirement, and we had the need to modify the packet header... resulting in the breaking of binary backwards compatibility.

Given that we were breaking back compat, I felt now was the time to also fix up the obscure XOR byte swizzling thing we were doing as a result of attempting to use UINT values on Windows. At this point, all the XOR stuff is done assuming a straight 4-byte array, and the Meterpreter code has been updated to reflect this as well.

Packet header changes

Prior to this PR, the packet header looked like this:

Values: [XOR KEY][packet length][packet type][ .... TLV packets go here .... ]
Size:   [   4   ][       4     ][     4     ][ ....          N          .... ]

After this PR, the packet header looks like this:

Values: [XOR KEY][session guid][encryption flags][packet length][packet type][ .... TLV packets go here .... ]
Size:   [   4   ][    16      ][      4       ][       4     ][     4     ][ ....          N          .... ]

• Session GUID: This contains the GUID that identifies the session. In the case of stageless payloads, this GUID is the null GUID (all null bytes), and a new one is generated and passed to the Meterpreter instance when the session is established. This GUID is now easily accessible outside of the contents of the TLV packets, and as said before this means we can route the packet to the correct handler prior to it being decrypted with the session-specific key.
• Encryption flags: This is a DWORD flag set that indicates whether this packet has been encrypted or not (and with what). If not, the structure of the packet from the length onwards is the same as it has always been. However, if the flag is set, the the value after the packet type is changed. For example, if the encryption flag is AES256 (ie. the value 1, it will look like this:

Values: ... [packet length][packet type][AES IV][ .... encrypted TLV .... ]
Size:   ... [       4     ][     4     ][  16  ][ ....        N      .... ]

There is a 16 byte initialisation vector that is used to being the AES256 decryption process. After those 16 bytes is the full encrypted TLV payload.

In short, the packet size has increased from 12 bytes to 32, and hence needs to be handled as follows:

1. The full 32 byte header should be downloaded first.
2. The first 4 bytes should be extracted, and then used to de-XOR the remaining 28 bytes as a single block.
3. The session GUID can (or will, in future) be used to identify the session handler. This isn't happening yet as we don't have the wiring in place, it just assumes the current session handler is going to handle it.
4. The encryption flags can be referenced to determine if the packet is encrypted and how.
5. The length value indicates how many more bytes should be read from the data stream associated with the transport.
6. Once fully downloaded, the entire packet should be XOR'd starting from byte 5 (again, to avoid packet alignment issues).
7. If encrypted, the IV should be extracted from the first 16 bytes, and then the decryption process should happen, converting the payload back into a set of TLV values.
8. The TLV "packet" can now be passed off for handling by the session handler with all the values decrypted/decoded/etc.

Packet encryption

As mentioned before, packets are now encrypted with AES256 in windows. But not immediately. The session needs to be established, and MSF will negotiate keys with Meterpreter on the fly. This is done for a few reasons:

• We don't want to have AES keys baked into payloads (and hence readable by someone untrusted).
• We want to have different keys for every session even if it's from executing the same payload repeatedly.
• We want to have the ability to change the keys on the fly at any time if we should wish.

This then begs the question: how do we exchange keys in a "secure" manner. At the end of this PR I'll post a rationale behind the approach that we're taking, particularly in an effort to address the "OMG THIS CAN BE MITM'd" crowd. The following shows the "handshake" process:

1. An exploit or payload fires in the usual way, resulting in a stager being executed.
2. The stager connects to MSF (or the other way, in the case of bind payloads).
3. MSF sends a (substantially smaller) metsrv DLL down the wire to the stager.
4. The stager downloads the stage, copies it to RWX memory and invokes it.
5. metsrv begins execution and takes over communications from the stager.
6. MSF then begins the AES key negotiation phase:
   1. MSF generates a 2048 bit RSA key pair (both public and private key).
   2. The public key is added to a packet in pem format, and the packet is sent to Meterpreter with the command core_negotiate_aes.
   3. Meterpreter receives the packet. If the AES negotiation feature does not exist, the "method not found" error happens as per usual.
   4. If the command is supported and public key handling is supported (which it is for Windows Meterpreter at this point), then:
      1. The public key is extracted, and parsed.
      2. A 256-bit AES key is generated using a cryptographically secure random number generator.
      3. The AES key is stored along with the session data in Meterpreter's memory.
      4. The AES key is then encrypted using the public key that was given as part of the initial message.
      5. Meterpreter responds to MSF with a packet containing the encrypted AES256 key.
   5. If the command is supported and public key handling is NOT supported (might not be the case, but might be for other Meterpreters):
      1. An AES key is generated and stored along side the Meterpreter session data.
      2. The AES key can be returned in clear text.
   6. MSF receives the result of the command:
      1. If the message was not supported, then no AES key is associated with the session.
      2. If the message was fully supported, MSF decrypts the AES key using the private key that was generated with the public key, and the AES key is stored with the session handler.
      3. If the message was partially supported, MSF copies the plain text AES key and stores it with the session handler.
      4. In all cases, the generated RSA key pair is forgotten.
7. If key negotiation was successful, then all future packets are encrypted using the shared AES key, and the encryption flag is set on the packet.
8. If key negotiation was not successful, future packets are sent in plain text just like they are usually (while still being XOR'd).

There are a couple of cases that needed to be considered when it came to packet encryption:

• The response for packet encryption negotiation couldn't be encrypted because MSF isn't yet aware of the key.
• After migration a session to another process, the negotiation phase needs to be re-run, because the AES key is not part of the configuration data (intentional). As a result, encrypt keys are cleared once migration is done and a new key is generated. This is all done prior to any extensions being reloaded, or any other important steps being taken.
• When a transport shuts down (as a result of sleep or transport switch), MSF believes the session to be dead. When the session comes back, MSF again negotiates a new AES key, as it doesn't have any tracking of the previous AES key. This might change down the track if the team decides to track AES across the session (personally I don't see the need -- more keys, more pain for IR!).

These cases have been considered and coded for.

Verification

I am sorry. I am so, so sorry for whoever has to do this. But, sometimes work sucks! This is one of those times.

These verification steps should be done for each of the Meterpreters that exist (yeah.. I know, right?), and for each transport:

• Create a staged handler and a stageless handler.
• Create a staged payload and a stageless payload.
• Execute the payloads and make sure that the sessions correctly establish, do not crash, and can be interacted with.
• Validate the channels still work by invoking a shell and interacting with bash/cmd.exe/whatever.
• Validate that the loading of extensions still works.
• On Windows, migrate across process boundaries to make sure the migration process still works.
• Make sure that the session comes back after sleep or transport switching.
• Make sure that stageless connections can connect to staged listeners and that the stage is ignored (the session should still connect correctly).
• Use both sessions -x and sessions -v and make sure the encryption indicator is accurate.
• Use both the powershell and python extensions, and make sure that the Meterpreter bindings still work.

For non-HTTPS payloads on Windows:

• Using wireshark, capture some traffic of the session and make sure that the content is not just XOR'd but encrypted as well (ie. is junk when you XOR the TLV packet with the first 4 bytes of the packet).
• using wireshark, capture some traffic from reestablished or migrated sessions and make sure the packets are still encrypted.

ERMAGHERD!! MAN IN THE MIDDLE!!

The most common question that has come up so far, and probably the most common that will come up down the track, is:

Isn't this vulnerable to MITM attacks?

I'm glad you asked. At leaset, I think I am.

The Short Answer

Yes! but this is no different to before ripping out OpenSSL.

If you don't believe me, read the next answer.

The Long Answer

There are two types of MITM attacks that should be considered if we're going to be completely thorough: active MITM and passive MITM.

• Active means that someone is between MSf and Meterpreter, and can interfere with the traffic as it goes across the wire.
• Passive means that someone is pcap'ing communications and could potentially decrypt the packets down the track.

Active MITM

The risk of active MITM is no different to how it was before. Meterpreter over TCP did nothing to validate that the STARTTLS was actually going to MSF, it just trusted that the 'thing on the other end' was MSF and did the handshake with it before sending it data. It didn't know for sure that MSF was listening, it could instead be someone malicious that is tapping the content of the communications in plain text. If active MITM is a concern now, it should have been a concern before. The risk is no different.

What can we do about it? Actually not a great deal. In order to avoid active MITM threats, we need a way of validation that the "thing on the other end" is what we expected, and validating that in the context of Meterp that was just brought to life via an exploit in an unknown network isn't exactly easy. It's fair to assume that the network is hostile, and that any means of validation we have isn't reliable. In the case of HTTPS connections, certificates are validated against a certificate store that references a hierarchy of authorities that can be verified. We just don't have this capability. The closest thing we have is baking a certificate of some kind into Meterpreter and validating the keys that come from MSF against that cert. This is again moot though, because if we have Active MITM going on, then that certificate can be replaced on the wire with a malicious one, and we're back to where we started.

It's fair to say that if you are currently being actively MITM'd, you have literally zero points of trust. You can choose to trust nothing, and shut yourself down, or you can do what you've always done, and accept the session and carry on with the assessment.

Passive MITM

This is a different beast, as it doesn't muck with the traffic. The concern here is that the packets captured can be analysed and decrypted later on. Assuming that we're not also being Actively MITM'd we're in a position to prevent the pcap from being decrypted, as we've got the equivalent of HTTPS with PFS enabled. The only differences are:

• We generate the ephemeral keys ourselves.
• We don't have the ability validate the keys like we do with HTTPS.

For someone to decrypt the traffic, they need to get hold of the AES keys. In order to get hold of the AES keys, they need to be able to get access to private key that was generated by MSF (again, we're assuming no active MITM where keys can be switched). For an attacker to get access to the private RSA key, the attacker's machine must be compromised. However, given that the attacker's machine does not store the RSA key pair for any longer than the duration of the request, compromsing the attacker's machine would be useless as the key is no longer there to be found. In the case of normal HTTPS, PSF is supported by generating ephemeral keys that are signed by the RSA private key associated with the HTTPS certificate. If the HTTPS certificate is later accessed, the private key can not be used to decrypt the traffic, as it's public key not the one that was used to encrypt it in the first place. We're doing the same thing, just without signing the key.

I honestly feel that the risk of passive MITM resulting in the compromise of data going across the wire is as small as it can be.

Thanks

As always, props to the crew behind the scenes that helped with this (even if it was just the thought process), including @busterb and @sempervictus. Thanks in advance to those people who get involved with the discussion and who help get this over the line.

Let the discussion commence!

Sorry!

I know, I lied about the memes and gifs!

Edit 28th June

Both sessions -x and sessions -v include an indication of whether or not the session is currently utilising packet level encryption. This should be tested as well!

[busterb]

I wonder if there's a simpler way to do this, since the 'reverse_tcp_ssl' payloads do SSL from the start anyway, even the stager.

[OJ]

Could well be! I think that'd be a good thing to revisit for sure.

[OJ]

I'll fix up payload cached sizes and Metasploit Payloads gem versions when the payloads have been landed.

[justinsteven]

Copypaste leftover?

[OJ]

Yes, thanks :)

[justinsteven]

Good place to mention - is it worth designating the ENCRYPTED_FLAG field as FLAGS with the bottom bit being ENCRYPTED_FLAG and the other 7 bits being reserved? Will 7 be enough for future use? In which case we should use boolean logic to extract the bottom bit rather than just == "\x01" to pave the way for general flag field use

[OJ]

To be honest, no. At the moment it's just a single bit. But it'll actually become an identifier which represents which encryption scheme is in use (such as something cheaper to run on smaller processors). So it'll stay as is for now :) Then later we'll probs change it to something indicative of the encryption scheme. Such as ENCRYPTION_SCHEME (where 00 will be none, and 01 will be AES256.. etc. Thanks!

[justinsteven]

Ahhh got it :)

[justinsteven]

Is it an issue that we're trusting packets to essentially declare whether the session is under encryption? I think it allows a passive MitM to spoof TLV responses. Would "knowing" that a session is under encryption and that all TLV's must arrive encrypted provide a measure of client authentication? Will noodle if it's worth considering.

[OJ]

Not really. The flag is an indication of whether incoming packets have been encrypted. If MITM switches the flag, then it doesn't decloak anything. If the endpoints have keys, they encrypt stuff.

We have to trust something in the protocol, this flag is there to 1) give us an idea that we can't just XOR, and 2) (in future) tell is which scheme to use.

So I'm comfortable with this flag being MITM-able, because there's no risk to current or future comms.

[justinsteven]

Hmm. But a passive MitM who knows the session GUID can spoof TLV packets that say "is ok, this packet isn't encrypted so you have no way of knowing that it came from the real client, don't bother decrypting it. Here have a fistful of cleartext TLV's maybe one of them is relevant to you right now?" Which would violate the integrity of the session?

We have to trust something in the protocol

Do we? Can't we track whether the session pertaining to a particular GUID is expected to be encrypted?

You know the architecture and the future plans, I'm just throwing ideas out there. I'm guessing there must be a reason why we can't go "Ah this GUID pertains to a session that I expect to get encrypted TLV's for. This packet has !encrypted_flag chuck it away."

[OJ]

There are scenarios, as per the PR description, where plain text TLVs are needed. There may be more.

The whole "trust the protocol" thing falls back into the "you have to trust something" category that I've already discussed. If you can't trust that byte, you can't trust any bytes. you can't trust the session GUID, or the length, or anything. So yes, I think trusting that byte is ok as far as an active 'toggle this byte' MITM is concerned.

I get the concern for having an unencrypted TLV during an encrypted session though, as far as passive replays go. It may be enough for MSF to say "if I have a key, I expect encryption" and go from there.

This will have to be considered more before settling on it though. Good question.

[justinsteven]

Now that I'm thinking, doing away with trusting the encrypted_flag for sessions that we expect encrypted packets for is a hard problem.

@OJ said:

After migration a session to another process, the negotiation phase needs to be re-run, because the AES key is not part of the configuration data (intentional). As a result, encrypt keys are cleared once migration is done and a new key is generated

This sounds like something a passive MitM can spoof at any stage and means we must not enforce the idea of "encrypted sessions must not play silly buggers with unencrypted packets". I shall noodle.

[OJ]

Yeah, it's a painful problem!

[justinsteven]

Is encrypted TLV replay a problem we should consider?

[OJ]

Great question @justinsteven. Perhaps! But perhaps not. Might be worth having a pow-wow about this in an open forum like IRC or something to see whether this is a concern. If it is, it's also a concern for unencrypted packets, and for any other packet we're sending, ever. We should definitely discuss and consider the negatives.

[justinsteven]

it's also a concern for unencrypted packets

Of course

and for any other packet we're sending, ever

Is it though? e.g. reverse_https you can't peek inside the HTTP stream so you can't steal the session GUID (unless we leak it outside of the HTTPS stream) and I'd think replaying individual TCP packets doesn't gain you anything under HTTPS.

[sempervictus]

@OJ: Working through it, had to convert named pipes to this, found out x64 doesnt like the HANDLE vs SOCKET piece :). Will get back with test results in a bit.

One note on active MITM: we can encrypt symmetric keys in stage1 by using rc4 in stage0 such as to prevent acquisition on the wire. If stage0 is delivered out of band, or uses contextual data (hostname, domain name, a low-precision time value) to decrypt stage1, we should be able to significantly increase our posture against this vector.

Basically stage0(shellcode which opens socket, recv, decrypt_in_place(via context key), load into memory) -> stage1(bring key and establish symmetric context, renegotiate under that context if needed). Migrations could bring the original key with them i suppose, or in the event of using context data for this, simply pull that ctx again (the time based approach could be a problem here, unless our stage handler keeps incrementing its clock too in order to maintain relative deltas which are acceptable).

Manual memory analysis will catch this approach, but i think we can rule out autonomous defenses currently (or in the near future) being able to do so.

Far as the "keep a certificate chain" piece, we may be able to use a simplified version of asymetric relationships to check a pair of small tokens instead of going through the rigmarole of a cert chain...

[sempervictus]

@OJ: migration from 32->64 is hosed. 32->32 works fine.

Attempting to tab-complete compatible sessions results in:

lib/msf/ui/console/command_dispatcher/core.rb:2280:in `option_values_sessions': undefined method `compatible_sessions' for #<Msf::Modules::Mod6578706c6f69742f77696e646f77732f736d622f707365786563::MetasploitModule:0x0002ce6e71b530> (NoMethodError)

Not sure if this happens upstream, lemme confirm this one before wasting your time and sanity

[OJ]

Interesting! I'll have to review that. I thought I'd tested that case. Thanks!

…

On Jun 28, 2017 08:36, "RageLtMan" ***@***.***> wrote: @OJ <https://github.com/oj>: migration from 32->64 is hosed. 32->32 works fine. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8625 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABw4PXzqsN0wsUAw-w05Ul5zLOscFU5ks5sIYPvgaJpZM4OGRF8> .

[sempervictus]

@OJ: this is on a win7 x64 box if it matters

Also, i broke something in the way named pipes handle things. Need to review that bit.

[sempervictus]

@OJ: we might have a problem - https://pastebin.com/gUUDXUDs
The first stack trace was trying to run DCERPC TCP enumeration over the switchboard via x64 meterp current (on 4f054d2), the second one came after, with the session dying along the way. Seems reproducible...

[OJ]

Thanks @sempervictus. Do you have a set of steps that I can follow to repro this please?

[OJ]

I can confirm that this is indeed a problem, but has nothing to do with this PR. From my testing on master:

1. Something is killing Meterpreter during the use of the tcp_dcerpc_auditor aux module.
2. With the session dying, the TCP channels in MSF are closed behind the scenes, and cid is set to nil.
3. Any interaction with the channel then breaks because the cid inside the channel class instance is nil, and packing nil into an Integer makes Ruby sad in the pants.

Investigating further, but as part of another PR, not this one :)

[sempervictus]

I remove my concerns regarding stability of this code. Turns out my payloads repo didn't keep the commented out version of unhooking call from jefftangs PR. That was causing my migration failures. So far with the comments back in place I'm only seeing a ~10% fail rate. Still too high for master/pro, but rational. I'm also cleaning out old named pipes code as I adopt OJs current work from the packet pivot branch, which seems to be stabilizing things more as I go. I need to do proof passes before official approval, but for all purposes of framework, I confirm this works

[OJ]

I think this is looking OK now to be honest. I haven't seen any issues with this that we didn't have with the current master.

@bcook-r7 should I change this PR so that it points to the metasploit-5 branch instead of master ?

[OJ]

Also @bcook-r7 as far as mettle goes, the main area of pain is the packet flushing code which analyses TLVs on the fly. That's going to have to change in some way given that we'll have encryption in place so we'll need to do something similar to what I'm doing, which is a bit of a hack. Can we tag team the work on that please?

[busterb]

Hi OJ - master is fine. Tag team engaged.

[OJ]

Thanks!

[busterb]

Quick progress report so this doesn't lose momentum. I'm about 50% through with the mettle bits (had an uninterrupted 3 hours on it yesterday, so I think the rest should come this weekend). I started updating the payloads branch a little too, because it has gotten a little convoluted with the merges. Once those bits are complete, we should be good to go. This is going to disable -aggregator for a while as we rethink how it will work.

As I was poking, I noticed that the xor patterns are really easy to identify due the large number of zeros in the first packet. I'm contemplating augmenting the xor obfuscation with an rng, using the xor instead as a seed.

[sempervictus]

Awesome, thank you sir. All on board with the xor piece, though that'll need to be implemented across the runtimes. Have some thoughts on aggregator and deep pivots, hoping to have a workable concept in the next few days.

[OJ]

@bcook-r7 I like that idea. Yes the first packet does indeed stand out given the NULL guid. The only concern would be making it consistent across platforms. We could easily solve that with a custom implementation though, right? It's not like it has to be cryptographically secure; it's just used for obfuscation.

[busterb]

Yep. Even a simple 32-bit lfsr would be sufficient, can be implemented in a couple of lines of code added to the xor function.

[OJ]

Got nailed again using A instead of a when unpacking important stuff. Sorry about that.

Unrelated: I hate (un)packing in Ruby.

[busterb]

note, bindata is now in tree and works a lot more like C structs than Ruby packing, if you're ever interested in doing some conversions :)

mettle PR is up now too: rapid7/mettle#91

[OJ]

Oh that's good to know! Is there a good ref for that lying around?

Thanks for helping with the mettle side mate.

[busterb]

https://github.com/dmendel/bindata

ruby_smb is done 100% with bindata, and I usually just bug @thelightcosine when I have questions or need to borrow examples. I converted a few modules to use it as well when we added Ruby 2.4 support.

[sempervictus]

Takes a moment to get used to it, but a hell of a lot neater than the packing semantics. Both python and ruby effectively make the dev keep a symbol lookup table in their head. Not everyone is wired like hdm and egypt :).

[OJ]

This is an abstraction I can get behind for sure. It's easy to mess things up, and I'm tired of seeing the same block of code in multiple spots (I'm a culprit).

[busterb]

OK, here goes nothing! We can continue stabilizing a bit in-tree as I just made a CYA release to cover any upcoming metasploit gem release needs.

[OJ]

This is awesome. Thanks so much @busterb !!

[busterb]

Release Notes

Transport-level encryption provided by OpenSSL has been replaced in some Meterpreters with application-level encryption provided per-message instead. This reduces the size of Windows Meterpreter in particular, while providing an encryption mechanism that is easier to implement in other Meterpreter implementations as well.