RFC: Bring Your Own Payloads (BYOP)

[smcintyre-r7]

Problem

Evasion has always been a problem for Metasploit and will remain a problem for the foreseeable future. This leads to user frustration as they can not leverage many of Metasploit payloads regardless of the exploit content the project provides, i.e. the exploit doesn't really matter if Meterpreter can't run. This has lead to a trend in the industry where users have written their own C2 servers and C2 payloads (see the ask.thec2matrix.com for a non-exhaustive list). They do this so their custom payloads are not detected by remote AVs/EDRs. This process often involves designing their own C2, writing their own C2 server, etc which has very little to do with the detection surface on the remote system.

Solution

The Metasploit framework could facilitate the current trends by allowing users to write their own payloads for Metasploit. This would remove the necessity for payload authors to write their own servers, protocols etc, thus enabling them to focus on the core of the problem which is the executable agent that runs on the target system.

Metasploit currently encourages uses to write exploits for the framework, and there's a lot of documentation on how to do it leading to a relatively low barrier to entry. An exploit module for Metasploit can for example be written in as little as 80 lines of Ruby. A payload on the other hand has a very high barrier to entry for anything resembling a modern implant (something like Meterpreter vs a standard shell). Metasploit could document a C2 protocol and provide code samples in a variety of languages such as Python, .NET, Go etc. that allow users to focus on how they build their implant and execute their various tasks.

The tasks provided by user payloads should be very limited and very well defined. While Meterpreter supports "everything and the kitchen sink" user payloads should be limited to a small subset of general functionality such as file I/O, code/command execution and host enumerating.

This would require a hefty amount of abstraction of the transports / protocols and handlers implemented by Metasploit to ideally allow users to also implement their own handlers through a sort of middleware. Users should be able to implement a handler in Ruby that accepts commands from an arbitrary source (an S3 bucket, a Dropbox file, a Git repo, a Twitter Tweet, etc.) and place them into a common format for Metasploit to handle and dispatch to.

Further Considerations

The success of this would be closely aligned with adoption. By the nature of the goal however, the Metasploit project may not see user-payloads being contributed. The value in these projects will often be that they are custom. This will make it difficult to measure the success.

To encourage adoption, the Metasploit project team members could provide training on how to develop payloads to encourage the community to get started.

[gwillcox-r7]

So my one question r.e this would be how would post modules that rely on RailGun or other Meterpreter specific functionality work? Would we not end up preventing a lot of this functionality from working? I imagine their is still the upgrade session route but that is more likely to be caught by AV.

[smcintyre-r7]

Yeah the custom payloads wouldn't have the vast majority of the functionality that Meterpreter has, thus preventing them from being an implementation. When a post-module or something that requires railgun is queued up, Metasploit would automatically upgrade the session to Meterpreter, run the necessary module with the temporary Meterpreter session with railgun, and whatever else is necessary and then close the Meterpreter session once the queue is complete.

Getting caught by AV when opening the new session is going to be problematic but we'll already be in memory at that point and thus have more options available to us for evasion that can be implemented by the custom payload. That is to say the user can implement whatever fancy shellcode injection technique they have that's effective in their environment.

If Meterpreter can't be used at all though, I don't think that would be a huge blocker because the target audience is already not using Meterpreter. They'd still be able to run OS commands, and of course by extension Powershell as well. Modules that are compatible with shell sessions would still work in this case.

[wolfthefallen]

I really like the idea of being able to write my own payload that would integrate with the framework. This would decrease the amount of time I take to create something new every time a blue team starts eating things I toss at them.

I could see the use of some exploits built into the framework not functioning as they were built for Meterpreter. My recommendation is to add a check for the current session to see if that payload is compatible for the required actions like ::has_railgun:: and if not then use @smcintyre-r7 recommendation of attempting to inject a Meterpreter session to utilize for the action.

[sempervictus]

This sort of necessitates those internal changes we discussed regarding disambiguation of platform, os, arch, runtime, and session semantics a few years ago at the hackathon. The internal wiring or how handlers->sessions happen is brittle, and needs to be deconstructed into relatively simple building blocks to permit composition of "head-ends" compatible with whatever payload/RAT/C2 users may want to leverage - sort of parallels the OpenSSL vs Noise programming trends. DNS tunneling for instance requires some serious gymnastics right now because you have to parse packets directly across TCP and UDP relayed from Tron knows where, to reassemble something that behaves like a synchronous session...
That said - i'm all for.

[smcintyre-r7]

So I've been thinking about this a bit more. One of the primary issues with this proposal is the massive scope of the work that would be involved. I think we could implement it using a more incremental approach if committed to using the existing Meterpreter TLV protocol for communication over our existing transports (i.e. reverse_tcp, reverse_http, etc.).

This would necessitate would-be payload developers to communicate with Metasploit in an identical way to how Meterpreter does so now and by extension would have all the same limitations from a network perspective. I think this would be acceptable though because most of the times a payload is blocked, it's being blocked by a host-based control like AV or EDR. Of course this is a generalization.

Now if we left the communications exactly the way they are, we could provide reusable and heavily documented libraries that could allow users to build their own Meterpreter. The process environment in which Meterpreter is running (x86, Java, PHP, Python, etc.) is really (AFAIK) only needed for extensions which I wouldn't imagine folks would be writing too many of anyways. So assuming they specified sysinfo correctly to Metasploit we could identify the OS environment on which it's running which is necessary for post modules etc.

Payload developers would then take a library we would write and provide to them in whatever language you want (Go, C#, Python, Rust, etc.). This library would provide:

• Defined constants for things like command IDs
• Utility functions to pack, unpack, iterate over TLVs
• Communicate with Metasploit over reverse_tcp, reverse_https, etc with the necessary encryption
• A command dispatcher

Users would then build whatever RAT they wanted with these, compiling it themselves (and breaking signatures of course through the process) and write whatever functionality they wanted it to have, registering it with the Meterpreter dispatcher. They could then implement the functions however they see fit using whatever leet methods to mask process creation they have up their sleeves.

From Metasploit's perspective, we'd spin up a handler for an unstaged Meterpreter using whatever transport like set PAYLOAD generic/meterpreter/reverse_tcp. We'd set the Meterpreter arch as Unknown since it doesn't matter, we'll never load any extensions. Metasploit would then load it and treat it as an unstaged Meterpreter session and we'd be off to the races with whatever commands the remote side exposes. This would just require us to finish the initiative we've already started to filter and detect functionality based on what Meterpreter says it supports (through command ID enumeration).

[wolfthefallen]

It is a massive project any way you look at it.

Try to break it down into is subtasks and where a competition of stubtasks completed can be used if released. While I would love this yesterday, these modifications are looking at a few years for full release across all protocols. I can agree that 90% of the time when my meterpeter payloads are caught it is at the host level with an EDR/AV suite. Being able to customize a meterpeter to change up items that the target environment's EDR/AV is detecting/acting on would help greatly and improve the survivability of meterpeter.

Starting with Meterpreter as the base, as it is already written and multiple communication channels are available for it would be a great place to start. Providing detailed template(s) in a few languages would be a great place to start. By staying with in the limitations that are already bound by the go-to RAT for this framework would give a working spot to help set guidelines for BYOR (Bring your own RAT). This would be a good stepping stone into creating a more flexible API on the serverside for end users to create their own listeners. To make this step easier focus on exposing on protocol at a time, with a set of core APIs all protocols will need to utilize. Such as Command ID, Sysinfo, etc.. Then just slowly expanding the project as time goes on based on interest from the public.

To help facilitate interest from the public provide classes/courses along the way. These classes would focus on building your custom Meterpreter then expand to custom c2 and RAT long X protocol in Y language. Not only would the classes help gauge community interest (How fast does the course fill up) but all provide feedback on how to expand and fix current APIs and templates.

Any way you cut it though this would be a long-haul project, and rate of release will depend on a number of developers on hand, creep, and arguing over the color of the bike. But it will solidify the Metasploit framework for the long haul as AV/EDR/NIDs continue to expand. Look they built a detection, and it was on X.. Whelp X is gone let me make Y. Faster turn around to get back into target environments for testing.

[sempervictus]

@wolfthefallen - being caught by EDR/AV and NIDS/proxies are different stories. The host-level stuff is generally watching for things like our ancient process injection or RWX allocations, that's at the execution tier. Throw in a different injector or change those RWX to RW->RX and you can walk around the static killchain they're looking for rather often. The linkage or transport problem is sort of what we're getting at in terms of "this is a giant PITA" :). There's this ghost of Christmas past problem where sessions and transports are grotesquely conjoined and none of us have ever had the time and relevant level of self-loathing to disambiguate, rewrite, survive review, and get up-streamed. Its probably the greatest sin within the codebase today.
@smcintyre-r7: i think that if we "librarize" the TLV transport, we might get some adoption, but more than likely we'll need to fork/PR other tools with it down the line. I think that the decomposition of sessions approach might be easier in the long run as we can pretty much break all sessions down to three major types:
1.Command sessions (with some differentiation - tty/raw/looping/etc)
2. C&C sessions (meterp/meterssh/mettle/pymet/phpmet/futuremet)
3. Application sessions (think irb repl remotely or more complex paradigms down the line)
Which would let us separate their transports into sync^async and single^multipath (out on UDP in on pastebin)

By having a transport handler and session handler as separate constructs running callbacks on recv until they produce a framework-compatible session output and back the other way on input (one of the 3 above), we could "absorb" other tools straight up, in plugins, etc. Think sessions through gdocs/gmaps or other complex encoding/decoding on long latency async mechanisms (hence that whole seqno effort on TLVs).

Half the effort here is to dig in deep enough to untie the cat's cradle in session+transport stack. If someone is going to cave-dive that mess, we might as well go all the way and fix the problem once and for all - or at least tear the functions apart to make it easier for others to come in behind the first rank and fix it piece by piece.

[Techno-Fox]

Just curious. Is there any update on this idea, if at all?

[smcintyre-r7]

Unfortunately we're not actively pursuing it at this time nor do we have plans to take it on soon. Things can change but at this point in time we are focusing our resources elsewhere.

[sempervictus]

I'm working on something to permit tunneling meterp over textual shells (STDIO on the target-side), so we might be able to leverage that (should i ever get it to anything useful) in conjunction with the shell equivalents of other advanced payloads to tunnel our traffic by popening-out to their handler from framework as a "channel" over which the TLVs will be b64-encoded to the stdio-transport-thingy in meterp (don't judge me, its rough-cut stuff right now) on the other side of the third-party comms channel.

[Techno-Fox]

I know it's been a year now. But I'll ask it again. As the idea is interesting. Any updates on this?

Most of the time (if an exploit ts found that can be used with Metasploit). It's gonna be using the download and execute payload to run our own implants on our own c2. This isn't bad per say. But would be a nice QOL improvement. Instead of switching between consistently.

P.S.

I need to get back to working on Metasploit again. I always made adjustments, and modules but never uploaded them as I felt they weren't good.

[zeroSteiner]

I think we still have unanswered questions regarding who would use it. Even if the project were entirely completed, it'd take users that want to write their own payloads to make it a success and that takes time and work on their part. It might be a tough sell to convince users to roll their own using Metasploit versus using an off the shelf product.

[EgeBalci]

I think Rapid7 should really consider investing more resources in this topic. Being able to integrate third-party implants into Metasploit would be a game changer for many folks in the industry. And it would also attract many new contributors to the project. @zeroSteiner I believe you can get a clear answer to "Who would use it?" if you just open a poll on Twitter. The number of different C2s available today is also another indicator that a lot of people are willing to make their own implants just for better evasion. None of the other C2s offer anything extra or new; none of them can compete with years of community work and thousands of modules inside Metasploit; most of them are light years behind Meterpreter in terms of features. But people still prefer their own C2s because, at the end of the day, if you can't evade detection, it all means nothing.

I think about creating my own C2 every day. But I always end up discouraging myself, thinking about the amount of necessary work for reimplementing everything already inside Metasploit. All the features, automation, post module, multi-platform, multi-user, support, etc. If there was an easy way to integrate a custom implant into Metasploit, I would start coding immediately. 😄

[wolfthefallen]

I think what @zeroSteiner and Rapid7 leadership as whole is trying to figure out how many people would actually utilize the "Roll your own payload". Unfortantly most people that would utilize this option would probably never open source what they created. Thus it would be hard to put a meteric to this, and if it would be worth the lift in the framework. In addition it would probably increase the number of support tickets of "How do I roll my own". For this I would recommend some base templates for people to utilize to roll there own from.

Now If Rapid7 really wanted to make some money, they could offer classes of how to make their own custom payload. Make it a 40hour class. Have the class cover the API that is exposed and how to incorperate it into the payload. Hey look this is x in the API this is what it is. Lab Add it to your payload. Hell I know I would drop the $$ to take this course so I could get a leg up on it. Even more so if @zeroSteiner was the teacher. Honestly Doing the whole course like this would probably be the best Meteric Rapid7 could use to determin the use of this feature and recoup the cost of development.

[EgeBalci]

I disagree on a few things. People shouldn't opensource their implants in the first place. Open-source implants will eventually get detected; there is no way around that. They also increase the possibility of malicious use by threat actors. By enabling other people to integrate their private implants with Metasploit, the project will eventually gain more usage and contributors. People will spend time developing modules and automation scripts around their implants.

Privilege escalation and post modules on Windows only mean something if you are able to get a session. Currently, all MSF payloads for Windows are getting detected by the Windows Defender. If this is the case, what is the point of developing such modules? Isn't this wasted resources?

I think it would be a huge mistake If Rapid7 decides to further privatize this with paid courses and "pro/premium" content, this is not a profit opportunity; the project simply needs to adapt to survive (IMO.) 😄 Even today, no one (I know) uses Metasploit on their engagements anymore, the industry will simply move on to other C2s and tools, and Metasploit will keep losing relevance.

[Gilks]

R7 is correct to be concerned about the target audience. There likely would be very little adoption in this feature as it's presented. A large majority of operators do not have the time or expertise to develop custom payloads. If they do it's likely they won't spend the time to figure out how to make it work with MSF. Typically a custom payload might be downloading a project and modifying a few lines. There's nothing wrong with this- sometimes you just have to get the job done.

What would truly make this a worthwhile offering would be something like a "meterpreter light" payload. It would be the most basic shell of a payload that interacts with MSF. It could be purchased by internal red teams who currently have to support their own C2. It should be coded in C/C++ and offer very little functionality. No, not some fancy new rust payload- but C/C++. The logic is that most POCs seem to be released in that language and would provide a platform for hacking your own payload.

R7 could sell additional "light payload dev" where developers could be contracted to implement specific functionality for smaller teams. It could be sold as a class. It could be some simple docs that explain how to add your own functionality. This is a much more flexible offering and would be successful as seen by the DSO malware dev courses. It's really up to R7 how they'd want to capitalize on it.

As a working example, let's say the light payload can't even migrate. Well users can simply go to that part of the code and pull it out of the open source meterpreter. They can hack on it to make it undetected and then they are done. Their custom payload will likely remain undetected for a lot longer as there's much fewer detection points but it does exactly what's needed for the operation.

Meterpreter is easily one of the most advanced pieces of open source malware. It's also useless to most internal teams because the detection points can be anywhere (at any time). If the thinking is reversed and internal teams can simply pull specific functionality that's needed and use it there would likely be a much more widespread adoption.

[sempervictus]

Nobody is preventing anything... We have had custom payloads since before r7 bought metasploit. If you're asking for commercial support from r7 for that, that's a sales discussion IMO.

Implementation here requires proper code integration with sockets, handlers, etc to actually use metasploit vs some hacky CLI wrapper over another tool... IMO

[EgeBalci]

Custom payloads only allow you to deploy your custom command or shellcode using the MSF exploit. That is not the exact topic of discussion here. Currently, there is no easy way to integrate your custom implant with the existing Meterpreter backend. MSF has lots of useful post/priv. esc. modules that you can only use with Meterpreter. There are a lot of features built into the framework, specifically for Meterpreter. Being able to utilize those features and modules with custom implants would be very helpful for users and it would also free the R7 from the burden of keeping the detection rate low.

[Techno-Fox]

That would be helpful. Giving the ability for custom implants to use what's already there. It would also allow others to share their post exploitation modules for other implants to use.

[wolfthefallen]

I agree that people generally should not opensource their own private payload. Thats how these antiviruses and EDR solutions quickly lock down payloads and C2s in general. While I personally dislike it if Rapid7 locked this feature behind "pro/premium" purchases. I was thinking a paid course could help the upper management decide to dedicate resources to putting this feature into play. That way it remains open source and open the community at large, but allows them to recoupe some of the costs of tossing resources at the feature. Really if Rapid7 provided documentation on the exposed features for "Bring your own payload" and simple templates to get people started, people would not have to take the course. In general practice people would still take the course just to get started. Just food for thought on how to get upper management to buy into the idea.

[Techno-Fox]

People could also think of new techniques to go along with their own implants. Of course they're isn't gonna be many open-source private payloads. But they're will be articles on how to do XYZ while making implants. And when people find out how to do a technique whilst making they're own implant. You'd be surprised how many people want to share that knowledge of the technique they're creating.

Whilst we're on the topic of techniques and tactics. People can make they're own implants good at doing particular things depending on the environment of the engagement. Meterpreter can do "anything under the kitchen sink", but whilst developing implants. You shouldn't have everything in one implant (the more suspicious it looks, the better chance AV/EDR will catch it). So people can get good at making implants do specific things.

[sempervictus]

That's a fine line - the SSM/ec2 stuff I pushed was 100% novel far as payloads go but they won't be detected by edr :). The PowerShell stuff I originally thought might get a few weeks of play... It all depends on whether you're talking about publishing some low entropy thing or engineering a truly dynamic and heuristically high entropy architecture to allow all sorts of payloads to be undetected. Some TTPs should be released only when industry is ready, but squirreling things away in corners and coveting them doesn't move the ball forward framework or industry-wise.