Responding to Request for Collaborators

[JustinTimperio]

Hi @brittonhayes,

I saw that you are currently looking for collaborators on this project and I have some interest in working on this. I recently built a tool (sadly a little too small for a repo) that could rapidly scan an entire disk's file tree into a compressed JSON object for exfiltration by an attacker. Rather than apply scanning rules on the client end, the object was to retrieve a list of all files as fast as possible, then let the attacker use their CPU resources to analyze the file paths, then return for the interesting files at a later point.

I'm not sure this work is directly relevant but it may provide you with some performance optimizations. I used (godirwalk)[https://github.com/karrick/godirwalk] which you may find useful. In benchmarks, it destroys pretty much everything else :)

Feel free to DM at:

• @jtimperio:matrix.org
• justintimperio@gmail.com

[brittonhayes]

Hey @JustinTimperio! I appreciate you reaching out boss. I'm totally open to contributions or even just chatting out ideas to make this tool useful and performant for many different scenarios.

I think in the spirit of openness, if you ever have a new idea or concept for the repository, I'd love to try and keep stuff as public GitHub discussion posts as much as possible. Gives more people an opportunity to participate in the discussion and keeps the evolution of the project very transparent 😄

As far as your idea goes, I think that's a great concept. For context, as of right now Pillager is backed entirely by Gitleaks' Non-git scanner so it functions much in the same way by just jumping straight into the files and digging for matches. I think there's a potential for a really nice hybrid or even just joining between your idea and the way it currently functions. The pre-scan approach almost sounds like it could be executed as a separate command, e.g. pillager gather - returning that compressed JSON object of the file tree. Then allow the user to follow up on that command with pillager hunt -f my_file_tree.json

[JustinTimperio]

Hi @brittonhayes,

Sure, I am still getting used to developing security tools in the open 😅

Yes, I think a pre-scanner solution would be optimal here for a number of reasons but mainly it would reduce the amount of pre-existing code that would need to be changed. I have read the code base lightly and while you are using a lot of really interesting go features and packages, it has obfuscated the control flow a little. I suspect I just need to put more time in the codebase and get my feet wet.

So a consideration here is resource usage. The reason my little tool is writing to a compressed buffer is because the size of the returned file list ballons to over 1GB of memory on machines with just 30GB of storage. The trade-off though is that at some point you need to convert it back to something usable with CPU cycles then also parse a massive list of entries with more cycles.

I'm not sure what your runtime target or use case is though which may changes things. Do you see this as a Red Team or Blue Team tool?

From my perspective that is basically the core design consideration. If this is a tool for attackers then runtime, resource usage, dependency management, and "embeddable-ness" become far more pressing concerns. As the defender, far more resources can be allocated to ensuring potential leaks have been closed with way less consideration for performance.

Best,
Justin

[JustinTimperio]

Just threw it into a repo so you can look at how it functions. It's a very tiny repo which I think is pretty straightforward. https://github.com/Project-Lernaean/goDiskDump

[brittonhayes]

I have read the code base lightly and while you are using a lot of really interesting go features and packages, it has obfuscated the control flow a little.

There are 1000% some piece of the hunter pkg specifically that I think need some love. Right now it's got some pretty clear moments of "shiny ball syndrome" where I've added a package without heavily considering the benefits of it's addition. I think the best example here is the afero.memfs.

Additionally, I think that the hound, templates, and output formats should be moved over into their own package. The functionality of printing the results is uncomfortably coupled with the scanning and I think that leads to some unnecessary complexity in my codebase.

So a consideration here is resource usage. The reason my...

Are you saying that the file list returned is 1GB when uncompressed or compressed? And if uncompressed, what is the resource allocation improvement you're seeing when writing to the compressed buffer?

edit: Just saw your repo link. taking a peek.

I'm not sure what your runtime target or use case is though which may changes things. Do you see this as a Red Team or Blue Team tool?

That's a damn good question and I really hadn't come to terms with the fact that it's probably important to decide this. I am really leaning towards red team tool. For the primary reason that Gitleaks does a phenomenal job at handling the Blue Team use-case. So Pillager has an opportunity to be turned into something very portable and flexible. Rather than something used as a git pre-commit checker, I see pillager as a tool belt addition for security researchers. The primary goals in that case IMO would be speed of execution, cross-platform compatibility, simple command line interface/experience, and ease of reporting findings.

I appreciate you asking this. Hadn't faced that fork in the road yet but being directly asked makes it much easier to pick a direction.

[JustinTimperio]

Right now it's got some pretty clear moments of "shiny ball syndrome" where I've added a package without heavily considering the benefits of it's addition.

I think this can be very difficult, as Go has a very active community with some amazing projects. I think the core team tho does a good job of following the KISS principle.

Are you saying that the file list returned is 1GB when uncompressed or compressed?

It ends up around 1GB uncompressed in mem. I think a lot of this is just poor go garbage collection. In any case, the compression brings it down into the 100-400MB range depending on the file system.

edit: Just saw your repo link. taking a peek.

It was a 1-hour hack job so it's not very clean

I am really leaning towards red team tool. For the primary reason that Gitleaks does a phenomenal job at handling the Blue Team use-case.

I think this is a fair assessment. I also assumed from the name that as a "pillager" the goal would be to raid a file system for pretty much anything of value. These days I am primarily a grey hat hacker and I rarely run into git repos in the wild. As an attacker, I am interested in far more generic and commonplace file types. For example, personal info such as pictures, accounting records, and other background info can be used to build extremely powerful social engineering attacks. These file types are much harder to sort through and require a far more detailed inventory and scanning methodology than say a gitleaks.

[JustinTimperio]

also really dig the idea of breaking it into categories based on their severity for easy post-mortem analysis and filtering.

I also like this because it would provide an API for what could be called a "static file system assessment tool". In the same way, we feed our code through static code analysis tools, "Pillager" could be made into a very compelling security assessment tool for both Red and Blud Team members. I also work on a project called ToRat (a red team tool) and adding the ability to call an external library to manage the analysis of an infected machine would be an amazing addition. From the other side, blue team members would be able to set up automated pipelines for collecting and logging potential holes created by misconfigured services.

This does raise a question for me though. How comfortable are you with rebasing/large commits? From the rough concept I have spec'ed in my head, it would be a fairly serious departure from your current codebase.

It would involve:

• A fairly complex asynchronous control surface
• An entirely custom content flagging system
• A tiered file analysis tool (tiered hunters for different things all with self-managing attributes)
• A smart routine dispatcher capable of managing the load (unbounded parallelism is pretty obvious as an attacker)
• An API for integration with other tools
• A reasonable report interface for admins to review results as blue team.

[brittonhayes]

I'm actually completely in favor of this. At the moment pillager feels very half-baked. The extent of its feature set is almost entirely in the customizable template output which isn't inherently unique.

I think that large sweeping changes to the tool and repository would be for the better of the project and be the most likely way to turn this into something with more character and utility.

The only things I'd like to try to really stick to are:

• The effective Go guidelines

• A simple command hierarchy. e.g. I'd like to keep the interface on the side of Docker or kubectl vs something like Hashcat where it can be overwhelming for new users.

• As much as possible, try to make PRs small and easily reviewable. It takes longer but since this is a side project outside of work, the more digestible a pull request is the more likely it's gonna get the love it needs.

• My current hope is to keep pillager on the neutral side of the security ethics line. So commands like torrat's shred command or similar, I'd hope to not do with this project. It should be purely for (like you said) file analysis, content flagging, and more things of that nature. Tldr: the information gathering stage of the penetration testing process.

Nothing different than really any other Go project but just putting those out as things that feel important to sustainable expansion of pillagers functionality. So all that being said, to answer your question yes. I am up for large overhauls of the code base given that the above bullets are kept in mind along the way 😊

[JustinTimperio]

I am in firm agreement on almost all these points. I stick to the Go guidelines pretty heavily. You'll also probably notice I am primarily a functional programmer and a huge nerd for KISS. Those are really my core axioms when writing software.

So commands like torrat's shred command or similar, I'd hope to not do with this project.

I actually had never considered that destructive (its just a pure go version of the unix util) but I see your point. I think this is something that we all grapple with. There are a number of features I have kept out of ToRat, mainly features that would make it trivial to control a botnet. Sadly though I think at some point you will have to face the fact that Pillager, as a project, will result in abuse by bad actors. It's pretty likely script kiddies and other bad actors will most likely benefit from our work here.

I think it is important though to keep in mind that the good these tools do in providing transparency to administrators and users far outweighs the cost. While this tool may be used by some kids to steal their friend's discord token, the administrators that add this as a risk detection pipeline will provide their users with substantially better security.

So I will probably spin up some discussions here so I can keep the docs and discussions organized for us and others in the future but my current thought process is the following:

• The template output is fairly helpful actually. The backend should be able to be swapped out without a huge amount of effort.
• The core should be pretty fast to frame and build. The key will be to build a highly reliable and performant frame we can extend on. Then as others join or make advances in filtering and detection, updates can easily be applied to existing rule sets and sub-routines.
• I think most of the work here is the actual analysis of the files. (IE in the chain of identity --> analyze --> respond, most of the work is the middle stage. I suspect like 60% of the repo will be the hunter subroutines

[brittonhayes]

Man, you are very good at addressing any concerns I have. You explained that beautifully. I'm totally with you on it and like the way you're going with this. Please feel free to make discussion posts and let me know if you'd like me to add a kanban to the projects tab as well.

This is a new side of the coin for me, having been exclusively working on the blue team side of things so if at any point my replies come through as reserved, it's not a perspective of disagreement so much as it is one of my unfamiliarity. Always willing to hear out the angle, so please feel free to share any thoughts or differences of opinion as this goes along.

I'll set up a contributors.md and append the README to point people towards the discussions section in case of ideas.

[JustinTimperio]

This is a new side of the coin for me, having been exclusively working on the blue team side of things so if at any point my replies come through as reserved, it's not a perspective of disagreement so much as it is one of my unfamiliarity.

This is totally justified. I have mostly lived in the grey areas of the industry, so I have had some time to deal with the ethics of these choices. I have built some tools I honestly never plan to release. On the flip side, by not releasing them, I leave potential holes in the applications and software I rely on. I have always found it a funny conundrum ethically. You can't make security good unless you break it, so it's always this race to break it fast and patch it before it becomes an issue. My opinion these days is that, for the most part, it's almost always worth publishing security vulnerabilities and hacking tools.

My core caveat is that I don't cater to script kiddies. I see a lot of people make the mistake of trying to cater to this audience because it garners stars and usage. I personally see this as a negative. While making software simple is always a good thing, creating tools that specifically allow inexperienced and ethically dubious indiviudals to benefit from academic work is kind of a waste of good work. I have found this to be a really weird line to walk but thankfully I think "pillager" as a concept posses little ethical dilemmas. If someone is already inside your FS, you have a bigger issue, that has nothing to do with the stuff you leave unsecured on the system.