Captcha support

[krautsource]

Hi,
As discussed in #69 , cl0p (and others) added a captcha to their blog and ransomwatch thus can't scrape it anymore.
It wouldn't surprise me if more ransomware groups were going to add captchas to their sites, so this issue may become more pressing with time.
I was wondering if we could work around this, either by using a commercial captcha solving service or with the help of volunteers.

Here's what I'm thinking:

• From what I understand (based on a quick Google search), commercial services usually charge less than 1 USD per 1000 captchas. cl0p's captchas are particularly hard to read, so it may take several attempts until one is correctly solved; but even then, a few USD go a long way.
• In order to keep cost down, it may be sufficient to scrape captcha-protected ransomware blogs only once a day or every second day, which is still far better than nothing.
• Captcha-solving could be implemented asynchronously, i.e. a separate job could - after solving the captcha - supply either the unlocked HTML code or the session cookie to the scraper/parser at a predefined location. According to my tests, the session cookie on the cl0p website is valid for one hour after solving the captcha.
• A simple image captcha like on cl0p's website should be fairly easy to handle using a commercial API. AvosLocker's custom clock captcha might be a bit more work to adapt to, but it should still be possible.
• If volunteer work is preferred over a commercial API, current captchas for all relevant groups could be presented to the user / volunteer on a single web page on the clear web. On the server side, a script acts as a proxy to fetch the captchas, forward the user's responses to the respective websites on the Tor network and receive the session cookie for use with the scraper. In the case of a custom captcha like AvosLocker's, the proxy method would probably also be the way to go when using a commercial API (in this case, the front-end user is a paid worker).

As a completely different approach, I was also wondering if cl0p and others would be willing to supply a current victim list using some kind of API or at least on a separate page without a captcha. They want people to know about their successes, don't they? So maybe they'd be willing to cooperate on this.
However, actively approaching a ransomware gang could become a moral or even a legal dilemma for the ransomwatch project.

Any thoughts on this?

[joshhighet]

hey, thanks for the ideas

yep - a number of groups have implemented various scraping countermeasures over the ~3 years this project has been running. five, to be exact

curl -sL ransomwhat.telemetry.ltd/groups | jq -r '.[] | select(.captcha==true) | .name'
avoslocker
grief
clop
doppelpaymer
entropy

the majority of groups that have implemented CAPTCHA's have done so in ways that can already be trivially bypassed, though incorporating these shortcomings into ransomwatch would be a net loss as far as I am concerned. given the open codebase, if a group has shown design choices that clearly suggest the detterance of scrapers, i've not added parsers to avoid inevitable games of cat and mouse.

ransomwatch at the core is a very simple framework. to incorporate post-dom-rendering actions and leverage 3rd party API's mid-flight would require an extensive rework to the current implementation, something I see more fit in a wholly new rewrite. this is something I'm open to if others are open to engagement, though I've ultimately parked a number of new features in the current model to avoid turning it into a frankenstein.

ransomwatch is a net loss for me financially - i'm happy to continue supporting it, though I would object the use of any commercial API's under the current model. there are certainly solver providers in this space that can tackle out-of-norm tests such as what you're accustomed to dealing with when viewing avoslocker. i don't feel right profiting from ransomwatch - though some kind of cost model would inevitably have to be introduced to go down this path. in full transparency, through two kind one-time GitHub sponsors over the past year, I have recovered about 40% of the operational costs to service this platform

I like your idea of offloading the solving to the visitor when required, a volunteer model certainly seems to be the only sustainable pathway. though implementing this would require changes in the infrastructure and threat model I have for this service. accepting user input and the use of server-side rendering has all been carefully avoided to date, and again I think such changes would be better suited towards a more efficient rewrite

you are right - morally, the ground gets shaky. I formed ransomwatch after the surge of DLS's that arose after Maze really kicked things off - in a time where it became unsustainable to visit, track and gain situational awareness across such a large and growing estate of web properties, where the only ones doing it were select CTI firms shrouded in secrecy, producing claims and reports that were simply un-attestable at the time.

a number of groups surface machine-consumable feeds through either JSON, XML or RSS - where these exist, I default to using them today. i have (and continue to) refuse direct offers from certain groups that wish to provide me with alternate locations or feeds to be included within ransomwatch

this speaks to some clear boundaries I have with this project

never being the first to broadcast a new service and/or group
never introducing non-publicly accessible information into the ransomwatch dataset

I believe ransomwatch is a beneficial utility to a number of industries and verticals - and I'd love it to remain that way whilst still being able to evolve as do our sources. That said, I believe there is a very fine line between what this project is today, the transparency and accountability it provides and an extortion amplification platform, something I'd hate for this to ever be compared to.

-j

[krautsource]

Hi Josh, thank you very much for elaborating and for the insight regarding your moral rationale for this project.

if a group has shown design choices that clearly suggest the detterance of scrapers

Well, on the respective web pages, it usually says "DDoS protection". So I'm not sure if this is actually targeted against scrapers or if the blocking of scrapers may in fact be collateral damage (considering they need publicity for maximum leverage against victims).

i've not added parsers to avoid inevitable games of cat and mouse

Yes, I thought so too.

to incorporate post-dom-rendering actions and leverage 3rd party API's mid-flight would require an extensive rework to the current implementation

I'm actually not sure if that's really necessary... probably I didn't make clear enough what I meant. The captcha solving could be done in a completely separate service/job, working out-of-band, mostly independent of the scraper / parser.
Maybe I'm missing something or I'm getting ahead of myself, but in its most simple form and using volunteer work instead of a commercial API, I think it could look something like the following. For simplicity, I'll call it the "solver service".

• The solver service serves a web page on the clear web. Access to that web page can be restricted to a trusted group of volunteers, if necessary.
• Volunteers visit this page whenever they wish, e.g. once every couple of days.
• Every time a volunteer visits the page, they are presented with captchas to solve. For this, the service ...
  • ... retrieves groups.json and extracts URLs for captcha-protected sites.
  • ... retrieves a captcha image in real time for each relevant group. (This might take some time depending on the length of the queue implemented on the respective site -- given we don't want to hack around such obstacles.)
  • ... displays the captcha images to the volunteer, along with input fields for the solution.
• After the volunteer has submitted a solution for a captcha, the solver service...
  • ... submits the solution to the original web site and checks if the solution is accepted or not.
  • ... writes the session cookie to a file called cookiejar.json (if the solution was accepted).

The ransomwatch.py script in turn retrieves the service's cookiejar.json at each scraper run and uses those cookies for the scraping process. As far as I can see, this (plus error handling) is the only thing that would need to be added to the existing script.
Every time a volunteer has solved the captchas, the next scheduled ransomwatch scrape run ought to be able to scrape the protected sites as well.
However, since the session cookies have a limited lifetime, the frequency of the github workflow running the scraper should probably be increased from bi-hourly to at least hourly, so no valuable cookies are lost due to session timeouts.

Alternatively -- in order to avoid timeout problems --, instead of providing a cookie jar for the scraper, the solver service could actually scrape the protected sites' HTML immediately and place it somewhere for the ransomwatch parsers to find later on. This would mean the scraper logic implemented in ransomwatch.py would have to be replicated for the solver service.
In the ransomwatch script itself, only the source locations for the parser stage would have to be made configurable and the scraping stage would have to be made optional (as the scraping of captcha-protected sites is offloaded to the solver service).

I hope I'm making sense.
For cl0p and AvosLocker, this should already do the job as both are working with plain old images. I haven't been able to find working URLs for grief, doppelpaymer or entropy to check their captcha mechanisms.

ransomwatch is a net loss for me financially [..] through two kind one-time GitHub sponsors over the past year, I have recovered about 40% of the operational costs

I'm curious: would you mind disclosing a ballbark figure for the current operational costs? Maybe (just maybe) I can arrange something with my employer.
In any case, I'd like to at least say "thank you" for developing and providing this service and for the continuing support. OSINT alternatives for commercial CTI are indeed very important.

If you think one of the "solver service" approaches described above makes sense, I'd be happy to do some more R&D and put together a PoC for you to review and adapt if you wish.
And -- if such a service is eventually implemented -- I'd of course be willing to routinely solve a couple of captchas manually once or twice every work day, if workload allows (or maybe I'll just have one of our trainees do it 😛 ).

[krautsource]

Just to further illustrate the idea, here's a highly sophisticated diagram of both the alternatives I mentioned.