Post Mortem for False Positive Trojan Alarm

[k9ert]

Post Mortem about the Trojan False Positive

We had a False Alarm for a Trojan in the v1.4.6 Windows Installer on 1st of July for about 24 hours. If you're in a hurry, read the twitter-storm.

WTF A Trojan

On 1st of July someone posted into the support-chat:

btw Windows is blocking install of Specter with warning "Detected: Trojan:Win32/FormBook.AVP!MTB"

So Kim discovered that message at around noon, first. So he used the online-antivirus-tool virustotal.com to check for that. Indeed, for the windows-installer of the newest v1.4.6 release, it found a "Trojan:Win32/Emali.A!cl". If you click this link now, everything will be green, as the algorithm from virustotal got adjusted within 24 hours. Unfortunately we don't have a screenshot from the original vulnerability shown.

So this might be a false positive. So Kim checked the 1.4.x versions before and indeed, all 1.4.x versions didn't had that alert: All of them were completely green.

Our Security Setup

There are many security threats to a Software-Project and we do a lot to draw out or mitigate security-threats:

• Almost all of our commits are signed ("verified")
• We have a quite hesitant informal merging-policy, especially if commits are coming from outsiders which we don't (unfortunately) not that many, yet.
• We're limiting ourself to as less dependencies as possible
• All our dependencies are pinned to a very exact version
• We're using the pip-hashes to validate binary hashes (pip3 install -r requirements.txt --require-hashes)
• We're using the dependabot-security-update-notification to get notified about security vulnerabilities in dependencies and upgrade them quite fast

In that case, very clearly, the malware seem to be introduced not via the source-code or one of its dependencies but via the build process.

Many OpenSource Projects use cloud-services like github-actions, travis or cirrus-ci to build our releases. We use them to test the software but we avoid them for creating our releases. Instead, we use private gitlab-runners. which are effectively physical machines at home.

As we're supporting 3 different platforms, the setup to create the releases is far from simple. We're using one fully dedicated Linux-machine and one not-fully-dedicated Windows-machine and for Mac, Stepan and Ben are building on their personal laptops. As the trojan had the name "Win32" in its names and almost all Malware is specific to a platform, on the first look, it seems that the windows-machine had been affected. Here is what non-dedicated means: It was occasionally used by a "younger relative" of Kim. The account on that machine building the binary is separated from the account occasionally used by that person, however ... who knows how Windows Account-shielding works exactly?

How could the malware gotten in (if it would not be a false positive anyway)?

So Kim's (wrong) assumption was, that this person did somethimg on that machine which dragged in that malware and that found it's way into the release. He (and also others on the team) knew just right from the start that it wasn't the best idea to use this non-dedicated machine but ... yeah, it was a point on the ToDo-List which always got pushed back again and again (and working on that windows stuff is horrible work anyway). Kim couldn't ask him what he did the last 3 days as he just needed to spend his time in the local mandatory "education facility".

So we checked the specterd-binaries (zip-files) and they were green as well. All of them! We should have investigated longer but Kim decided that it was confirmed enough to warn people in the chat and via twitter:

If you're running on Windows AND you're running the latest version (v1.4.6 created 3 days ago)
i've suggest that you do not use that version and deinstall.
Apologize. A full Post_Mortem will follow. In short: The windows-build-server was not fully
dedicated to building.

This was clearly to fast. Kim didn't consulted with anyone else with the team. To him, in that situation, it was very clear that the users needed to be informed and so he did.

Immediately he had to answer a lot of questions as people were concerned, while resfreshing the Avira malware-scanner on the windows-machine and doing a full-machine-scan there.

In the meantime Ben was also up, currently travelling in south-america. One thing from his point-of-view were strange: It's counterintuitive but the specterd-binary for windows is built on the windows-machine but the specter-installer (the exe-file for windows) is built (via docker) on the Linux-machine. So, is the specterd really not infected but the windows-machine is? Doesn't make much sense? Or is the Linux-server infected (with a windows-virus/trojan)? Doesn't make much sense either.

So Kim refreshed the clamav-scanner on the Linux machine and started scanning there as well. So testing the Linux-machine's builddirectory with clamav didn't resulted in any detection. Later on, the windows-scan won't have any positive result (apart from "coinminer") either.

Things are no longer adding up

However doublechecking the specterd-binaries did reveal that virustotal is not able to detect anything in a zip-file. Unpacking it and checking it unpacked revealed that the most recent v1.4.6 binary was "Malicious" by one of the different scanner-engines. However, that Maliciousness doesn't looked very specific. It doesn't name the malware which got detected. In addition to that, testing further versions of that binary (also VERY old versions) all were malicious as well. That looked clearly like a false-positive. So at that point in time, we started to google for false positives.

So googling for virustotal and pyinstaller (the framework we use to make specter-desktop installable on the desktop) revealed a bunch of Links which suggests that false positives seem to be detected quite often within pyinstaller installation-packages.

We also now considered alternatives like https://virusscan.jotti.org and checked binaries there. The specterd-binary for MacOS seem to have also "adware" in it and (not surprising) coinmining-detections. Due to the way we build binaries, this finding definitely had NOTHING to do with our first finding which made the false-positive scenario much likier than before.

We've already used some kind of false positive as travis (when we still used the service) is more than often blocking our builds because it assumes that we would do malicious mining. That's the nature of footprint-testing most malware scanners are working which are resulting in a lot of false-positives.

Was that a correct detection? Is it all false positives? How could you know if different services are already accumulating different results from different engines but then have different kind of results?

In the afternoon we've been contacted by Charles Guillemet the CTO of Ledger. Ledger has a security-team which asked for the binaries. In the evening, they started to investigate.

Stepping back, we had many more detections than we ever thought we had but, honestly, we never tested our binaries. At least not systematically. We just assumed that our build-system is secure.

The all clear signal

On the next day in the morning these three things made us to qualify the threat as a false-positive and gave an all clear signal:

• Our investigations above didn't fit together with a real threat-scenario but pointed to a false positive
• the security-researcher told us that he couldn't find anything suspicious in the binaries we sent him
• Virustotal had adjusted it's detection-mechanism and no longer flagged our binary as malicious

Learnings

Puh, that were not so nice 24h. Clearly we have some leare and some things on our improvement-list as well:

• Don't cry wolf too early. Consult with at least one more team-member.
• Don't use non-dedicated (especially windows) machines. They might still be safe but they will lurk at the back of your mind making you not clearly thinking when it's most necessary. (We already created a dedicated windows-build-machine)
• Don't use Personal-Laptops to build at all. Use dedicated machines not only for Linux and Windows but also MacOS (still to do Creating dedicated MacOS building machine #1262 )

Things we won't do (for now)

Priorities are always about the things you won't do. So here are two which people recommend but we can't do it for various reasons.

Windows support

There were a lot of comments from people blaming it on windows.
Should we stop supporting windows? We probably have a huge windows-user-base. And the windows-marketshare is huge. And that's probably also the reason that it's more lucrative to create windows-malware rather then MacOS or Linu-Malware.
So even though we recommend every Win-User to consider to use a non-Win-system lile MacOS or Linux, we won't for now stop Windows-support.

Reproducible Builds

We agree that reproducible builds are the very best thing to do from a binary-releasing-security point-of-view and for sure that would be a long term goal for us. However, we're running python and the pyinstaller and reproducible builds are even a huge effort for projects running with a better suited foundations as ours.
So we're very welcome if anyone would work on the short- or midterm on that but we won't be able to do that with our current core-team, currently.