Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SAV reported bitcoind infected w Silly.218, crashed bitcoind (likely false positive) #4069

Closed
ghost opened this issue Apr 19, 2014 · 87 comments
Closed
Labels

Comments

@ghost
Copy link

@ghost ghost commented Apr 19, 2014

OS: Win 7 x64
Bitcoin core: 0.9.1

While running bitcoind.exe --reindex:

c:\Program Files\Bitcoin\daemon>bitcoind.exe --reindex
Error: System error: Database I/O error

Symantec Anti Virus detected Silly.218 in chainstate\052878.sst directory (other names under which the malware is known: Virus.DOS.Dutch_Tiny.163.a (Kaspersky), Silly.218 (Symantec), Tiny-Family #3 (Avira)). Hash of the pattern detected: E272F4FF4AD99D1C48C4888990893FC6193DB1CB9849C69B1710069BBD047E0D

As the file was automatically quarantined and deleted (I can't change SAV default settings), that crashed bitcoind and corrupted the bitcoin DB.

Internet search on this yielded no results. This looks like a false positive.

@ghost ghost changed the title SAV reported bitcoind to have Silly.218, crashed bitcoind (likely false positive) SAV reported bitcoind infected w Silly.218, crashed bitcoind (likely false positive) Apr 19, 2014
@gmaxwell

This comment has been minimized.

Copy link
Contributor

@gmaxwell gmaxwell commented Apr 19, 2014

Please report the misbehavior to your anti-virus vendor.

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Apr 19, 2014

Ugh. I really hate this.

The only workaround (without the AV vendors cleaning up their act) would be to start obfuscating the database and block files, but that looks even more suspicious.

@laanwj laanwj added the Windows label Apr 19, 2014
@gmaxwell

This comment has been minimized.

Copy link
Contributor

@gmaxwell gmaxwell commented Apr 19, 2014

The alternative seems to be software randomly corrupting the installs, not much of a tradeoff. We could have a setting at least to disable the obfuscation (we'll need to support obfuscation off for 'legacy files' in any case). At least then they'd have to be specifically targeting Bitcoin.

If it makes you feel better, obfuscating the block files would address some fringe risks. E.g. say someone discovers some disk sector value that that a popular brand of disk always stores incorrectly (don't laugh, I've had silicon flaws in switches result in packets that they cannot forward, due to the bit pattern making it lose sync with internal busses)... and exploit it to fork the network.

@KyrosKrane

This comment has been minimized.

Copy link

@KyrosKrane KyrosKrane commented Apr 19, 2014

I got hit with this same issue just a bit ago. The file it hit in my directory was a different file, but it was also a .sst file in the chainstate directory. I ended up telling Norton Antivirus to ignore my bitcoin data directory entirely. Is the right way to recover from this now to reindex? Or what's the right way to restore the .sst files?

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Apr 19, 2014

Yes, reindex.

We should at least add a note to the next release notes that people must add the bitcoin data directory to the ignored directories of their AV software.

@leofidus

This comment has been minimized.

Copy link

@leofidus leofidus commented Apr 19, 2014

that people must add the bitcoin data directory to the ignored directories of their AV software

Couldn't that lead to malware installing itself in the data directory? Bitcoin (indirectly) helping malware is certainly a headline we want to avoid.

@KyrosKrane

This comment has been minimized.

Copy link

@KyrosKrane KyrosKrane commented Apr 19, 2014

Ideally, I guess you'd want to exclude *.sst files in the chainstate directory only (as that's what seems to be triggering this alert). Norton, stupidly, can't do that. It can exclude all .sst files, everywhere on your PC, OR it can exclude a particular folder/directory. It can't do both.

@ghost

This comment has been minimized.

Copy link
Author

@ghost ghost commented Apr 19, 2014

Well that might be task the Bitcoin Foundation could get involved with administratively IMO, so contact AV vendors and discuss whitelisting the database files. In any case, they are not executable so I cant see why they would be scanning and flagging these files.

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Apr 20, 2014

@leofidus Indeed, they could. Making the virus scanner ignore .sst files sounds like a better suggestion. It's a fairly rare extension and is never used for executables.

That would be enough. Somehow the blk*.dat and rev*.dat files currently avoid 'detection' as they are padded to large sizes.

@c64e5e9e-08ce-4f7d-be39-92cf89188b45

This comment has been minimized.

Copy link

@c64e5e9e-08ce-4f7d-be39-92cf89188b45 c64e5e9e-08ce-4f7d-be39-92cf89188b45 commented Apr 21, 2014

Something arguably needs to be done, the 30+ virus vendors in common use won't all whitelist these files.

@gmaxwell
If you go ahead with obfuscating the blocks in some way, be sure to be in contact with the Armory folks as this will completely break their platform on Windows.

@laanwj
Virus scanners ignore large files while scanning. Padding out the .sst to 32MB would maintain compatibility with current parsers while escaping the virus scanners.

The files not being executable isn't actually anything to do with it, malware authors get around this on Windows by creating file links that attempt to execute any file. This was used on Reddit recently, with a ".txt" file containing a "leaked Bitstamp passwords" and a link to it containing a command that executed the ".txt" file. It's a fairly unpleasant trick that gets even seasoned users, so I'm not sure a global whitelist of those files would be advisable. Even apparently safe files can be dangerous on Windows.

In reality virus scanners are the dumbest of the dumb. Even just XORing the files would do. There's no comfortable way of doing it though, either every platform obfuscates the blocks with the next version at the expense of massive IO for the whole network, or Windows users have incompatible block files with Linux and OSX. In my opinion just padding out the sst files is the least destructive change.

@c64e5e9e-08ce-4f7d-be39-92cf89188b45

This comment has been minimized.

Copy link

@c64e5e9e-08ce-4f7d-be39-92cf89188b45 c64e5e9e-08ce-4f7d-be39-92cf89188b45 commented Apr 22, 2014

Just for fun, there's about 8000 reachable nodes on the network at the time of writing. Assuming that a large portion of the network is unreachable (NAT, filtering, intermittent, just not listening), it's probably safe to assume there's probably at least 50,000 nodes with the complete blockchain. If we XOR just the chainstate, we cause 50000_430 MB of disk writes, 50000_430_2 MB read and write combined, somewhere in the region of 43TB. If we XOR the entire blockchain on disk we cause 50000_21000*2 MB of IO, around 1.95PB of RW across the wider Bitcoin network. Incredible.

@gmaxwell

This comment has been minimized.

Copy link
Contributor

@gmaxwell gmaxwell commented Apr 22, 2014

My assumption is that we'd store the chainstate key as a record in chainstate, and if there is no key or its size zero we do nothing. On newly created or reindexed chainstates we'd set it to a random value... so we wouldn't rewrite it for anyone. If someone has problems with their AV they'd be told to reindex and doing so would set a key.

@c64e5e9e-08ce-4f7d-be39-92cf89188b45

This comment has been minimized.

Copy link

@c64e5e9e-08ce-4f7d-be39-92cf89188b45 c64e5e9e-08ce-4f7d-be39-92cf89188b45 commented Apr 22, 2014

Some sort of sane error when the inevitable "oh crap some of my databases are gone" moment happens wouldn't go astray either. Currently it's not clear what has even happened, just that Bitcoin Core broke and the virus protection went nuts deleting "viruses".

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Apr 22, 2014

@c64e5e9e-08ce-4f7d-be39-92cf89188b45 I don't believe padding is a sustainable solution. As time moves on and storage media become faster and larger, viruses may use this trick too and AV vendors will likely increase the maximum file size. That's a cat-and-mouse game we don't want to play. So that leaves XORing.

@gmaxwell Good point on only doing it for new databases/reindexes. That answers the eternal 'what about legacy support' question.

@leo-bogert

This comment has been minimized.

Copy link

@leo-bogert leo-bogert commented Apr 23, 2014

Please think about what a virus is: Executable code. There are only 3 ways in which data in a file can be executable on any operating system:

  • The file contains a program header. Notice that the file extension is not a program header. The header is contained inside of the file. On Windows its typically a PE-Header (http://en.wikipedia.org/wiki/Portable_Executable). The trick for executing .TXT files likely only works if they contain such a header. [But then it is not even an exploit, if I recall it correctly you are for example free to use the regular Windows CreateProcess() API to execute files without EXE extension.]. So as our SST files do not contain a program header, this case does not match.
  • The file does not contain a program header but the operating system still executes its content automatically due to exploitable bugs. This has been used in viruses for sure. But it is again not our problem, the bug in the OS is to blame and to be fixed.
  • The file does not contain a program header but there is a different file, commonly called a "loader", on the computer which does contain a program header, thereby gets executed, and contains code to load the non-executable SST file into its own address space and execute it. This is the only case which in theory could match here. But there is no reason for a loader-program on a computer to randomly pick up our SST files and load the very particular area which contains the virus for execution. This would have to be done intentionally, you don't just execute random file content. So the loader is the malicious root of execution. And as long as the virus scanner detects and deletes the loader, the non-executable SST file is completely harmless. There is no reason for the virus scanner to care about the non-executable file. So even in this only matching case, there still would be a large indication that the virus scanner is just wrong here, it is a false positive.

Conclusion: You are wasting your time discussing this. Contact Symantec to fix the false positive. They are obliged to do so: After all a virus scanner divides the world into "good" and "bad" software, which is a really judgmental thing to do, and someone who is that judgmental has a moral obligation to only judge against something which is really malicious, which Bitcoin isn't in any way. It is their job to fix this.

@c64e5e9e-08ce-4f7d-be39-92cf89188b45

This comment has been minimized.

Copy link

@c64e5e9e-08ce-4f7d-be39-92cf89188b45 c64e5e9e-08ce-4f7d-be39-92cf89188b45 commented Apr 23, 2014

@leo-bogert There's more than Symantec giving false positives. At my last count there's at least 50 virus vendors in common use (looking at virustotal.com's scan database) most of which will probably have these incredibly old virus definitions in them somewhere. I doubt even the smallest portion of them will bother to respond.

@gmaxwell

This comment has been minimized.

Copy link
Contributor

@gmaxwell gmaxwell commented Apr 23, 2014

AV companies have also show absolutely no hesitance in explicitly and intentionally marking as malware general bitcoin miner software that some malware has simply taken along for the ride.

When you're talking about software written by people who think its acceptable to corrupt a database because they didn't like a particular 16 byte sequence nested inside of it you can't make too many assumptions. (I can't wait until someone legally changes their name to one of these sequences and we find out that all sorts of government databases didn't have functioning backups… :) ) Of course, people should report the false positives in any case, but I don't think false positive reporting is a long term solution.

@c64e5e9e-08ce-4f7d-be39-92cf89188b45

This comment has been minimized.

Copy link

@c64e5e9e-08ce-4f7d-be39-92cf89188b45 c64e5e9e-08ce-4f7d-be39-92cf89188b45 commented Apr 24, 2014

#4086 was a dupe also occurring with Microsoft Security Essentials antivirus package.

@pygy

This comment has been minimized.

Copy link

@pygy pygy commented May 16, 2014

But why XOR the whole block chain for everyone?

Rather than just storing the random XOR key, you could store a (block, key) tuple in a custom file.

The first time the new client is run, it would start XORing at the block that is the source of the current issue.

Fresh installs would XOR the whole chain.

@ghost

This comment has been minimized.

Copy link
Author

@ghost ghost commented May 16, 2014

i dont imagine XORing will help since a joker could engineer some data to look wrong when XOR'd too. Surely this is something the up to now useless Bitcoin Foundation can start to deal with and actually lobby these companies. They have the funds to employ staff to do just that.

@leofidus

This comment has been minimized.

Copy link

@leofidus leofidus commented May 16, 2014

If the value you XOR against is chosen randomly and stored e.g. at the beginning of the block file, it would work.

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented May 16, 2014

Yes, the xor key would need to be different for everyone, otherwise it can
be trivially worked around.

@whitslack

This comment has been minimized.

Copy link

@whitslack whitslack commented May 16, 2014

Whatever you guys decide to do, would you please put in a build option to disable your workaround. I have no problems with virus scanners, as I don't run one (I use Linux), and I value being able to scan my block chain data files with external tools. I don't want the data obfuscated.

@wyager

This comment has been minimized.

Copy link
Contributor

@wyager wyager commented May 16, 2014

In fact, if this must be implemented, please keep this disabled by default. Either only build this into windows builds, or leave it as a checkbox (if checked by default, only on windows machines).

We should not compromise the usefulness of our own software to facilitate the awfulness of some other software.

@gmaxwell

This comment has been minimized.

Copy link
Contributor

@gmaxwell gmaxwell commented May 16, 2014

@wyager Doing this in no way compromises the usefulness of the software. Its generally protective for users for the same reason freenet does the same thing, and this kind of encoding was previously proposed back in 2011— long before people were complaining about crazy AV on windows.

@wyager

This comment has been minimized.

Copy link
Contributor

@wyager wyager commented May 16, 2014

It compromises the usefulness of the software for the same reason @whitslack mentioned. If the chain files are XORed, it disrupts the ability of external analysis tools to work on the blockchain.

Can you explain what you mean re. freenet? I was under the impression that freenet files were not easily decryptable by the user, which gave them plausible deniability. That isn't what we're trying to do here.

@gmaxwell

This comment has been minimized.

Copy link
Contributor

@gmaxwell gmaxwell commented May 16, 2014

What needs to be adjusted here is not the blockchain, it's already ignored by AV tools, but the chainstate indexes which already cannot be read by external tools. Any such tools could also decode the data. In any case, increasing the complexity for 'accidental' decodes can reduce some problems in jurisdictions with strict liability for possessing some kinds of data. (though generally our anti-spam/anti-dos rules are also protective there, people would like to relax them in the future…)

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented May 19, 2014

@luke-jr Yes we compile without it. That saves some memory as we don't use it.

@gmaxwell

This comment has been minimized.

Copy link
Contributor

@gmaxwell gmaxwell commented May 19, 2014

My recollection is that snappy makes the chainstate larger. The reason you get compression on the blocks with a stream compressor is because the compression spans many transactions and blocks and you get savings primarily from pubkey reuse, the blocks also are raw as compared to chainstate's hyper optimized serialization. (I also expect that the figures are bloated up somewhat due to a single now mostly defunct notorious pubkey reuser. :) )

@jafaber

This comment has been minimized.

Copy link

@jafaber jafaber commented May 19, 2014

Just wondering what common practice is for XORring. Obviously you don't want to XOR with 0. Would you just pick a random non zero mask? Or would it be useful to require at least a few 1 bits in there? Or maybe at least one 1 bit in each byte of the mask? (Also considering the NSA attack of a magic number triggering the harddisk into doing something nasty.)

@leofidus

This comment has been minimized.

Copy link

@leofidus leofidus commented May 19, 2014

If you want full protection against advanced pattern recognitions (which could be done by a harddisk firmware), you would probably want to XOR against the bytestream for a fast pseudorandom number generator and save the seed of that generator.

But I suspect anything beyond XORing against a random non-zero 64-bit mask is overengineering this. Requiring at least one 1 bit per byte is a nice bonus though to protect against random short pattern matches.

@sipa

This comment has been minimized.

Copy link
Member

@sipa sipa commented May 19, 2014

What do we protect against? Accidental matches with overzealous byte sequence matchers, or intentional attempts to insert data that matches?

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented May 19, 2014

@sipa Both, really; people have put virus signatures in the block chain on purpose, and will likely see it as a challenge.

@jgarzik

This comment has been minimized.

Copy link
Contributor

@jgarzik jgarzik commented May 19, 2014

Both, by necessity

@dscotese

This comment has been minimized.

Copy link

@dscotese dscotese commented Jun 4, 2014

Sorry to dredge up this old thread, but I think the user should be allowed to tell the bitcoin client that one of its files needs to be retrieved from the network, perhaps in a from-random-peers way. So after the AV software breaks a file because it thought it was bad (which it may have been), the user can get that file again. This means adding a halt-and-repair-file function to the bitcoin client. The function would request the path to the bad file, request a new version from peers, and then use some blockchain data to validate the new file. It could then compare the new file with the old to tell the user whether the AV was a false positive (no difference) or the file had actually been altered.

I haven't (yet) thought about this long enough to see what problems it might cause, but I can't see any off the top of my head.

@leofidus

This comment has been minimized.

Copy link

@leofidus leofidus commented Jun 4, 2014

@dscotese You can already restore the file by running the Bitcoin Core with the -rescan option (you might have to delete the bad file first). I don't think anything more specific is required, and there are more important features to add.

Is somebody working on obfuscating the files?

@dscotese

This comment has been minimized.

Copy link

@dscotese dscotese commented Jun 4, 2014

@leofidus I see what you mean. I was under the mistaken impression that what my client was doing (right now, because I got that silly.218 problem) was retrieving the blockchain again. Actually, it's just re-indexing, but it is taking a few hours. If there were a protocol to get a chainstate file from someone else, then that would take a couple minutes, and then the client could do the same thing it does at every startup (validate all the data). That would take a few minutes instead of a few hours. If --rescan would accomplish that, then why wasn't it mentioned before? Laanwj suggested the --reindex was the best way to fix it once the .sst file was deleted.

@sipa

This comment has been minimized.

Copy link
Member

@sipa sipa commented Jun 4, 2014

-rescan is an option to scan the blockchain for missing wallet transactions. It is only needed in case there's (only) something wrong with the wallet, and since many versions ago almost always automatically detected. Unless people are manually editing their wallet files, -rescan is not a useful suggestions to give.

-reindex rebuilds the verification database, by processing the block files you have locally already as if they were received from network, redoing the validation.

Yes, you can copy that database from someone (just copy the chainstate/ directory, while the client is not running), but be aware that you must absolutely trust the person who is giving it to you (ideally: it is yourself), as they may make your client believe anything (up to and including making you believe you received payments that conjure coins out of thin air).

@whitslack

This comment has been minimized.

Copy link

@whitslack whitslack commented Jun 5, 2014

@dscotese: You can't just download a missing/corrupted chainstate file from someone else. The only units of data that are guaranteed to be identical across peers are the Bitcoin blocks themselves. The block chain data files, the block index files, the transaction index files, and the chainstate database files are all going to vary from one machine to the next, just due to the variation in the messages they've received from the network, their uptime, their particular schedule of software upgrades, etc. Your indices and chainstate files are not interchangeable with those of another peer.

Also, what @sipa said about being able to copy the entire database from another node is only true if the other node is the same architecture. You can't copy the database from an x86 to an ARM, for instance.

@rebroad

This comment has been minimized.

Copy link
Contributor

@rebroad rebroad commented Jun 10, 2014

In the same way that bitcoin leaves things to third party tools to rotate its log files, it can be left to third party tools (e.g. truecrypt) to encrypt the blockchain files.

Web caches also will contain CP - how do they deal with this?

@sipa

This comment has been minimized.

Copy link
Member

@sipa sipa commented Jun 10, 2014

Truecrypt will hide the data from someone who gets physical offline access to your harddrive.

It does not hide anything from running processes (how would you see the blockfiles in your OS, if it can't access them)?

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Jun 10, 2014

@rebroad How does that solve the issue here? If you use third party tools to encrypt the files, the virus scanner will still pick it up (at least if it is some kind of mounted fs). And requiring every user with a virus scanner to run a third-party tool is ridiculous anyway.

@ghost

This comment has been minimized.

Copy link
Author

@ghost ghost commented Jun 18, 2014

@rebroad, Web caches they simply exclude from scanning

@rebroad

This comment has been minimized.

Copy link
Contributor

@rebroad rebroad commented Jul 5, 2014

Ok, so the answer is to mention during setup that virus scanners need to exclude the directory containing the blockchain - ideally giving assistance to those less technically savy.

@ghost

This comment has been minimized.

Copy link
Author

@ghost ghost commented Jul 5, 2014

Personally I don't think that the answer is to document to exclude the directory (I would have offered it at the beginning if I did) but if others I agree that's fine by me - let's call it an answer and the issue can be closed.
It's more of a workaround than an answer and in many environments it's simply not possible to exclude (ignore) anything at will, for example my system is a corporate notebook on which I ocassionally run bitcoind (when I'm at home) and I'm not allowed to make any modifications to AV client settings.

@voisine

This comment has been minimized.

Copy link

@voisine voisine commented Jul 5, 2014

Hopefully including virus signatures in the blockchain wasn't part of some
nefarious plan to make it easier to hide bitcoin stealing malware on
windows desktops.

Aaron Voisine
breadwallet.com

On Sat, Jul 5, 2014 at 11:59 AM, rippler notifications@github.com wrote:

Personally I don't think that the answer is to document to exclude the
directory (I would have offered it at the beginning if I did) but if others
I agree that's fine by me - let's call it an answer and the issue can be
closed.
It's more of a workaround than an answer and in many environments it's
simply not possible to exclude (ignore) anything at will, for example my
system is a corporate notebook on which I ocassionally run bitcoind (when
I'm at home) and I'm not allowed to make any modifications to AV client
settings.


Reply to this email directly or view it on GitHub
#4069 (comment).

@dexX7

This comment has been minimized.

Copy link
Contributor

@dexX7 dexX7 commented Sep 6, 2015

@wyager: In fact, if this must be implemented, please keep this disabled by default. Either only build this into windows builds, or leave it as a checkbox (if checked by default, only on windows machines).

Just as slightly related side note: in July I received a report that ClamXav on OS X appears to cause DB curruptions as well. Based on the logs, the synchronization failed after block 294007 due to an error in the block header. I haven't verified the report, but appearingly the DB corruption was not a one-time event, and resolved after disabling the AV.

@jamesob jamesob mentioned this issue Sep 7, 2015
2 of 2 tasks complete
@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Oct 6, 2015

Should be fixed by #6650, which is merged fo r 0.12

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.