Skip to content
Find file
Fetching contributors…
Cannot retrieve contributors at this time
executable file 323 lines (273 sloc) 16.2 KB
################################################################################
# #
# Copyright (C) 2011, Alan C. Reiner <alan.reiner@gmail.com> #
# Distributed under the GNU Affero General Public License (AGPL v3) #
# See LICENSE or http://www.gnu.org/licenses/agpl.html #
# #
################################################################################
********************************************************************************
*
* Project: PyBtcEngine
* Author: Alan Reiner
* Date: 11 July, 2011
* Donate: 1Gffm7LKXcNFPrtxy6yF4JBoe5rVka4sn1
*
* Descr: PyBtcEngine is a full computational backend for Bitcoin software.
* It combines C++ and Python to enable *all* the functionality
* necessary to write Bitcoin software, *except for networking*.
* The networking is more application-oriented, whereas this library
* is more focused on enabling you to read/interpret/construct the
* packets you would be exchanging on the Bitcoin network.
*
* For a demo of all the features of the library, you can see the
* PyQt block-explorer I made in the pyqt directory. If it doesn't
autodetect your blkfile, you can provide it as a CLI argument
*
* This library was originally based on Sam Rushing "caesure" code,
* but is a complete rewrite and no longer resembles or includes any
* of the code he wrote. So, I have removed his license/copyright.
* Of course, he still gets a ton of credit for providing such good
* reference code. "Lis" from the Bitcoin forums also deserves a ton
* of credit for writing a fantastic, dependency-less version of ECDSA.
* This may enable more platforms where this code can be used, as they
* don't need to have any particular crypto libraries installed
*
* Note: I am not a "real" programmer, I'm an engineer that writes
* algorithms to solve problems really quickly. There's a few
* places I should've used inheritance but didn't. And although
* I usually am good about const correctness, it turned into a
* nightmare here, so I've mostly removed it
*
********************************************************************************
********************************************************************************
* Please take a moment to donate 1.0 BTC! 1Gffm7LKXcNFPrtxy6yF4JBoe5rVka4sn1
********************************************************************************
********************************************************************************
* STATUS: Last Updated - 28 Oct, 2011
* Legend:
* _ not implemented
* . implemented but not tested
* + implemented and partially tested
* X implemented and tested
*
* C++ Python SWIG
* ---------------------------------------------------------------
* (01) Ser/Unser Block Objects X X X
* (02) Hash160/Hash256 X X X
* (03) Difficulty calcs X X X
* (04) Address Generation X X
* (05) Address Verify/Manip X X
* ---------------------------------------------------------------
* (06) BlkHeaders read/scan/org X X X
* (07) BlkHeaders reorgs X X
* (08) Blockchain read/scan/org X X
* (09) Blockchain reorgs X X
* (10) Blockchain verify integrity X X
* ---------------------------------------------------------------
* (11) NonStd Tx Detection + + +
* (12) Arbitrary script parsing X X X
* (13) OP_CHECKSIG/CHECKMULTISIG X X X
* (14*) Arbitrary script eval X X
* (15) ECDSA Sign/Verify X X X
* ---------------------------------------------------------------
* (16) Address/Wallet tracking X X X
* (17) Scan blkchain for Tx X X
* (18) Scan blkchain for NonStd X X
* (19) Reorg w/ double-spend X X
* (20) Add new blockdata real-time X X
* ---------------------------------------------------------------
(20) SelectCoins for tx X X
* (21) Tx construct given inputs X X
* (22*) Distr Proposals/multisig
* (23) Tx broadcast
* (24) Tx fee calculations X X
* (25) Blockchain download
*
*
*
*
* Notes:
* (09) I have THOROUGHLY tested the blockchain reorg code. and making
* sure everything looks exactly right at each step. After a reorg,
* invalidated Tx's show up in the address ledger with a flag
* that identifies it has been removed. This is really important in
* the event of oddities in the blockchain -- normally the transaction
* would just disappear without evidence it ever existed, which can
* cause a lot of user confusion.
*
* (11) Is "mostly" tested because there are a few TxIn scripts that I fail
* to identify as "standard." This is because it appears that a few
* TxIns (less than 50 of 1,500,000) have a couple extra bytes in the
* DER signature that throws me off. I'm not sure whether I need to
* turn around and reimplement the signauture reading based on a real
* DER library, or just hack it to accept extra 0x00 bytes at the end.
*
* (14*) I have tested a variety of very complicated scripts, mostly from
* the testnet, including OP_CHECKSIG and OP_CHECKMULTISIG. Both the
* C++ and python code can parse all these scripts, and the python
* code can execute these scripts successfully. HOWEVER, I have
* NOT implemented the OP_IF/OP_NOTIF/OP_ELSE/OP_ENDIF op-codes.
* These codes require a bit of work to get right, and not currently
* part of any near-term isStandard() transaction types.
*
*
* (22*) In anticipation of supporting multi-signature transactions, I have
* created a specification for handling multi-sig transactions, which
* also happens to work well for offline wallets/transactions. I
* have called these "Distribution Proposals". See the following BIP:
* https://gist.github.com/1321518
*
*
* More Notes:
* FullRAM implementation only in C++ code. However,
* modern computers can handle the full blockchain in RAM right now, and
* probably for the next year, so there's no hurry to do this.
* Short/Medium-term future:
* *** NETWORKING & BLOCKCHAIN DOWNLOAD ***
* Deterministic address generation
* Wallet serialization/storage to file
* Convert wallet files to/from official BTC wallets (pywallet)
*
********************************************************************************
********************************************************************************
For information on dependencies and compiling, see Using_PyBtcEngine.README
********************************************************************************
#####
28 Oct, 2011
*** Implemented SelectCoins and started "Tx Distribution Proposals."
Actually spent a bit of time writing the following specification instead
of coding:
https://gist.github.com/1321518
I figured that ironing out a good way to implement it before I actually do,
would be a good investment of time.
Fantastically, this method of dealing with multi-signature transactions,
also works surprisingly well for offline transactions: you can use the
exact same mechanism, as a 1-of-1 tx that needs to be signed. The way
the TxDP works, this can be done by an offline computer without access to
the blockchain, and thus only needs the ECDSA private keys.
The only problem is, this complicates my TxDP code, because most TxDPs will
only involve signing a single TxIn, whereas this is a "regular" transaction
which will potentially have multiple TxIns. It's going to get done, but it
is extra complexity...
18 Oct, 2011
*** Working blockchain reorgs with double-spend detection, GUI works in 10.04
I added a createTestChain.py script that can be modified to create any kind
of difficulty-1 blockchain, starting with the actual genesis block. I have
used it to create a reorg-double-spend test, and crushed some pretty heavy
bugs in that code. Next I will be working on the Tx/Ledger handling, so
that any transactions that were, at any point, marked valid, will remain in
the DB/memorypool for some amount of time, to let the user know what
happened.
Along the way, also had to iron out my tx-construction code in python. It's
a complete pain in the ass (endianness is a bitch), but it's finally done
now and the python code can construct Txs from a list of Addr-Txout-pairs
and a list of recipient-value-pairs. (I do not have TxOut selection
implemented, but it should be straightforward compared to the ECDSA and
OP_CHECKSIG stuff).
#####
06 Oct, 2011
*** A working block-explorer demo in PyQt4! (Linux/Ubuntu, only, right now)
Got enough of the interface working in SWIG to be able to create a full
blockchain explorer. It does not include anything to do with addresses,
besides identifying the addresses involved in the particular block you
are browsing. In the immediate future I plan to include a separate
tab/view for doing address searches and display.
I had implemented a feature to auto-update the GUI in realtime if you
are simultaneously running the BTC client and it updates blk0001.dat.
Unfortunately, there's a bug in the addBlockData() method which causes
a segfault. This feature will be back soon, but I think I need the help
of MSVS to thoroughly test/debug addBlockData().
#####
25 Sep, 2011:
*** Address scanning, Tx ledgers, wallet balance, etc, ADDED
The directory cppForSwig contains a C++ implementation of the block objects
with everything that doesn't require a bignum library. The goal was to
create C++ code that could be accessed via SWIG/python to do the heavy
lifting, since python isn't very good at doing it quickly/efficiently.
At the moment, the BlockDataManager_FullRAM will store the entire blockchain
in RAM, and do all calculations on it there. I anticipate in the future, to
convert this to a file-based approach with an initial scan.
Below is a list of the operations the C++ code supports, as well as timings
based on my decent AMD Phenom II X4 840 CPU and WD 1TB HDD, using a file
containing the blockchain up to about block 145,000: 600 MB.
-- Reading blockchain from disk: 5.4s
-- Populating header and tx maps: 10.2s
-- Organizing headers into chain: 0.6s
-- Scan blkchain for 3 addrs' tx: 2.9s
Note that these operations are all from scratch, meaning you don't need to
keep any other files on your HDD except for your address list, and the
client could start up, read the entire blkchain file, and find all of your
transactions and compute your balances from scratch, in under 30s. I'm
very pleased with these results! (NOTE: This does not involve any kind
of verification, or crypto operations).
Some additional features:
-- Will maintain a tx ledger for each address individually, as well
as for the entire wallet as a whole (this should aggregate multiple
TxIn objects in a single Tx into a single ledger entry, though I
haven't tested this specific feature yet).
-- The ledgers can be sorted by order in the blockchain
-- A separate list of your TxOut-TxIn pairs provides a simple way
to access your unspent TxOuts, and prune off TxOuts that have
already been spent
-- The organize-blockchain should detect reorganizations and invalid
blocks, but does not yet do anything other than return a flag
I will eventually go through and populate the previouslyValid list.
-- The basics of the SWIG interface are in place: I've made sure that
std::strings and std::vectors are handled properly and that the
blockchain reading works from python. But that was a few commits
ago and the interface needs to be updated.
#####
20 Aug, 2011:
PyBtcEngine is a finally a fairly complete set of tools that addresses most
of the calculations, manipulations and crypto that is involved in analyzing
the blockchain. It's not particularly fast, which is why I started creating
a C++ backend to handle all the block data, to be imported via SWIG. This
is not yet complete, but believe will be the most efficient method possible
to process the entire blockchain at once.
Due to many requests on the forum, as well as personal desire to identify
large chunks of binary/hex data, I have created a separate script called
mysteryHex.py which will try to completely break down a chunk of data and
find BTC-related strings in it, using PyBtcEngine. It should be a fairly
versatile tool, allowing for quick ID of headers, transactions, public keys,
etc. Also allows the user to read their blockfile store to construct a
dictionary of 1-2 million hashes to include in the block file identification
(use -s to enable hash searching from knownHashes.bin, and -u to create or
update your knownHashes.bin file from blk0001.dat in your BTC directory).
In fact, here's the output of ./mysteryHex.py --help:
Try to identify Bitcoin-related strings in a block of data
Usage: mysteryHex.py [--binary|-b] -f FILE
or: mysteryHex.py unidentifiedHex
Options:
-h, --help show this help message and exit
-f FILENAME, --file=FILENAME
Get unidentified data from this file
-k BLK0001FILE, --blkfile=BLK0001FILE
Update hashlist from this file (default
~/.bitcoin/blk0001.dat)
-g HASHFILE, --hashfile=HASHFILE
The file to store and retrieve header/tx hashes
-b, --binary Specified file is in binary
-s, --usehashes Import header/tx hashes to be used in searching
-u, --updatehashes Search blk0001.dat to update hashlist
-r, --rescanhashes Rescan blkfile for header/tx hashes
#####
16 July, 2011:
After a ton of work, I have finally completed much of the calculation engine!
All number/binary conversions work, with easy endianness swapping.
Complete, recursive serialization and unserialization of all block/tx data
MerkleTree calculation and verification done
Cryptography library from "Lis" wrapped and tested on arbitrary data
Most script OP_CODEs implemented, and OP_CHECKSIG works on example tx data!
Basically, everything else that I need to do (besides networking), should
be very easy to implement. With all the endianness, serialization and
crypto methods tested, packet construction, block verification, data
structures, and crypto anything should be a breeze.
Of course, this does not include networking. The networking engine is going
to be a whole different beast... but luckily Sam's code already has a lot of
that written. When it comes to networking, I might just stick to using other
people's code, instead of re-writing it in my own way.
Some immeidate things that I need to do:
SIGHASH_ALL/NONE/SINGLE/ANYONCANPAY
Wallet/Account file formats and implementation
Long term:
Networking engine (ugh)
Jump to Line
Something went wrong with that request. Please try again.