blkdat: Add Bitcoin blk.dat decoder

[wader]

TODO:
seems there are various verisons of the format?

[wader]

@mutatrum added inital script support also, maybe could even always decode every scriptsig but might be slow, could be optional.

$ go run fq.go -o line_bytes=10 '.[0].transactions[0].inputs[0].scriptsig | blkdat_script | d' format/bitcoin/testdata/genesis.dat
    │00 01 02 03 04 05 06 07 08 09│0123456789│.[0:3]: (blkdat_script)
    │                             │          │  [0]{}: optcode
0x82│   04                        │ .        │    opcode: "OP_PUSHSELF" (4)
0x82│      ff ff 00 1d            │  ....    │    arg: raw bits
    │                             │          │  [1]{}: optcode
0x82│                  01         │      .   │    opcode: "OP_PUSHSELF" (1)
0x82│                     04      │       .  │    arg: raw bits
    │                             │          │  [2]{}: optcode
0x82│                        45   │        E │    opcode: "OP_PUSHSELF" (69)
0x82│                           54│         T│    arg: raw bits
0x8c│68 65 20 54 69 6d 65 73 20 30│he Times 0│
0x96│33 2f 4a 61 6e 2f 32 30 30 39│3/Jan/2009│
*   │until 0xcf.7 (69)            │          │

$ go run fq.go -o line_bytes=10 -n '"4c4730440220206fea7b21dbda9db2227c2d028d4e014c02fa1f29775eccea41fe70aeedb5aa022047e8e8d2b7983baa84a48f85ea55edfdbc5aae7001d46e2e6657a8d86c0c60d6014c21033ca9b849063327597b13fa4da8c98053302118ffa4d194090d636ba22906d65776a91469dec09e9b32ffd447c80d413d58f0413e99208e88ac" | hex  | blkdat_script | dd'
    │00 01 02 03 04 05 06 07 08 09│0123456789│.[0:7]: (blkdat_script)
    │                             │          │  [0]{}: optcode
0x00│4c                           │L         │    opcode: "OP_PUSHDATA1" (76)
0x00│   47                        │ G        │    arg_length: 71
0x00│      30 44 02 20 20 6f ea 7b│  0D.  o.{│    arg: raw bits
0x0a│21 db da 9d b2 22 7c 2d 02 8d│!...."|-..│
0x14│4e 01 4c 02 fa 1f 29 77 5e cc│N.L...)w^.│
0x1e│ea 41 fe 70 ae ed b5 aa 02 20│.A.p..... │
0x28│47 e8 e8 d2 b7 98 3b aa 84 a4│G.....;...│
0x32│8f 85 ea 55 ed fd bc 5a ae 70│...U...Z.p│
0x3c│01 d4 6e 2e 66 57 a8 d8 6c 0c│..n.fW..l.│
0x46│60 d6 01                     │`..       │
    │                             │          │  [1]{}: optcode
0x46│         4c                  │   L      │    opcode: "OP_PUSHDATA1" (76)
0x46│            21               │    !     │    arg_length: 33
0x46│               03 3c a9 b8 49│     .<..I│    arg: raw bits
0x50│06 33 27 59 7b 13 fa 4d a8 c9│.3'Y{..M..│
0x5a│80 53 30 21 18 ff a4 d1 94 09│.S0!......│
0x64│0d 63 6b a2 29 06 d6 57      │.ck.)..W  │
    │                             │          │  [2]{}: optcode
0x64│                        76   │        v │    opcode: "OP_DUP" (118)
    │                             │          │  [3]{}: optcode
0x64│                           a9│         .│    opcode: "OP_HASH160" (169)
    │                             │          │  [4]{}: optcode
0x6e│14                           │.         │    opcode: "OP_PUSHSELF" (20)
0x6e│   69 de c0 9e 9b 32 ff d4 47│ i....2..G│    arg: raw bits
0x78│c8 0d 41 3d 58 f0 41 3e 99 20│..A=X.A>. │
0x82│8e                           │.         │
    │                             │          │  [5]{}: optcode
0x82│   88                        │ .        │    opcode: "OP_EQUALVERIFY" (136)
    │                             │          │  [6]{}: optcode
0x82│      ac│                    │  .│      │    opcode: "OP_CHECKSIG" (172)

[wader]

Thinking of renaming and splitting the formats into. bitcoin_blocks (just iterates bitcoin_block), bitcoin_block and bitcoin_script. Does that make sense? the block format is generic and not just for a specific bitcoin client?

Im not that familiar with bitcoin so any comments what names and format fields should have are welcome. Things like hex, oct, symbol mappings etc.

Also should scripts be decoded by default when decoding a block? it will probably slow down things and use memory. Hopefully soon there will be format options to control things like this

[mutatrum]

That's quite some progress! Decoding script should probably be optional, that's a different subset of tasks that one might be doing. Splitting between block and script makes a lot of sense.

As for naming/formats. some ideas:

block_header.time: timestamp, but don't know if timestamp fields normally are decoded by default;
block_header.previous_block_hash is normally shown as hex, same for txid and merkle_root;
transactions.outputs.value is an int;

What makes it tricky going further is a lot of things are done by reference. For example, the blocks are ordered by the previous_block_hash field. This points to sha256(sha256(block_header)) of the block before it. I would assume this might be achievable by keeping a list of hashes of blocks, and numbering the blocks accordingly. However, this gets really complicated when starting somewhere in the middle, or if it's traversing multiple files.

Similar for example for getting the value of a transaction. The outgoing values are specified, but incoming values are only retrievable by looking at the referenced input transactions.

A cross-reference tool that might come in handy is https://mempool.space/.

[mutatrum]

Some things I noticed, the 2nd output value (.block_header.previous_block_hash | hex) is in reverse order of what I expect, while the first argument (the raw hex of the blockheader), is in the correct order.

And - but way outside the scope of this issue - I'm confused why the array range is not working?

[Roasbeef]

If you're comfortable w/ the added dependency, then you can re-use the existing btcd libraries to handle:

• Wire format parsing: https://pkg.go.dev/github.com/btcsuite/btcd/wire#MsgBlock
• Script decompilation: https://pkg.go.dev/github.com/btcsuite/btcd/txscript#DisasmString

We also have an old tool that'll import an existing blk.dat field into btcd's database (a different format is used), it does so in a streaming manner: https://github.com/btcsuite/btcd/blob/master/cmd/addblock/import.go#L149-L181 + https://github.com/btcsuite/btcd/blob/master/cmd/addblock/import.go#L96-L100

[wader]

Some things I noticed, the 2nd output value (.block_header.previous_block_hash | hex) is in reverse order of what I expect, while the first argument (the raw hex of the blockheader), is in the correct order.

Not sure i follow, you mean for example previous_block_hash 6fe2.... is shown to be the bytes ....e26f? in the screenshot it has same?

Nice that someone else i poking around with the binary stuff in fq :) sorry it is a bit hard to follow, has been one of the hardest things to design to fit into jq... and i'm still not that happy with it, so feedback is very welcome.

And - but way outside the scope of this issue - I'm confused why the array range is not working?

Slicing will return an array so try do .[0:10][] | ...

[wader]

@mutatrum @Roasbeef Thanks for the feedback and links, will be very useful. It usually a bit hard to use other decoders directly in fq as it want to keep track of bit position, but will be a good references to peek at, also will probably copy the base58 thingy.

@mutatrum interesting comment about going further and to more interesting things. haven't look closer at it but for some other formats i've added format specific jq functions to do more high level things and then let the go decoder just be a "dumb" decoder. Eg the sqlite3 decoder (has not been merged yet before of some other issues) has these helpers https://github.com/wader/fq/blob/7c0662e80eebd6e7bf82e302690bb566cf2da005/format/sqlite3/sqlite3.jq i wonder if that could work for this?

[wader]

Did some updates, split into separate decoders and fixed some nicer presentation. Also think i know why the bytes were reversed now, from https://learnmeabitcoin.com/technical/txid

Due to historical accident, the tx and block hashes that bitcoin core uses are byte-reversed. I’m not entirely sure why. May be something like using openssl bignum to store hashes or something like that, then printing them as a number. – Wladimir van der Laan (Bitcoin Core developer)

😄

A bit of a bitcoin noob, how would one get the in/out addresses, have to run the script and look at the output?

[Roasbeef]

A bit of a bitcoin noob, how would one get the in/out addresses, have to run the script and look at the output?

So the output script is in the output itself, typically called a pkScript, or scriptPubKey. There're a few standardized pkScripts that are commonly mapped to addresses. You can use this function to extract the known address for a given script.

Input scripts are the same, but a transaction references the previous input by it's txid:output_index tuple. In order to get this script efficiently, an index is required that maps the txid to the transaction itself. Once you have the transaction, you can locate the output by the referenced index, and use the function above to extract the addresses.

[wader]

@Roasbeef Thanks for the info, still a bit confused but that gave me a good starting point to understand it better! hope i can get back to getting the decoder into a merge state. I notice that i have to take care of coinbase transcations also as it seems the script are just some opaque data in those? are there more special cases?

@mutatrum About sha256, i have some vague future plan to add more/cleanup crypto and hashing stuff in fq, for example it already has md5 and aes_ctr but mostly for experimenting with api design. Maybe should be called tomd5 and would be nice if block ciphers config were more flexible.

[Roasbeef]

I notice that i have to take care of coinbase transcations also as it seems the script are just some opaque data in those? are there more special cases?

So coinbases are special since the previous input reference (prev out) is actually just all zeroes, and it can only have a single input. In terms of outputs, usually there's a normal pay out address for the output. With segwit, a prior update to Bitcoin, it can also contain a commitment to some special data (a merkle tree of all the signature/witness data).

One thing people do with the coinbase inputs is sort of use them as block space advertisement or a special marker. IN this case it might not actually be a real script.

[wader]

@mutatrum Hi again, did a rebase, some small fixes and cleanups. I'm a bit busy with some other things (some other fq things!), but do you think what's in this PR now is useful enough to merge?
To build more useful thing on top of this (as jq scripts ex) i guess some more hash/crypto functions would be useful?

[wader]

@mutatrum hey, worth include current state in next release?

[mutatrum]

@mutatrum hey, worth include current state in next release?

I think it definitely is. I'm testing it, and it looks good. Added one comment.