Implement find-satoshi command #20

casey · 2021-12-29T01:53:20Z

cberner · 2021-12-29T06:18:05Z

cberner/redb#107 should fix the data corruption in redb :)

casey · 2021-12-29T06:25:58Z

Updated to the corruption branch, and it does indeed, all tests pass!

casey · 2021-12-29T06:30:05Z

Oh right, gotta figure out how to test this on CI. I guess we'll have to create some fake .blk files and commit them to the repository.

casey · 2021-12-31T01:36:31Z

I just updated to latest redb, and it is way faster! It is also crashing a little bit:

: RUST_BACKTRACE=1 cargo run --release find-satoshi 0 0
    Finished release [optimized] target(s) in 0.01s
     Running `target/release/bitcoin-atoms find-satoshi 0 0`
Inserted 119959 blocks…
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /Users/rodarmor/.cargo/git/checkouts/redb-24e44532b0b35edd/8a2d19d/src/tree_store/page_store/page_manager.rs:702:74
stack backtrace:
   0: rust_begin_unwind
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14
   2: core::panicking::panic
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:50:5
   3: redb::tree_store::page_store::page_manager::TransactionalMemory::allocate
   4: redb::tree_store::btree_utils::tree_insert_helper
   5: redb::tree_store::btree_utils::tree_insert_helper
   6: redb::tree_store::btree_utils::tree_insert_helper
   7: redb::tree_store::btree_utils::tree_insert_helper
   8: redb::tree_store::btree_utils::tree_insert
   9: redb::tree_store::storage::Storage::insert
  10: bitcoin_atoms::arguments::Arguments::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }', /Users/rodarmor/.cargo/git/checkouts/redb-24e44532b0b35edd/8a2d19d/src/tree_store/page_store/page_manager.rs:463:48
stack backtrace:
   0:        0x102f1b23c - std::backtrace_rs::backtrace::libunwind::trace::hd7ea89f96dbcea2f
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
   1:        0x102f1b23c - std::backtrace_rs::backtrace::trace_unsynchronized::h9b871e5dd80d4afa
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x102f1b23c - std::sys_common::backtrace::_print_fmt::hed76ee81d5e4a477
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:67:5
   3:        0x102f1b23c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8483effd3e9ec544
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:46:22
   4:        0x102f30eec - core::fmt::write::ha34752bcd39bca36
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/fmt/mod.rs:1163:17
   5:        0x102f18770 - std::io::Write::write_fmt::hfc57a6e48a9ddae2
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/io/mod.rs:1696:15
   6:        0x102f1cac8 - std::sys_common::backtrace::_print::h9e9cb382fe2b19e4
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:49:5
   7:        0x102f1cac8 - std::sys_common::backtrace::print::hc8209d6be1d0b41b
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:36:9
   8:        0x102f1cac8 - std::panicking::default_hook::{{closure}}::he9aaba83dd453f35
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:210:50
   9:        0x102f1c6a8 - std::panicking::default_hook::hc87ff103cfd8260f
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:227:9
  10:        0x102f1d108 - std::panicking::rust_panic_with_hook::hf890018b6cf19ef5
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:624:17
  11:        0x102f1cc54 - std::panicking::begin_panic_handler::{{closure}}::hc0918e64b5d68f06
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:521:13
  12:        0x102f1b6fc - std::sys_common::backtrace::__rust_end_short_backtrace::hb99d925d54d4af60
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:139:18
  13:        0x102f1cbbc - rust_begin_unwind
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5
  14:        0x102f3a42c - core::panicking::panic_fmt::hf9e85df46ac02a7d
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14
  15:        0x102f3a488 - core::result::unwrap_failed::h3b4d980f8618ad59
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/result.rs:1616:5
  16:        0x102ef45b4 - <redb::tree_store::page_store::page_manager::TransactionalMemory as core::ops::drop::Drop>::drop::h78c02efdbe176abd
  17:        0x102e9d170 - core::ptr::drop_in_place<redb::db::Database>::h977035e118987c7a
  18:        0x102ea0ba4 - bitcoin_atoms::arguments::Arguments::run::h1998cbb5cb04a2b6
  19:        0x102e9aafc - std::sys_common::backtrace::__rust_begin_short_backtrace::h8689b482946b236e
  20:        0x102e9b2d0 - std::rt::lang_start::{{closure}}::h8556eaf28075f894
  21:        0x102f1af98 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::haa2614d8b36c2daa
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/ops/function.rs:259:13
  22:        0x102f1af98 - std::panicking::try::do_call::hed608bbfbbf4bef8
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:403:40
  23:        0x102f1af98 - std::panicking::try::hbb4553a4f1bd94fc
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:367:19
  24:        0x102f1af98 - std::panic::catch_unwind::hfa517dc4dbf0fd9c
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panic.rs:133:14
  25:        0x102f1af98 - std::rt::lang_start_internal::{{closure}}::h07f4a8c2fba5706e
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/rt.rs:128:48
  26:        0x102f1af98 - std::panicking::try::do_call::h0f7528a4a32d01ca
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:403:40
  27:        0x102f1af98 - std::panicking::try::h3ae59fba589ee65b
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:367:19
  28:        0x102f1af98 - std::panic::catch_unwind::hc74e853193d89d19
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panic.rs:133:14
  29:        0x102f1af98 - std::rt::lang_start_internal::hc245c4266d131202
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/rt.rs:128:20
  30:        0x102e9aae4 - _main
thread panicked while panicking. aborting.
zsh: trace trap  RUST_BACKTRACE=1 cargo run --release find-satoshi 0 0

cberner · 2021-12-31T04:42:28Z

Could you have run out of space? The TODO on the line above that is ominous lol: // TODO: handle out-of-memory and return an error

You probably need about 3*(key_size * log(n) * n + value_size * n) bytes of space in the database where key_size & value_size are the average sizes, and n is the number of entries.

If not, good chance I have a memory leak. I already fixed one in the delete code path this week :)

casey · 2022-01-03T00:33:17Z

That was definitely it! I updated to latest master and got Error: OutOfSpace, and when I increase the database size 10x it works.

# This is the 1st commit message: Add kafka feature and stream # The commit message #2 will be skipped: # Add more fields in stream event # The commit message #3 will be skipped: # Add more fields in stream event # The commit message #4 will be skipped: # Add new_output_value # The commit message #5 will be skipped: # Gracefully handle owner issue # The commit message #6 will be skipped: # Add txid # The commit message #7 will be skipped: # Add more fields and consolidate the schema of the events # The commit message #8 will be skipped: # Update sat details name # The commit message #9 will be skipped: # Add sync producer flush # The commit message #10 will be skipped: # We probably only care about Text for now for the content body # The commit message #11 will be skipped: # Add STREM_HTTP_URL and also with kafka poll for extra care # The commit message #12 will be skipped: # Add auth header # The commit message #13 will be skipped: # Fix auth header typo # The commit message #14 will be skipped: # Use rustls # The commit message #15 will be skipped: # disable default reqwest features # The commit message #16 will be skipped: # disable default reqwest features # The commit message #17 will be skipped: # revert http client # The commit message #18 will be skipped: # Add block_hash for reorg # The commit message #19 will be skipped: # Rebase latest ord # The commit message #20 will be skipped: # graceful handle 0-value inscription # The commit message #21 will be skipped: # Add BLOCKS_BEHIND # The commit message #22 will be skipped: # Add BLOCKS_BEHIND # The commit message #23 will be skipped: # Change to use threadedproducer # The commit message #24 will be skipped: # Add brc20 indexing and versioning # The commit message #25 will be skipped: # Update dockerfile # The commit message #26 will be skipped: # Add sats domain # The commit message #27 will be skipped: # Fix clippy # The commit message #28 will be skipped: # Make domain generic # The commit message #29 will be skipped: # Fix name validation # The commit message #30 will be skipped: # Add tx_block_index # The commit message #31 will be skipped: # Add cursed handling of number and unbound satpoint # The commit message #32 will be skipped: # Add Dockerfile

brc20 rpc

casey added 3 commits December 28, 2021 15:35

Find satoshi 0

066e44d

Working on indexing block heights, broken

fd35ad2

Implement find-satoshi command

4c0d7fb

Update to fixed redb

7fd2750

casey marked this pull request as ready for review December 29, 2021 06:26

casey enabled auto-merge (squash) December 29, 2021 06:26

Update to latest redb

153ebb0

casey and others added 8 commits January 5, 2022 19:06

Update redb and increase database size

18ea48c

Add blockdirs command line argument

8e89afb

Update .gitignore to ignore Intellij metadata

29bbafd

Rename blockdir variable to blockfile

d215cec

Rename a bunch of variables

b52044b

Use released version of redb

4c4f0f1

Fix clippy error

f0159c7

rustfmt

97d1d43

casey merged commit 394fad3 into master Jan 6, 2022

casey deleted the find-satoshi branch January 6, 2022 05:38

cberner mentioned this pull request Feb 14, 2022

Overhead expecations cberner/redb#171

Closed

yann-sjtu referenced this pull request in okx/ord Dec 18, 2023

Merge pull request #20 from okx/wc/rpc-dev

e635ac4

brc20 rpc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement find-satoshi command #20

Implement find-satoshi command #20

casey commented Dec 29, 2021

cberner commented Dec 29, 2021

casey commented Dec 29, 2021

casey commented Dec 29, 2021

casey commented Dec 31, 2021

cberner commented Dec 31, 2021

casey commented Jan 3, 2022

Implement find-satoshi command #20

Implement find-satoshi command #20

Conversation

casey commented Dec 29, 2021

cberner commented Dec 29, 2021

casey commented Dec 29, 2021

casey commented Dec 29, 2021

casey commented Dec 31, 2021

cberner commented Dec 31, 2021

casey commented Jan 3, 2022