Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement find-satoshi command #20

Merged
merged 13 commits into from
Jan 6, 2022
Merged

Implement find-satoshi command #20

merged 13 commits into from
Jan 6, 2022

Conversation

casey
Copy link
Collaborator

@casey casey commented Dec 29, 2021

👀 @cberner

@cberner
Copy link
Contributor

cberner commented Dec 29, 2021

cberner/redb#107 should fix the data corruption in redb :)

@casey
Copy link
Collaborator Author

casey commented Dec 29, 2021

Updated to the corruption branch, and it does indeed, all tests pass!

@casey casey marked this pull request as ready for review December 29, 2021 06:26
@casey casey enabled auto-merge (squash) December 29, 2021 06:26
@casey
Copy link
Collaborator Author

casey commented Dec 29, 2021

Oh right, gotta figure out how to test this on CI. I guess we'll have to create some fake .blk files and commit them to the repository.

@casey
Copy link
Collaborator Author

casey commented Dec 31, 2021

I just updated to latest redb, and it is way faster! It is also crashing a little bit:

: RUST_BACKTRACE=1 cargo run --release find-satoshi 0 0
    Finished release [optimized] target(s) in 0.01s
     Running `target/release/bitcoin-atoms find-satoshi 0 0`
Inserted 119959 blocks…
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /Users/rodarmor/.cargo/git/checkouts/redb-24e44532b0b35edd/8a2d19d/src/tree_store/page_store/page_manager.rs:702:74
stack backtrace:
   0: rust_begin_unwind
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14
   2: core::panicking::panic
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:50:5
   3: redb::tree_store::page_store::page_manager::TransactionalMemory::allocate
   4: redb::tree_store::btree_utils::tree_insert_helper
   5: redb::tree_store::btree_utils::tree_insert_helper
   6: redb::tree_store::btree_utils::tree_insert_helper
   7: redb::tree_store::btree_utils::tree_insert_helper
   8: redb::tree_store::btree_utils::tree_insert
   9: redb::tree_store::storage::Storage::insert
  10: bitcoin_atoms::arguments::Arguments::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }', /Users/rodarmor/.cargo/git/checkouts/redb-24e44532b0b35edd/8a2d19d/src/tree_store/page_store/page_manager.rs:463:48
stack backtrace:
   0:        0x102f1b23c - std::backtrace_rs::backtrace::libunwind::trace::hd7ea89f96dbcea2f
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
   1:        0x102f1b23c - std::backtrace_rs::backtrace::trace_unsynchronized::h9b871e5dd80d4afa
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x102f1b23c - std::sys_common::backtrace::_print_fmt::hed76ee81d5e4a477
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:67:5
   3:        0x102f1b23c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8483effd3e9ec544
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:46:22
   4:        0x102f30eec - core::fmt::write::ha34752bcd39bca36
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/fmt/mod.rs:1163:17
   5:        0x102f18770 - std::io::Write::write_fmt::hfc57a6e48a9ddae2
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/io/mod.rs:1696:15
   6:        0x102f1cac8 - std::sys_common::backtrace::_print::h9e9cb382fe2b19e4
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:49:5
   7:        0x102f1cac8 - std::sys_common::backtrace::print::hc8209d6be1d0b41b
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:36:9
   8:        0x102f1cac8 - std::panicking::default_hook::{{closure}}::he9aaba83dd453f35
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:210:50
   9:        0x102f1c6a8 - std::panicking::default_hook::hc87ff103cfd8260f
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:227:9
  10:        0x102f1d108 - std::panicking::rust_panic_with_hook::hf890018b6cf19ef5
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:624:17
  11:        0x102f1cc54 - std::panicking::begin_panic_handler::{{closure}}::hc0918e64b5d68f06
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:521:13
  12:        0x102f1b6fc - std::sys_common::backtrace::__rust_end_short_backtrace::hb99d925d54d4af60
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:139:18
  13:        0x102f1cbbc - rust_begin_unwind
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5
  14:        0x102f3a42c - core::panicking::panic_fmt::hf9e85df46ac02a7d
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14
  15:        0x102f3a488 - core::result::unwrap_failed::h3b4d980f8618ad59
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/result.rs:1616:5
  16:        0x102ef45b4 - <redb::tree_store::page_store::page_manager::TransactionalMemory as core::ops::drop::Drop>::drop::h78c02efdbe176abd
  17:        0x102e9d170 - core::ptr::drop_in_place<redb::db::Database>::h977035e118987c7a
  18:        0x102ea0ba4 - bitcoin_atoms::arguments::Arguments::run::h1998cbb5cb04a2b6
  19:        0x102e9aafc - std::sys_common::backtrace::__rust_begin_short_backtrace::h8689b482946b236e
  20:        0x102e9b2d0 - std::rt::lang_start::{{closure}}::h8556eaf28075f894
  21:        0x102f1af98 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::haa2614d8b36c2daa
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/ops/function.rs:259:13
  22:        0x102f1af98 - std::panicking::try::do_call::hed608bbfbbf4bef8
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:403:40
  23:        0x102f1af98 - std::panicking::try::hbb4553a4f1bd94fc
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:367:19
  24:        0x102f1af98 - std::panic::catch_unwind::hfa517dc4dbf0fd9c
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panic.rs:133:14
  25:        0x102f1af98 - std::rt::lang_start_internal::{{closure}}::h07f4a8c2fba5706e
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/rt.rs:128:48
  26:        0x102f1af98 - std::panicking::try::do_call::h0f7528a4a32d01ca
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:403:40
  27:        0x102f1af98 - std::panicking::try::h3ae59fba589ee65b
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:367:19
  28:        0x102f1af98 - std::panic::catch_unwind::hc74e853193d89d19
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panic.rs:133:14
  29:        0x102f1af98 - std::rt::lang_start_internal::hc245c4266d131202
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/rt.rs:128:20
  30:        0x102e9aae4 - _main
thread panicked while panicking. aborting.
zsh: trace trap  RUST_BACKTRACE=1 cargo run --release find-satoshi 0 0

@cberner
Copy link
Contributor

cberner commented Dec 31, 2021

Could you have run out of space? The TODO on the line above that is ominous lol: // TODO: handle out-of-memory and return an error

You probably need about 3*(key_size * log(n) * n + value_size * n) bytes of space in the database where key_size & value_size are the average sizes, and n is the number of entries.

If not, good chance I have a memory leak. I already fixed one in the delete code path this week :)

@casey
Copy link
Collaborator Author

casey commented Jan 3, 2022

That was definitely it! I updated to latest master and got Error: OutOfSpace, and when I increase the database size 10x it works.

@casey casey merged commit 394fad3 into master Jan 6, 2022
@casey casey deleted the find-satoshi branch January 6, 2022 05:38
nothing0012 referenced this pull request in magicoss/ord-kafka Jun 13, 2023
# This is the 1st commit message:

Add kafka feature and stream

# The commit message #2 will be skipped:

# Add more fields in stream event

# The commit message #3 will be skipped:

# Add more fields in stream event

# The commit message #4 will be skipped:

# Add new_output_value

# The commit message #5 will be skipped:

# Gracefully handle owner issue

# The commit message #6 will be skipped:

# Add txid

# The commit message #7 will be skipped:

# Add more fields and consolidate the schema of the events

# The commit message #8 will be skipped:

# Update sat details name

# The commit message #9 will be skipped:

# Add sync producer flush

# The commit message #10 will be skipped:

# We probably only care about Text for now for the content body

# The commit message #11 will be skipped:

# Add STREM_HTTP_URL and also with kafka poll for extra care

# The commit message #12 will be skipped:

# Add auth header

# The commit message #13 will be skipped:

# Fix auth header typo

# The commit message #14 will be skipped:

# Use rustls

# The commit message #15 will be skipped:

# disable default reqwest features

# The commit message #16 will be skipped:

# disable default reqwest features

# The commit message #17 will be skipped:

# revert http client

# The commit message #18 will be skipped:

# Add block_hash for reorg

# The commit message #19 will be skipped:

# Rebase latest ord

# The commit message #20 will be skipped:

# graceful handle 0-value inscription

# The commit message #21 will be skipped:

# Add BLOCKS_BEHIND

# The commit message #22 will be skipped:

# Add BLOCKS_BEHIND

# The commit message #23 will be skipped:

# Change to use threadedproducer

# The commit message #24 will be skipped:

# Add brc20 indexing and versioning

# The commit message #25 will be skipped:

# Update dockerfile

# The commit message #26 will be skipped:

# Add sats domain

# The commit message #27 will be skipped:

# Fix clippy

# The commit message #28 will be skipped:

# Make domain generic

# The commit message #29 will be skipped:

# Fix name validation

# The commit message #30 will be skipped:

# Add tx_block_index

# The commit message #31 will be skipped:

# Add cursed handling of number and unbound satpoint

# The commit message #32 will be skipped:

# Add Dockerfile
yann-sjtu referenced this pull request in okx/ord Dec 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants