Index getting bloated when the commit interval is set to 10 or some lower value

[krrish-sehgal]

Issue: High Storage Usage While Syncing Ordinals Index

System Specifications:

• RAM: 16GB
• Storage: 200GB

Command Used:

sudo /usr/local/bin/ord \
  --bitcoin-rpc-username=btc_rpc_user \
  --bitcoin-rpc-password=pass \
  --bitcoin-rpc-url=http://ip:8332 \
  --index-runes \
  --index /blockchain/ord-index/mainnet/index-0.22-without-878500.redb \
  --commit-interval 10 \
  --index-cache-size=15032385536 \
  server --http-port 9003

Context:

I am using a prebuilt index https://ordstuff.info/indexes/0.22/index-0.22-without-878500.redb.gz.torrent, which unpacks to around 61GB. After starting with this, I sync the remaining blocks. This approach is based on discussions regarding the looping issue (e.g., Issue #3804)

However, at around block 879,200, I encountered the same looping issue. To mitigate this, I set --commit-interval to 10. Upon reaching block 880,100, I noticed a significant increase in storage usage.

Question:

• The index file has grown significantly. Could the low commit interval be causing excessive disk usage?
• Given my 200GB storage limit, is there an optimal way to manage storage while ensuring the index syncs successfully?

[raphjaph]

The looping could come from your disk being full and then the fsync not being to write the data to disk. The large index size is definitely concerning and shouldn't happen. Could you post the output of ord index info?

[felipelincoln]

Upon reaching block 880,100, I noticed a significant increase in storage usage.

The same is happening to me.
Im running ord 0.22.2 on my own rust script with the flags --index-sats and --commit-interval=50 after block 800,000.

let index_options = Options::parse();
let index_settings = Settings::load(index_options).unwrap();
let (sender, receiver) = tokio::sync::mpsc::channel::<Event>(MAX_EVENTS);
let index = Arc::new(Index::open_with_event_sender(&index_settings, Some(sender)).unwrap());

but recently Im not able to complete the index as the index.redb file is reaching 1 TB and is consuming all my EBS volume from my AWS EC2 instance.

I restarted again the indexing but this time Ill try running ord index info before it crashes my instance.
@raphjaph any idea on what's going on here?

[krrish-sehgal]

The looping could come from your disk being full and then the fsync not being to write the data to disk. The large index size is definitely concerning and shouldn't happen. Could you post the output of ord index info?

Unfortunately due to resource usage constraints for my org , i had to turn off the ord server, so i cannot run the ord index info now.

So ,when the disk still had around 50-20GB left , the looping started and that was due to some block so i reduced the commit interval to maybe 10-50 from 500, and then when it took significant space but didn't exhaust complete disk space , i restarted the ord server with 500 interval.
(This happened like 4-5 times: looping , reduce commit interval, let the looping block get committed, revert the commit interval back to 500)

I am certain the lower commit interval was increasing the size of the index because i was constantly monitoring it thru df -h command, although i was able to run it with just 10Gb left on my VM thru the method I described above( it was brutal ,and needed constant monitoring for the looping block)