Publish on Docker Hub

[jimmysong]

I built the Docker image locally for v1.0 (master seems to end with an error). It would be great if you could create an account on Docker Hub so people can pull and run this easily instead of building from scratch.

BTW, great job! Super slick and works well with the bitcoind node I'm also running on Docker. If it's not too much to ask, maybe a docker-compose file with the right containers communicating with each other (including a separate nginx image) would be even better.

[wiz]

Hey @jimmysong !

Thanks, actually I've just registered the mempool organization on Docker Hub a week ago and we are planning to do it soon. I should have it published within the next few days.

[maltokyo]

Great stuff. I'll be keen to try when dockerised!

[wiz]

@jimmysong I sent you a DM on Keybase, but it's probably best if I explain here as well:

The V1 version of mempool is now published at https://hub.docker.com/r/mempool/mempool - can you please test it and let me know how it works? For V1, the only requirement is Bitcoin Core RPC, and has basic explorer functionality.

The V2 version of mempool, which has full explorer functionality (currently running on mempool.space), is not yet published on Docker Hub because the main issue is that it requires a fully indexed electrs instance and it seems Blockstream doesn't publish electrs as a standalone Docker image yet (only together with their esplora block explorer) so we would have to make our own Docker for this first. It's also quite beefy hardware requirements for running electrs - indexing it on a Raspberry Pi 4 would easily take a week or longer.

I'm going to leave this issue open until we figure out the V2 docker image and hear back from users that it works well for their setup. Also at some point we will need documentation for the Dockerfile configuration https://github.com/mempool/mempool/blob/master/Dockerfile

[jimmysong]

@wiz can confirm mempool/mempool:v1.0 works fine. I do wish I could just pull mempool/mempool and it would get me the latest.

I can't wait for v2.0! My own block explorer with a fee API is exactly what I would want in order to not have to depend on a third party.

[wiz]

Cool. For the "latest" tag built from our master branch (aka v2), I'll keep working towards getting this on Docker, but again the issue is having to run a very heavy electrs instance index of the entire blockchain, which requires over 1 TB of SSD storage and beefy hardware. Can you start preparing an electrs backend to run mempool V2 on your own hardware? Also I am waiting to hear back from Blockstream if they're going to publish electrs on their Docker Hub, and if so we can integrate with that image. And please come chat with us on Keybase 😉

[jimmysong]

I'll log an issue on Blockstream's electrs repo and see if they'll publish something. In the meantime, I'll build the image and start syncing.

Also, I don't do KeyBase. Happy to chat on TG, though. Same username as here.

[maltokyo]

Would it not be possible to use ElectrumX instead of Electrs? This is already fully dockerised and we have it running and integrated with BTCPay server. Please consider - and I'd be fully willing to help out including integration into BTCPay server if @NicolasDorier agrees!

[wiz]

@jimmysong I already pushed them internally and also published electrs on the mempool docker http://hub.docker.com/r/mempool/electrs

@maltokyo nah we need the blockstream fork

[jimmysong]

@wiz, the docker pull command seems to be missing the manifest. Maybe I need to wait?

[wiz]

Ah sorry you're right, the build status is still "in progress" so please try again in a few minutes. Also I don't have Telegram because I don't have a phone number, wanna DM me on Twitter?

[wiz]

It took over an hour to build but the docker image is finally available for mempool/electrs

[maltokyo]

Can this be also used as a standard Electrs implementation for electrum wallet?

[jimmysong]

@wiz, confirmed that the image is downloadable.

I have the .bitcoin directory mounted in my bitcoind docker, where do I mount that for electrs?

[wiz]

The documentation for electrs says default location is ~/.bitcoin - here is the full help

Electrum Rust Server 0.4.1

USAGE:
    electrs [FLAGS] [OPTIONS]

FLAGS:
        --disable-prevout    Don't attach previous output details to inputs
    -h, --help               Prints help information
        --jsonrpc-import     Use JSONRPC instead of directly importing blk*.dat files. Useful for remote full node or
                             low memory system
        --timestamp          Prepend log lines with a timestamp
    -V, --version            Prints version information
    -v                       Increase logging verbosity

OPTIONS:
        --bulk-index-threads <bulk_index_threads>
            Number of threads used for bulk indexing (default: use the # of CPUs) [default: 0]

        --cookie <cookie>
            JSONRPC authentication cookie ('USER:PASSWORD', default: read from ~/.bitcoin/.cookie)

        --cors <cors>                                Origins allowed to make cross-site requests
        --daemon-dir <daemon_dir>                    Data directory of Bitcoind (default: ~/.bitcoin/)
        --daemon-rpc-addr <daemon_rpc_addr>
            Bitcoin daemon JSONRPC 'addr:port' to connect (default: 127.0.0.1:8332 for mainnet, 127.0.0.1:18332 for
            testnet and 127.0.0.1:18443 for regtest)
        --db-dir <db_dir>                            Directory to store index database (default: ./db/)
        --electrum-rpc-addr <electrum_rpc_addr>
            Electrum server JSONRPC 'addr:port' to listen on (default: '127.0.0.1:50001' for mainnet, '127.0.0.1:60001'
            for testnet and '127.0.0.1:60401' for regtest)
        --electrum-txs-limit <electrum_txs_limit>
            Maximum number of transactions returned by Electrum history queries. Lookups with more results will fail.
            [default: 100]
        --http-addr <http_addr>
            HTTP server 'addr:port' to listen on (default: '127.0.0.1:3000' for mainnet, '127.0.0.1:3001' for testnet
            and '127.0.0.1:3002' for regtest)
        --index-batch-size <index_batch_size>
            Number of blocks to get in one JSONRPC request from bitcoind [default: 100]

        --monitoring-addr <monitoring_addr>
            Prometheus monitoring 'addr:port' to listen on (default: 127.0.0.1:4224 for mainnet, 127.0.0.1:14224 for
            testnet and 127.0.0.1:24224 for regtest)
        --network <network>                          Select Bitcoin network type (mainnet, testnet, regtest)
        --precache-scripts <precache_scripts>        Path to file with list of scripts to pre-cache
        --tx-cache-size <tx_cache_size>
            Number of transactions to keep in for query LRU cache [default: 10000]

[jimmysong]

I managed to get electrs docker to run, but it keeps panicking:

2020-03-30T21:17:23.166+00:00 - INFO - BlockchainInfo { chain: "main", blocks: 623660, headers: 623660, bestblockhash: "00000000000000000001a717380f6778770d09fbe5fcde731fd3aafe9a8ac840", pruned: false, initialblockdownload: false }
2020-03-30T21:17:23.167+00:00 - DEBUG - opening DB at "./db/mainnet"
2020-03-30T21:17:23.415+00:00 - TRACE - lastest indexed blockhash: 0000000000000000000000000000000000000000000000000000000000000000
2020-03-30T21:17:23.415+00:00 - INFO - listing block files at "/home/user/.bitcoin/blocks/blk*.dat"
2020-03-30T21:17:23.415+00:00 - INFO - indexing 0 blk*.dat files
2020-03-30T21:17:23.415+00:00 - DEBUG - found 0 indexed blocks
2020-03-30T21:17:23.418+00:00 - TRACE - downloading 100000 block headers
2020-03-30T21:17:26.489+00:00 - TRACE - downloading 100000 block headers
2020-03-30T21:17:33.214+00:00 - TRACE - downloading 100000 block headers
2020-03-30T21:17:45.080+00:00 - TRACE - downloading 100000 block headers
2020-03-30T21:17:48.026+00:00 - TRACE - downloading 100000 block headers
2020-03-30T21:17:51.570+00:00 - TRACE - downloading 100000 block headers
2020-03-30T21:17:54.794+00:00 - TRACE - downloading 23661 block headers
2020-03-30T21:17:57.199+00:00 - DEBUG - applying 623661 new headers from height 0
2020-03-30T21:17:57.344+00:00 - DEBUG - no more blocks to index
2020-03-30T21:17:57.345+00:00 - DEBUG - no more blocks to index
2020-03-30T21:17:57.345+00:00 - DEBUG - no more blocks to index
2020-03-30T21:17:57.345+00:00 - DEBUG - no more blocks to index
2020-03-30T21:17:57.345+00:00 - DEBUG - no more blocks to index
2020-03-30T21:17:57.345+00:00 - DEBUG - no more blocks to index
2020-03-30T21:17:57.345+00:00 - DEBUG - no more blocks to index
2020-03-30T21:17:57.345+00:00 - DEBUG - no more blocks to index
thread 'bulk_writer' panicked at 'no indexed header found', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/electrs-0.8.3/src/bulk.rs:66:27
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
2020-03-30T21:17:57.345+00:00 - TRACE - closing DB at "./db/mainnet"
thread 'main' panicked at 'writer panicked: Any', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/electrs-0.8.3/src/bulk.rs:244:5

[jimmysong]

ok this was a permission problem to the bitcoin data directory. it's no longer an issue.

[windsok]

@jimmysong could you share your docker-compose setup? I'm looking at doing something similar with a home node

[jimmysong]

I think my electrs instance is running and synced? What's next for 2.0?

[jimmysong]

@windsok I didn't use docker-compose, I used this:

$ docker create --name mempool -p 8999:80 -e BITCOIN_NODE_HOST=mainnet -e BITCOIN_NODE_USER=**** -e BITCOIN_NODE_PASS=**** --link mainnet:mainnet mempool/mempool:v1.0

[wiz]

@jimmysong just to give you an update, the reason this has been taking longer than originally expected is due to the advanced setup and configuration required to run 3 electrs instances with 3 fullnode instances, but I'm making progress and have a non-docker setup guide here https://github.com/mempool/mempool/tree/master/production

[wiz]

@jimmysong We finally shipped mempool v2.1 with docker support, and the images are published https://hub.docker.com/u/mempool

I will add some instructions to our repo's top-level README with an example docker-compose.yml for how to set this up shortly