Latest blocks not shown

[Derrick-]

Mempool v2.4.0
Bitcoin Core version v22.0.0
Fulcrum 1.7.0 (compiled: gcc 7.5.0)

Transactions are shown, Blocks are shown correctly in top right, but "Latest blocks" is empty on dashboard, mining and /blocks page

Block details do not show transactions, but transaction do show which block they were confirmed in. Other block details are missing as well.

[softsimon]

• In what environment are you running. Custom installation on a Linux machine?
• What is the frontend HTTP error response for fetching the block data
• Can you see the backend NodeJS log for errors

[Derrick-]

Operating System: Debian GNU/Linux 11 (bullseye)
Kernel: Linux 5.10.0-14-amd64

Yes, core and Fulcrum installed on host Mempool running in three docker containers

I do not see any HTTP errors in dashboard, blocks or mining page

I didn't not think to look at the response data on the pages. It does look like blocks (xhr) is returning the data, it's just not displaying:

[{"extras":{"reward":645776327,"coinbaseTx":{"vin":[{"scriptsig":"03cf9a0b315c20506f7765726564206279204c75786f722054656368205c0000000036dee631409cd1cc009f16076701000000000000"}],"vout":[{"scriptpubkey_address":"1MkCDCzHpBsYQivp8MxjY5AkTGG1f2baoe","value":645776327}]},"coinbaseRaw":"03cf9a0b315c20506f7765726564206279204c75786f722054656368205c0000000036dee631409cd1cc009f16076701000000000000","medianFee":17,"feeRange":[1,13,13,17,19,24,1002],"totalFees":20776327,"avgFee":10080,"avgFeeRate":20,"matchRate":96},"id":"0000000000000000000702936ac05a2db125ae18475743ddfb5f8a8adf427f21","height":760527,"version":800333824,"timestamp":1666879917,"bits":386376745,"nonce":2666957403,"difficulty":36835682546787.98,"merkle_root":"b5227898ef30169e15f69c59c51e3767f254da970046b91b72c31a799452f75c","tx_count":2062,"size":1297676,"weight":3992861,"previousblockhash":"0000000000000000000762f8acee42a6faad461b670a3097e4432f171ec21e28"},{"extras":{"reward":646295590,"coinbaseTx":{"vin":[{"scriptsig":"03ce9a0b1b4d696e656420627920416e74506f6f6c383732830001010f335b4afabe6d6d74fbd62e76d2830a4431585f...

Using Brave browser. I did not have "Shields" turned off, but I do now, no difference.
Tested now in Chrome and Edge, same results.

I do not know how to view the NodeJS logs on the backend container.

[Derrick-]

Here are my config and log files:

docker-compose.yml

version: "3.7"

services:
web:
environment:
FRONTEND_HTTP_PORT: "8080"
BACKEND_MAINNET_HTTP_HOST: "api"
image: mempool/frontend:latest
user: "1000:1000"
restart: on-failure
stop_grace_period: 1m
command: "./wait-for db:3306 --timeout=720 -- nginx -g 'daemon off;'"
ports:
- 80:8080
api:
environment:
MEMPOOL_BACKEND: "electrum"
ELECTRUM_HOST: "172.17.0.1" #"172.27.0.1"
ELECTRUM_PORT: "50001"
ELECTRUM_TLS_ENABLED: "false"
CORE_RPC_HOST: "172.17.0.1"
CORE_RPC_PORT: "8332"
CORE_RPC_USERNAME: "mempool"
CORE_RPC_PASSWORD: "mempool"
DATABASE_ENABLED: "true"
DATABASE_HOST: "db"
DATABASE_DATABASE: "mempool"
DATABASE_USERNAME: "mempool"
DATABASE_PASSWORD: "mempool"
STATISTICS_ENABLED: "true"
image: mempool/backend:latest
user: "1000:1000"
restart: on-failure
stop_grace_period: 1m
command: "./wait-for-it.sh db:3306 --timeout=720 --strict -- ./start.sh"
volumes:
- ./data:/backend/cache
db:
environment:
MYSQL_DATABASE: "mempool"
MYSQL_USER: "mempool"
MYSQL_PASSWORD: "mempool"
MYSQL_ROOT_PASSWORD: "admin"
image: mariadb:10.5.8
user: "1000:1000"
restart: on-failure
stop_grace_period: 1m
volumes:
- ./mysql/data:/var/lib/mysql

bitcoin.conf

rpcuser=mempool
rpcpassword=mempool
rpcallowip=127.0.0.1
rpcallowip=172.0.0.0/8
rpcbind=127.0.0.1
rpcbind=172.17.0.1
txindex=1
server=1

Backend container logs: (Are these the NodeJS logs you asked about?)

git not found, cannot parse git hash
Oct 25 14:53:33 [84] INFO: Connected to Electrum Server at 172.17.0.1:50001 (["Fulcrum 1.7.0","1.4"])
Oct 25 14:53:33 [84] INFO: Database connection established.
Oct 25 14:53:33 [84] INFO: Starting statistics service
Oct 25 14:53:33 [84] INFO: Starting currency rates service
Oct 25 14:53:33 [84] INFO: Currency rates service will be queried over clearnet using https://price.bisq.wiz.biz/getAllMar>
Oct 25 14:53:33 [84] INFO: Updating latest mining pools from Github
Oct 25 14:53:33 [84] INFO: List of public pools will be queried over clearnet
Oct 25 14:54:10 [84] INFO: Indexing 9992 blocks from #760249 to #750250
Oct 25 15:13:33 [84] INFO: Disconnected from Electrum Server at 172.17.0.1:50001
Oct 25 15:13:43 [84] INFO: Connected to Electrum Server at 172.17.0.1:50001 (["Fulcrum 1.7.0","1.4"])
Oct 25 15:22:27 [84] INFO: Indexing 1000 blocks from #750249 to #749250
Oct 25 15:25:02 [84] INFO: Indexed 10992 blocks
Oct 25 15:25:02 [84] INFO: Some blocks are not indexed, skipping missing blocks during chain validation
Oct 25 15:25:02 [84] INFO: 11001 blocks hash validated in 123 ms
Oct 25 15:25:02 [84] INFO: Indexed 72 day of network hashrate
Oct 25 16:08:43 [84] INFO: Disconnected from Electrum Server at 172.17.0.1:50001
Oct 25 16:08:53 [84] INFO: Connected to Electrum Server at 172.17.0.1:50001 (["Fulcrum 1.7.0","1.4"])
Oct 25 16:38:53 [84] INFO: Disconnected from Electrum Server at 172.17.0.1:50001

...

Oct 27 13:50:03 [84] INFO: Disconnected from Electrum Server at 172.17.0.1:50001
Oct 27 13:50:13 [84] INFO: Connected to Electrum Server at 172.17.0.1:50001 (["Fulcrum 1.7.0","1.4"])
Oct 27 13:50:38 [84] WARN: Cannot assign pool to block 760525 and 'unknown' pool does not exist. Check your "pools" table entries
Oct 27 14:02:43 [84] WARN: Cannot assign pool to block 760526 and 'unknown' pool does not exist. Check your "pools" table entries
Oct 27 14:05:13 [84] INFO: Disconnected from Electrum Server at 172.17.0.1:50001
Oct 27 14:05:23 [84] INFO: Connected to Electrum Server at 172.17.0.1:50001 (["Fulcrum 1.7.0","1.4"])
Oct 27 14:07:36 [84] WARN: Cannot assign pool to block 760032 and 'unknown' pool does not exist. Check your "pools" table entries
Oct 27 14:12:06 [84] WARN: Cannot assign pool to block 760527 and 'unknown' pool does not exist. Check your "pools" table entries
Oct 27 14:20:23 [84] INFO: Disconnected from Electrum Server at 172.17.0.1:50001
Oct 27 14:20:33 [84] INFO: Connected to Electrum Server at 172.17.0.1:50001 (["Fulcrum 1.7.0","1.4"])
Oct 27 14:40:33 [84] INFO: Disconnected from Electrum Server at 172.17.0.1:50001
Oct 27 14:40:43 [84] INFO: Connected to Electrum Server at 172.17.0.1:50001 (["Fulcrum 1.7.0","1.4"])
Oct 27 14:53:47 [84] WARN: Cannot assign pool to block 760528 and 'unknown' pool does not exist. Check your "pools" table entries

Frontend "docker container logs" are empty

[wiz]

@knorrium can you please look into this bug report?

[knorrium]

ELECTRUM_TLS_ENABLED: "false"

@Derrick- can you try changing this to ELECTRUM_TLS_ENABLED: "true" and use port 50002?

[Derrick-]

Currently Fulcrum does not have this enabled, per default config.
For TLS in Fulcrum, I need to set up cert and key in Fulcrum config , I will figure out how to do that over the weekend.

[Derrick-]

Now connecting TLS to Fulcrum on port 50002

Same behavior on web UI, blocks not displaying (no change).

New backend container log:

Oct 28 16:38:56 [82] NOTICE: Starting Mempool Server... (?)
Oct 28 16:38:56 [82] INFO: Restoring mempool and blocks data from disk cache
Oct 28 16:38:57 [82] INFO: Connected to Electrum Server at 172.17.0.1:50002 (["Fulcrum 1.7.0","1.4"])
Oct 28 16:38:57 [82] INFO: Database connection established.
Oct 28 16:38:57 [82] INFO: Starting statistics service
Oct 28 16:38:57 [82] INFO: Starting currency rates service
Oct 28 16:38:57 [82] INFO: Currency rates service will be queried over clearnet using https://price.bisq.wiz.biz/getAllMarketPrices
Oct 28 16:38:57 [82] NOTICE: Mempool Server is running on port 8999
Oct 28 16:38:57 [82] INFO: Updating latest mining pools from Github
Oct 28 16:38:57 [82] INFO: List of public pools will be queried over clearnet
Oct 28 16:38:57 [82] WARN: Pools.json is outdated, fetch latest from github
Oct 28 16:39:07 [82] ERR: Could not connect to Github. Reason: timeout of 10000ms exceeded
Oct 28 16:39:10 [82] INFO: Indexed 0 blocks
Oct 28 16:39:11 [82] INFO: Some blocks are not indexed, skipping missing blocks during chain validation
Oct 28 16:39:11 [82] INFO: 11439 blocks hash validated in 35 ms
Oct 28 16:39:57 [82] NOTICE: The mempool is now in sync!

[knorrium]

Oct 28 16:38:57 [82] WARN: Pools.json is outdated, fetch latest from github
Oct 28 16:39:07 [82] ERR: Could not connect to Github. Reason: timeout of 10000ms exceeded

@Derrick- do you have anything outside the docker setup that could be blocking requests to https://raw.githubusercontent.com/mempool/mining-pools/master/pools.json ?

@nymkappa would a missing or corrupt pools.json file cause the indexing to break or end up incomplete?

[nymkappa]

@Derrick- @knorrium

I think this is the problem here. It looks like your frontend is not able to connect to the websocket, which is used to display real time data like latest blocks.

[Derrick-]

@knorrium I don't see anything outside the dockersetup that's blocking github requests. My initial guess on this was that it was a security measure intentionally implemented in the docker container, I would like to fix this as well, but separate issue I figured.

The docker containers I am using were directly from https://github.com/mempool/mempool/tree/master/docker
In these, I only modified the docker-compose.yml as posted above per instructions.
The Debian VM this is running on is very barebones. It's on private network with no firewall configured

@nymkappa
The websocket seems to be passing data, even with the 101. Here are the details from browser.

Please see also the same 101 on https://mempool.space, but blocks page works there.

[knorrium]

@Derrick- can you try using the images curated for Umbrel instead of the latest tags?

image: mempool/backend:v2.4.0@sha256:6036cc6866b782efadc765e8189958940202b3fa1770eb35c35210c86efd2f47

image: mempool/frontend:v2.4.0@sha256:f71722f1e3abfb3d8a3df6f2a32e384d39434e311bb83d810629077b4fa6ffaf

[Derrick-]

No change using Umbrel images.

I have discovered that my VM host OS cannot curl https://raw.githubusercontent.com/mempool/mining-pools/master/pools.json
Don't understand why yet, other sites curl fine. It's a Canadian datacenter IP, maybe on a blocklist. Working on that.

[knorrium]

@wiz @nymkappa @softsimon I remember we talked about bundling pools.json with the build and update later - it looks like it would have prevented this issue from happening

[Derrick-]

Solved an issue with the soft router gateway on the server's network, permitting the connection to download pools.json and everything is working now.

Yes, seeding the pool table would be nice, but in the end the problem I had was a networking issue.
I was surprised to find that the empty pools table prohibited block display though, so maybe there is something still there to be looked at.

Thanks!