On new block, stats information switches to another network

[diega]

We are running the instance behind https://blockscout.com/etc/kotti from the production-classic-stg branch. When you first load the home page, the information displayed is correct, but at the time a new block should be displayed, all the information changes to display values coming from some other chain (ETC mainnet apparently).

This is the command line we use to start the Blockscout instance.

ETHEREUM_JSONRPC_VARIANT='geth' \
ETHEREUM_JSONRPC_WS_URL='ws://127.0.0.1:8546' \
ETHEREUM_JSONRPC_HTTP_URL='http://127.0.0.1:8545' \
ETHEREUM_JSONRPC_TRACE_URL='http://127.0.0.1:8545' \
MIX_ENV='prod' \
DATABASE_URL='*******' \
SOCKET_ROOT='/etc/kotti' \
PORT='80' \
SUPPORTED_CHAINS='[ { "title": "Ethereum Classic", "url": "https://blockscout.com/etc/mainnet"}, { "title": "ETC Mordor", "url": "https://blockscout.com/etc/mordor", "test_net?": true }, { "title": "POA Core", "url": "https://blockscout.com/poa/core" }, { "title": "POA Sokol", "url": "https://blockscout.com/poa/sokol", "test_net?": true }, { "title": "xDai Chain", "url": "https://blockscout.com/poa/dai" }, { "title": "Kovan Testnet", "url": "https://blockscout.com/eth/kovan", "test_net?": true }, { "title": "RSK Mainnet", "url": "https://blockscout.com/rsk/mainnet", "other?": true }, { "title": "LUKSO L14 testnet", "url": "https://blockscout.com/lukso/l14", "test_net?": true }, { "title": "Ethereum Mainnet", "url": "https://blockscout.com/eth/mainnet" }]' \
TXS_COUNT_CACHE_PERIOD='7200' \
BLOCKSCOUT_HOST='blockscout.com' \
ELIXIR_VERSION='v1.9.1' \
NETWORK_PATH='/etc/kotti' \
API_PATH='/etc/kotti' \
SUBNETWORK='Kotti' \
BLOCK_COUNT_CACHE_PERIOD='7200' \
LINK_TO_OTHER_EXPLORERS='true' \
LOGO='/images/classic_ethereum_logo.svg' \
LOGO_FOOTER='/images/classic_ethereum_logo.svg' \
NETWORK_ICON='_test_network_icon.html' \
COIN='ETC' \
NETWORK='ETC' \
POOL_SIZE='150' \
CHAIN_SPEC_PATH='https://raw.githubusercontent.com/etclabscore/core-geth/master/params/parity.json.d/kotti.json' \
BLOCKSCOUT_VERSION='v3.7.3-beta' \
make start

And our local core-geth:1.12.2 instance is running with the following options
--http --http.api debug,web3,eth,txpool,net,trace --ws --ws.api debug,web3,eth,txpool,net,trace --kotti --syncmode full --gcmode archive

[vbaranov]

@diega please, try to deploy from the branch for Kotti https://github.com/blockscout/blockscout/tree/production-etc-kotti-stg. The path to Kotti instance websocket adjusted there.

Please, also
Change:

1. BLOCKSCOUT_VERSION=v.4.1.0-beta
2. POOL_SIZE='100'
3. POOL_SIZE_API='50'

Add:
CHAIN_ID=6

Remove:
ELIXIR_VERSION env var - it is not used by the application anymore.

I also suggest updating ETC Mainnet instance from https://github.com/blockscout/blockscout/tree/production-classic-stg. It has been updated with release 4.1.0. The recommendations for environment variables changes are the same except CHAIN_ID=61.

Waiting for feedback from you.

[diega]

Hello Viktor. Thank you for your updating the branches!
I think there was an unsolved merge conflict that went at least into the Makefile @ 220008e87

[vbaranov]

@diega fixed, please pull the latest updates in both branches.

[diega]

@vbaranov excellent, https://blockscout.com/etc/kotti has been fixed. I'm rebuilding https://blockscout.com/etc/mainnet, so it's down at the moment, but in the meantime would be possible to get a branch for https://blockscout.com/etc/mordor as well? It has the same problem about using the websocket for ETC mainnet

[vbaranov]

Kotti instance returns an error for me:

What is in the logs?

[diega]

That's strange, I'm accessing it just fine here, even in an incognito session. The logs seems to be clean. This is the only issue that appears from time to time

Request: GET /favicon.ico
** (exit) an exception was raised:
    ** (ArgumentError) cookie store expects conn.secret_key_base to be set
        (plug 1.12.1) lib/plug/session/cookie.ex:203: Plug.Session.COOKIE.validate_secret_key_base/1
        (plug 1.12.1) lib/plug/session/cookie.ex:198: Plug.Session.COOKIE.derive/3
        (plug 1.12.1) lib/plug/session/cookie.ex:98: Plug.Session.COOKIE.get/3
        (plug 1.12.1) lib/plug/session.ex:78: anonymous fn/5 in Plug.Session.fetch_session/1
        (block_scout_web 0.0.1) BlockScoutWeb.WebRouter.browser/2
        (block_scout_web 0.0.1) lib/block_scout_web/web_router.ex:1: BlockScoutWeb.WebRouter.__pipe_through0__/1
        (phoenix 1.5.13) lib/phoenix/router.ex:347: Phoenix.Router.__call__/2
        (phoenix 1.5.13) lib/phoenix/router/route.ex:41: Phoenix.Router.Route.call/2

but it doesn't seems to be the reason for an outage.

[vbaranov]

It was a cache issue on my side. Kotti instance works fine.

In order to eliminate an error in the logs, please, follow these steps:

4. Generate a new secret_key_base for the DB by setting a corresponding ENV var:
   export SECRET_KEY_BASE=VTIB3uHDNbvrY0+60ZWgUoUBKDn9ppLR8MI4CpRz4/qLyEFs54ktJfaNT6Z221No

from https://docs.blockscout.com/for-developers/manual-deployment#deployment-steps

The branch for Mordor is ready: https://github.com/blockscout/blockscout/tree/production-etc-mordor-stg

[diega]

@vbaranov I'm getting this error while migrating mainnet DB

11:52:38.968 [info]  == Running 20211206071033 Explorer.Repo.Migrations.ModifyAddressGasUsedBignit.up/0 forward

11:52:38.974 [info]  alter table addresses
** (Postgrex.Error) ERROR 53100 (disk_full) could not extend file "base/16396/664295.2": No space left on device

    hint: Check free disk space.
    (ecto_sql 3.7.0) lib/ecto/adapters/sql.ex:756: Ecto.Adapters.SQL.raise_sql_call_error/1
    (elixir 1.12.2) lib/enum.ex:1582: Enum."-map/2-lists^map/1-0-"/2
    (ecto_sql 3.7.0) lib/ecto/adapters/sql.ex:844: Ecto.Adapters.SQL.execute_ddl/4
    (ecto_sql 3.7.0) lib/ecto/migration/runner.ex:343: Ecto.Migration.Runner.log_and_execute_ddl/3
    (ecto_sql 3.7.0) lib/ecto/migration/runner.ex:117: anonymous fn/6 in Ecto.Migration.Runner.flush/0
    (elixir 1.12.2) lib/enum.ex:2385: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto_sql 3.7.0) lib/ecto/migration/runner.ex:116: Ecto.Migration.Runner.flush/0
    (ecto_sql 3.7.0) lib/ecto/migration/runner.ex:280: Ecto.Migration.Runner.perform_operation/3
make[2]: *** [Makefile:450: migrate_only] Error 1
make[2]: Leaving directory '/home/admin/blockscout/docker'
make[1]: *** [Makefile:455: migrate] Error 2
make[1]: Leaving directory '/home/admin/blockscout/docker'
make: *** [Makefile:460: start] Error 2

But the instance has doesn't seems to be running out of space

$ df -h
Filesystem       Size  Used Avail Use% Mounted on
udev             7.6G     0  7.6G   0% /dev
tmpfs            1.6G  153M  1.4G  10% /run
/dev/nvme0n1p1   148G   12G  131G   9% /
tmpfs            7.7G     0  7.7G   0% /dev/shm
tmpfs            5.0M     0  5.0M   0% /run/lock
tmpfs            7.7G     0  7.7G   0% /sys/fs/cgroup
/dev/nvme0n1p15  124M  278K  124M   1% /boot/efi
tmpfs            1.6G     0  1.6G   0% /run/user/1000
/dev/nvme2n1p1   962G  650G  274G  71% /data/besu-classic

[vbaranov]

@diega it looks like you have no space on RDS instance, not on the application instance (EC2).

[diega]

Thanks Viktor. Yes, that was exactly the case. https://blockscout.com/etc/mainnet/ is up and running again

[vbaranov]

@diega could you please set v4.1.0-beta in BLOCKSCOUT_VERSION? Currently, v4.1.3-beta is set, which doesn't exist.

[diega]

@diega could you please set v4.1.0-beta in BLOCKSCOUT_VERSION? Currently, v4.1.3-beta is set, which doesn't exist.

Good catch! fixed

[diega]

https://blockscout.com/etc/mordor is up and running too without the original issue. I'm closing this. Thanks @vbaranov for your help! Have a great year

[bobsummerwill]

Thanks very much, @vbaranov!

Best wishes to you and your family.