[BUG] Erigon startup EROR: mdbx_env_open: MDBX_TOO_LARGE #7367

cppfuns · 2023-04-23T01:17:20Z

System information

Erigon version: `./erigon --version`

erigon version 2.43.0-dev-ef71909c

OS & Version: Windows/Linux/OSX

ubuntu:20.04

Commit hash:

ef71909

Erigon Command (with flags/config):

./erigon --datadir=./data --private.api.addr=0.0.0.0:9090 --http.addr=0.0.0.0 --http.vhosts=* --http.corsdomain=* --http.api=eth,erigon,web3,net,debug,trace,txpool --ws --log.dir.path=./logs/

Specific implementation steps

 ./erigon init --datadir ./data ./config/genesis.json
./erigon --datadir=./data --private.api.addr=0.0.0.0:9090 --http.addr=0.0.0.0 --http.vhosts=* --http.corsdomain=* --http.api=eth,erigon,web3,net,debug,trace,txpool --ws --log.dir.path=./logs/

Problem Description

In the test, the current dev branch 2.43.0-dev-ef71909c, after initializing with genesis.json, an error is reported at startup
./erigon --datadir=./data --private.api.addr=0.0.0.0:9090 --http.addr=0.0.0.0 --http.vhosts=* --http.corsdomain=* --http.api=eth,erigon,web3,net,debug,trace,txpool --ws --log.dir.path=./logs/

INFO[04-23|08:59:32.892] logging to file system                   log dir=./logs/ file prefix=erigon log level=info json=false
INFO[04-23|08:59:32.907] Build info                               git_branch=devel git_tag=v2.31.0-rc.4-919-gef71909c3 git_commit=ef71909c34ad7d9ff8841c7fcd121f09165bd985
INFO[04-23|08:59:32.907] Starting Erigon on Ethereum mainnet...
INFO[04-23|08:59:32.913] Maximum peer count                       ETH=100 total=100
INFO[04-23|08:59:32.913] starting HTTP APIs                       APIs=eth,erigon,web3,net,debug,trace,txpool
INFO[04-23|08:59:32.913] torrent verbosity                        level=WRN
INFO[04-23|08:59:35.030] Set global gas cap                       cap=50000000
INFO[04-23|08:59:35.120] [Downloader] Runnning with               ipv6-enabled=true ipv4-enabled=true download.rate=16mb
 upload.rate=4mb
INFO[04-23|08:59:35.122] Opening Database                         label=chaindata path=/mnt/d/github/Opside/other/test-genesis/data/chaindata
mdbx_setup_dxb:15897 could not apply preconfigured db-geometry: err -30417
EROR[04-23|08:59:35.247] Erigon startup                           err="mdbx_env_open: MDBX_TOO_LARGE: Database is too large for current system, e.g. could NOT be mapped into RAM, label: chaindata, trace: [kv_mdbx.go:262 node.go:323 node.go:326 backend.go:203 node.go:106 main.go:59 command.go:274 app.go:332 app.go:309 main.go:36 proc.go:250 asm_amd64.s:1594]"
mdbx_env_open: MDBX_TOO_LARGE: Database is too large for current system, e.g. could NOT be mapped into RAM, label: chaindata, trace: [kv_mdbx.go:262 node.go:323 node.go:326 backend.go:203 node.go:106 main.go:59 command.go:274 app.go:332 app.go:309 main.go:36 proc.go:250 asm_amd64.s:1594]

Probability of recurrence

100%

Version comparison test

In version: 2.42.0-stable, this problem does not occur

Why the test for this question

We are the technical team from opside.network (A Decentralized ZK-RaaS Network.), we are exploring related technologies

The text was updated successfully, but these errors were encountered:

AskAlexSharov · 2023-04-23T04:25:00Z

hi.

try latest devel branch
if 1. will not help, share machine spec plz: os/fs/ram/etc..
temporary workaround: set explicit --db.size.limit

paul-serdiuk · 2023-04-26T09:13:15Z

Can confirm that I encountered the same problem on 2.43 even after setting db.size.limit, the machine has 32 cores and 128 GB of RAM, I'm running Erigon in a Docker container.

edit: the ulimit inside the docker container is as follows:

/ $ ulimit -a
core file size (blocks)         (-c) unlimited
data seg size (kb)              (-d) unlimited
scheduling priority             (-e) 0
file size (blocks)              (-f) unlimited
pending signals                 (-i) 514281
max locked memory (kb)          (-l) 64
max memory size (kb)            (-m) unlimited
open files                      (-n) 1048576
POSIX message queues (bytes)    (-q) 819200
real-time priority              (-r) 0
stack size (kb)                 (-s) 8192
cpu time (seconds)              (-t) unlimited
max user processes              (-u) unlimited
virtual memory (kb)             (-v) unlimited
file locks                      (-x) unlimited

Could one of these values be the problem?

AskAlexSharov · 2023-04-27T03:40:18Z

what is --db.pagesize ?
what is size of db now?
it's not about "ram", it's about "virtual memory"

paul-serdiuk · 2023-05-03T23:42:42Z

--db.pagesize=16k
DB size is 7.6T

AskAlexSharov · 2023-05-04T02:52:34Z

make db-tools
./build/bin/mdbx_stat -ef /chaindata

AskAlexSharov · 2023-05-10T05:10:42Z

seems i fixed it by #7482

paul-serdiuk · 2023-05-17T05:28:46Z

Still getting this on devel:

[EROR] [05-17|08:27:47.480] Erigon startup                           err="mdbx_env_open: MDBX_TOO_LARGE: Database is too large for current system, e.g. could NOT be mapped into RAM, label: chaindata, trace: [kv_mdbx.go:262 node.go:325 node.go:328 backend.go:207 node.go:107 main.go:66 make_app.go:42 command.go:274 app.go:332 app.go:309 main.go:37 proc.go:250 asm_amd64.s:1598]"
mdbx_env_open: MDBX_TOO_LARGE: Database is too large for current system, e.g. could NOT be mapped into RAM, label: chaindata, trace: [kv_mdbx.go:262 node.go:325 node.go:328 backend.go:207 node.go:107 main.go:66 make_app.go:42 command.go:274 app.go:332 app.go:309 main.go:37 proc.go:250 asm_amd64.s:1598]

paul-serdiuk · 2023-05-17T05:31:52Z

Output of stat

mdbx_stat v0.12.0-71-g1cac6536 (2022-07-28T09:57:31+07:00, T-9a6d7e5b917e5fbd14dc51835fa749d092aa1d72)
Running for ../../data/erigon-data/chaindata/...
Environment Info
  Pagesize: 4096
  Dynamic datafile: 49152..8796093022208 bytes (+2147483648/-4294967296), 12..2147483648 pages (+524288/-1048576)
  Current mapsize: 8796093022208 bytes, 2147483648 pages
  Current datafile: 8441758220288 bytes, 2060976128 pages
  Last transaction ID: 22591502
  Latter reader transaction ID: 22591502 (0)
  Max readers: 116
  Number of reader slots uses: 1
Garbage Collection
  Pagesize: 4096
  Tree depth: 2
  Branch pages: 1
  Leaf pages: 70
  Overflow pages: 12494
  Entries: 12501
Page Usage
  Total: 2147483648 100%
  Backed: 2060976128 96.0%
  Allocated: 2060859216 96.0%
  Remained: 86624432 4.0%
  Used: 2048139679 95.4%
  GC: 12719537 0.6%
  Retained: 7 0.0%
  Reclaimable: 12719530 0.6%
  Available: 99343962 4.6%
Status of Main DB
  Pagesize: 4096
  Tree depth: 2
  Branch pages: 1
  Leaf pages: 3
  Overflow pages: 0
  Entries: 107

I'm not sure why it shows a pagesize of 4096 considering I am running Erigon v2.42.0 fine with --db.pagesize=16k with no errors.

AskAlexSharov · 2023-05-17T07:23:21Z

@paul-serdiuk
8Tb is max what you can get on 4Kb pageSize.

--db.pagesize=16k can set this flag only at startup

To change db pagesize:
you will need manually create new db - by starting erigon with new flags in new data dir.
then move all data to new db, by "integration mdbx_to_mdbx": https://github.com/ledgerwatch/erigon/tree/devel/cmd/integration#copy-data-to-another-db

paul-serdiuk · 2023-05-17T08:21:10Z

Doesn't --db.pagesize show an error when trying to open a database that was created with a different pagesize?

Also, will Erigon show an error if the limit is reached? I hope the current DB is not borked by that.

artjoma · 2023-05-22T12:21:11Z

seems i fixed it by #7482

This fix works fine! Tested on BSC Chapel.

WhymustIhaveaname · 2024-01-02T10:53:24Z

I still met this issue in version 2.56.0. My system info is as follows

$ ./erigon --version
erigon version 2.56.0-dev-415b17ca
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 320986
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 320986
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
$ ./mdbx_stat -ef /Data8T/ethmainnet/chaindata/
mdbx_stat v0.12.0-71-g1cac6536 (2022-07-28T09:57:31+07:00, T-9a6d7e5b917e5fbd14dc51835fa749d092aa1d72)
Running for /Data8T/ethmainnet/chaindata/...
Environment Info
  Pagesize: 4096
  Dynamic datafile: 24576..3298534883328 bytes (+2147483648/-4294967296), 6..805306368 pages (+524288/-1048576)
  Current mapsize: 3298534883328 bytes, 805306368 pages 
  Current datafile: 2087354105856 bytes, 509607936 pages
  Last transaction ID: 8379750
  Latter reader transaction ID: 8379750 (0)
  Max readers: 116
  Number of reader slots uses: 1
Garbage Collection
  Pagesize: 4096
  Tree depth: 2
  Branch pages: 1
  Leaf pages: 54
  Overflow pages: 9754
  Entries: 9755
Page Usage
  Total: 805306368 100%
  Backed: 509607936 63.3%
  Allocated: 509397685 63.3%
  Remained: 295908683 36.7%
  Used: 499469399 62.0%
  GC: 9928286 1.2%
  Retained: 956 0.0%
  Reclaimable: 9927330 1.2%
  Available: 305836013 38.0%
Status of Main DB
  Pagesize: 4096
  Tree depth: 2
  Branch pages: 1
  Leaf pages: 3
  Overflow pages: 0
  Entries: 106

AskAlexSharov · 2024-01-02T12:56:29Z

Your db is “ Pagesize: 4096”

WhymustIhaveaname · 2024-01-02T13:28:42Z

Your db is “ Pagesize: 4096”

I know how large my Pagesize is. I was always using the default setting until I upgraded to the latest version. I also tried set db.pagesize : '4096' or db.pagesize : '4KB'. Both do not help. Exactly the same error as cppfuns happened on version 2.56.0.

Erigon startup err="mdbx_env_set_geometry: MDBX_TOO_LARGE: Database is too large for current system, e.g. could NOT be mapped into RAM"

My server has 80GB of RAM and 4x2T RAID SSD.

AskAlexSharov · 2024-01-03T01:16:04Z

@WhymustIhaveaname see breaking change section in release notes: https://github.com/ledgerwatch/erigon/releases/tag/v2.55.0 and #7367 (comment)

WhymustIhaveaname · 2024-01-03T02:51:05Z

Thank you! It works after I set db.size.limit : '8TB'.

[db] open lable=chaindata sizeLimit=8TB pageSize=4096

cppfuns changed the title ~~Erigon startup EROR: mdbx_env_open: MDBX_TOO_LARGE~~ [BUG] Erigon startup EROR: mdbx_env_open: MDBX_TOO_LARGE Apr 23, 2023

setunapo mentioned this issue May 17, 2023

mdbx_env_open: MDBX_TOO_LARGE node-real/bsc-erigon#38

Closed

AskAlexSharov closed this as completed May 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Erigon startup EROR: mdbx_env_open: MDBX_TOO_LARGE #7367

[BUG] Erigon startup EROR: mdbx_env_open: MDBX_TOO_LARGE #7367

cppfuns commented Apr 23, 2023 •

edited

Loading

AskAlexSharov commented Apr 23, 2023 •

edited

Loading

paul-serdiuk commented Apr 26, 2023 •

edited

Loading

AskAlexSharov commented Apr 27, 2023 •

edited

Loading

paul-serdiuk commented May 3, 2023

AskAlexSharov commented May 4, 2023

AskAlexSharov commented May 10, 2023

paul-serdiuk commented May 17, 2023

paul-serdiuk commented May 17, 2023 •

edited

Loading

AskAlexSharov commented May 17, 2023 •

edited

Loading

paul-serdiuk commented May 17, 2023 •

edited

Loading

artjoma commented May 22, 2023

WhymustIhaveaname commented Jan 2, 2024

AskAlexSharov commented Jan 2, 2024

WhymustIhaveaname commented Jan 2, 2024 •

edited

Loading

AskAlexSharov commented Jan 3, 2024 •

edited

Loading

WhymustIhaveaname commented Jan 3, 2024

[BUG] Erigon startup EROR: mdbx_env_open: MDBX_TOO_LARGE #7367

[BUG] Erigon startup EROR: mdbx_env_open: MDBX_TOO_LARGE #7367

Comments

cppfuns commented Apr 23, 2023 • edited Loading

System information

Erigon version: ./erigon --version

OS & Version: Windows/Linux/OSX

Commit hash:

Erigon Command (with flags/config):

Specific implementation steps

Problem Description

Probability of recurrence

Version comparison test

Why the test for this question

AskAlexSharov commented Apr 23, 2023 • edited Loading

paul-serdiuk commented Apr 26, 2023 • edited Loading

AskAlexSharov commented Apr 27, 2023 • edited Loading

paul-serdiuk commented May 3, 2023

AskAlexSharov commented May 4, 2023

AskAlexSharov commented May 10, 2023

paul-serdiuk commented May 17, 2023

paul-serdiuk commented May 17, 2023 • edited Loading

AskAlexSharov commented May 17, 2023 • edited Loading

paul-serdiuk commented May 17, 2023 • edited Loading

artjoma commented May 22, 2023

WhymustIhaveaname commented Jan 2, 2024

AskAlexSharov commented Jan 2, 2024

WhymustIhaveaname commented Jan 2, 2024 • edited Loading

AskAlexSharov commented Jan 3, 2024 • edited Loading

WhymustIhaveaname commented Jan 3, 2024

cppfuns commented Apr 23, 2023 •

edited

Loading

Erigon version: `./erigon --version`

AskAlexSharov commented Apr 23, 2023 •

edited

Loading

paul-serdiuk commented Apr 26, 2023 •

edited

Loading

AskAlexSharov commented Apr 27, 2023 •

edited

Loading

paul-serdiuk commented May 17, 2023 •

edited

Loading

AskAlexSharov commented May 17, 2023 •

edited

Loading

paul-serdiuk commented May 17, 2023 •

edited

Loading

WhymustIhaveaname commented Jan 2, 2024 •

edited

Loading

AskAlexSharov commented Jan 3, 2024 •

edited

Loading