cmd/geth: add db commands: stats, compact, put, get, delete

[holiman]

This PR supersedes #21991 and #21887 (those PRs contain more info and examples.

This PR introduces

• db.put to put a value into the database
• db.get to read a value from the database
• db.delete to delete a value from the database
• db.stats to check compaction info from the database
• db.compact to trigger a db compaction

It also moves inspectdb to db.inspect

[holiman]

Rebased + squashed, PTAL

[holiman]

Suggestions from triage:

geth db put xx
geth db get xx
geth db compact
geth db delete xx
geth db inspect
geth removedb

[holiman]

One thing that I'm having some problems with.. Three cases:

./build/bin/geth db stats --goerli  
./build/bin/geth db --goerli  stats 
./build/bin/geth --goerli db  stats 

What should happen in these cases?

A) Only one works, the others show some kind of error + help (which one should work?)
B) All three works, with identical result, resolving the goerli datadir
C) Some of them just ignore the --goerli but otherwise "works"

RIght now, I'm at C, which is obviously incorrect. But should A or B be the target?

[holiman]

Ok, nevermind, I'll go with

A). Canon form is geth --goerli db stats, and the other variations show error + help

[holiman]

I added a commit which makes the ArgsUsage show:
Before:

[user@work go-ethereum]$ ./build/bin/geth --goerli export -h
export [command options] [arguments...]

After:

[user@work go-ethereum]$ ./build/bin/geth --goerli export -h
export [command options] <filename> [<blockNumFirst> <blockNumLast>]

And

[user@work go-ethereum]$ geth db put -h
put <hex-encoded key> <hex-encoded value>
This command sets a given database key to the given value. 
WARNING: This is a low-level operation which may cause database corruption!

[holiman]

Before this PR:

./build/bin/geth --goerli db inspect
...
INFO [02-08|13:30:14.738] Inspecting database                      count=56301000 elapsed=41.019s
+-----------------+--------------------+------------+----------+
|    DATABASE     |      CATEGORY      |    SIZE    |  ITEMS   |
+-----------------+--------------------+------------+----------+
| Key-Value store | Headers            | 70.02 MiB  |   113420 |
| Key-Value store | Bodies             | 212.04 MiB |   113420 |
| Key-Value store | Receipt lists      | 145.44 MiB |   113420 |
| Key-Value store | Difficulties       | 5.04 MiB   |   114067 |
| Key-Value store | Block number->hash | 4.59 MiB   |   113665 |
| Key-Value store | Block hash->number | 163.71 MiB |  4187014 |
| Key-Value store | Transaction index  | 294.63 MiB |  8581639 |
| Key-Value store | Bloombit index     | 146.44 MiB |  2093056 |
| Key-Value store | Contract codes     | 247.72 MiB |    43117 |
| Key-Value store | Trie nodes         | 4.27 GiB   | 42450160 |
| Key-Value store | Trie preimages     | 46.79 MiB  |   672798 |
| Key-Value store | Account snapshot   | 0.00 B     |        0 |
| Key-Value store | Storage snapshot   | 0.00 B     |        0 |
| Key-Value store | Clique snapshots   | 2.23 MiB   |     2433 |
| Key-Value store | Singleton metadata | 185.00 B   |        6 |
| Ancient store   | Headers            | 1.34 GiB   |  4096595 |
| Ancient store   | Bodies             | 1.94 GiB   |  4096595 |
| Ancient store   | Receipt lists      | 1.01 GiB   |  4096595 |
| Ancient store   | Difficulties       | 39.03 MiB  |  4096595 |
| Ancient store   | Block number->hash | 148.46 MiB |  4096595 |
| Light client    | CHT trie nodes     | 0.00 B     |        0 |
| Light client    | Bloom trie nodes   | 0.00 B     |        0 |
+-----------------+--------------------+------------+----------+
|                         TOTAL        | 10.04 GIB  |          |
+-----------------+--------------------+------------+----------+

After this PR:

INFO [02-08|13:40:44.535] Inspecting database                      count=55761000 elapsed=40.177s
+-----------------+--------------------+------------+----------+
|    DATABASE     |      CATEGORY      |    SIZE    |  ITEMS   |
+-----------------+--------------------+------------+----------+
| Key-Value store | Headers            | 70.02 MiB  |   113420 |
| Key-Value store | Bodies             | 212.04 MiB |   113420 |
| Key-Value store | Receipt lists      | 145.44 MiB |   113420 |
| Key-Value store | Difficulties       | 5.04 MiB   |   114067 |
| Key-Value store | Block number->hash | 4.59 MiB   |   113665 |
| Key-Value store | Block hash->number | 163.71 MiB |  4187014 |
| Key-Value store | Transaction index  | 294.63 MiB |  8581639 |
| Key-Value store | Bloombit index     | 146.44 MiB |  2093056 |
| Key-Value store | Contract codes     | 247.72 MiB |    43117 |
| Key-Value store | Trie nodes         | 4.27 GiB   | 42450160 |
| Key-Value store | Trie preimages     | 46.79 MiB  |   672798 |
| Key-Value store | Account snapshot   | 0.00 B     |        0 |
| Key-Value store | Storage snapshot   | 0.00 B     |        0 |
| Key-Value store | Clique snapshots   | 2.23 MiB   |     2433 |
| Key-Value store | Singleton metadata | 185.00 B   |        6 |
| Ancient store   | Headers            | 1.34 GiB   |  4096595 |
| Ancient store   | Bodies             | 1.94 GiB   |  4096595 |
| Ancient store   | Receipt lists      | 1.01 GiB   |  4096595 |
| Ancient store   | Difficulties       | 39.03 MiB  |  4096595 |
| Ancient store   | Block number->hash | 148.46 MiB |  4096595 |
| Light client    | CHT trie nodes     | 0.00 B     |        0 |
| Light client    | Bloom trie nodes   | 0.00 B     |        0 |
+-----------------+--------------------+------------+----------+
|                         TOTAL        | 10.04 GIB  |          |
+-----------------+--------------------+------------+----------+

Checking the c (contract codes):

[user@work go-ethereum]$ ./build/bin/geth --goerli db inspect 0x63 0x00
...
+-----------------+--------------------+------------+---------+
|    DATABASE     |      CATEGORY      |    SIZE    |  ITEMS  |
+-----------------+--------------------+------------+---------+
| Key-Value store | Headers            | 0.00 B     |       0 |
| Key-Value store | Bodies             | 0.00 B     |       0 |
| Key-Value store | Receipt lists      | 0.00 B     |       0 |
| Key-Value store | Difficulties       | 0.00 B     |       0 |
| Key-Value store | Block number->hash | 0.00 B     |       0 |
| Key-Value store | Block hash->number | 0.00 B     |       0 |
| Key-Value store | Transaction index  | 0.00 B     |       0 |
| Key-Value store | Bloombit index     | 0.00 B     |       0 |
| Key-Value store | Contract codes     | 247.72 MiB |   43117 |
| Key-Value store | Trie nodes         | 17.12 MiB  |  165946 |
| Key-Value store | Trie preimages     | 0.00 B     |       0 |
| Key-Value store | Account snapshot   | 0.00 B     |       0 |
| Key-Value store | Storage snapshot   | 0.00 B     |       0 |
| Key-Value store | Clique snapshots   | 2.23 MiB   |    2433 |
| Key-Value store | Singleton metadata | 0.00 B     |       0 |
| Ancient store   | Headers            | 1.34 GiB   | 4096595 |
| Ancient store   | Bodies             | 1.94 GiB   | 4096595 |
| Ancient store   | Receipt lists      | 1.01 GiB   | 4096595 |
| Ancient store   | Difficulties       | 39.03 MiB  | 4096595 |
| Ancient store   | Block number->hash | 148.46 MiB | 4096595 |
| Light client    | CHT trie nodes     | 0.00 B     |       0 |
| Light client    | Bloom trie nodes   | 0.00 B     |       0 |
+-----------------+--------------------+------------+---------+
|                         TOTAL        |  4.72 GIB  |         |
+-----------------+--------------------+------------+---------+

Checking clique- prefix:

[user@work go-ethereum]$  ./build/bin/geth --goerli db inspect 0x636c697175652d 0x00
...
+-----------------+--------------------+------------+---------+
|    DATABASE     |      CATEGORY      |    SIZE    |  ITEMS  |
+-----------------+--------------------+------------+---------+
| Key-Value store | Headers            | 0.00 B     |       0 |
| Key-Value store | Bodies             | 0.00 B     |       0 |
| Key-Value store | Receipt lists      | 0.00 B     |       0 |
| Key-Value store | Difficulties       | 0.00 B     |       0 |
| Key-Value store | Block number->hash | 0.00 B     |       0 |
| Key-Value store | Block hash->number | 0.00 B     |       0 |
| Key-Value store | Transaction index  | 0.00 B     |       0 |
| Key-Value store | Bloombit index     | 0.00 B     |       0 |
| Key-Value store | Contract codes     | 0.00 B     |       0 |
| Key-Value store | Trie nodes         | 0.00 B     |       0 |
| Key-Value store | Trie preimages     | 0.00 B     |       0 |
| Key-Value store | Account snapshot   | 0.00 B     |       0 |
| Key-Value store | Storage snapshot   | 0.00 B     |       0 |
| Key-Value store | Clique snapshots   | 2.23 MiB   |    2433 |
| Key-Value store | Singleton metadata | 0.00 B     |       0 |
| Ancient store   | Headers            | 1.34 GiB   | 4096595 |
| Ancient store   | Bodies             | 1.94 GiB   | 4096595 |
| Ancient store   | Receipt lists      | 1.01 GiB   | 4096595 |
| Ancient store   | Difficulties       | 39.03 MiB  | 4096595 |
| Ancient store   | Block number->hash | 148.46 MiB | 4096595 |
| Light client    | CHT trie nodes     | 0.00 B     |       0 |
| Light client    | Bloom trie nodes   | 0.00 B     |       0 |
+-----------------+--------------------+------------+---------+
|                         TOTAL        |  4.47 GIB  |         |
+-----------------+--------------------+------------+---------+

The last example brings down 41s to:

real    0m0.157s
user    0m0.119s
sys     0m0.029s

[holiman]

If possible, open in read-only

[holiman]

Fixed most of the concerns, with the exception of not opening the db in readonly mode for inspect, which would be quite an invasive change.

put, get, delete

Check unclean shutdown marker (note: now in readonly)

[user@work go-ethereum]$ ./build/bin/geth --goerli  db get 0x756e636c65616e2d73687574646f776e
INFO [02-10|09:26:06.705] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:26:06.705] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:26:06.706] Set global gas cap                       cap=25000000
INFO [02-10|09:26:06.706] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=16.00MiB handles=500 readonly=true
key 0x756e636c65616e2d73687574646f776e:
        0xc780c58460217a8d

Overwrite unclean shutdown marker

[user@work go-ethereum]$ ./build/bin/geth --goerli  db put 0x756e636c65616e2d73687574646f776e 0x1337
INFO [02-10|09:27:41.871] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:27:41.871] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:27:41.871] Set global gas cap                       cap=25000000
INFO [02-10|09:27:41.871] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=512.00MiB handles=262144
INFO [02-10|09:27:41.913] Opened ancient database                  database=/home/user/.ethereum/goerli/geth/chaindata/ancient
Previous value:
0xc780c58460217a8d
INFO [02-10|09:27:41.914] Freezer shutting down

Check it

[user@work go-ethereum]$ ./build/bin/geth --goerli  db get 0x756e636c65616e2d73687574646f776e
INFO [02-10|09:28:14.451] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:28:14.451] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:28:14.452] Set global gas cap                       cap=25000000
INFO [02-10|09:28:14.452] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=16.00MiB handles=500 readonly=true
key 0x756e636c65616e2d73687574646f776e:
        0x1337

Put it back

[user@work go-ethereum]$ ./build/bin/geth --goerli  db put 0x756e636c65616e2d73687574646f776e 0xc780c58460217a8d
INFO [02-10|09:28:46.107] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:28:46.107] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:28:46.108] Set global gas cap                       cap=25000000
INFO [02-10|09:28:46.108] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=512.00MiB handles=262144
INFO [02-10|09:28:46.156] Opened ancient database                  database=/home/user/.ethereum/goerli/geth/chaindata/ancient
Previous value:
0x1337
INFO [02-10|09:28:46.156] Freezer shutting down

Delete it

[user@work go-ethereum]$ ./build/bin/geth --goerli  db delete 0x756e636c65616e2d73687574646f776e
INFO [02-10|09:33:02.186] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:33:02.186] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:33:02.186] Set global gas cap                       cap=25000000
INFO [02-10|09:33:02.188] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=512.00MiB handles=262144
INFO [02-10|09:33:02.222] Opened ancient database                  database=/home/user/.ethereum/goerli/geth/chaindata/ancient
INFO [02-10|09:33:02.223] Freezer shutting down

Verify delete:

[user@work go-ethereum]$ ./build/bin/geth --goerli  db get  0x756e636c65616e2d73687574646f776e
INFO [02-10|09:33:05.494] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:33:05.495] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:33:05.495] Set global gas cap                       cap=25000000
INFO [02-10|09:33:05.496] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=16.00MiB handles=500 readonly=true
INFO [02-10|09:33:05.508] Get operation failed                     error="leveldb: not found"
leveldb: not found

Restore it

[user@work go-ethereum]$ ./build/bin/geth --goerli  db put 0x756e636c65616e2d73687574646f776e 0xc780c58460217a8d
INFO [02-10|09:33:12.390] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:33:12.390] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:33:12.390] Set global gas cap                       cap=25000000
INFO [02-10|09:33:12.391] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=512.00MiB handles=262144
INFO [02-10|09:33:12.437] Opened ancient database                  database=/home/user/.ethereum/goerli/geth/chaindata/ancient
INFO [02-10|09:33:12.438] Freezer shutting down

inspect

Added uncleanShutdownMarker to the stats. This is still not readonly, since it's relying on the whole make utils.MakeChain(ctx, stack, true).

[user@work go-ethereum]$ ./build/bin/geth --goerli  db inspect
INFO [02-10|09:31:40.672] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:31:40.672] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:31:40.673] Set global gas cap                       cap=25000000
INFO [02-10|09:31:40.673] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=512.00MiB handles=262144
INFO [02-10|09:31:40.706] Opened ancient database                  database=/home/user/.ethereum/goerli/geth/chaindata/ancient
INFO [02-10|09:31:40.711] Persisted trie from memory database      nodes=361 size=51.17KiB time="742.775µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
...
| Key-Value store | Singleton metadata | 139.00 B   |       4 |
| Key-Value store | Shutdown metadata  | 24.00 B    |       1 |
| Ancient store   | Headers            | 1.34 GiB   | 4095953 |

stats

Note readonly

[user@work go-ethereum]$ ./build/bin/geth --goerli  db stats
INFO [02-10|09:32:28.114] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:32:28.114] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:32:28.115] Set global gas cap                       cap=25000000
INFO [02-10|09:32:28.115] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=16.00MiB handles=500 readonly=true
Compactions
 Level |   Tables   |    Size(MB)   |    Time(sec)  |    Read(MB)   |   Write(MB)
-------+------------+---------------+---------------+---------------+---------------
   0   |          2 |       0.00060 |       0.00000 |       0.00000 |       0.00000
   1   |          2 |       0.00059 |       0.00000 |       0.00000 |       0.00000
   2   |        320 |     655.45603 |       0.00000 |       0.00000 |       0.00000
-------+------------+---------------+---------------+---------------+---------------
 Total |        324 |     655.45723 |       0.00000 |       0.00000 |       0.00000

Read(MB):0.03220 Write(MB):0.00000

compact

[user@work go-ethereum]$ ./build/bin/geth --goerli  db compact
INFO [02-10|09:37:32.313] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-10|09:37:32.313] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-10|09:37:32.313] Set global gas cap                       cap=25000000
INFO [02-10|09:37:32.313] Allocated cache and file handles         database=/home/user/.ethereum/goerli/geth/chaindata cache=512.00MiB handles=262144
Compactions
 Level |   Tables   |    Size(MB)   |    Time(sec)  |    Read(MB)   |   Write(MB)
-------+------------+---------------+---------------+---------------+---------------
   0   |          3 |       0.00101 |       0.00000 |       0.00000 |       0.00000
   1   |          2 |       0.00059 |       0.00000 |       0.00000 |       0.00000
   2   |        320 |     655.45603 |       0.00000 |       0.00000 |       0.00000
-------+------------+---------------+---------------+---------------+---------------
 Total |        325 |     655.45763 |       0.00000 |       0.00000 |       0.00000

Read(MB):0.03209 Write(MB):0.03208
INFO [02-10|09:37:32.348] Triggering compaction
Compactions
 Level |   Tables   |    Size(MB)   |    Time(sec)  |    Read(MB)   |   Write(MB)
-------+------------+---------------+---------------+---------------+---------------
   1   |          0 |       0.00000 |       0.01392 |       0.00160 |       0.00060
   2   |        318 |     655.34744 |       2.93187 |     186.54778 |     186.43859
-------+------------+---------------+---------------+---------------+---------------
 Total |        318 |     655.34744 |       2.94579 |     186.54938 |     186.43919

Read(MB):183.20840 Write(MB):186.48059
INFO [02-10|09:37:35.314] Closing db
INFO [02-10|09:37:35.316] Exiting