Cannot start a node with cleveldb enabled #636
Comments
|
Sorry not too familiar with cleveldb, what is the reason to use it instead of goleveldb? |
|
We are exploring it for block generation performance gains for our node during epoch settlements. It is the general consensus within the ecosystem that goleveldb is the least performant DB option for running validator nodes; other options being cleveldb, rocksdb. An example of going as far as making Rocksdb the default option. |
|
Have you tried with rocksdb? I have been experimenting with badgerdb. |
|
@faddat Same result but at least I can get a stripped tendermint node work with cleveldb. I haven't been successful in getting a v34 tendermint node launch with either v5.18.3 or latest v6 Rocksdb built in Ubuntu 20.04, Go 1.17. |
|
@blewater sooooooo -- I did a bunch of work with RocksDB back in the day. Can confirm that it is very, very performant.
export CXX=clang -- and you probably want to make sure it is the very latest clang. You might need to install additional libraries for this. Our build process for Blurt was always pretty complex. I think I've seen that badger is a pretty good set of tradeoffs between native Goness and speed. That's what I am evaluating and here is how I am compiling-- note that I do not mess with the Makefile: go install -tags badgerdb ./... here's how I run osmosis: ulimit -n 50000
osmosisd start --pruning nothing --db_backend badgerdb --p2p.seeds 1b077d96ceeba7ef503fb048f343a538b2dcdf1b@136.243.218.244:26656,2308bed9e096a8b96d2aa343acc1147813c59ed2@3.225.38.25:26656,902bdfe51b6a97cc9369664a21c87ed61d471d2a@136.243.218.243:26656,085f62d67bbf9c501e8ac84d4533440a1eef6c45@95.217.196.54:26656,f515a8599b40f0e84dfad935ba414674ab11a668@osmosis.blockpane.com:26656,63aba59a7da5197c0fbcdc13e760d7561791dca8@162.55.132.230:2000Also I need to note that the ulimit thing is very important. I now have a node syncing faster than I've ever seen before. |
Screen.Recording.2021-12-05.at.3.47.49.AM.mov |
|
go build -mod=readonly -tags "netgo ledger cleveldb gcc" -ldflags '-X github.com/cosmos/cosmos-sdk/version.Name=osmosis -X github.com/cosmos/cosmos-sdk/version.AppName=osmosisd -X github.com/cosmos/cosmos-sdk/version.Version=3.1.0 -X github.com/cosmos/cosmos-sdk/version.Commit=13916d1e10bca718b6ea7f4b84715710bc319e6d -X "github.com/cosmos/cosmos-sdk/version.BuildTags=netgo,ledger,gcc" -X github.com/cosmos/cosmos-sdk/types.DBBackend=cleveldb -w -s' -trimpath -o /home/mar/go/src/github.com/osmosis-labs/osmosis/build/ ./... |
|
your errors resemble this? github.com/jmhodges/levigo../go/pkg/mod/github.com/jmhodges/levigo@v1.0.0/batch.go:4:11: fatal error: 'leveldb/c.h' file not found |
|
To make-happy goRocksdb, do exactly, on exactly arch linux: pacman -Syyu rocksdb base-devel snappy cmakeyour go.mod needs to look like, for v4.2.0 w/ speed boosts: For 3.1.0 w/o speed boosts: If you value your time & sanity, do use arch linux. |
|
I think that we can eliminate badgerdb from the competition. It is really, really slow doing the update from v3.1.0 to v4.2.0, even with the caching changes. I will still run this archive node to the tip, check the size of the node when it is finished updating, and the like. Rocksdb seems like it is off to a very strong start indeed. Our relayers run against full archive nodes, so when rocks, goleveldb, and badger are fully synced, it'll be possible to have a compeition of sorts. I don't really see any advantages to using cleveldb-- are there any>? |
|
@faddat thanks for all the detailed input!
ACK
Based on older community feedback at least modest performance improvements with a relatively easy build process.
They still offer it as an option for validators to choose. https://docs.desmos.network/fullnode/rocksdb-installation/ I distilled installation instructions for a functional Ubuntu (tested in v20.04) rocksdb installation from this issue git clone https://github.com/facebook/rocksdb.git
cd rocksdb
git checkout v5.18.3
make clean
make shared_lib -j4
sudo cp --preserve=links ./librocksdb.* /usr/local/lib/
sudo cp -r ./include/rocksdb/ /usr/local/include/
sudo ldconfigTo test a successful check the following commands whether they link with the cd /tmp
g++ -std=c++11 -o rocksdbtest test.cpp -lrocksdb -lpthread -ldl
./rocksdbtest
get bar success!!Yet CGO_CFLAGS="-I/path/to/rocksdb/include" \
CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4 -lzstd" \
go get github.com/tecbot/gorocksdb
# github.com/tecbot/gorocksdb
../../../../pkg/mod/github.com/roysc/gorocksdb@v1.1.1/transactiondb.go:255:17: could not determine kind of name for C.rocksdb_optimistictransactiondb_checkpoint_object_create
../../../../pkg/mod/github.com/roysc/gorocksdb@v1.1.1/transactiondb.go:208:12: could not determine kind of name for C.rocksdb_optimistictransactiondb_property_value
../../../../pkg/mod/github.com/roysc/gorocksdb@v1.1.1/transactiondb.go:236:2: could not determine kind of name for C.rocksdb_optimistictransactiondb_write
../../../../pkg/mod/github.com/roysc/gorocksdb@v1.1.1/transactiondb.go:59:12: could not determine kind of name for C.rocksdb_transactiondb_property_value
Unfortunately, not an available option. |
|
I built https://github.com/notional-labs/sos around arch because it was the only place I could get C++ builds to work correctly, consistently. ah, I think you missed something in my directions-- check out the info on go.mod morpheus voice What if I told you Arch is always an option, but you just didn't know how bad the version soup is elsewhere? |
|
Sorry, behind on following up on this. Will read through in a bit more detail in a few days, currently trying to get v5 testing things through before focusing much more time on performance issues. LevelDB is a pretty bad fit for our access patterns, so it seems like we should not use this. I'd heavily recommend not focusing on CLevelDB. Rocks DB and BadgerDB are the places we should put more focus on imo. @ValNodes is looking into the data structures / db configurations in more detail to get better opinions on which is better for our use case. We're also working on getting much better data access patterns in IAVL for most usecases. |
|
We decided to use rocksdb for now. There are significant issues with leveldb that will not work with our use case. Thank you for the suggestion! |
Hi,
Osmosis version:
git checkout v3.1.0Error in Ubuntu 20.04 with the cleveldb option enabled
build Instruction
Start script
Clevel db build instructions ref
What am I missing?
The text was updated successfully, but these errors were encountered: