Support mounting remote storage

[sambacha]

There are several ways to do this, the benefit is that (for me at least) I can access geth archive state from our production replication service. Geth requires a beacon client to sync to mainnet, which makes it more cumbersome, at least for mainnet use case.

I am refering to SSHFS, etc. Not S3.

[fxfactorial]

Agree - this is a good idea. I think as a fastest solution , I could do like what emacs tramps kind of does. that is, is you provide the path to the db (and I assume you know you can't use this on a running chain, just a snapshot hence I realize why you said replications service aka you can temp stop it while using the db)

you provide the path to the db starting with ssh, so like ssh:///home/repl1/eth-mainnet/geth/chaindata and then I from then on will do ssh reads for whenever the golang code needs a file read operation . I think that's the fastest, easiest way cause I don't want a second binary/daemon/websocket. Everyone already has ssh access to their servers

[fxfactorial]

also are you mainly using pebble backed or leveldb backed machines? I assume probably leveldb?

[fxfactorial]

actually I just looked at this again https://github.com/libfuse/sshfs - did you try it, maybe it should work already as is?

[fxfactorial]

@sambacha Okay so I tried this and while it does work , its painfully slow. Like you'll be waiting over 15 minutes (and this was for a snap synced dir, so I imagine archive one will be much worse) to do initial dir load (as there's a dir scan that happens) and sshfs will do a copy to in memory of each file. Chaindata itself will have many hundred of files so it's just not feasible.

[sambacha]

@sambacha Okay so I tried this and while it does work , its painfully slow. Like you'll be waiting over 15 minutes (and this was for a snap synced dir, so I imagine archive one will be much worse) to do initial dir load (as there's a dir scan that happens) and sshfs will do a copy to in memory of each file. Chaindata itself will have many hundred of files so it's just not feasible.

What are you using for remote FS, FUSE?

[fxfactorial]

yes sshfs , and this was on local network too so I imagine remote on different network will be even worse

sshfs -o debug,sshfs_debug,loglevel=debug,allow_other,kill_on_unmount,reconnect,allow_other,direct_io,auto_cache thelio-archive:/eth-archive/goerli temp-mount-2

where sshfs running on my MacBook and thelio-archive is my local network linux machine.

[sambacha]

OK OK, how about something similar to this?

ethereum/go-ethereum#26621

This PR is allows users to export their chain into an an archive format called Era1. It is formulated similarly to the Era1 format, which is optimized for reading and distribution CL data. The Era and Era1 format are stricter subsets of a simple type-length-value scheme called e2store2, both developed by the Nimbus team.

[fxfactorial]

Interesting - I didn't know about that will look up