-
-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ethdb/s3: segment storage #235
Conversation
semaphore: semaphore.NewWeighted(int64(maxOpenCount)), | ||
segments: make(map[string]Segment), | ||
} | ||
ss.cache, _ = lru.NewWithEvict(maxOpenCount, ss.onEvicted) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it safe to ignore this error?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it only returns an error if maxOpenCount
is less than 1. I added a default at the top of this function: https://github.com/gochain-io/gochain/pull/235/files/0a6dbc9ae552b875cd0450627b9c8f1c89976a23#diff-4a5dc09daf342ff846651415bb5cc7c2R22
Overview
This pull request adds support for S3-backed tiered storage for
ethdb
. It works by injecting anethdb.SegmentOpener
ðdb.SegmentCompactor
intoethdb.DB
so that segments can be backed up upon compaction and fetched as necessary. Theethdb.Table
holds a reference to thes3.Segment
which will open/fetch segments on request.This pull request does not include the LRU cache to limit the maximum space used. That will come in a later PR.
Usage
The S3 configuration can be set from the CLI or configuration:
The
archive
tool has been removed. The configuration fields are now in thenode.Config
.Testing
An integration test has been added to the
ethdb/s3
package to test the uploading & fetching. It can be run by specifying theintegration
build tag and passing in the appropriate flags:$ test -tags integration ./ethdb/s3 -endpoint ENDPOINT -bucket BUCKET -access-key-id ACCESSKEYID -secret-access-key SECRETACCESSKEY