ethdb/s3: segment storage #235
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jmank88
reviewed
Jul 27, 2018
| semaphore: semaphore.NewWeighted(int64(maxOpenCount)), | ||
| segments: make(map[string]Segment), | ||
| } | ||
| ss.cache, _ = lru.NewWithEvict(maxOpenCount, ss.onEvicted) |
There was a problem hiding this comment.
Is it safe to ignore this error?
There was a problem hiding this comment.
Yes, it only returns an error if maxOpenCount is less than 1. I added a default at the top of this function: https://github.com/gochain-io/gochain/pull/235/files/0a6dbc9ae552b875cd0450627b9c8f1c89976a23#diff-4a5dc09daf342ff846651415bb5cc7c2R22
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This pull request adds support for S3-backed tiered storage for
ethdb. It works by injecting anethdb.SegmentOpenerðdb.SegmentCompactorintoethdb.DBso that segments can be backed up upon compaction and fetched as necessary. Theethdb.Tableholds a reference to thes3.Segmentwhich will open/fetch segments on request.This pull request does not include the LRU cache to limit the maximum space used. That will come in a later PR.
Usage
The S3 configuration can be set from the CLI or configuration:
The
archivetool has been removed. The configuration fields are now in thenode.Config.Testing
An integration test has been added to the
ethdb/s3package to test the uploading & fetching. It can be run by specifying theintegrationbuild tag and passing in the appropriate flags:$ test -tags integration ./ethdb/s3 -endpoint ENDPOINT -bucket BUCKET -access-key-id ACCESSKEYID -secret-access-key SECRETACCESSKEY