Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ethdb, eth, core: archive old chaindata to s3 compatible bucket #105

Merged
merged 1 commit into from
Mar 31, 2018

Conversation

jmank88
Copy link
Contributor

@jmank88 jmank88 commented Mar 30, 2018

This PR adds the option to archive old chaindata from leveldb to an s3 compatible bucket. The archive is enabled by setting the following flags:

ARCHIVE OPTIONS:
  --archive value        S3 compatible archive endpoint.
  --archivebucket value  Name of archive bucket. Must already exist.
  --archiveid value      Archive access key ID.
  --archivesecret value  Archive access key secret.
  --archiveage value     Archive age. The number of blocks from head before archiving. (default: 100000)
  --archiveperiod value  How often the archive process runs. (default: 5m0s)

When the archive is enabled, a background process periodically sweeps leveldb for data to archive. Currently this is header, body, and receipts data older than 100000 blocks from the current head.

The archive emits metrics (if enabled) as db/archive/chaindata/*, similar to the existing levedb metrics under db/chaindata/*, for comparison.

#100

@jmank88
Copy link
Contributor Author

jmank88 commented Mar 30, 2018

I've been running a local devnet node, and archiving to this bucket: https://cloud.digitalocean.com/spaces/devnet-archive?i=782cce


const (
// archiveThreshold is the distance from latest at which data is eligible for archival.
archiveThreshold = 10000
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this should be ~120000, for a week of 5 second blocks (#100).

@jmank88 jmank88 force-pushed the db-archive branch 3 times, most recently from deed28a to 0ca8d4f Compare March 30, 2018 20:37
@jmank88 jmank88 changed the title WIP: ethdb, eth, core: archive old chaindata to s3 compatible bucket ethdb, eth, core: archive old chaindata to s3 compatible bucket Mar 30, 2018
@jmank88 jmank88 requested a review from treeder March 30, 2018 22:53
@jmank88
Copy link
Contributor Author

jmank88 commented Mar 30, 2018

Removing WIP, since this is stable and behind a feature flag. Needs further testing after merge.

@jmank88 jmank88 merged commit ad872e6 into master Mar 31, 2018
@jmank88 jmank88 deleted the db-archive branch March 31, 2018 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant