Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce dynamoDB as a Database implementation #598

Merged
merged 6 commits into from Aug 12, 2020

Conversation

aidan-kwon
Copy link
Member

Proposed changes

  • DynamoDB store data into AWS dynamoDB.
  • If the data is bigger than 400 KB, it will be stored in S3.

Types of changes

Please put an x in the boxes related to your change.

  • Bugfix
  • New feature or enhancement
  • Others

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

  • I have read the CONTRIBUTING GUIDELINES doc
  • I have signed the CLA
  • Lint and unit tests pass locally with my changes ($ make test)
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)
  • Any dependent changes have been merged and published in downstream modules

@@ -1090,6 +1095,7 @@ func (bc *BlockChain) writeStateTrie(block *types.Block, state *state.StateDB) e
nodesSize, preimagesSize = trieDB.Size()
nodesSizeLimit = common.StorageSize(bc.cacheConfig.CacheSize) * 1024 * 1024
)
// TODO-Aidan: increase size limit
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// TODO-Aidan: increase size limit
// TODO-Klaytn: consider to increase size limit or parameterize

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the comment

}

logger.Info("creating s3FileDB ", "bucket", config.TableName+"-bucket")
s3FileDB, err := newS3FileDB(config.Region, "https://s3.ap-northeast-2.amazonaws.com", config.TableName+"-bucket")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it ok that url includes ap-northeast-2?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, as far as I know the region should be specified in the endpoint URL.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the value will be replaced to the user-configurable value.

cmd/utils/flags.go Outdated Show resolved Hide resolved
storage/database/partitioned_database.go Outdated Show resolved Hide resolved
storage/database/interface.go Outdated Show resolved Hide resolved
node/cn/config.go Outdated Show resolved Hide resolved
}

logger.Info("creating s3FileDB ", "bucket", config.TableName+"-bucket")
s3FileDB, err := newS3FileDB(config.Region, "https://s3.ap-northeast-2.amazonaws.com", config.TableName+"-bucket")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, as far as I know the region should be specified in the endpoint URL.

storage/database/dynamodb.go Show resolved Hide resolved
@aidan-kwon
Copy link
Member Author

This PR was a draft which refines some code from https://github.com/winnie-byun/klaytn/tree/dynamo2-noBatch4
I am gonna change this PR to be mergeable version, please wait to add comments.

@aidan-kwon
Copy link
Member Author

Reviewers, please take a look. Consider that dynamoDB config values will be configurable later.

ehnuje
ehnuje previously approved these changes Aug 11, 2020
@KimKyungup KimKyungup self-requested a review August 12, 2020 01:24
Copy link
Contributor

@KimKyungup KimKyungup left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. :)
Merge this.
@winnie-byun please refine more with some comments.

Copy link
Contributor

@joowon-byun joowon-byun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :)
@aidan-kwon For work efficiency, I will refine myself after this PR.

@KimKyungup KimKyungup merged commit 5bbd345 into klaytn:dev Aug 12, 2020
@aidan-kwon aidan-kwon deleted the 0810-dynamo branch February 22, 2021 04:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants