Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

republishing and storage #21

Merged
merged 7 commits into from
Oct 23, 2023
Merged

republishing and storage #21

merged 7 commits into from
Oct 23, 2023

Conversation

decentralgabe
Copy link
Member

@decentralgabe decentralgabe commented Oct 21, 2023

need to add tests

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This intentional?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removing it for now in a follow up


type PKARRRecord struct {
// Up to an 1000 byte base64URL encoded string
V string `json:"v" validate:"required"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possible to add length validation for these 3 fields?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is validated up the call stack, will open an issue for further validation

@@ -83,3 +132,54 @@ func (s *PKARRService) GetPKARR(ctx context.Context, id string) (*GetPKARRRespon
Sig: got.Sig,
}, nil
}

// TODO(gabe) make this more efficient. create a publish schedule based on each individual record, not all records
func (s *PKARRService) republish() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this meant to scale? How will sharding work?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no sharding planned. it will scale better with individual publishing schedules


// TODO(gabe) make this more efficient. create a publish schedule based on each individual record, not all records
func (s *PKARRService) republish() {
allRecords, err := s.db.ListRecords()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This call may simply return a subset of all the results. Might need to do some cursor tracking

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree, will open up an issue for this

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#23

dht: d,
scheduler: &scheduler,
}
if err = scheduler.Schedule(cfg.PKARRConfig.RepublishCRON, service.republish); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way this is setup, there will be a republish job for each instance that's running did-dht. Is that intentional?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intended to run as a single instance

@decentralgabe decentralgabe merged commit f93a1e2 into main Oct 23, 2023
4 checks passed
@decentralgabe decentralgabe deleted the republishing branch October 23, 2023 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants