Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support standalone miner-market process #5983

Closed
wants to merge 9 commits into from

Conversation

magik6k
Copy link
Contributor

@magik6k magik6k commented Apr 7, 2021

This PR adds support for running the miner market module as a separate process

  • A lotus-miner market process connects to the "main" lotus process for all sealing/storage interactions
  • It can pull data directly from workers
  • It has it's own libp2p node

TODO:

  • Fix api deps
    • (don't depend on ffi)
  • Fix node tests
  • Delegate market node auth to the main node (storage subsystem assumes that)
    • Put API Secret behind an interface, in DI replace the implementation by the main node API
  • Figure out what to do about local market-node storage
    • Either drop completely (likely preferred initially)
      • This likely just means making the local storage implementation ignore paths, or (a bit cleaner) making it possible to run Remote store without a Local store
    • Or register storage properly (and handle disconnects, a bit like workers do)
  • Use http-mode jsonrpc for talking to the main Sterage/Sealing node (for reliability)
  • Return nicer errors from unsupported api calls when subsystems are disabled
    • Not sure if we can do much better than if x == nil { error(module x is disabled) }
  • Don't construct miner Libp2p node when not needed
    • Mostly just shifting things around in the node builder
  • Bump API versions which need bumping
    • Miner version at least
  • Check API versions on reconnect
  • Setup docs
  • Write integration test
    • Copy one of the api/test/deal.go tests, and run in this setup

@magik6k magik6k requested review from raulk and Kubuxu April 7, 2021 19:17
Comment on lines +409 to +433
log.Infof("Read local %s (+%d,%d)", path, offset, size)
ssize, err := s.ProofType.SectorSize()
if err != nil {
return nil, err
}

pf, err := ffiwrapper.OpenPartialFile(abi.PaddedPieceSize(ssize), path)
if err != nil {
return nil, xerrors.Errorf("opening partial file: %w", err)
}

has, err := pf.HasAllocated(storiface.UnpaddedByteIndex(offset.Unpadded()), size.Unpadded())
if err != nil {
return nil, xerrors.Errorf("has allocated: %w", err)
}

if !has {
if err := pf.Close(); err != nil {
return nil, xerrors.Errorf("close partial file: %w", err)
}

return nil, nil
}

return pf.Reader(storiface.PaddedByteIndex(offset), size)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part technically should live on the Local store

@jennijuju jennijuju added this to the v1.9.x milestone Apr 30, 2021
@jennijuju jennijuju modified the milestones: v1.9.x, v1.11.x May 18, 2021
@magik6k
Copy link
Contributor Author

magik6k commented Jun 14, 2021

Superseded by #6356

@magik6k magik6k closed this Jun 14, 2021
@Kubuxu Kubuxu deleted the feat/miner-microservices branch November 25, 2021 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants