An implementation of the graphsync protocol in go!
GraphSync is a protocol for synchronizing IPLD graphs among peers. It allows a host to make a single request to a remote peer for all of the results of traversing an IPLD selector on the remote peer's local IPLD graph.
go-graphsync
provides an implementation of the Graphsync protocol in go.
go-graphsync
relies on go-ipld-prime
to traverse IPLD Selectors in an IPLD graph. go-ipld-prime
implements the IPLD specification in go and is an alternative to older implementations such as go-ipld-format
and go-ipld-cbor
. In order to use go-graphsync
, some understanding and use of go-ipld-prime
concepts is necessary.
If your existing library (i.e. go-ipfs
or go-filecoin
) uses these other older libraries, you can largely use go-graphsync without switching to go-ipld-prime
across your codebase, but it will require some translations
go-graphsync
requires Go >= 1.11 and can be installed using Go modules
import (
graphsync "github.com/TRON-US/go-graphsync/impl"
gsnet "github.com/TRON-US/go-graphsync/network"
gsbridge "github.com/TRON-US/go-graphsync/ipldbridge"
ipld "github.com/ipld/go-ipld-prime"
)
var ctx context.Context
var host libp2p.Host
var loader ipld.Loader
var storer ipld.Storer
network := gsnet.NewFromLibp2pHost(host)
exchange := graphsync.New(ctx, network, loader, storer)
Parameter Notes:
context
is just the parent context for all of GraphSyncnetwork
is a network abstraction provided to Graphsync on top of libp2p. This allows graphsync to be tested without the actual networkloader
is used to load blocks from content ids from the local block store. It's used when RESPONDING to requests from other clients. It should conform to the IPLD loader interface: https://github.com/ipld/go-ipld-prime/blob/master/linking.gostorer
is used to store incoming blocks to the local block store. It's used when REQUESTING a graphsync query, to store blocks locally once they are validated as part of the correct response. It should conform to the IPLD storer interface: https://github.com/ipld/go-ipld-prime/blob/master/linking.go
GraphSync provides two convenience functions in the storeutil
package for
integrating with BlockStore's from IPFS.
import (
graphsync "github.com/TRON-US/go-graphsync/impl"
gsnet "github.com/TRON-US/go-graphsync/network"
gsbridge "github.com/TRON-US/go-graphsync/ipldbridge"
gsbridge "github.com/TRON-US/go-graphsync/storeutil"
ipld "github.com/ipld/go-ipld-prime"
blockstore "github.com/ipfs/go-ipfs-blockstore"
)
var ctx context.Context
var host libp2p.Host
var bs blockstore.Blockstore
network := gsnet.NewFromLibp2pHost(host)
loader := storeutil.LoaderForBlockstore(bs)
storer := storeutil.StorerForBlockstore(bs)
exchange := graphsync.New(ctx, network, loader, storer)
If you are using a traditional go-ipfs-blockstore, your link loading function looks like this:
type BlockStore interface {
Get(lnk cid.Cid) (blocks.Block, error)
}
or, more generally:
type Cid2BlockFn func (lnk cid.Cid) (blocks.Block, error)
in go-ipld-prime
, the signature for a link loader is as follows:
type Loader func(lnk Link, lnkCtx LinkContext) (io.Reader, error)
go-ipld-prime
intentionally keeps its interfaces as abstract as possible to limit dependencies on other ipfs/filecoin specific packages. An IPLD Link is an abstraction for a CID, and IPLD expects io.Reader's rather than an actual block. IPLD provides a cidLink
package for working with Links that use CIDs as the underlying data, and it's safe to assume that's the type in use if your code deals only with CIDs. A conversion would look something like this:
import (
ipld "github.com/ipld/go-ipld-prime"
cidLink "github.com/ipld/go-ipld-prime/linking/cid"
)
func LoaderFromCid2BlockFn(cid2BlockFn Cid2BlockFn) ipld.Loader {
return func(lnk ipld.Link, lnkCtx ipld.LinkContext) (io.Reader, error) {
asCidLink, ok := lnk.(cidlink.Link)
if !ok {
return nil, fmt.Errorf("Unsupported Link Type")
}
block, err := cid2BlockFn(asCidLink.Cid)
if err != nil {
return nil, err
}
return bytes.NewReader(block.RawData()), nil
}
}
If you are using a traditional go-ipfs-blockstore, your storage function looks like this:
type BlockStore interface {
Put(blocks.Block) error
}
or, more generally:
type BlockStoreFn func (blocks.Block) (error)
in go-ipld-prime
, the signature for a link storer is a bit different:
type StoreCommitter func(Link) error
type Storer func(lnkCtx LinkContext) (io.Writer, StoreCommitter, error)
go-ipld-prime
stores in two parts to support streaming -- the storer is called and returns an IO.Writer and a function to commit changes when finished. Here's how you can write a storer from a traditional block storing signature.
import (
blocks "github.com/ipfs/go-block-format"
ipld "github.com/ipld/go-ipld-prime"
cidLink "github.com/ipld/go-ipld-prime/linking/cid"
)
func StorerFromBlockStoreFn(blockStoreFn BlockStoreFn) ipld.Storer {
return func(lnkCtx ipld.LinkContext) (io.Writer, ipld.StoreCommitter, error) {
var buffer bytes.Buffer
committer := func(lnk ipld.Link) error {
asCidLink, ok := lnk.(cidlink.Link)
if !ok {
return fmt.Errorf("Unsupported Link Type")
}
block := blocks.NewBlockWithCid(buffer.Bytes(), asCidLink.Cid)
return blockStoreFn(block)
}
return &buffer, committer, nil
}
}
var exchange graphsync.GraphSync
var ctx context.Context
var p peer.ID
var selector ipld.Node
var rootLink ipld.Link
var responseProgress <-chan graphsync.ResponseProgress
var errors <-chan error
responseProgress, errors = exchange.Request(ctx context.Context, p peer.ID, root ipld.Link, selector ipld.Node)
Paramater Notes:
ctx
is the context for this request. To cancel an in progress request, cancel the context.p
is the peer you will send this request tolink
is an IPLD Link, i.e. a CID (cidLink.Link{Cid})selector
is an IPLD selector node. Recommend using selector builders from go-ipld-prime to construct these
type ResponseProgress struct {
Node ipld.Node // a node which matched the graphsync query
Path ipld.Path // the path of that node relative to the traversal start
LastBlock struct { // LastBlock stores the Path and Link of the last block edge we had to load.
ipld.Path
ipld.Link
}
}
The above provides both immediate and relevant metadata for matching nodes in a traversal, and is very similar to the information provided by a local IPLD selector traversal in go-ipld-prime
PRs are welcome!
Before doing anything heavy, checkout the Graphsync Architecture
This library is dual-licensed under Apache 2.0 and MIT terms.
Copyright 2020. TRON-US.