Skip to content
Initial Implementation Of GraphSync Wire Protocol
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
ci ci(Jenkins): Add Jenkinsfile Feb 6, 2019
docs docs(architecture): flesh out architecture docs Apr 30, 2019
ipldbridge feat(loader): loader for request manager Apr 19, 2019
linktracker refactor(linktracker): move to top level package Apr 19, 2019
message feat(message): Modify message API Apr 19, 2019
messagequeue feat(message): Modify message API Apr 19, 2019
metadata refactor(metadata): convert to array structure Apr 19, 2019
network feat(graphsync): can make roundtrip! Apr 19, 2019
peermanager
requestmanager feat(graphsync): can make roundtrip! Apr 19, 2019
responsemanager refactor(peertaskqueue): extract to package May 10, 2019
testbridge feat(requestmanager): perform traversals Apr 19, 2019
testutil refactor(types): Move ResponseProgress Apr 19, 2019
COPYRIGHT docs(license): update to dual license Mar 20, 2019
LICENSE-APACHE docs(license): update to dual license Mar 20, 2019
LICENSE-MIT docs(license): update to dual license Mar 20, 2019
README.md docs(README): update initialization steps Apr 19, 2019
go.mod refactor(peertaskqueue): extract to package May 10, 2019
go.sum refactor(peertaskqueue): extract to package May 10, 2019
graphsync.go refactor(peertaskqueue): extract to package May 10, 2019
graphsync_test.go

README.md

go-graphsync

Coverage Status Travis CI

An implementation of the graphsync protocol in go!

Table of Contents

Background

GraphSync is a protocol for synchronizing IPLD graphs among peers. It allows a host to make a single request to a remote peer for all of the results of traversing an IPLD selector on the remote peer's local IPLD graph.

go-graphsync provides an implementation of the Graphsync protocol in go.

Go-IPLD-Prime

go-graphsync relies on go-ipld-prime to traverse IPLD Selectors in an IPLD graph. go-ipld-prime implements the IPLD specification in go and is an alternative to older implementations such as go-ipld-format and go-ipld-cbor. In order to use go-graphsync, some understanding and use of go-ipld-prime concepts is necessary.

If your existing library (i.e. go-ipfs or go-filecoin) uses these other older libraries, go-graphsync provide translation layers so you can largely use it without switching to go-ipld-prime across your codebase.

Install

go-graphsync requires Go >= 1.11 and can be installed using Go modules

Usage

Initializing a GraphSync Exchange

import (
  graphsync "github.com/ipfs/go-graphsync"
  gsnet "github.com/ipfs/go-graphsync/network"
  gsbridge "github.com/ipfs/go-graphsync/ipldbridge"
  ipld "github.com/ipfs/go-ipld-prime"
)

var ctx context.Context
var host libp2p.Host
var loader ipld.Loader

network := gsnet.NewFromLibp2pHost(host)
ipldBridge := gsbridge.NewIPLDBridge()
exchange := graphsync.New(ctx, network, ipldBridge, loader)

Parameter Notes:

  1. context is just the parent context for all of GraphSync
  2. network is a network abstraction provided to Graphsync on top of libp2p. This allows graphsync to be tested without the actual network
  3. ipldBridge is an IPLD abstraction provided to Graphsync on top of go-ipld-prime. This makes the graphsync library testable in isolation
  4. loader is used to load blocks from content ids from the local block store. It's used when RESPONDING to requests from other clients. It should conform to the IPLD loader interface: https://github.com/ipld/go-ipld-prime/blob/master/linking.go

Write A Loader From The Stuff You Know

Coming from a pre-go-ipld-prime world, you probably expect a link loading function signature to look like this:

type Cid2BlockFn func (lnk cid.Cid) (blocks.Block, error)

in go-ipld-prime, the signature for a link loader is as follows:

type Loader func(lnk Link, lnkCtx LinkContext) (io.Reader, error)

go-ipld-prime intentionally keeps its interfaces as abstract as possible to limit dependencies on other ipfs/filecoin specific packages. An IPLD Link is an abstraction for a CID, and IPLD expects io.Reader's rather than an actual block. IPLD provides a cidLink package for working with Links that use CIDs as the underlying data, and it's safe to assume that's the type in use if your code deals only with CIDs. Anyway, a conversion would look something like this:

import (
   ipld "github.com/ipld/go-ipld-prime"
   cidLink "github.com/ipld/go-ipld-prime/linking/cid"
)

func LoaderFromCid2BlockFn(cid2BlockFn Cid2BlockFn) ipld.Loader {
	return func(lnk ipld.Link, lnkCtx ipld.LinkContext) (io.Reader, error) {
		asCidLink, ok := lnk.(cidlink.Link)
		if !ok {
			return nil, fmt.Errorf("Unsupported Link Type")
		}
		block, err := cid2BlockFn(asCidLink.Cid)
		if err != nil {
			return nil, err
		}
		return bytes.NewReader(block.RawData()), nil
	}
}

Alternatively, you can just call:

loader := graphsync.LoaderFromCid2BlockFn(cid2BlockFn)

Calling Graphsync

var exchange graphsync.GraphSync
var ctx context.Context
var p peer.ID
var cidRootedSelector ipld.Node

var responseProgress <-chan graphsync.ResponseProgress
var errors <-chan error

responseProgress, errors = exchange.Request(ctx context.Context, p peer.ID, rootedSelector Node)

Paramater Notes:

  1. ctx is the context for this request. To cancel an in progress request, cancel the context.
  2. p is the peer you will send this request to
  3. rootedSelector is the a go-ipld-prime node the specifies a rooted selector

Building a path selector

A rooted selector is a go-ipld-prime node that follows the spec outlined here: https://github.com/ipld/specs/blob/master/selectors/selectors.md

go-ipld-prime provides a series of builder interfaces for building this kind of structured data into a node. If your library simply wants to make a selector from CID and a path, represented by an array of strings, you could construct the node as follows:

import (
	ipld "github.com/ipld/go-ipld-prime"
	free "github.com/ipld/go-ipld-prime/impl/free"
	fluent "github.com/ipld/go-ipld-prime/fluent"
	cidLink "github.com/ipld/go-ipld-prime/linking/cid"
)

func SelectorSpecFromCidAndPath(lnk cid.Cid, pathSegments []string) (ipld.Node, error) {
	var node ipld.Node
	err := fluent.Recover(func() {
		builder := fluent.WrapNodeBuilder(free.NodeBuilder())
		node = builder.CreateMap(func (mb fluent.MapBuilder, knb fluent.NodeBuilder, vnb fluent.NodeBuilder) {
			mb.Insert(knb.CreateString("root"), vnb.CreateLink(cidLink.Link{lnk}))
			mb.Insert(knb.CreateString("selectors"), 
			vnb.CreateList(func (lb fluent.ListBuilder, vnb fluent.NodeBuilder) {
				for _, pathSegment := range pathSegments {
					lb.Append(CreateMap(
						func (mb fluent.MapBuilder, knb fluent.NodeBuilder, vnb fluent.NodeBuilder) {
							mb.Insert(knb.CreateString("selectPath"), vnb.CreateString(pathSegment))
						},
					))
				}
			}))
		});
	})
	if err != nil {
		return nil, err
	}
	return node, nil
}

Alternatively, just call:

rootedSelector := graphsync.SelectorSpecFromCidAndPath(lnk, pathSegments)

Response Type

type ResponseProgress struct {
  Node      ipld.Node // a node which matched the graphsync query
  Path      ipld.Path // the path of that node relative to the traversal start
	LastBlock struct {  // LastBlock stores the Path and Link of the last block edge we had to load. 
		ipld.Path
		ipld.Link
	}
}

The above provides both immediate and relevant metadata for matching nodes in a traversal, and is very similar to the information provided by a local IPLD selector traversal in go-ipld-prime

Compatibility: Block Requests

While the above is extremely useful if your library already uses go-ipld-prime, if your library uses an older version of go ipld libraries, working with these types of go-ipld-prime data may prove challenging.

To support these clients, Graphsync provides a compatibility version of the above function that returns just blocks that were traversed:

var blocksChan <-chan blocks.Block
var errors <-chan error

blocksChan, errors = exchange.GetBlocks(ctx context.Context, p peer.ID, rootedSelector Node)

This is provided as a transitional layer and go-graphsync may drop support for this format in the future.

Contribute

PRs are welcome!

Small note: If editing the Readme, please conform to the standard-readme specification.

License

This library is dual-licensed under Apache 2.0 and MIT terms.

Copyright 2019. Protocol Labs, Inc.

You can’t perform that action at this time.