go-libp2p documentation effort

README.md

@@ -6,44 +6,85 @@
 [![GoDoc](https://godoc.org/github.com/ipfs/go-libp2p?status.svg)](https://godoc.org/github.com/libp2p/go-libp2p)
 [![Build Status](https://travis-ci.org/ipfs/go-libp2p.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p)
 
-![](https://raw.githubusercontent.com/diasdavid/specs/libp2p-spec/protocol/network/figs/logo.png)
+<p align="center">
+  <img src="https://github.com/libp2p/libp2p/raw/master/logo/libp2p-logo.png" />
+</p>
 
 > libp2p implementation in Go
 
 # Description
 
 [libp2p](https://github.com/libp2p/specs) is a networking stack and library modularized out of [The IPFS Project](https://github.com/ipfs/ipfs), and bundled separately for other tools to use.
->
-libp2p is the product of a long, and arduous quest of understanding -- a deep dive into the internet's network stack, and plentiful peer-to-peer protocols from the past. Building large scale peer-to-peer systems has been complex and difficult in the last 15 years, and libp2p is a way to fix that. It is a "network stack" -- a protocol suite -- that cleanly separates concerns, and enables sophisticated applications to only use the protocols they absolutely need, without giving up interoperability and upgradeability. libp2p grew out of IPFS, but it is built so that lots of people can use it, for lots of different projects.
->
-> We will be writing a set of docs, posts, tutorials, and talks to explain what p2p is, why it is tremendously useful, and how it can help your existing and new projects. But in the meantime, check out
->
-> - [**The IPFS Network Spec**](https://github.com/libp2p/specs), which grew into libp2p
-> - [**go-libp2p implementation**](https://github.com/libp2p/go-libp2p)
-> - [**js-libp2p implementation**](https://github.com/libp2p/js-libp2p)
 
-# Contribute
+> libp2p is the product of a long, and arduous quest of understanding -- a deep dive into the internet's network stack, and plentiful peer-to-peer protocols from the past. Building large scale peer-to-peer systems has been complex and difficult in the last 15 years, and libp2p is a way to fix that. It is a "network stack" -- a protocol suite -- that cleanly separates concerns, and enables sophisticated applications to only use the protocols they absolutely need, without giving up interoperability and upgradeability. libp2p grew out of IPFS, but it is built so that lots of people can use it, for lots of different projects.
 
-libp2p implementation in Go is a work in progress. As such, there's a few things you can do right now to help out:
- - Go through the modules below and **check out existing issues**. This would be especially useful for modules in active development. Some knowledge of IPFS/libp2p may be required, as well as the infrasture behind it - for instance, you may need to read up on p2p and more complex operations like muxing to be able to help technically.
- - **Perform code reviews**.
- - **Add tests**. There can never be enough tests.
+This is the entry point for the Go implementation of `libp2p`, which is formed by multiple modules, usually accessible under the `github.com/libp2p/go-libp2p-*` namespace.
 
 # Usage
 
-`go-libp2p` repo will be a place holder for the list of Go modules that compose Go libp2p, as well as its entry point.
+The vast majority of `go-libp2p` modules live in their own repositories. This repository is the apex of the Go `libp2p` stack, and primarily serves to track the working-version set of all the different modules.
 
-## Install
+This is done using [Gx](https://github.com/whyrusleeping/gx) and its companion [Gx-go](https://github.com/whyrusleeping/gx-go).
+
+The [`package.json`](https://github.com/libp2p/go-libp2p/blob/master/package.json) file maintains the current list of dependencies which, when used, provide a working version of the stack.
+
+**Any `go-libp2p` application (including the examples in this repository) need to have their imports rewritten with Gx, otherwise compilation will fail.**
+
+The process to get `go-libp2p` working in your project is as follows:
+
+* Make sure you install Gx and Gx-go:
 
-```bash
-$ go get -d github.com/libp2p/go-libp2p
-$ cd $GOPATH/src/github.com/libp2p/go-libp2p
-$ make
 ```
+go get -u github.com/whyrusleeping/gx
+go get -u github.com/whyrusleeping/gx-go
+```
+
+* Initialize Gx in the base folder of your project's repository. This will create a `package.json` file:
+
+```
+gx init
+```
+
+* Import `go-libp2p` to your project. This will import `go-libp2p` on its latest published version:
+
+```
+gx import github.com/libp2p/go-libp2p
+```
+
+* Download and install all the `go-libp2p` dependencies:
+
+```
+gx install --verbose --global
+```
+
+* Write your `go-libp2p` code normally and use regular imports (i.e. `import "github.com/libp2p/go-libp2p-net"`)
+* Before compiling, run a Gx-go-rewrite command in the base folder of your repository: You will see that your `go-libp2p-*` imports have been rewritten to something like `gx/ipfs/QmVtMT3fD7DzQNW7hdm6Xe6KPstzcggrhNpeVZ4422UpKK/go-libp2p-net`:
+
+```
+gx-go rewrite
+```
+
+* With the imports rewritten, you can now compile your `go-libp2p`. If you don't rewrite your imports, your build will error as `go-libp2p` does not build on master.
+* Before committing, it is customary to undo the rewrites with:
+
+```
+gx-go rewrite --undo
+```
+
+Most projects include these steps in a `Makefile` for convenience. [Here is an example](https://github.com/libp2p/go-libp2p-raft/blob/master/Makefile).
+
+
+# Contribute
+
+libp2p implementation in Go is a work in progress. As such, there's a few things you can do right now to help out:
+ - Go through the modules below and **check out existing issues**. This would be especially useful for modules in active development. Some knowledge of IPFS/libp2p may be required, as well as the infrasture behind it - for instance, you may need to read up on p2p and more complex operations like muxing to be able to help technically.
+ - **Perform code reviews**.
+ - **Add tests**. There can never be enough tests.
+
 
 # Examples
 
-Examples can be found on the [examples folder](examples).
+Examples and information on how to run them can be found on the [examples folder](examples) and their respective READMEs.
 
 # Run tests
 
@@ -53,7 +94,6 @@ $ make deps
 $ go test ./p2p/<path of module you want to run tests for>
 ```
 
-
 ## Links
 - [**Specs**](https://github.com/libp2p/specs)
 - [**Website**](https://github.com/libp2p/website)

p2p/discovery/mdns.go

@@ -1,3 +1,6 @@
+// Package discovery provides mDNS announcements and discovery for libp2p hosts.
+// It can be used to auto-discover peers in networks where mDNS (multicast)
+// is not blocked.
 package discovery
 
 import (
@@ -21,14 +24,20 @@ import (
 
 var log = logging.Logger("mdns")
 
+// ServiceTag is used to identify the mDNS service
 const ServiceTag = "_ipfs-discovery._udp"
 
+// Service provides an interface for mDNS service implementations.
+// Currently they support registration and removal of notifees, which
+// are notified when a new peer is found.
 type Service interface {
 	io.Closer
 	RegisterNotifee(Notifee)
 	UnregisterNotifee(Notifee)
 }
 
+// Notifee is an interface which allows to be notified
+// when a new peer is found by a mDNS service.
 type Notifee interface {
 	HandlePeerFound(pstore.PeerInfo)
 }
@@ -61,6 +70,10 @@ func getDialableListenAddrs(ph host.Host) ([]*net.TCPAddr, error) {
 	return out, nil
 }
 
+// NewMdnsService creates and initializes a new Service. It receives a
+// cancellable context (used to cancel mDNS polling), a Host (whose information
+// is multicasted to the local network), and an interval which specifies how
+// often to perform mDNS queries.
 func NewMdnsService(ctx context.Context, peerhost host.Host, interval time.Duration) (Service, error) {
 
 	// TODO: dont let mdns use logging...

p2p/host/basic/basic_host.go

@@ -1,3 +1,4 @@
+// Package basichost provides a basic libp2p Host implementation.
 package basichost
 
 import (
@@ -21,6 +22,8 @@ import (
 
 var log = logging.Logger("basichost")
 
+// NegotiateTimeout sets a deadline to perform stream handling
+// negotiation (exchange of protocol IDs)
 var NegotiateTimeout = time.Second * 60
 
 // Option is a type used to pass in options to the host.
@@ -53,7 +56,9 @@ type BasicHost struct {
 	bwc metrics.Reporter
 }
 
-// New constructs and sets up a new *BasicHost with given Network
+// New constructs and sets up a new *BasicHost with given Network. It can take
+// additional options. Currently NATPortMap (see "Constants"
+// documentation) and a custom metrics.Reporter are supported.
 func New(net inet.Network, opts ...interface{}) *BasicHost {
 	h := &BasicHost{
 		network:          net,
@@ -167,12 +172,12 @@ func (h *BasicHost) Network() inet.Network {
 	return h.network
 }
 
-// Mux returns the Mux multiplexing incoming streams to protocol handlers
+// Mux returns the Mux multiplexing incoming streams to protocol handlers.
 func (h *BasicHost) Mux() *msmux.MultistreamMuxer {
 	return h.mux
 }
 
-// IDService returns
+// IDService returns the IDService for this host.
 func (h *BasicHost) IDService() *identify.IDService {
 	return h.ids
 }
@@ -201,7 +206,7 @@ func (h *BasicHost) SetStreamHandlerMatch(pid protocol.ID, m func(string) bool,
 	})
 }
 
-// RemoveStreamHandler returns ..
+// RemoveStreamHandler removes the handler matching the given protocol ID.
 func (h *BasicHost) RemoveStreamHandler(pid protocol.ID) {
 	h.Mux().RemoveHandler(string(pid))
 }

p2p/host/routed/routed.go

@@ -1,3 +1,6 @@
+// Package routedhost provides a wrapper for libp2p Host and a generic Routing
+// mechanism, which allow them to discover hosts which are not in their
+// peerstores.
 package routedhost
 
 import (
@@ -31,10 +34,13 @@ type RoutedHost struct {
 	route Routing
 }
 
+// The Routing interface allows to wrap any peer discovery implementations
 type Routing interface {
 	FindPeer(context.Context, peer.ID) (pstore.PeerInfo, error)
 }
 
+// Wrap creates a RoutedHost by wrapping a regular Host and a Routing
+// implementation.
 func Wrap(h host.Host, r Routing) *RoutedHost {
 	return &RoutedHost{h, r}
 }
@@ -85,41 +91,59 @@ func logRoutingErrDifferentPeers(ctx context.Context, wanted, got peer.ID, err e
 	log.Event(ctx, "routingError", lm)
 }
 
+// ID returns the (local) peer.ID associated with this Host
 func (rh *RoutedHost) ID() peer.ID {
 	return rh.host.ID()
 }
 
+// Peerstore returns the Host's repository of Peer Addresses and Keys.
 func (rh *RoutedHost) Peerstore() pstore.Peerstore {
 	return rh.host.Peerstore()
 }
 
+// Addrs returns all the addresses of BasicHost at this moment in time.
 func (rh *RoutedHost) Addrs() []ma.Multiaddr {
 	return rh.host.Addrs()
 }
 
+// Network returns the Network interface of the Host
 func (rh *RoutedHost) Network() inet.Network {
 	return rh.host.Network()
 }
 
+// Mux returns the Mux multiplexing incoming streams to protocol handlers.
 func (rh *RoutedHost) Mux() *msmux.MultistreamMuxer {
 	return rh.host.Mux()
 }
 
+// SetStreamHandler sets the protocol handler on the Host's Mux.
+// This is equivalent to:
+//   host.Mux().SetHandler(proto, handler)
+// (Threadsafe)
 func (rh *RoutedHost) SetStreamHandler(pid protocol.ID, handler inet.StreamHandler) {
 	rh.host.SetStreamHandler(pid, handler)
 }
 
+// SetStreamHandlerMatch sets the protocol handler on the Host's Mux
+// using a matching function to do protocol comparisons
 func (rh *RoutedHost) SetStreamHandlerMatch(pid protocol.ID, m func(string) bool, handler inet.StreamHandler) {
 	rh.host.SetStreamHandlerMatch(pid, m, handler)
 }
 
+// RemoveStreamHandler removes the handler matching the given protocol ID.
 func (rh *RoutedHost) RemoveStreamHandler(pid protocol.ID) {
 	rh.host.RemoveStreamHandler(pid)
 }
 
+// NewStream opens a new stream to given peer p, and writes a p2p/protocol
+// header with given protocol.ID. If there is no connection to p, attempts
+// to create one. If ProtocolID is "", writes no header.
+// (Threadsafe)
 func (rh *RoutedHost) NewStream(ctx context.Context, p peer.ID, pids ...protocol.ID) (inet.Stream, error) {
 	return rh.host.NewStream(ctx, p, pids...)
 }
+
+// Close shuts down the Host's services (network, etc).
 func (rh *RoutedHost) Close() error {
 	// no need to close IpfsRouting. we dont own it.
 	return rh.host.Close()

p2p/net/README.md

@@ -13,5 +13,5 @@ The IPFS Network package handles all of the peer-to-peer networking. It connects
 It looks a bit like this:
 
 <center>
-![](https://docs.google.com/drawings/d/1FvU7GImRsb9GvAWDDo1le85jIrnFJNVB_OTPXC15WwM/pub?h=480)
+![](https://ipfs.io/ipfs/Qme7oC5qvYh7F2KvyNArHjz9B1dFEsXRxVEofaGm86EUve)
 </center>

p2p/net/mock/interface.go

@@ -19,6 +19,10 @@ import (
 	ma "github.com/multiformats/go-multiaddr"
 )
 
+// Mocknet is an interface representing a network of peers. Each peer
+// has its own net.Network. Mocknet can add and remove peers,
+// link different networks and provide a representation of the state
+// of the system via LinkMaps.
 type Mocknet interface {
 
 	// GenPeer generates a peer and its inet.Network in the Mocknet

p2p/net/mock/mock_net.go

@@ -38,6 +38,7 @@ type mocknet struct {
 	sync.Mutex
 }
 
+// New initializes and returns a default implementation of Mocknet.
 func New(ctx context.Context) Mocknet {
 	return &mocknet{
 		nets:  map[peer.ID]*peernet{},

p2p/net/mock/mock_stream.go

@@ -26,6 +26,7 @@ type transportObject struct {
 	arrivalTime time.Time
 }
 
+// NewStream returns a new mock stream, which implements net.Stream.
 func NewStream(p net.Conn) *stream {
 	s := &stream{
 		Pipe:      p,

p2p/net/mock/ratelimiter.go

@@ -5,8 +5,8 @@ import (
 	"time"
 )
 
-//  A ratelimiter is used by a link to determine how long to wait before sending
-//  data given a bandwidth cap.
+// A ratelimiter is used by a link to determine how long to wait before sending
+// data given a bandwidth cap.
 type ratelimiter struct {
 	lock         sync.Mutex
 	bandwidth    float64       // bytes per nanosecond
@@ -17,7 +17,7 @@ type ratelimiter struct {
 	duration     time.Duration // total delay introduced due to rate limiting
 }
 
-//  Creates a new ratelimiter with bandwidth (in bytes/sec)
+// NewRatelimiter creates a new ratelimiter with bandwidth (in bytes/sec)
 func NewRatelimiter(bandwidth float64) *ratelimiter {
 	//  convert bandwidth to bytes per nanosecond
 	b := bandwidth / float64(time.Second)
@@ -29,7 +29,7 @@ func NewRatelimiter(bandwidth float64) *ratelimiter {
 	}
 }
 
-//  Changes bandwidth of a ratelimiter and resets its allowance
+// Changes bandwidth of a ratelimiter and resets its allowance
 func (r *ratelimiter) UpdateBandwidth(bandwidth float64) {
 	r.lock.Lock()
 	defer r.lock.Unlock()
@@ -42,12 +42,12 @@ func (r *ratelimiter) UpdateBandwidth(bandwidth float64) {
 	r.lastUpdate = time.Now()
 }
 
-//  Returns how long to wait before sending data with length 'dataSize' bytes
+// Returns how long to wait before sending data with length 'dataSize' bytes
 func (r *ratelimiter) Limit(dataSize int) time.Duration {
 	r.lock.Lock()
 	defer r.lock.Unlock()
 	//  update time
-	var duration time.Duration = time.Duration(0)
+	var duration = time.Duration(0)
 	if r.bandwidth == 0 {
 		return duration
 	}