v0.37.x: backport of new content on spec/p2p (cometbft#1004)

This is backport of three PRs originally merged against main:

* spec/p2p: Specify the operation of a Reactor (cometbft#714)
* spec/p2p: document the p2p API used by Reactors (cometbft#851)
* spec/p2p: new structure for the p2p specification (cometbft#966)

Co-authored-by: Josef Widder <44643235+josef-widder@users.noreply.github.com>
Co-authored-by: Lasaro <lasaro@informal.systems>
Co-authored-by: Sergio Mena <sergio@informal.systems>

p2p/README.md

@@ -4,7 +4,7 @@ The p2p package provides an abstraction around peer-to-peer communication.
 
 Docs:
 
-- [Connection](https://github.com/cometbft/cometbft/blob/v0.37.x/spec/p2p/connection.md) for details on how connections and multiplexing work
-- [Peer](https://github.com/cometbft/cometbft/blob/v0.37.x/spec/p2p/node.md) for details on peer ID, handshakes, and peer exchange
-- [Node](https://github.com/cometbft/cometbft/blob/v0.37.x/spec/p2p/node.md) for details about different types of nodes and how they should work
-- [Config](https://github.com/cometbft/cometbft/blob/v0.37.x/spec/p2p/config.md) for details on some config option
+- [Connection](../spec/p2p/legacy-docs/connection.md) for details on how connections and multiplexing work
+- [Peer](../spec/p2p/legacy-docs/node.md) for details on peer ID, handshakes, and peer exchange
+- [Node](../spec/p2p/legacy-docs/node.md) for details about different types of nodes and how they should work
+- [Config](../spec/p2p/legacy-docs/config.md) for details on some config option

spec/README.md

@@ -35,12 +35,12 @@ please submit them to our [bug bounty](https://cometbft.com/security)!
 
 ### P2P and Network Protocols
 
-- [The Base P2P Layer](./p2p/node.md): multiplex the protocols ("reactors") on authenticated and encrypted TCP connections
-- [Peer Exchange (PEX)](./p2p/messages/pex.md): gossip known peer addresses so peers can find each other
-- [Block Sync](./p2p/messages/block-sync.md): gossip blocks so peers can catch up quickly
-- [Consensus](./p2p/messages/consensus.md): gossip votes and block parts so new blocks can be committed
-- [Mempool](./p2p/messages/mempool.md): gossip transactions so they get included in blocks
-- [Evidence](./p2p/messages/evidence.md): sending invalid evidence will stop the peer
+- [The Base P2P Layer](./p2p/legacy-docs/node.md): multiplex the protocols ("reactors") on authenticated and encrypted TCP connections
+- [Peer Exchange (PEX)](./p2p/legacy-docs/messages/pex.md): gossip known peer addresses so peers can find each other
+- [Block Sync](./p2p/legacy-docs/messages/block-sync.md): gossip blocks so peers can catch up quickly
+- [Consensus](./p2p/legacy-docs/messages/consensus.md): gossip votes and block parts so new blocks can be committed
+- [Mempool](./p2p/legacy-docs/messages/mempool.md): gossip transactions so they get included in blocks
+- [Evidence](./p2p/legacy-docs/messages/evidence.md): sending invalid evidence will stop the peer
 
 ### RPC
 

spec/abci/abci++_basic_concepts.md

@@ -177,7 +177,7 @@ call sequences of these methods.
 
 State sync allows new nodes to rapidly bootstrap by discovering, fetching, and applying
 state machine (application) snapshots instead of replaying historical blocks. For more details, see the
-[state sync documentation](../p2p/messages/state-sync.md).
+[state sync documentation](../p2p/legacy-docs/messages/state-sync.md).
 
 New nodes discover and request snapshots from other nodes in the P2P network.
 A CometBFT node that receives a request for snapshots from a peer will call

spec/abci/abci++_methods.md

@@ -346,7 +346,7 @@ title: Methods
     can be spoofed by adversaries, so applications should employ additional verification schemes
     to avoid denial-of-service attacks. The verified `AppHash` is automatically checked against
     the restored application at the end of snapshot restoration.
-    * For more information, see the `Snapshot` data type or the [state sync section](../p2p/messages/state-sync.md).
+    * For more information, see the `Snapshot` data type or the [state sync section](../p2p/legacy-docs/messages/state-sync.md).
 
 ### ApplySnapshotChunk
 
@@ -864,7 +864,7 @@ Most of the data structures used in ABCI are shared [common data structures](../
     | metadata | bytes  | Arbitrary application metadata, for example chunk hashes or other verification data.                                                                                              | 5            |
 
 * **Usage**:
-    * Used for state sync snapshots, see the [state sync section](../p2p/messages/state-sync.md) for details.
+    * Used for state sync snapshots, see the [state sync section](../p2p/legacy-docs/messages/state-sync.md) for details.
     * A snapshot is considered identical across nodes only if _all_ fields are equal (including
     `Metadata`). Chunks may be retrieved from all nodes that have the same snapshot.
     * When sent across the network, a snapshot message can be at most 4 MB.

spec/p2p/README.md

@@ -0,0 +1,46 @@
+---
+order: 1
+parent:
+  title: P2P
+  order: 6
+---
+
+# Peer-to-Peer Communication
+
+A CometBFT network is composed of multiple CometBFT instances, hereafter called
+`nodes`, that interact by exchanging messages.
+
+The CometBFT protocols are designed under the assumption of a partially-connected network model.
+This means that a node is not assumed to be directly connected to every other
+node in the network.
+Instead, each node is directly connected to only a subset of other nodes,
+hereafter called its `peers`.
+
+The peer-to-peer (p2p) communication layer is then the component of CometBFT that:
+
+1. establishes connections between nodes in a CometBFT network
+2. manages the communication between a node and the connected peers
+3. intermediates the exchange of messages between peers in CometBFT protocols
+
+The specification the p2p layer is a work in progress,
+tracked by [issue #19](https://github.com/cometbft/cometbft/issues/19).
+The current content is organized as follows:
+
+- [`implementation`](./implementation/README.md): documents the current state
+  of the implementation of the p2p layer, covering the main components of the
+  `p2p` package. The documentation covers, in a fairly comprehensive way,
+   the items 1. and 2. from the list above.
+- [`reactor-api`](./reactor-api/README.md): specifies the API offered by the
+  p2p layer to the protocol layer, through the `Reactor` abstraction.
+  This is a high-level specification (i.e., it should not be implementation-specific)
+  of the p2p layer API, covering item 3. from the list above.
+- [`legacy-docs`](./legacy-docs/): We keep older documentation in 
+  the `legacy-docs` directory, as overall, it contains useful information. 
+  However, part of this content is redundant,
+  being more comprehensively covered in more recent documents,
+  and some implementation details might be outdated
+  (see [issue #981](https://github.com/cometbft/cometbft/issues/981)).
+
+In addition to this content, some unfinished, work in progress, and auxiliary
+material can be found in the
+[knowledge-base](https://github.com/cometbft/knowledge-base/tree/main/p2p) repository.

spec/p2p/implementation/README.md

@@ -0,0 +1,38 @@
+# Implementation of the p2p layer
+
+This section documents the implementation of the peer-to-peer (p2p)
+communication layer in CometBFT.
+
+The documentation was [produced](https://github.com/tendermint/tendermint/pull/9348)
+using the `v0.34.*` releases
+and the branch [`v0.34.x`](https://github.com/cometbft/cometbft/tree/v0.34.x)
+of this repository as reference.
+As there were no substancial changes in the p2p implementation, the
+documentation also applies to the releases `v0.37.*` and `v0.38.*` [^v35].
+
+[^v35]: The releases `v0.35.*` and `v0.36.*`, which included a major
+  refactoring of the p2p layer implementation, were [discontinued][v35postmorten].
+
+[v35postmorten]: https://interchain-io.medium.com/discontinuing-tendermint-v0-35-a-postmortem-on-the-new-networking-layer-3696c811dabc
+
+## Contents
+
+The documentation follows the organization of the
+[`p2p` package](https://github.com/cometbft/cometbft/tree/v0.34.x/p2p),
+which implements the following abstractions:
+
+- [Transport](./transport.md): establishes secure and authenticated
+   connections with peers;
+- [Switch](./switch.md): responsible for dialing peers and accepting
+   connections from peers, for managing established connections, and for
+   routing messages between the reactors and peers,
+   that is, between local and remote instances of the CometBFT protocols;
+- [PEX Reactor](./pex.md): due to the several roles of this component, the
+  documentation is split in several parts:
+    - [Peer Exchange protocol](./pex-protocol.md): enables nodes to exchange peer addresses, thus implementing a peer discovery service;
+    - [Address Book](./addressbook.md): stores discovered peer addresses and
+  quality metrics associated to peers with which the node has interacted;
+    - [Peer Manager](./peer_manager.md): defines when and to which peers a node
+  should dial, in order to establish outbound connections;
+- [Types](./types.md) and [Configuration](./configuration.md) provide a list of
+  existing types and configuration parameters used by the p2p package.