feat: add markdown linting and fix errors (#1091)

Co-authored-by: Hugo Dias <hugomrdias@gmail.com>

.github/workflows/main.yml

@@ -20,6 +20,7 @@ jobs:
       - run: sudo apt install bzr
 
       - run: npm install
+      - run: npm test
       - run: npm run build
 
       # Pin the built site to ipfs-cluster, output the cid as `steps.ipfs.outputs.cid`

.remarkrc.yaml

@@ -0,0 +1,15 @@
+plugins:
+  # make remark aware of fontmatter.
+  remark-frontmatter:
+  # sensible linter defaults see: https://github.com/remarkjs/remark-lint/tree/main/packages/remark-preset-lint-recommended#rules
+  remark-preset-lint-recommended:
+  # add extra rules
+  remark-lint-heading-increment:
+  remark-lint-fenced-code-flag:
+  remark-lint-first-heading-level:
+  remark-lint-no-hr-after-heading:
+  # disable some of the sensible defaults given where we are today.
+  remark-lint-list-item-indent: false
+  remark-lint-final-newline: false
+  remark-lint-no-literal-urls: false
+  remark-lint-list-item-bullet-indent: false

README.md

@@ -33,6 +33,19 @@ yarn start
 # open http://localhost:1313/ in the browser
 ```
 
+### Check your markdown
+
+We have a markdown linter set up to check for common errors like incorrectly nested headers. It runs in CI and you can run it locally with:
+
+```bash
+npm test
+content/algorithms/crypto/randomness.md
+  15:39-15:46  warning  Found reference to undefined definition  no-undefined-references  remark-lint
+  54:24-54:31  warning  Found reference to undefined definition  no-undefined-references  remark-lint
+
+⚠ 2 warnings
+```
+
 ### Solving Common problems
 
 **Problem** - Site fails to build with an error that states it faled to download modules on macos
@@ -85,15 +98,14 @@ hugo mod get github.com/filecoin-project/specs-actors@v0.7.2
 ```
 
 ## Page Header
-The first heading should be `# Page Title` with `---` like below and should refer to the overall title of the document.
+The first heading should be an atx style heading `# Head` and should refer to the overall title of the document.
 
 ```md
 ---
 title: Storage Power Actor
 ---
 
 # Storage Power Actor
----
 
 ## Header for a section in this document
 Some text

content-pt/intro/_index.md

@@ -6,7 +6,6 @@ bookCollapseSection: true
 ---
 
 # Introdução
----
 
 Filecoin is a distributed storage network based on a blockchain mechanism.
 Filecoin *miners* can elect to provide storage capacity for the network, and thereby

content/algorithms/block_sync.md

@@ -8,7 +8,6 @@ dashboardTests: 0
 ---
 
 # BlockSync
----
 
 {{< hint info >}}
 **Name**: Block Sync  

content/algorithms/crypto/_index.md

@@ -9,12 +9,11 @@ dashboardTests: 0
 ---
 
 # Cryptographic Primitives
----
 
-  - Merkle tree/DAG
-  - Vector commitment scheme
-  - zkSNARK
-  - Reliable broadcast channel (libp2p)
+- Merkle tree/DAG
+- Vector commitment scheme
+- zkSNARK
+- Reliable broadcast channel (libp2p)
 
 
 - TODO: Add more detail and include references to relevant papers.

content/algorithms/crypto/randomness.md

@@ -12,7 +12,7 @@ dashboardTests: 0
 TODO: clean up stale .id/.go files
 
 Randomness is used throughout the protocol in order to generate values and extend the blockchain.
-Random values are drawn from a (drand)[drand] beacon and appropriately formatted for usage.
+Random values are drawn from a [drand](drand) beacon and appropriately formatted for usage.
 We describe this formatting below.
 
 ## Encoding Random Beacon randomness for on-chain use
@@ -51,7 +51,7 @@ The beacon entry is combined with a few elements for use as part of the protocol
 
 While all elements are not needed for every use of entropy (e.g. the inclusion of the round number is not necessary prior to genesis or outside of leader election, other entropy is only used sometimes, etc), we draw randomness as follows for the sake of uniformity/simplicity in the overall protocol.
 
-In all cases, a (drand)[drand] signature is used as the base of randomness: it is hashed using blake2b in order to obtain a usable randomness seed. In order to make randomness seed creation uniform, the protocol derives all such seeds in the same way, using blake2b as a hash function to generate a 256-bit output as follows:
+In all cases, a [drand](drand) signature is used as the base of randomness: it is hashed using blake2b in order to obtain a usable randomness seed. In order to make randomness seed creation uniform, the protocol derives all such seeds in the same way, using blake2b as a hash function to generate a 256-bit output as follows:
 
 In round `n`, for a given randomness lookback `l`, and serialized entropy `s`:
 

content/algorithms/crypto/signatures.md

@@ -8,7 +8,6 @@ dashboardTests: 0
 ---
 
 # Signatures
----
 
 Signatures are cryptographic functions that attest to the origin of a particular
 message. In the context of Filecoin, signatures are used to send and receive
@@ -61,7 +60,7 @@ key locally from the signature rather than specifying an ID of the public key.
 as described below. For more details on how the Filecoin `Signature` type is
 serialized, see [Signature](signatures).
 
-```
+```text
 SignatureBytes = [0x30][len][0x02][r][indicator][s][indicator][recovery]
 ```
 
@@ -82,7 +81,7 @@ SignatureBytes = [0x30][len][0x02][r][indicator][s][indicator][recovery]
 
 ### BLS Signatures
 
-Filecoin uses the [BLS signature scheme](https://datatracker.ietf.org/doc/draft-boneh-bls-signature/) over the [BLS12-381](BLS12-381](https://electriccoin.co/blog/new-snark-curve/) group of elliptic curves. You can find the default Rust implementation in [Filecoin's repo](https://github.com/filecoin-project/bls-signatures/).
+Filecoin uses the [BLS signature scheme](https://datatracker.ietf.org/doc/draft-boneh-bls-signature/) over the [BLS12-381](https://electriccoin.co/blog/new-snark-curve/) group of elliptic curves. You can find the default Rust implementation in [Filecoin's repo](https://github.com/filecoin-project/bls-signatures/).
 
 {{<embed src="bls.id" lang="go">}}
 {{<embed src="bls.go" lang="go">}}

content/algorithms/crypto/vrf.md

@@ -8,7 +8,6 @@ dashboardTests: 0
 ---
 
 # Verifiable Random Function
----
 
 Filecoin uses the notion of a [Verifiable Random
 Function](https://people.csail.mit.edu/silvio/Selected%20Scientific%20Papers/Pseudo%20Randomness/Verifiable_Random_Functions.pdf)

content/algorithms/cryptoecon/_index.md

@@ -8,7 +8,6 @@ dashboardTests: 0
 ---
 
 # Cryptoecon
----
 
 The Filecoin network is a complex multi-agent economic system. This section aims to explain some mechanisms and parameters in the system that can help achieve network-level goals. For now, just lists some key mechanisms and parameters that are still subject to changes during testnet but that must be resolved before mainnet launch. Note that this is a list of economic levers that are in consideration and not all of them will be used in mainnet. Some may be added or changed after mainnet launch, through the FIP process, requiring a **network upgrade**.
 

content/algorithms/expected_consensus/_index.md

@@ -8,7 +8,6 @@ dashboardTests: 0
 ---
 
 # Expected Consensus
----
 
 TODO: remove stale .id/.go files
 
@@ -106,12 +105,12 @@ w[r+1] = w[r] + (wPowerFactor[r+1] + wBlocksFactor[r+1]) * 2^8
 For a given tipset `ts` in round `r+1`, we define:
 
 - `wPowerFactor[r+1]  = wFunction(totalPowerAtTipset(ts))`
-- wBlocksFactor[r+1] =  `wPowerFactor[r+1] * wRatio * t / e`
+- `wBlocksFactor[r+1] =  wPowerFactor[r+1] * wRatio * t / e`
   - with `t = |ticketsInTipset(ts)|`
   - `e = expected number of tickets per round in the protocol`
   - and `wRatio in ]0, 1[`
 Thus, for stability of weight across implementations, we take:
-- wBlocksFactor[r+1] =  `(wPowerFactor[r+1] * b * wRatio_num) / (e * wRatio_den)`
+- `wBlocksFactor[r+1] =  (wPowerFactor[r+1] * b * wRatio_num) / (e * wRatio_den)`
 
 We get:
 
@@ -146,7 +145,7 @@ In the case where two Tipsets of equal weight have the same min ticket, the mine
 
 The above case may happen in situations under certain block propagation conditions. Assume three blocks B, C, and D have been mined (by miners 1, 2, and 3 respectively) off of block A, with minTicket(B) < minTicket(C) < minTicket(D).
 
-Miner 1 outputs their block B and shuts down. Miners 2 and 3 both receive B but not each others' blocks. We have miner 2 mining a Tipset made of B and C and miner 3 mining a Tipset made of B and D. If both succesfully mine blocks now, other miners in the network will receive new blocks built off of Tipsets with equal weight and the same smallest ticket (that of block B). They should select the block mined atop [B, C] since minTicket(C) < minTicket(D).
+Miner 1 outputs their block B and shuts down. Miners 2 and 3 both receive B but not each others' blocks. We have miner 2 mining a Tipset made of B and C and miner 3 mining a Tipset made of B and D. If both succesfully mine blocks now, other miners in the network will receive new blocks built off of Tipsets with equal weight and the same smallest ticket (that of block B). They should select the block mined atop `[B, C]` since `minTicket(C) < minTicket(D)`.
 
 The probability that two Tipsets with different blocks would have all the same tickets can be considered negligible: this would amount to finding a collision between two 256-bit (or more) collision-resistant hashes.
 

content/algorithms/gossip_sub.md

@@ -10,7 +10,6 @@ dashboardAuditURL: https://gateway.ipfs.io/ipfs/QmWR376YyuyLewZDzaTHXGZr7quL5LB1
 ---
 
 # GossipSub
----
 
 Transaction messages and block headers alongside the message references are propagated using the [libp2p GossipSub router](https://github.com/libp2p/specs/tree/master/pubsub/gossipsub). **In order to guarantee interoperability between different implementations, all filecoin full nodes must implement and use this protocol.** All pubsub messages are authenticated and must be [syntactically validated](message#message-syntax-validation) before being propagated further.
 

content/algorithms/payment_channels.md

@@ -8,7 +8,6 @@ dashboardTests: 0
 ---
 
 # Payment Channels
----
 
 In order for the Filecoin Markets to work in a timely manner, we need to be able to have off-chain payments. This is a solved problem (at least, for our purposes in v0). Payment channels have been implemented and used in bitcoin, ethereum and many other networks.
 

content/algorithms/porep/_index.md

@@ -6,7 +6,6 @@ bookhidden: true
 ---
 
 # Proof-of-Replication
----
 
 _Proof-of-Replication(PoRep)_, is a new kind of _Proof-of-Storage_, that can be used to prove that some data _D_ has been replicated to its own uniquely dedicated physical storage.  Enforcing unique physical copies enables a verifier to check that a prover is not deduplicating multiple copies of _D_ into the same storage space.  This construction is particularly useful in Cloud Computing and Decentralized Storage Networks, which must be transparently verifiable, resistant to Sybil attacks, and unfriendly to outsourcing.
 

content/algorithms/porep/porep_commitments/_index.md

@@ -4,7 +4,6 @@ weight: 2
 ---
 
 # PoRep Commitments
----
 
 {{< hint warning >}}
 TODO: Description

content/algorithms/porep/stacked_drg.md

@@ -4,7 +4,6 @@ weight: 1
 ---
 
 # Stacked DRG PoRep
----
 
 
 This section describes *Stacked DRG PoRep* (SDR), the specific Proof-of-Replication (PoRep) used in Filecoin. In this construction, the prover encodes the original data into a replica and commits to it. An offline PoRep proves that the commitment to the replica is a valid commitment of the encoded original data.
@@ -137,7 +136,7 @@ We have describe three hash functions:
 
 `RepHash` is a vector commitment used to generate commitments to sealed sectors, unsealed sectors, piece commitments and intermediate stepds of the Proof-of-Replication. Filecoin uses a balanced binary Merkle tree for `RepHash`. The leaves of the Merkle tree are pairs of adjacent nodes.
 
-`RepHash` inputs MUST respect a valid Storage Format. [TODO: What does this mean?]
+`RepHash` inputs MUST respect a valid Storage Format. (TODO: What does this mean?)
 
 ## Stacked DRG Graph
 
@@ -252,12 +251,12 @@ The Replication Algorithm  proceeds as follows:
 - `RandomSeed` is a 32-byte array of randomness extracted from the chain.
 - `CommD` is the Merkle root obtained by performing `RepHash` on the original data represented in `paddedfr32v1`.
 
-```
+```text
 ReplicaID := Hash(ProverID || SectorNumber || RandomSeed || CommD)
 ```
 
 - Perform `RepHash` on `Data` to yield `CommD` and `TreeD`:
-```
+```text
 CommD, TreeD = RepHash(data)
 ```
 

content/algorithms/porep/stacked_drg_circuit.md

@@ -4,10 +4,6 @@ weight: 3
 ---
 
 # Stacked DRG - Offline PoRep Circuit Spec
----
-
-
-### Stacked DRG Overview
 
 Stacked DRG PoRep is based on layering DRG graphs `LAYERS` times. The data represented in each DRG layer is a labeling based on previously labeled nodes. The final labeled layer is the SDR key, and the 'final layer' of replication the replica, an encoding of the original data using the generated key.
 
@@ -21,7 +17,7 @@ The (offline) proof size in SDR is too large for blockchain usage (~3MB). We use
 
 This circuit proves that given a Merkle root `CommD`, `CommRLast`, and `commRStar`, that the prover knew the correct replicated data at each layer.
 
-### Spec notation
+## Spec notation
 
 - **Fr**: Field element of BLS12-381
 - **UInt**: Unsigned integer
@@ -31,9 +27,9 @@ This circuit proves that given a Merkle root `CommD`, `CommRLast`, and `commRSta
   - otherwise, execute the function
 - **Inclusion path**: Binary representation of the Merkle tree path that must be proven packed into a single `Fr` element.
 
-# Offline PoRep circuit
+## Offline PoRep circuit
 
-## Public Parameters
+### Public Parameters
 
 *Parameters that are embeded in the circuits or used to generate the circuit*
 
@@ -44,7 +40,7 @@ This circuit proves that given a Merkle root `CommD`, `CommRLast`, and `commRSta
 - `TREE_DEPTH: UInt`: Depth of the Merkle tree. Note, this is (log_2(Size of original data in bytes/32 bytes per leaf)).
 - `PARENT_COUNT : UInt`: Defined as `EXPANSION_DEGREE+BASE_DEGREE`.
 
-## Public Inputs
+### Public Inputs
 
 *Inputs that the prover uses to generate a SNARK proof and that the verifier uses to verify it*
 
@@ -57,10 +53,10 @@ This circuit proves that given a Merkle root `CommD`, `CommRLast`, and `commRSta
 Design notes:
 
 - `CommRLast` is a private input used during during Proof-of-Spacetime.
-   To enable this, the prover must store `CommC` and use it to prove that `CommRLast` is included in `CommR` [TODO: define 'included' language.]
+   To enable this, the prover must store `CommC` and use it to prove that `CommRLast` is included in `CommR` (TODO: define 'included' language.)
 - `InclusionPath` and `ParentInclusionPath`: Each layer `l` has `LAYER_CHALLENGES[l]` inclusion paths.
 
-## Private Inputs
+### Private Inputs
 
 *Inputs that the prover uses to generate a SNARK proof, these are not needed by the verifier to verify the proof*
 
@@ -80,7 +76,7 @@ Design notes:
 
 - `ParentValue : [LAYERS][][PARENT_COUNT]Fr`: Value of the parent leaves for each challenged leaf at layer `l`.
 
-## Circuit
+### Circuit
 
 In high level, we do 4 checks:
 
@@ -158,11 +154,9 @@ for l in range LAYERS {
 ```
 
 
-
-## Verification of offline porep proof
+## Verification of Offline PoRep Proof
 
 - SNARK proof check: **Check** that given the SNARK proof and the public inputs, the SNARK verification outputs true
 - Parent checks: For each `leaf = InclusionPath[l][c]`:
   - **Check** that all `ParentsInclusionPaths_[l][c][0..PARENT_COUNT}` are the correct parent leaves of `leaf` in the DRG graph, if a leaf has less than `PARENT_COUNT`, repeat the leaf with the highest label in the graph.
   - **Check** that the parent leaves are in ascending numerical order.
-

content/algorithms/post/_index.md

@@ -9,7 +9,6 @@ dashboardTests: 0
 ---
 
 # Proof-of-Spacetime
----
 _Proof-of-Storage_ schemes allow a user to check if a storage provider is storing the outsourced data at the time
 of the challenge. How can we use **PoS** schemes to prove that some data was being stored throughout a period
 of time? A natural answer to this question is to require the user to repeatedly (e.g. every minute) send

content/algorithms/post/election_post.md

@@ -7,7 +7,6 @@ dashboardTests: 0
 ---
 
 # Election PoSt
----
 
 
 This document describes Election-PoSt, the Proof-of-Spacetime used in Filecoin.
@@ -21,7 +20,7 @@ At a high-level it marries `ElectionPoSt` with a `SurprisePoSt` fallback:
 
 Election PoSt couples the PoSt process with block production, meaning that in order to produce a block, the miner must produce a valid PoSt proof (snark output). Specifically, a subset of non-faulted sectors the miner is storing (i.e. eligible sectors) allows them to attempt a leader election using a PartialTicket any of which could yield a valid ChallengeTicket for leader election. The probability of scratching a winning ChallengeTicket is dependent on sector size and total storage. Miners are rewarded in proportion to the quantity of winning ChallengeTickets they generate in a given epoch, thereby incentivizing a miner to check as much of their storage as allowed in order to express their full power in a leader election. The number of election proofs a miner can generate in a given epoch will determine the block reward it earns.
 
-An election proof is generated from a given PartialTicket by Hashing it, and using that hash to generate a value in [0,1]. Specifically `ChallengeTicket = H(PartialTicket)/2^len(H)`. 
+An election proof is generated from a given PartialTicket by Hashing it, and using that hash to generate a value in `[0,1]`. Specifically `ChallengeTicket = H(PartialTicket)/2^len(H)`. 
 
 This does mean that, in a given round, a lucky miner may succeed in generating a block by proving only a single sector, but again on expectation, a miner will have to prove their sectors at every single round in order for their full power to contribute to block generation. In the event one of the miner’s sectors cannot be proven (i.e. the miner does not have access to the nodes from that sector), no tickets will be returned. In order to prevent this, a miner can declare faults on their faulty sectors to avoid having to include them in the eligible sector set. Their power will be reduced accordingly.
 
@@ -210,7 +209,7 @@ Once issued a surprise, a miner will have to challenge the sectors in their `Pro
 
 The miner's response must be a PoSt proof over the PartialTickets generated by the challenge which clear a given target `SURPRISE_TARGET` to be defined.
 
-```
+```text
 submitSurpriseTickets(partialTickets):
     eligibleTickets = []
     for tix in partialTickets:

content/algorithms/sdr/_index.md

@@ -12,6 +12,7 @@ dashboardAudit: done
 dashboardTests: 0
 ---
 
+<!--lint disable -->
 
 {{< plain hidden >}}
 $$
@@ -374,8 +375,9 @@ $$
 $$
 {{< /plain >}}
 
+<!--lint enable -->
+
 # Stacked DRG Proof of Replication
----
 
 ## Merkle Proofs