all: implement milestones (#961)

* Milestone Implementation * Merge branch 'POS-347' into reciept-e2e-test * Changes for testing, will be removed after testing * Debugged the error * Just for testing purpose * refactor debug api methods, rename whitelist -> checkpoint * remove first iteration based vars * fix linters * Rewind Changes * Error changes * RewindBack function in bor_checkpoint_verifier * Testcases * Added the fetch test for milestone and checkpoint * Debugged the lint changes * Debugged the lint changes * Debugged the lint changes * Debugged the lint changes * Improved the error in Miner test file * Improved the error of pointing to the wrong function * Locking the sprint after the vote has been made on it. * Adding more logs for testing * Adding more logs for testing * Adding more logs for testing * Implemented the NoAckMilestone fetching mechanism * Testcases for milestone implementation * Testing code for fetchNoAckMilestone and fetchLastNoAckMilestone * Testing changes * refactor else-if * Corrected the number of params in bor_ext.go * Dummy API for testing * Defined the GetVoteOnRootHash in interface * Defined the GetVoteOnRootHash in interface * Made changes in the web3ext file * Added the GetVoteOnRootHash in PublicBlockChain API * Added the GetVoteOnRootHash in filterBackend * Added the log of Root and RooHash * Removed the 0x from rootHash * Just for testing purpose * "GetVoteOnRootHash" mock implementation * bor_test.go * Added the test for milestone implementation * Added service for fetching milestone by ID * Improved the comments * Removed the duplicate code * use setter for borVerifier * use setter for borVerifier * refactor handleNoAckMilestone * remove code repetition with retry function * Converged the repetitive code * after CR * persistence * persistence implementation * feature flag * Persistence Changes * cr * initial * fix * fix * Whitelist Flag * 1 Add:Included the milestone flag 2.Add:Hardlimit the rewind to maximum of 255 blocks * Chg:Updated go.mod file * Remove:Dubai Hardfork code * Add:checked errors for call functions to the Db, Rmv: Remote Header variable from the IsValidPeer() function * Fix:Linting issues' * Add:MilestoneGRPC functions * Fix:Lint issues * Fix:Lint issues * Fix: TestFetchMilestoneFromMockHeimdall * Fix:Integrations tests * Add:Test for sprint length and milestone changes * Add:Functionality to fetch the finalized block * Chg:Changed default val of TriesInmemory to 1024 * fix:Some functions of heimdallGRPC client * Restored the GRPC functionality, was commented out for developing purpose * Fix:Bor_checkpoint_Verfier function * Test:Added the chain Rewinding test * Test:Added the Sprint Length + Milestone merge test * Add:Implemented the future milestone * Add:Future milestone changes * Add:Future milestone changes * Chg: Voting on endBlockHash rather than rootHash * Chg: Changed the logic of future milestone from rootHash checking to endBlockHash checking * Fix:Using endBockHash while verifying the incoming milestone * Chg:Variable names for better readiblity * Fix:Testing changes * Add:metrics for milestone implementation * Add:Metrics for milestone implementatian * Fix:Order of statements in a function for better optimization * Chg:Removed unrequired file * Fix:new variable intialization * Add:Comment to increase readiblity * Fix:Logs * Chg:Name of GetVoteOnRootHash to GetVoteOnHash * Fix:Linting issues * Fixed linting issues * Rmv: Unnecessary logs and Add:Skip test for long tests * Fix:Checking current chain with whitelisted milestone or checkpoint in Finalized block function * Fix:Test * Fix:Whitelisting of Milestone and Checkpoint process * Fix: Milestone JSON structure * Chg:Testcases changes * Fix:Change from VoteOnRootHash to VoteOnHash * Fix:Variable name fix * Fix:Finalized API * internal/jsre/deps: update web3.js bundle * Fix:milestone verifier * Chg:Handling the long future chain import issue * Fix:Lint issues * Fix:TestLowDiffLongChain and TestPrunedImportSide tests, used hardcoded value 128 instead of DefaultTriesInMemory value * Chg:Testcode for producing metrics * Chg:Milestong polling value to 32 secs * Add:Testcases * Add:Implemented the check to fetch the milestoneId from heimdall before locking the fork * Added GRPC method for FetchMilestoneID * Fix:lint issue * Fix:lint issue * Skiped out the tests which were mainly used to produce the supporting data * remove vcs build when running snyk * Add:Improved the logs and comments * fix linters * Skipped some test as they are panic due to timeout issue in github * Chg:Variable name LockerSprintNumber to LockedMilestoneNumber for better readablity and clarity * Chg:Conflicting variable names in milestone test file * Chg:Conflicting function names in milestone test file * fix : minor fix in TestInsertingSpanSizeBlocks * Fix:Mocking issue in TestInsertingSpanSizeBlocks * Fix:GRPC Polyproto Version * eth/downloader: skip peer drop due to whitelisting err * eth, tests/bor: bug fixes and minor refactor * Add:Implemented the milestone related functions in the HeimdallApp * Fix:Lint Errors & Remove:Redundant Code * Fix:Testing Errors * Fix:Bor integeration tests * Fix:Test errors * update heimdall client mock files * remove unused arguments * remove redundant code * Chg:Changed the milestone polling intervals * Add: added block finality from whitelisted checkpoint * skip future chain validation * Add:confirmation check of 16 blocks over the end block while voting for the milestone in GetVoteHash() function * Chg:Included endBlockNum in UnlockMutex function * Add:Property based test for milestone * Fix:Opening the lock while processing future milestone * Add:Property based test for futureMilestone * Defined the value of TempTriesInMemory * Fixed the finalized api * Fixed lint issues * eth: add logs while fetching and rewinding * fix linters: use default returns instead of recursive calls * Fix:Milestone intergration test * Add:GetVoteHash fn in mock backend * tests/bor: fix mock span * tests/bor: remove t.Parallel() * use bor namespace in ethclient, fix mock function --------- Co-authored-by: Vaibhav Jindal <vaibhavjindal29@gmail.com> Co-authored-by: VaibhavJindal <74560896+VAIBHAVJINDAL3012@users.noreply.github.com> Co-authored-by: Manav Darji <manavdarji.india@gmail.com> Co-authored-by: Evgeny Danienko <6655321@bk.ru> Co-authored-by: Shivam Sharma <shivam691999@gmail.com> Co-authored-by: Anshal Shukla <shukla.anshal85@gmail.com>
maticnetwork · Aug 28, 2023 · b48b89f · b48b89f
1 parent b38412d
commit b48b89f
Show file tree

Hide file tree

Showing 58 changed files with 5,614 additions and 587 deletions.
diff --git a/accounts/abi/bind/backends/bor_simulated.go b/accounts/abi/bind/backends/bor_simulated.go
@@ -24,6 +24,10 @@ func (fb *filterBackend) GetBorBlockReceipt(ctx context.Context, hash common.Has
 	return receipt, nil
 }
 
+func (fb *filterBackend) GetVoteOnHash(ctx context.Context, starBlockNr uint64, endBlockNr uint64, hash string, milestoneId string) (bool, error) {
+	return false, nil
+}
+
 func (fb *filterBackend) GetBorBlockLogs(ctx context.Context, hash common.Hash) ([]*types.Log, error) {
 	receipt, err := fb.GetBorBlockReceipt(ctx, hash)
 	if err != nil || receipt == nil {

diff --git a/common/flags/milestone.go b/common/flags/milestone.go
@@ -0,0 +1,3 @@
+package flags
+
+const Milestone = true
diff --git a/common/gererics/empty.go b/common/gererics/empty.go
@@ -0,0 +1,5 @@
+package gererics
+
+func Empty[T any]() (t T) {
+	return
+}
diff --git a/consensus/bor/api.go b/consensus/bor/api.go
@@ -341,6 +341,10 @@ func (api *API) GetRootHash(start uint64, end uint64) (string, error) {
 	return root, nil
 }
 
+func (api *API) GetVoteOnHash(starBlockNr uint64, endBlockNr uint64, hash string, milestoneId string) (bool, error) {
+	return false, nil
+}
+
 func (api *API) initializeRootHashCache() error {
 	var err error
 	if api.rootHashCache == nil {

diff --git a/consensus/bor/heimdall.go b/consensus/bor/heimdall.go
@@ -5,6 +5,7 @@ import (
 
 	"github.com/ethereum/go-ethereum/consensus/bor/clerk"
 	"github.com/ethereum/go-ethereum/consensus/bor/heimdall/checkpoint"
+	"github.com/ethereum/go-ethereum/consensus/bor/heimdall/milestone"
 	"github.com/ethereum/go-ethereum/consensus/bor/heimdall/span"
 )
 
@@ -14,5 +15,10 @@ type IHeimdallClient interface {
 	Span(ctx context.Context, spanID uint64) (*span.HeimdallSpan, error)
 	FetchCheckpoint(ctx context.Context, number int64) (*checkpoint.Checkpoint, error)
 	FetchCheckpointCount(ctx context.Context) (int64, error)
+	FetchMilestone(ctx context.Context) (*milestone.Milestone, error)
+	FetchMilestoneCount(ctx context.Context) (int64, error)
+	FetchNoAckMilestone(ctx context.Context, milestoneID string) error //Fetch the bool value whether milestone corresponding to the given id failed in the Heimdall
+	FetchLastNoAckMilestone(ctx context.Context) (string, error)       //Fetch latest failed milestone id
+	FetchMilestoneID(ctx context.Context, milestoneID string) error    //Fetch the bool value whether milestone corresponding to the given id is in process in Heimdall
 	Close()
 }
diff --git a/consensus/bor/heimdall/client.go b/consensus/bor/heimdall/client.go
@@ -13,6 +13,7 @@ import (
 
 	"github.com/ethereum/go-ethereum/consensus/bor/clerk"
 	"github.com/ethereum/go-ethereum/consensus/bor/heimdall/checkpoint"
+	"github.com/ethereum/go-ethereum/consensus/bor/heimdall/milestone"
 	"github.com/ethereum/go-ethereum/consensus/bor/heimdall/span"
 	"github.com/ethereum/go-ethereum/log"
 	"github.com/ethereum/go-ethereum/metrics"
@@ -23,6 +24,8 @@ var (
 	ErrShutdownDetected      = errors.New("shutdown detected")
 	ErrNoResponse            = errors.New("got a nil response")
 	ErrNotSuccessfulResponse = errors.New("error while fetching data from Heimdall")
+	ErrNotInRejectedList     = errors.New("milestoneID doesn't exist in rejected list")
+	ErrNotInMilestoneList    = errors.New("milestoneID doesn't exist in Heimdall")
 )
 
 const (
@@ -66,8 +69,16 @@ func NewHeimdallClient(urlString string) *HeimdallClient {
 const (
 	fetchStateSyncEventsFormat = "from-id=%d&to-time=%d&limit=%d"
 	fetchStateSyncEventsPath   = "clerk/event-record/list"
-	fetchCheckpoint            = "/checkpoints/%s"
-	fetchCheckpointCount       = "/checkpoints/count"
+
+	fetchCheckpoint      = "/checkpoints/%s"
+	fetchCheckpointCount = "/checkpoints/count"
+
+	fetchMilestone      = "/milestone/latest"
+	fetchMilestoneCount = "/milestone/count"
+
+	fetchLastNoAckMilestone = "/milestone/lastNoAck"
+	fetchNoAckMilestone     = "/milestone/noAck/%s"
+	fetchMilestoneID        = "/milestone/ID/%s"
 
 	fetchSpanFormat = "bor/span/%d"
 )
@@ -144,6 +155,23 @@ func (h *HeimdallClient) FetchCheckpoint(ctx context.Context, number int64) (*ch
 	return &response.Result, nil
 }
 
+// FetchMilestone fetches the checkpoint from heimdall
+func (h *HeimdallClient) FetchMilestone(ctx context.Context) (*milestone.Milestone, error) {
+	url, err := milestoneURL(h.urlString)
+	if err != nil {
+		return nil, err
+	}
+
+	ctx = withRequestType(ctx, milestoneRequest)
+
+	response, err := FetchWithRetry[milestone.MilestoneResponse](ctx, h.client, url, h.closeCh)
+	if err != nil {
+		return nil, err
+	}
+
+	return &response.Result, nil
+}
+
 // FetchCheckpointCount fetches the checkpoint count from heimdall
 func (h *HeimdallClient) FetchCheckpointCount(ctx context.Context) (int64, error) {
 	url, err := checkpointCountURL(h.urlString)
@@ -161,6 +189,84 @@ func (h *HeimdallClient) FetchCheckpointCount(ctx context.Context) (int64, error
 	return response.Result.Result, nil
 }
 
+// FetchMilestoneCount fetches the milestone count from heimdall
+func (h *HeimdallClient) FetchMilestoneCount(ctx context.Context) (int64, error) {
+	url, err := milestoneCountURL(h.urlString)
+	if err != nil {
+		return 0, err
+	}
+
+	ctx = withRequestType(ctx, milestoneCountRequest)
+
+	response, err := FetchWithRetry[milestone.MilestoneCountResponse](ctx, h.client, url, h.closeCh)
+	if err != nil {
+		return 0, err
+	}
+
+	return response.Result.Count, nil
+}
+
+// FetchLastNoAckMilestone fetches the last no-ack-milestone from heimdall
+func (h *HeimdallClient) FetchLastNoAckMilestone(ctx context.Context) (string, error) {
+	url, err := lastNoAckMilestoneURL(h.urlString)
+	if err != nil {
+		return "", err
+	}
+
+	ctx = withRequestType(ctx, milestoneLastNoAckRequest)
+
+	response, err := FetchWithRetry[milestone.MilestoneLastNoAckResponse](ctx, h.client, url, h.closeCh)
+	if err != nil {
+		return "", err
+	}
+
+	return response.Result.Result, nil
+}
+
+// FetchNoAckMilestone fetches the last no-ack-milestone from heimdall
+func (h *HeimdallClient) FetchNoAckMilestone(ctx context.Context, milestoneID string) error {
+	url, err := noAckMilestoneURL(h.urlString, milestoneID)
+	if err != nil {
+		return err
+	}
+
+	ctx = withRequestType(ctx, milestoneNoAckRequest)
+
+	response, err := FetchWithRetry[milestone.MilestoneNoAckResponse](ctx, h.client, url, h.closeCh)
+	if err != nil {
+		return err
+	}
+
+	if !response.Result.Result {
+		return fmt.Errorf("%w: milestoneID %q", ErrNotInRejectedList, milestoneID)
+	}
+
+	return nil
+}
+
+// FetchMilestoneID fetches the bool result from Heimdal whether the ID corresponding
+// to the given milestone is in process in Heimdall
+func (h *HeimdallClient) FetchMilestoneID(ctx context.Context, milestoneID string) error {
+	url, err := milestoneIDURL(h.urlString, milestoneID)
+	if err != nil {
+		return err
+	}
+
+	ctx = withRequestType(ctx, milestoneIDRequest)
+
+	response, err := FetchWithRetry[milestone.MilestoneIDResponse](ctx, h.client, url, h.closeCh)
+
+	if err != nil {
+		return err
+	}
+
+	if !response.Result.Result {
+		return fmt.Errorf("%w: milestoneID %q", ErrNotInMilestoneList, milestoneID)
+	}
+
+	return nil
+}
+
 // FetchWithRetry returns data from heimdall with retry
 func FetchWithRetry[T any](ctx context.Context, client http.Client, url *url.URL, closeCh chan struct{}) (*T, error) {
 	// request data once
@@ -266,10 +372,34 @@ func checkpointURL(urlString string, number int64) (*url.URL, error) {
 	return makeURL(urlString, url, "")
 }
 
+func milestoneURL(urlString string) (*url.URL, error) {
+	url := fetchMilestone
+
+	return makeURL(urlString, url, "")
+}
+
 func checkpointCountURL(urlString string) (*url.URL, error) {
 	return makeURL(urlString, fetchCheckpointCount, "")
 }
 
+func milestoneCountURL(urlString string) (*url.URL, error) {
+	return makeURL(urlString, fetchMilestoneCount, "")
+}
+
+func lastNoAckMilestoneURL(urlString string) (*url.URL, error) {
+	return makeURL(urlString, fetchLastNoAckMilestone, "")
+}
+
+func noAckMilestoneURL(urlString string, id string) (*url.URL, error) {
+	url := fmt.Sprintf(fetchNoAckMilestone, id)
+	return makeURL(urlString, url, "")
+}
+
+func milestoneIDURL(urlString string, id string) (*url.URL, error) {
+	url := fmt.Sprintf(fetchMilestoneID, id)
+	return makeURL(urlString, url, "")
+}
+
 func makeURL(urlString, rawPath, rawQuery string) (*url.URL, error) {
 	u, err := url.Parse(urlString)
 	if err != nil {

diff --git a/consensus/bor/heimdall/client_test.go b/consensus/bor/heimdall/client_test.go
@@ -15,6 +15,7 @@ import (
 	"github.com/ethereum/go-ethereum/common"
 	"github.com/ethereum/go-ethereum/common/network"
 	"github.com/ethereum/go-ethereum/consensus/bor/heimdall/checkpoint"
+	"github.com/ethereum/go-ethereum/consensus/bor/heimdall/milestone"
 
 	"github.com/stretchr/testify/require"
 )
@@ -23,7 +24,10 @@ import (
 // requests to the mock heimdal server for specific functions. Add more handlers
 // according to requirements.
 type HttpHandlerFake struct {
-	handleFetchCheckpoint http.HandlerFunc
+	handleFetchCheckpoint         http.HandlerFunc
+	handleFetchMilestone          http.HandlerFunc
+	handleFetchNoAckMilestone     http.HandlerFunc
+	handleFetchLastNoAckMilestone http.HandlerFunc
 }
 
 func (h *HttpHandlerFake) GetCheckpointHandler() http.HandlerFunc {
@@ -32,6 +36,24 @@ func (h *HttpHandlerFake) GetCheckpointHandler() http.HandlerFunc {
 	}
 }
 
+func (h *HttpHandlerFake) GetMilestoneHandler() http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		h.handleFetchMilestone.ServeHTTP(w, r)
+	}
+}
+
+func (h *HttpHandlerFake) GetNoAckMilestoneHandler() http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		h.handleFetchNoAckMilestone.ServeHTTP(w, r)
+	}
+}
+
+func (h *HttpHandlerFake) GetLastNoAckMilestoneHandler() http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		h.handleFetchLastNoAckMilestone.ServeHTTP(w, r)
+	}
+}
+
 func CreateMockHeimdallServer(wg *sync.WaitGroup, port int, listener net.Listener, handler *HttpHandlerFake) (*http.Server, error) {
 	// Create a new server mux
 	mux := http.NewServeMux()
@@ -41,6 +63,21 @@ func CreateMockHeimdallServer(wg *sync.WaitGroup, port int, listener net.Listene
 		handler.GetCheckpointHandler()(w, r)
 	})
 
+	// Create a route for fetching milestone
+	mux.HandleFunc("/milestone/latest", func(w http.ResponseWriter, r *http.Request) {
+		handler.GetMilestoneHandler()(w, r)
+	})
+
+	// Create a route for fetching milestone
+	mux.HandleFunc("/milestone/noAck/{id}", func(w http.ResponseWriter, r *http.Request) {
+		handler.GetNoAckMilestoneHandler()(w, r)
+	})
+
+	// Create a route for fetching milestone
+	mux.HandleFunc("/milestone/lastNoAck", func(w http.ResponseWriter, r *http.Request) {
+		handler.GetLastNoAckMilestoneHandler()(w, r)
+	})
+
 	// Add other routes as per requirement
 
 	// Create the server with given port and mux
@@ -118,9 +155,59 @@ func TestFetchCheckpointFromMockHeimdall(t *testing.T) {
 	wg.Wait()
 }
 
+// TestFetchMilestoneFromMockHeimdall tests the heimdall client side logic
+// to fetch milestone from a mock heimdall server.
+// It can be used for debugging purpose (like response fields, marshalling/unmarshalling, etc).
+func TestFetchMilestoneFromMockHeimdall(t *testing.T) {
+	t.Parallel()
+
+	// Create a wait group for sending across the mock server
+	wg := &sync.WaitGroup{}
+	wg.Add(1)
+
+	// Initialize the fake handler and add a fake milestone handler function
+	handler := &HttpHandlerFake{}
+	handler.handleFetchMilestone = func(w http.ResponseWriter, _ *http.Request) {
+		err := json.NewEncoder(w).Encode(milestone.MilestoneResponse{
+			Height: "0",
+			Result: milestone.Milestone{
+				Proposer:   common.Address{},
+				StartBlock: big.NewInt(0),
+				EndBlock:   big.NewInt(512),
+				Hash:       common.Hash{},
+				BorChainID: "15001",
+				Timestamp:  0,
+			},
+		})
+
+		if err != nil {
+			w.WriteHeader(500) // Return 500 Internal Server Error.
+		}
+	}
+
+	// Fetch available port
+	port, listener, err := network.FindAvailablePort()
+	require.NoError(t, err, "expect no error in finding available port")
+
+	// Create mock heimdall server and pass handler instance for setting up the routes
+	srv, err := CreateMockHeimdallServer(wg, port, listener, handler)
+	require.NoError(t, err, "expect no error in starting mock heimdall server")
+
+	// Create a new heimdall client and use same port for connection
+	client := NewHeimdallClient(fmt.Sprintf("http://localhost:%d", port))
+	_, err = client.FetchMilestone(context.Background())
+	require.NoError(t, err, "expect no error in fetching milestone")
+
+	// Shutdown the server
+	err = srv.Shutdown(context.TODO())
+	require.NoError(t, err, "expect no error in shutting down mock heimdall server")
+
+	// Wait for `wg.Done()` to be called in the mock server's routine.
+	wg.Wait()
+}
+
 // TestFetchShutdown tests the heimdall client side logic for context timeout and
-// interrupt handling while fetching checkpoints (latest for the scope of test)
-// from a mock heimdall server.
+// interrupt handling while fetching data from a mock heimdall server.
 func TestFetchShutdown(t *testing.T) {
 	t.Parallel()
 
@@ -135,7 +222,7 @@ func TestFetchShutdown(t *testing.T) {
 	// greater than `retryDelay`. This should cause the request to timeout and trigger shutdown
 	// due to `ctx.Done()`. Expect context timeout error.
 	handler.handleFetchCheckpoint = func(w http.ResponseWriter, _ *http.Request) {
-		time.Sleep(6 * time.Second)
+		time.Sleep(100 * time.Millisecond)
 
 		err := json.NewEncoder(w).Encode(checkpoint.CheckpointResponse{
 			Height: "0",
@@ -165,7 +252,7 @@ func TestFetchShutdown(t *testing.T) {
 	// Create a new heimdall client and use same port for connection
 	client := NewHeimdallClient(fmt.Sprintf("http://localhost:%d", port))
 
-	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
 
 	// Expect this to fail due to timeout
 	_, err = client.FetchCheckpoint(ctx, -1)
@@ -182,7 +269,7 @@ func TestFetchShutdown(t *testing.T) {
 		w.WriteHeader(500) // Return 500 Internal Server Error.
 	}
 
-	ctx, cancel = context.WithTimeout(context.Background(), 10*time.Second) // Use some high value for timeout
+	ctx, cancel = context.WithTimeout(context.Background(), 50*time.Millisecond) // Use some high value for timeout
 
 	// Cancel the context after a delay until we make request
 	go func(cancel context.CancelFunc) {