Skip to content

Commit

Permalink
Leo/merge master to feature consensus (#1438)
Browse files Browse the repository at this point in the history
* rename root block vote

* pass in identities separate from participant data

* add log

* Extracted rootBlock command from finalize command for bootstrap util. Added encoding and writing to disk encoded root block and votes

* unwrap encodable type of key for partners

* Implemented QC creation as part of finalizing cmd. Updated docs. Implementing reading/writing of DKG and root block

* Fixed typo, updated tests

* Updated QC building logic. Fixed signing of QC

* Updated godoc

* Updated comments. Reverted last commit

* Updated how root block is created. Separated root block and votes

* Added encoding of full DKG to construct QC in finilize step

* Updated reading of votes, dkg data in finalizer. Simplified logic for generating QC

* Updated tests

* Fixed tests

* Linted. Fixed tests

* Update cmd/bootstrap/run/qc.go

* renamed methods for reading node infos to reflect that fact that we are _reading_ the infos and not generating them

* consolidated consistency checks

* cleanup

* format

* update localnet/integration tests to generate votes separately

* put votes in same directory

* fix path join

* change file names

* Update finalize_test.go

* fix filename

* fixing makefile and other fixes (#1378)

* fixing makefile and other fixes

* removing special nocgo docker target for the transit script

* fix test (lint)

* Update README.md

* address comments

* LInted and fixed unit tests

* Fixed network test

* Updated godoc

* add todo to enforce encryption

* [sync] Add (very) basic rapid test for sync engine

When a block B at height h is requested through RequestBlock(B.ID()) and then received through HandleBlock(B):
- an interleaving request for a different block B' at height h' == h may have happened,
- if it is handled (through HandleBlock(B')) before the HandleBlock(B),
- then it will populate c.heights[h']
- and return a status of Received on the getRequestStatus(h, B.ID()) in HandleBlock(B)
- as a consequence HandleBlock(B) will return false

* [sync] Fix height accounting from forks

Makes getRequestStatus(height, ID) return a height status only if there's no superseding ID status
and the height status isn't a mismatch for the query ID.

* [sync] add height tracking logic to rapid tests

Model height request tracking

* lint

* Clarify godoc
Add extra check for error type in a test

* split max delay and sampled delay

add test case for default delay base values

* fixing the rootblock.json download path

* fixing log message

* lint

* Update module/dkg/controller.go

Co-authored-by: Leo Zhang <zhangchiqing@gmail.com>

* restrict fees to mainnet testnet and canary

* Update core.go

* split sign and upload into two steps

* update localnet

- update local net flags for LN/SN nodes

- update default access node count to 2

-  put flow client config prep in common function

* add comments

* added comments and added AddComponent method to component manager

* Networking and scaffold changes

* fix build errors

* fix build errors

* remove unnecessary contexts.

* fix tests

* generate mocks

* fix more tests

* Update peerManager.go

* fix bug

* Update peerManager.go

* fix tests

* more test fixes

* Update network.go

* remove mocks

* fix more tests

* update integration tests

- update integration tests flags and container configs (from #1323)

* fix tesets

* add missing error return when node fails to prepare flow client conn opts

* Update flow_client.go

* update dkg tests controller factory client arg

* modify component implementations

* fix build error

* fix lint

* address comments

* go mod tidy

* update comments and log statement formatting

* Update flow_client.go

* add fallback to poll/submitResult

* add log

* fix bug creating flow client opts

we always used the public key from the first AN, making any fallback
attempts useless (wrong key)

* [Consensus] fix startup time (#1402)

* fix startup time

* remove log from debugging

* maintain ordering of ANs

* update var names

* update flag help string, improve logging,

- use trim prefix

* [crypto] Move ADX detection to crypto Makefile

Fixes #1229

* [crypto] De-quarantine BLST tests

Their flakiness was fixed by #1227

* add blockID logging
rename reference block for clarity

* Update broker.go

* [Consensus and Collection] Refactors guarantee dissemination  (#1406)

* refactors pusher engine multicast

* refactors ingestion engine publish mechanism on consensus node side

* fixes lint

* fixes a comment

* Add insanely long rapid test

* added comments

* update test

* better channel closed checks

* better error checking

* fix subtle race condition

* reduce channel close latency allowance

* [Consensus] Revert unsealed reason (#1332)

* revert unsealed reason

* fix linter

Co-authored-by: Alexander Hentschel <alex.hentschel@axiomzen.co>

* Update component-interface.md

* update indentation

* docs: fix dead links

* Add details about RunComponent

* Add comments and fix indentation

* add period

* fix syntax error

* add '*' default value for --access-node-ids flag for LN/SN nodes

* fix lint

* Update command_runner_test.go

* Update integration/testnet/client.go

Co-authored-by: Jordan Schalm <jordan@dapperlabs.com>

* add sentinel for disabled dkg client

* ensure emulator link is enabled at end of test

* fix typo

* fix cadence type assertion

the test assume a particular ordering of Cadence maps, which no longer
can be guaranteed by the implementation

* update var names, remove unused funcs, refactor

* Merge branch 'khalil/1595-access-node-ids-default' of github.com:onflow/flow-go into khalil/1583-stake-node-util-func

* grammar, add sanity check if 0 node ids returned for default, remove obsolete struct field ContainerConfig.Unstaked

* Update cmd/util/cmd/common/validation.go

Co-authored-by: Jordan Schalm <jordan@dapperlabs.com>

* fix linter

* update signaler implementation and runcomponent

* Update irrecoverable API to match semantics of Context API

* Update component manager

* Update startable

* update network and middleware

* fix command runner and scaffold

* fixing subtle race condition

* Update run_component_test.go

* Delete failures

* fix component tests

* fix test failures

* Update irrecoverable_example_test.go

* fix lint

* Implement log level admin command

* Update command_runner_test.go

* fix test bug

* update client funcs, use normal flow account, add TestEpochJoin placeholder test

- add CreateAccount and ExecuteScriptBytes funcs to client

- remove locked tokens from staking flow, use normal flow account

- add placeholder tests for epoch join

* test(sync-engine): mode HandleHeight in rapid tests of sync engine

FIxes dapperlabs/flow-go#5894

* test(sync engine): Improvements in heigth modeling for the sync engine tests

This now models the canceling of height requests by other requests.

* [test-only, sync-engine] Simplify request tracking

* Update component.go

* Update component-interface.md

* use get node info script and compare node id info

* address comments

* Move command into new file

* re-point links from content/ to the rendered page

* add comment describing test

* Update integration/tests/epochs/suite.go

Co-authored-by: Jordan Schalm <jordan@dapperlabs.com>

* Update epoch_test.go

* fix lint

* tidy

* run_component_test minor updates to tests and  documentation

* remove unused func

* remove redundant key generation code

* remove error return, minor updates

* Update component-interface.md

* indentation

* add hotstuff view to ping route

* address comments

* remove redundant redirects

* fix lint

* applies fix (#1416)

Co-authored-by: Leo Zhang <zhangchiqing@gmail.com>

* fix tests

* fix engine unit

Co-authored-by: Simon Zhu <simon.zsiyan@gmail.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
Co-authored-by: Alexander Hentschel <alex.hentschel@axiomzen.co>
Co-authored-by: Jordan Schalm <jordan@dapperlabs.com>
Co-authored-by: Yurii Oleksyshyn <yuraolex@gmail.com>
Co-authored-by: Vishal <1117327+vishalchangrani@users.noreply.github.com>
Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>
Co-authored-by: François Garillot <francois.garillot@dapperlabs.com>
Co-authored-by: François Garillot <4142+huitseeker@users.noreply.github.com>
Co-authored-by: Maks Pawlak <120831+m4ksio@users.noreply.github.com>
Co-authored-by: Kay-Zee <kan@axiomzen.co>
Co-authored-by: Janez Podhostnik <janez.podhostnik@gmail.com>
Co-authored-by: Khalil Claybon <khalil.claybon@dapperlabs.com>
Co-authored-by: Janez Podhostnik <67895329+janezpodhostnik@users.noreply.github.com>
Co-authored-by: ramtinms <ramtin@axiomzen.co>
Co-authored-by: Ramtin M. Seraj <ramtinms@users.noreply.github.com>
Co-authored-by: Martin Gallagher <martin@martingallagher.com>
Co-authored-by: Yahya Hassanzadeh <yhassanzadeh13@ku.edu.tr>
Co-authored-by: Mangirdas <mangirdas@judeikis.lt>
Co-authored-by: gomisha <misha@gomisha.com>
  • Loading branch information
21 people committed Oct 18, 2021
1 parent dd55cdd commit c356b1d
Show file tree
Hide file tree
Showing 312 changed files with 12,464 additions and 2,922 deletions.
38 changes: 1 addition & 37 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,40 +97,4 @@ jobs:
with:
timeout_minutes: 15
max_attempts: 2
command: make ci-integration

cross-blst-test:
name: cross-testing internal BLS implementation with BLST
strategy:
fail-fast: false
runs-on: ubuntu-latest
steps:
- name: CPU information
run: |
lscpu
- name: Setup Go
uses: actions/setup-go@v2
with:
go-version: 1.16
- name: Checkout repo
uses: actions/checkout@v2
- name: Build relic
run: make crypto/relic/build
- name: Run tests
if: github.actor != 'bors[bot]'
run: |
if ! (grep -q -e '^flags.*\badx\b' /proc/cpuinfo) 2>/dev/null; then
export CGO_CFLAGS="-D__BLST_PORTABLE__"
fi
make -C crypto cross-blst-test
- name: Run tests (Bors)
if: github.actor == 'bors[bot]'
uses: nick-invision/retry@v2
with:
timeout_minutes: 25
max_attempts: 2
command: |
if ! (grep -q -e '^flags.*\badx\b' /proc/cpuinfo) 2>/dev/null; then
export CGO_CFLAGS="-D__BLST_PORTABLE__"
fi
make -C crypto cross-blst-test
command: make ci-integration
9 changes: 6 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ install-tools: crypto/relic/build check-go-version
cd ${GOPATH}; \
GO111MODULE=on go get github.com/golang/protobuf/protoc-gen-go@v1.3.2; \
GO111MODULE=on go get github.com/uber/prototool/cmd/prototool@v1.9.0; \
GO111MODULE=on go get github.com/gogo/protobuf/protoc-gen-gofast; \
GO111MODULE=on go get github.com/vektra/mockery/cmd/mockery@v1.1.2; \
GO111MODULE=on go get github.com/golang/mock/mockgen@v1.3.1; \
GO111MODULE=on go get golang.org/x/tools/cmd/stringer@master;
Expand All @@ -66,6 +67,7 @@ unittest:
# test all packages with Relic library enabled
GO111MODULE=on go test -coverprofile=$(COVER_PROFILE) -covermode=atomic $(if $(JSON_OUTPUT),-json,) --tags relic ./...
$(MAKE) -C crypto test
$(MAKE) -C crypto cross-blst-test
$(MAKE) -C integration test

.PHONY: test
Expand Down Expand Up @@ -99,10 +101,11 @@ generate-proto:

.PHONY: generate-mocks
generate-mocks:
GO111MODULE=on mockery -name '(ReadyDoneAwareNetwork|Connector|PingInfoProvider)' -dir=network/p2p -case=underscore -output="./network/mocknetwork" -outpkg="mocknetwork"
GO111MODULE=on mockery -name '(Connector|PingInfoProvider)' -dir=network/p2p -case=underscore -output="./network/mocknetwork" -outpkg="mocknetwork"
GO111MODULE=on mockgen -destination=storage/mocks/storage.go -package=mocks github.com/onflow/flow-go/storage Blocks,Headers,Payloads,Collections,Commits,Events,ServiceEvents,TransactionResults
GO111MODULE=on mockgen -destination=module/mocks/network.go -package=mocks github.com/onflow/flow-go/module Network,Local,Requester
GO111MODULE=on mockgen -destination=module/mocks/network.go -package=mocks github.com/onflow/flow-go/module Local,Requester
GO111MODULE=on mockgen -destination=network/mocknetwork/engine.go -package=mocknetwork github.com/onflow/flow-go/network Engine
GO111MODULE=on mockgen -destination=network/mocknetwork/mock_network.go -package=mocknetwork github.com/onflow/flow-go/network Network
GO111MODULE=on mockery -name 'ExecutionState' -dir=engine/execution/state -case=underscore -output="engine/execution/state/mock" -outpkg="mock"
GO111MODULE=on mockery -name 'BlockComputer' -dir=engine/execution/computation/computer -case=underscore -output="engine/execution/computation/computer/mock" -outpkg="mock"
GO111MODULE=on mockery -name 'ComputationManager' -dir=engine/execution/computation -case=underscore -output="engine/execution/computation/mock" -outpkg="mock"
Expand Down Expand Up @@ -308,7 +311,7 @@ tool-bootstrap: docker-build-bootstrap
.PHONY: docker-build-bootstrap-transit
docker-build-bootstrap-transit:
docker build -f cmd/Dockerfile --build-arg TARGET=bootstrap/transit --build-arg COMMIT=$(COMMIT) --build-arg VERSION=$(VERSION) --no-cache \
--target production-transit-nocgo \
--target production \
-t "$(CONTAINER_REGISTRY)/bootstrap-transit:latest" -t "$(CONTAINER_REGISTRY)/bootstrap-transit:$(SHORT_COMMIT)" -t "$(CONTAINER_REGISTRY)/bootstrap-transit:$(IMAGE_TAG)" .

PHONY: tool-transit
Expand Down
225 changes: 225 additions & 0 deletions admin/admin/admin.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit c356b1d

Please sign in to comment.