Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - API phase 9: finish implementing grpc services #2071

Closed
wants to merge 145 commits into from
Closed
Show file tree
Hide file tree
Changes from 142 commits
Commits
Show all changes
145 commits
Select commit Hold shift + click to select a range
675f713
Implement MeshService.CurrentEpoch
lrettig Jun 29, 2020
7413293
Implement and add tests for 3 more endpoints
lrettig Jun 29, 2020
32159dd
Slightly refactor tests
lrettig Jun 29, 2020
623f20e
Move max tx per block into config
lrettig Jun 29, 2020
5026d04
Implement MaxTransactionsPerSecond
lrettig Jun 29, 2020
3085565
WIP implementing LayersQuery
lrettig Jun 29, 2020
46dd378
Complete, untested implementation of LayersQuery
lrettig Jun 30, 2020
1e39060
Trivial version of test of new endpoint passing
lrettig Jun 30, 2020
7a34df6
Implement more comprehensive test
lrettig Jun 30, 2020
3f72fad
Make lint happy
lrettig Jun 30, 2020
137c9a9
Handling and testing of errors
lrettig Jul 1, 2020
61c494a
Add test for missing inputs
lrettig Jul 1, 2020
8b4dc88
WIP: implement AccountMeshDataQuery
lrettig Jul 1, 2020
02c7652
Add mock mempool
lrettig Jul 1, 2020
868406c
Overhaul tests
lrettig Jul 2, 2020
4326ca2
Run grpc server on localhost for test
lrettig Jul 2, 2020
e982dd4
Pass txpool to new MeshService
lrettig Jul 2, 2020
81a994d
Make lint happy
lrettig Jul 2, 2020
67df6b6
Very minor cleanup
lrettig Jul 7, 2020
a152971
Bugfix
lrettig Jul 10, 2020
675657a
Update modules after rebase
lrettig Jul 14, 2020
2c28083
Tidy modules after rebase
lrettig Jul 16, 2020
d9521fc
Basic E2E support for first stream
lrettig Jul 2, 2020
0c428f1
Add activations stream and test for same
lrettig Jul 2, 2020
9a1e428
WIP: sending events
lrettig Jul 3, 2020
a0b73e1
Properly handle filter in AccountMeshDataStream
lrettig Jul 6, 2020
aa9fa0a
Make tests much more robust
lrettig Jul 6, 2020
2308093
WIP cleaning up comprehensive stream test
lrettig Jul 6, 2020
02abfa7
Lots of cleanup
lrettig Jul 6, 2020
224330d
More test cleanup
lrettig Jul 6, 2020
bddf782
Streamline all events reporting
lrettig Jul 6, 2020
794edce
Basic LayerStream implementation
lrettig Jul 6, 2020
7da1620
Add todo note about stream context
lrettig Jul 7, 2020
a7dcc65
Minor test cleanup
lrettig Jul 7, 2020
8956927
Fully validate received layer from LayerStream
lrettig Jul 7, 2020
13e6bd9
Move test function
lrettig Jul 7, 2020
003ddfc
Rename streamer -> reporter
lrettig Jul 7, 2020
09aa1da
Make linter happy
lrettig Jul 7, 2020
a1800a8
Subsume pubsub init into new reporter
lrettig Jul 7, 2020
ed34f3f
Some minor cleanup
lrettig Jul 8, 2020
ddb686f
Add error stream
lrettig Jul 8, 2020
35eaea3
Set up Status channel infrastructure
lrettig Jul 8, 2020
2489a5d
Subscribe to all relevant status events
lrettig Jul 8, 2020
749a7df
Hook into zapcore logging to report errors
lrettig Jul 8, 2020
fa984e6
Add basic test for ReportNodeStatus
lrettig Jul 8, 2020
2487229
WIP writing a node test for the new streams
lrettig Jul 8, 2020
6269e47
Bugfix: reporter must subscribe in a goroutine
lrettig Jul 9, 2020
d01ae88
Cleanup comments
lrettig Jul 9, 2020
ca07971
Improve NodeService stream tests
lrettig Jul 9, 2020
fc173f5
Add cmdline flag for tcp interface
lrettig Jul 9, 2020
c694a8d
Add cmdline flag for grpc server interface
lrettig Jul 9, 2020
69f838b
Add channel to notify that app is started
lrettig Jul 9, 2020
4fd3139
Improve test
lrettig Jul 9, 2020
e308be5
More sophisticated handling of errors
lrettig Jul 9, 2020
c760388
Implement AccountMeshDataQuery MaxResults, Offset
lrettig Jul 9, 2020
b7bbc9c
Handle some lingering TODO items
lrettig Jul 10, 2020
3b44b44
Bugfix in Reporter
lrettig Jul 14, 2020
93d7e0d
Make linter happy
lrettig Jul 16, 2020
cdd729b
Make linter great again
lrettig Jul 16, 2020
393f972
Fix test
lrettig Jul 16, 2020
5a1353d
Fix test for CI and Docker
lrettig Jul 17, 2020
956e816
Small tweaks to fix tests
lrettig Jul 17, 2020
cc403db
Skip two defunct tests for now
lrettig Jul 17, 2020
e93994a
Register all services before starting grpc server
lrettig Jul 18, 2020
6e512a6
Make tests less fragile
lrettig Jul 18, 2020
2b02f87
Try fixing the leaking another way
lrettig Jul 18, 2020
3a57f40
Add config for all services
lrettig Jul 7, 2020
87758e2
Finish GlobalStateService mocks
lrettig Jul 8, 2020
6560560
Finish TxService, SmesherService stubs
lrettig Jul 8, 2020
e34f5dd
Get state root hash for all layers
lrettig Jul 9, 2020
31974a6
Update to latest api build
lrettig Jul 9, 2020
928ad87
Remove two unused interface methods
lrettig Jul 9, 2020
c712d75
Implement GlobalStateHash
lrettig Jul 9, 2020
6ee34ba
Implement and test Account endpoint
lrettig Jul 9, 2020
448cf00
Minor cleanup of filters in MeshService
lrettig Jul 9, 2020
01501ab
WIP: partially implemented AccountDataQuery
lrettig Jul 9, 2020
180382e
Update comment
lrettig Jul 9, 2020
690e6ee
Update txProcessor interface in test
lrettig Jul 10, 2020
3ff559b
Update and clean up tests
lrettig Jul 10, 2020
d51526c
Finish implementing AccountDataQuery
lrettig Jul 10, 2020
4b0b6ba
Mark SmesherDataQuery as unimplemented
lrettig Jul 10, 2020
1d037d8
WIP implementing AccountDataStream
lrettig Jul 10, 2020
7282067
Finish AccountDataStream
lrettig Jul 13, 2020
1dfd9ff
Add blocking option to Reporter
lrettig Jul 14, 2020
72fb55d
Update to latest api module version
lrettig Jul 14, 2020
5e77cf3
Update api module again
lrettig Jul 14, 2020
b3e53b5
Rename to match latest API module
lrettig Jul 14, 2020
f95e463
Finish GlobalStateService, add basic test
lrettig Jul 14, 2020
7843f6a
Add comprehensive test for same
lrettig Jul 14, 2020
20082f6
Revert "Remove two unused interface methods"
lrettig Jul 14, 2020
ed0b689
Update to latest API module
lrettig Jul 14, 2020
ad825ac
Rename things to match latest API build
lrettig Jul 14, 2020
f73faf0
Finish alpha implementation of SmesherService
lrettig Jul 14, 2020
6c34b4b
Add basic SmesherService tests and stubs
lrettig Jul 14, 2020
c0f9ed3
Properly handle input errors
lrettig Jul 14, 2020
601a0e7
Add more tests for unimplemented endpoints
lrettig Jul 15, 2020
7e32692
Minor cleanup
lrettig Jul 15, 2020
00ecda3
Initial complete implementation of TxService
lrettig Jul 15, 2020
47b05b6
More comprehensive tests
lrettig Jul 15, 2020
fcd0a8d
Finish adding TxService tests
lrettig Jul 16, 2020
ad8afd0
Add another more robust stream test
lrettig Jul 16, 2020
9313076
Update mocked object to match interface change
lrettig Jul 16, 2020
0dbdcde
Do not allow reporter to be reinitialized
lrettig Jul 16, 2020
4dfca3d
Update service arguments
lrettig Jul 16, 2020
27d839b
Add AlwaysListen config
lrettig Jul 16, 2020
e71dc7f
Add E2E test of tx service
lrettig Jul 16, 2020
6114530
Add more bad input tests
lrettig Jul 16, 2020
9ee01da
Make status reporting much simpler
lrettig Jul 16, 2020
501e015
Add issue links for TODO items
lrettig Jul 16, 2020
fd1f624
Register all services to gateway server
lrettig Jul 18, 2020
4695f7c
Make lint happy
lrettig Jul 20, 2020
4b982a3
Fix merge conflicts, update API version
lrettig Jul 20, 2020
d63c552
Always close reporter before reinit in tests
lrettig Jul 20, 2020
ef9582d
Re-enable and fix failing tests
lrettig Jul 21, 2020
e8cb54b
Bugfix: run http server in a goroutine
lrettig Jul 21, 2020
4f6abeb
Bugfix: make sure context isn't closed too early
lrettig Jul 22, 2020
6da5f5b
Minor improvements to tests
lrettig Jul 22, 2020
b46492a
Make test less flakey
lrettig Jul 22, 2020
fc2e391
Small fix to test
lrettig Jul 22, 2020
e9a37c7
Make thread-safe
lrettig Jul 22, 2020
13a098a
Remove an unnecessary check
lrettig Jul 22, 2020
e56ef41
Fix test
lrettig Jul 22, 2020
0530600
Improve concurrency in test
lrettig Jul 22, 2020
f74aa22
Better isolate tests
lrettig Jul 23, 2020
48ef2c4
Remove unused args from tx service
lrettig Jul 23, 2020
4ca2167
Remove a redundant report
lrettig Jul 28, 2020
628ce8f
Small tweaks to improve logging
lrettig Jul 28, 2020
6901fd2
WIP troubleshoot LayerStream issue
lrettig Jul 29, 2020
1fa7a2b
Fix error about missing layer state root
lrettig Jul 29, 2020
fdc01c7
Fix debug message
lrettig Jul 29, 2020
97eac4a
Add link to issue for TODO
lrettig Jul 30, 2020
224326e
Fix lingering merge conflict
lrettig Jul 30, 2020
8adf6d4
Bugfix: correctly calculate CommitmentSize
lrettig Jul 30, 2020
5346114
Make linter happy, fix a test
lrettig Jul 30, 2020
176b46d
Fix another test
lrettig Jul 30, 2020
dec0a01
Fix test
lrettig Jul 30, 2020
2f60c6e
Bugfix, add debug output
lrettig Aug 11, 2020
146c5c1
Final cleanup, per review
lrettig Aug 11, 2020
c21de74
More cleanup post refactor
lrettig Aug 11, 2020
5108f75
One more for the refactor
lrettig Aug 11, 2020
616cb9b
Fix more imports post-rebase
lrettig Aug 11, 2020
a47f6db
Fix tests post-rebase
lrettig Aug 11, 2020
dc5e05e
Make endpoint function description clearer
lrettig Aug 12, 2020
b50d919
Fix small typo
lrettig Aug 20, 2020
1eff553
Merge branch 'develop' into api-phase9
lrettig Sep 9, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
9 changes: 7 additions & 2 deletions activation/activation.go
Expand Up @@ -148,6 +148,11 @@ func (b *Builder) Stop() {
close(b.stop)
}

// GetSmesherID returns the ID of the smesher that created this activation
func (b *Builder) GetSmesherID() types.NodeID {
return b.nodeID
}

// SignAtx signs the atx and assigns the signature into atx.Sig
// this function returns an error if atx could not be converted to bytes
func (b *Builder) SignAtx(atx *types.ActivationTx) error {
Expand Down Expand Up @@ -191,7 +196,7 @@ func (b *Builder) loop() {
if _, stopRequested := err.(StopRequestedError); stopRequested {
return
}
events.Publish(events.AtxCreated{Created: false, Layer: uint64(b.currentEpoch())})
events.ReportAtxCreated(false, uint64(b.currentEpoch()), "")
<-b.layerClock.AwaitLayer(b.layerClock.GetCurrentLayer() + 1)
}
}
Expand Down Expand Up @@ -424,7 +429,7 @@ func (b *Builder) PublishActivationTx() error {
}

b.log.Event().Info("atx published!", atx.Fields(size)...)
events.Publish(events.AtxCreated{Created: true, ID: atx.ShortString(), Layer: uint64(b.currentEpoch())})
events.ReportAtxCreated(true, uint64(b.currentEpoch()), atx.ShortString())

select {
case <-atxReceived:
Expand Down
11 changes: 11 additions & 0 deletions activation/activationdb_test.go
Expand Up @@ -84,6 +84,17 @@ func (MockState) AddressExists(types.Address) bool {
return true
}

func (MockState) GetLayerStateRoot(layer types.LayerID) (types.Hash32, error) {
panic("implement me")
}

func (MockState) GetBalance(addr types.Address) uint64 {
panic("implement me")
}
func (MockState) GetNonce(addr types.Address) uint64 {
panic("implement me")
}

type ATXDBMock struct {
mock.Mock
counter int
Expand Down
4 changes: 2 additions & 2 deletions activation/atxdb.go
Expand Up @@ -338,7 +338,7 @@ func (db *DB) deleteLock(viewHash types.Hash12) {
// - ATX LayerID is NipstLayerTime or less after the PositioningATX LayerID.
// - The ATX view of the previous epoch contains ActiveSetSize activations.
func (db *DB) SyntacticallyValidateAtx(atx *types.ActivationTx) error {
events.Publish(events.NewAtx{ID: atx.ShortString(), LayerID: uint64(atx.PubLayerID.GetEpoch())})
events.ReportNewActivation(atx)
pub, err := ExtractPublicKey(atx)
if err != nil {
return fmt.Errorf("cannot validate atx sig atx id %v err %v", atx.ShortString(), err)
Expand Down Expand Up @@ -760,7 +760,7 @@ func (db *DB) HandleGossipAtx(data service.GossipMessage, syncer service.Syncer)
}

err = db.SyntacticallyValidateAtx(atx)
events.Publish(events.ValidAtx{ID: atx.ShortString(), Valid: err == nil})
events.ReportValidActivation(atx, err == nil)
if err != nil {
db.log.Warning("received syntactically invalid ATX %v: %v", atx.ShortString(), err)
// TODO: blacklist peer
Expand Down
46 changes: 40 additions & 6 deletions api/api_test.go
Expand Up @@ -19,7 +19,6 @@ import (
"github.com/spacemeshos/go-spacemesh/common/types"
"github.com/spacemeshos/go-spacemesh/common/util"
config2 "github.com/spacemeshos/go-spacemesh/config"
"github.com/spacemeshos/go-spacemesh/mesh"
"github.com/spacemeshos/go-spacemesh/p2p/p2pcrypto"
"github.com/spacemeshos/go-spacemesh/p2p/service"
"github.com/spacemeshos/go-spacemesh/priorityq"
Expand Down Expand Up @@ -181,6 +180,33 @@ func (t *TxAPIMock) AddressExists(types.Address) bool {
return true
}

func (t *TxAPIMock) GetATXs([]types.ATXID) (map[types.ATXID]*types.ActivationTx, []types.ATXID) {
return nil, nil
}

func (t *TxAPIMock) GetTransactions([]types.TransactionID) ([]*types.Transaction, map[types.TransactionID]struct{}) {
return nil, nil
}

func (t *TxAPIMock) GetLayer(types.LayerID) (*types.Layer, error) {
return nil, nil
}

func (t *TxAPIMock) ProcessedLayer() types.LayerID {
panic("implement me")
}

func (t *TxAPIMock) GetLayerStateRoot(layer types.LayerID) (types.Hash32, error) {
panic("implement me")
}

func (t *TxAPIMock) GetBalance(addr types.Address) uint64 {
panic("implement me")
}
func (t *TxAPIMock) GetNonce(addr types.Address) uint64 {
panic("implement me")
}

// MiningAPIMock is a mock for mining API
type MiningAPIMock struct{}

Expand All @@ -199,6 +225,14 @@ func (*MiningAPIMock) StartPost(types.Address, string, uint64) error {

func (*MiningAPIMock) SetCoinbaseAccount(types.Address) {}

func (*MiningAPIMock) GetSmesherID() types.NodeID {
panic("implement me")
}

func (*MiningAPIMock) Stop() {
panic("implement me")
}

type OracleMock struct{}

func (*OracleMock) GetEligibleLayers() []types.LayerID {
Expand Down Expand Up @@ -399,22 +433,22 @@ func TestJsonWalletApi(t *testing.T) {
// test get txs per account:

// add incoming tx to mempool
mempoolTxIn, err := mesh.NewSignedTx(1337, addr, 420, 3, 42, signing.NewEdSigner())
mempoolTxIn, err := types.NewSignedTx(1337, addr, 420, 3, 42, signing.NewEdSigner())
r.NoError(err)
txMempool.Put(mempoolTxIn.ID(), mempoolTxIn)

// add outgoing tx to mempool
mempoolTxOut, err := mesh.NewSignedTx(1337, types.BytesToAddress([]byte{1}), 420, 3, 42, signer)
mempoolTxOut, err := types.NewSignedTx(1337, types.BytesToAddress([]byte{1}), 420, 3, 42, signer)
r.NoError(err)
txMempool.Put(mempoolTxOut.ID(), mempoolTxOut)

// add incoming tx to mesh
meshTxIn, err := mesh.NewSignedTx(1337, addr, 420, 3, 42, signing.NewEdSigner())
meshTxIn, err := types.NewSignedTx(1337, addr, 420, 3, 42, signing.NewEdSigner())
r.NoError(err)
txAPI.returnTx[meshTxIn.ID()] = meshTxIn

// add outgoing tx to mesh
meshTxOut, err := mesh.NewSignedTx(1337, types.BytesToAddress([]byte{1}), 420, 3, 42, signer)
meshTxOut, err := types.NewSignedTx(1337, types.BytesToAddress([]byte{1}), 420, 3, 42, signer)
r.NoError(err)
txAPI.returnTx[meshTxOut.ID()] = meshTxOut

Expand Down Expand Up @@ -597,7 +631,7 @@ func genTx(t *testing.T) *types.Transaction {
require.NoError(t, err)
signer, err := signing.NewEdSignerFromBuffer(key)
require.NoError(t, err)
tx, err := mesh.NewSignedTx(1111, [20]byte{}, 1234, 11, 321, signer)
tx, err := types.NewSignedTx(1111, [20]byte{}, 1234, 11, 321, signer)
require.NoError(t, err)

return tx
Expand Down
79 changes: 49 additions & 30 deletions api/config/config.go
Expand Up @@ -6,30 +6,38 @@ import (
)

const (
defaultStartGRPCServer = false
defaultGRPCServerPort = 9091
defaultNewGRPCServerPort = 9092
defaultStartJSONServer = false
defaultStartNewJSONServer = false
defaultJSONServerPort = 9090
defaultNewJSONServerPort = 9093
defaultStartNodeService = false
defaultStartMeshService = false
defaultStartGRPCServer = false
defaultGRPCServerPort = 9091
defaultNewGRPCServerPort = 9092
defaultNewGRPCServerInterface = ""
defaultStartJSONServer = false
defaultStartNewJSONServer = false
defaultJSONServerPort = 9090
defaultNewJSONServerPort = 9093
defaultStartNodeService = false
defaultStartMeshService = false
defaultStartGlobalStateService = false
defaultStartTransactionService = false
defaultStartSmesherService = false
)

// Config defines the api config params
type Config struct {
StartGrpcServer bool `mapstructure:"grpc-server"`
StartGrpcServices []string `mapstructure:"grpc"`
GrpcServerPort int `mapstructure:"grpc-port"`
NewGrpcServerPort int `mapstructure:"grpc-port-new"`
StartJSONServer bool `mapstructure:"json-server"`
StartNewJSONServer bool `mapstructure:"json-server-new"`
JSONServerPort int `mapstructure:"json-port"`
NewJSONServerPort int `mapstructure:"json-port-new"`
StartGrpcServer bool `mapstructure:"grpc-server"`
StartGrpcServices []string `mapstructure:"grpc"`
GrpcServerPort int `mapstructure:"grpc-port"`
NewGrpcServerPort int `mapstructure:"grpc-port-new"`
NewGrpcServerInterface string `mapstructure:"grpc-interface-new"`
StartJSONServer bool `mapstructure:"json-server"`
StartNewJSONServer bool `mapstructure:"json-server-new"`
JSONServerPort int `mapstructure:"json-port"`
NewJSONServerPort int `mapstructure:"json-port-new"`
// no direct command line flags for these
StartNodeService bool
StartMeshService bool
StartNodeService bool
StartMeshService bool
StartGlobalStateService bool
StartTransactionService bool
StartSmesherService bool
}

func init() {
Expand All @@ -39,16 +47,20 @@ func init() {
// DefaultConfig defines the default configuration options for api
func DefaultConfig() Config {
return Config{
StartGrpcServer: defaultStartGRPCServer, // note: all bool flags default to false so don't set one of these to true here
StartGrpcServices: nil, // note: cannot configure an array as a const
GrpcServerPort: defaultGRPCServerPort,
NewGrpcServerPort: defaultNewGRPCServerPort,
StartJSONServer: defaultStartJSONServer,
StartNewJSONServer: defaultStartNewJSONServer,
JSONServerPort: defaultJSONServerPort,
NewJSONServerPort: defaultNewJSONServerPort,
StartNodeService: defaultStartNodeService,
StartMeshService: defaultStartMeshService,
StartGrpcServer: defaultStartGRPCServer, // note: all bool flags default to false so don't set one of these to true here
StartGrpcServices: nil, // note: cannot configure an array as a const
GrpcServerPort: defaultGRPCServerPort,
NewGrpcServerPort: defaultNewGRPCServerPort,
NewGrpcServerInterface: defaultNewGRPCServerInterface,
StartJSONServer: defaultStartJSONServer,
StartNewJSONServer: defaultStartNewJSONServer,
JSONServerPort: defaultJSONServerPort,
NewJSONServerPort: defaultNewJSONServerPort,
StartNodeService: defaultStartNodeService,
StartMeshService: defaultStartMeshService,
StartGlobalStateService: defaultStartGlobalStateService,
StartTransactionService: defaultStartTransactionService,
StartSmesherService: defaultStartSmesherService,
lrettig marked this conversation as resolved.
Show resolved Hide resolved
}
}

Expand All @@ -61,14 +73,21 @@ func (s *Config) ParseServicesList() error {
s.StartMeshService = true
case "node":
s.StartNodeService = true
case "globalstate":
s.StartGlobalStateService = true
case "transaction":
s.StartTransactionService = true
case "smesher":
s.StartSmesherService = true
default:
return errors.New("unrecognized GRPC service requested: " + svc)
}
}

// If JSON gateway server is enabled, make sure at least one
// GRPC service is also enabled
if s.StartNewJSONServer && !s.StartNodeService {
if s.StartNewJSONServer && !s.StartNodeService && !s.StartMeshService &&
!s.StartGlobalStateService && !s.StartTransactionService && !s.StartSmesherService {
return errors.New("must enable at least one GRPC service along with JSON gateway service")
}

Expand Down