Skip to content
This repository has been archived by the owner on Aug 2, 2021. It is now read-only.

Network simulation for swarm tests #769

Merged
merged 50 commits into from
Jul 17, 2018
Merged

Network simulation for swarm tests #769

merged 50 commits into from
Jul 17, 2018

Conversation

janos
Copy link
Member

@janos janos commented Jul 3, 2018

This is a proof of concept.

acud and others added 30 commits June 22, 2018 00:58
@acud
cmd/swarm: minor cli flag text adjustments
945fb96
@janos
cmd/swarm, swarm/storage, swarm: fix mingw on windows test issues
f3484ce
@acud
pr changes
0b65a9b
@zelig
Merge pull request #736 from ethersphere/cli-command-documentation
fc98052 
cmd/swarm: minor cli flag text adjustments
@zelig
Merge pull request #737 from ethersphere/fix-tests-mingw
d96d997 
Fix mingw test issues on windows
@nonsense
cmd/swarm: support for smoke tests on the production swarm cluster
a4ed238
@nonsense
cmd/swarm/swarm-smoke: simplify cluster logic as per suggestion
0f7de24
@nonsense
Merge pull request #746 from ethersphere/prod_smoketest
7e33832 
cmd/swarm: support for smoke tests on the production swarm cluster
changed colour of landing page
9001a9c
landing page reacts to enter keypress
10ea10f
@janos
swarm/api/http: sticky footer for swarm landing page using flex
195ca55
@janos
swarm/api/http: sticky footer for error pages and fix for multiple ch…
0a578d7 
…oices
@zelig
Merge pull request #752 from ethersphere/landing-page-fix-2
55f0857 
Landing page fix 2
@nonsense @gbalint
swarm: propagate ctx to internal apis (#754)
a499b68
@janos
swarm/simnet: add basic node/service functions
f81cbe9
@janos
swarm/netsim: add buckets for global state and kademlia health check
ee78635
@janos
Use sync.Map as bucket and provide cleanup function for ServiceFunc
80551d8
@janos
swarm, swarm/netsim: adjust SwarmNetworkTest
a764aa5
@janos
swarm/netsim: fix tests
3905ae1
@holisticode
swarm: added visualization option to sim net redesign
4aa94f7
@janos
swarm/netsim: support multiple services per node
f74ccd2
@janos
swarm/netsim: remove redundant return statement
5e4c54c
@janos
swarm/netsim: add comments
6485d8e
@holisticode
swarm: shutdown HTTP in Simulation.Close
4810b29
@holisticode
swarm: swarm specific HTTP sim server routes
378dfb1
@holisticode
swarm: swarm specific HTTP sim server routes
a28af69
@holisticode
swarm: merged upstream simnet-redesign
6c4831a
@holisticode
swarm: sim HTTP server timeout
39a1ac3
@janos
swarm/netsim: add more simulation methods and peer events examples
8d6907a
@janos
swarm/netsim: add WaitKademlia example
4fa997b
@janos janos requested review from gbalint and lmars July 10, 2018 15:02
@janos
Copy link
Member Author

janos commented Jul 10, 2018

@zelig, thank you for review. the code is now updated with more functions, tests and with @holisticode's work. Could you review it again? And if I remember correctly, you suggested to call this package swarm/simulation, should swarm/netsim be renamed?

holisticode
holisticode approved these changes Jul 10, 2018
View reviewed changes
Copy link
Contributor

@holisticode holisticode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments. The package is still /netsim though, did we reach a conclusion for the name?

swarm/netsim/bucket.go Outdated
)

// BucketKey is the type that should be used for keys in simulation buckets.
type BucketKey string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't you say you removed this type in favor of keys being strings only?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it is required for BucketKeyKademlia, but strings can be used for keys in specific tests.

swarm/netsim/events_test.go Outdated
"time"
)

func TestPeerEvents(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a short description for what the test does if the test does a number of steps, e.g.

  • Create simulation
  • Add 2 nodes
  • Register for PeerEvents
  • Connect peers in chain
  • Check amount of expected events received, should match

Not needed for all tests of course (there are tests which really it is easy to understand, like ExampleSimulation_WaitTillHealthy

swarm/netsim/example_test.go Outdated

// Every node can have a Kademlia associated using the node bucket under
// BucketKeyKademlia key. This allows to use WaitTillHealthy to block until
// all nodes have the their Kadmlias healthy.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a short description of how to run this tests, as they are not "normal" tests (the same way you explained to me in the chat)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, this text can not be run, as it does not fulfill requirements to be executed (no // Output: comment). There is another test fo testing kademlia TestWaitTillHealthy. Using examples as test is well described here https://blog.golang.org/examples. But in this case, it is better to test it properly in a dedicated test, and have examples for documentation.

@janos
Copy link
Member Author

janos commented Jul 10, 2018

@holisticode One suggestion is to call it simulation #769 (comment).

@GitCop
Copy link

GitCop commented Jul 13, 2018

There were the following issues with your Pull Request

  • Commit: 0b65a9b

  • Commits must be in the following format: %{scope}: %{description}

  • Commit: 9001a9c

  • Commits must be in the following format: %{scope}: %{description}

  • Commit: 10ea10f

  • Commits must be in the following format: %{scope}: %{description}

  • Commit: 0a578d7

  • Your subject line is longer than 72 characters

  • Commit: 80551d8

  • Commits must be in the following format: %{scope}: %{description}

Guidelines are available at https://github.com/ethersphere/go-ethereum

This message was auto-generated by https://gitcop.com

@janos
Copy link
Member Author

janos commented Jul 13, 2018

As we discussed and decided, the package has been renamed to swarm/network/simulation.

@zelig zelig merged commit 872defc into develop Jul 17, 2018
nonsense pushed a commit that referenced this pull request Jul 23, 2018
@janos @nonsense
Network simulation for swarm tests (#769)
118c2b2 
* cmd/swarm: minor cli flag text adjustments

* cmd/swarm, swarm/storage, swarm: fix  mingw on windows test issues

* cmd/swarm: support for smoke tests on the production swarm cluster

* cmd/swarm/swarm-smoke: simplify cluster logic as per suggestion

* changed colour of landing page

* landing page reacts to enter keypress

* swarm/api/http: sticky footer for swarm landing page using flex

* swarm/api/http: sticky footer for error pages and fix for multiple choices

* swarm: propagate ctx to internal apis (#754)

* swarm/simnet: add basic node/service functions

* swarm/netsim: add buckets for global state and kademlia health check

* swarm/netsim: Use sync.Map as bucket and provide cleanup function for ServiceFunc

* swarm, swarm/netsim: adjust SwarmNetworkTest

* swarm/netsim: fix tests

* swarm: added visualization option to sim net redesign

* swarm/netsim: support multiple services per node

* swarm/netsim: remove redundant return statement

* swarm/netsim: add comments

* swarm: shutdown HTTP in Simulation.Close

* swarm: sim HTTP server timeout

* swarm/netsim: add more simulation methods and peer events examples

* swarm/netsim: add WaitKademlia example

* swarm/netsim: fix comments

* swarm/netsim: terminate peer events goroutines on simulation done

* swarm, swarm/netsim: naming updates

* swarm/netsim: return not healthy kademlias on WaitTillHealthy

* swarm: fix WaitTillHealthy call in testSwarmNetwork

* swarm/netsim: allow bucket to have any type for a key

* swarm: Added snapshots to new netsim

* swarm/netsim: add more tests for bucket

* swarm/netsim: move http related things into separate files

* swarm/netsim: add AddNodeWithService option

* swarm/netsim: add more tests and Start* methods

* swarm/netsim: add peer events and kademlia tests

* swarm/netsim: fix some tests flakiness

* swarm/netsim: improve random nodes selection, fix TestStartStop* tests

* swarm/netsim: remove time measurement from TestClose to avoid flakiness

* swarm/netsim: builder pattern for netsim HTTP server (#773)

* swarm/netsim: add connect related tests

* swarm/netsim: add comment for TestPeerEvents

* swarm: rename netsim package to network/simulation
nonsense pushed a commit that referenced this pull request Jul 23, 2018
@janos @nonsense
Network simulation for swarm tests (#769)
2743b69 
* cmd/swarm: minor cli flag text adjustments

* cmd/swarm, swarm/storage, swarm: fix  mingw on windows test issues

* cmd/swarm: support for smoke tests on the production swarm cluster

* cmd/swarm/swarm-smoke: simplify cluster logic as per suggestion

* changed colour of landing page

* landing page reacts to enter keypress

* swarm/api/http: sticky footer for swarm landing page using flex

* swarm/api/http: sticky footer for error pages and fix for multiple choices

* swarm: propagate ctx to internal apis (#754)

* swarm/simnet: add basic node/service functions

* swarm/netsim: add buckets for global state and kademlia health check

* swarm/netsim: Use sync.Map as bucket and provide cleanup function for...

* swarm, swarm/netsim: adjust SwarmNetworkTest

* swarm/netsim: fix tests

* swarm: added visualization option to sim net redesign

* swarm/netsim: support multiple services per node

* swarm/netsim: remove redundant return statement

* swarm/netsim: add comments

* swarm: shutdown HTTP in Simulation.Close

* swarm: sim HTTP server timeout

* swarm/netsim: add more simulation methods and peer events examples

* swarm/netsim: add WaitKademlia example

* swarm/netsim: fix comments

* swarm/netsim: terminate peer events goroutines on simulation done

* swarm, swarm/netsim: naming updates

* swarm/netsim: return not healthy kademlias on WaitTillHealthy

* swarm: fix WaitTillHealthy call in testSwarmNetwork

* swarm/netsim: allow bucket to have any type for a key

* swarm: Added snapshots to new netsim

* swarm/netsim: add more tests for bucket

* swarm/netsim: move http related things into separate files

* swarm/netsim: add AddNodeWithService option

* swarm/netsim: add more tests and Start* methods

* swarm/netsim: add peer events and kademlia tests

* swarm/netsim: fix some tests flakiness

* swarm/netsim: improve random nodes selection, fix TestStartStop* tests

* swarm/netsim: remove time measurement from TestClose to avoid flakiness

* swarm/netsim: builder pattern for netsim HTTP server (#773)

* swarm/netsim: add connect related tests

* swarm/netsim: add comment for TestPeerEvents

* swarm: rename netsim package to network/simulation
nonsense pushed a commit that referenced this pull request Jul 23, 2018
@janos @nonsense
swarm: network simulation for swarm tests (#769)
dcaaa3c 
* cmd/swarm: minor cli flag text adjustments

* cmd/swarm, swarm/storage, swarm: fix  mingw on windows test issues

* cmd/swarm: support for smoke tests on the production swarm cluster

* cmd/swarm/swarm-smoke: simplify cluster logic as per suggestion

* changed colour of landing page

* landing page reacts to enter keypress

* swarm/api/http: sticky footer for swarm landing page using flex

* swarm/api/http: sticky footer for error pages and fix for multiple choices

* swarm: propagate ctx to internal apis (#754)

* swarm/simnet: add basic node/service functions

* swarm/netsim: add buckets for global state and kademlia health check

* swarm/netsim: Use sync.Map as bucket and provide cleanup function for...

* swarm, swarm/netsim: adjust SwarmNetworkTest

* swarm/netsim: fix tests

* swarm: added visualization option to sim net redesign

* swarm/netsim: support multiple services per node

* swarm/netsim: remove redundant return statement

* swarm/netsim: add comments

* swarm: shutdown HTTP in Simulation.Close

* swarm: sim HTTP server timeout

* swarm/netsim: add more simulation methods and peer events examples

* swarm/netsim: add WaitKademlia example

* swarm/netsim: fix comments

* swarm/netsim: terminate peer events goroutines on simulation done

* swarm, swarm/netsim: naming updates

* swarm/netsim: return not healthy kademlias on WaitTillHealthy

* swarm: fix WaitTillHealthy call in testSwarmNetwork

* swarm/netsim: allow bucket to have any type for a key

* swarm: Added snapshots to new netsim

* swarm/netsim: add more tests for bucket

* swarm/netsim: move http related things into separate files

* swarm/netsim: add AddNodeWithService option

* swarm/netsim: add more tests and Start* methods

* swarm/netsim: add peer events and kademlia tests

* swarm/netsim: fix some tests flakiness

* swarm/netsim: improve random nodes selection, fix TestStartStop* tests

* swarm/netsim: remove time measurement from TestClose to avoid flakiness

* swarm/netsim: builder pattern for netsim HTTP server (#773)

* swarm/netsim: add connect related tests

* swarm/netsim: add comment for TestPeerEvents

* swarm: rename netsim package to network/simulation
@nonsense nonsense deleted the simnet-redesign branch August 7, 2018 11:45
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@holisticode holisticode holisticode approved these changes

@zelig zelig zelig approved these changes

@gbalint gbalint Awaiting requested review from gbalint

@lmars lmars Awaiting requested review from lmars

Assignees

@janos janos

Labels
ready for review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@janos @GitCop @holisticode @zelig @acud @nonsense