From a03aa77ab286b4847d25f7258abadccd1f0b7355 Mon Sep 17 00:00:00 2001 From: fernandalavalle Date: Thu, 19 Nov 2015 11:41:13 -0500 Subject: [PATCH 1/5] Second round of documentation and code changes to make client paths generalizable. --- README.md | 47 ++++++++++++++- SETUP.md | 48 ---------------- client.cfg | 6 -- client/SETUP.md | 85 ++++++++++++++++++++++++++++ acps-service => client/acps-service | 0 client/client.cfg | 7 +++ client.py => client/client.py | 8 ++- web100clt => client/web100clt | Bin server/SETUP.md | 68 ++++++++++++++++++++++ governor.cfg => server/governor.cfg | 1 + governor.py => server/governor.py | 2 +- 11 files changed, 213 insertions(+), 59 deletions(-) delete mode 100644 SETUP.md delete mode 100755 client.cfg create mode 100644 client/SETUP.md rename acps-service => client/acps-service (100%) create mode 100755 client/client.cfg rename client.py => client/client.py (98%) rename web100clt => client/web100clt (100%) create mode 100644 server/SETUP.md rename governor.cfg => server/governor.cfg (75%) rename governor.py => server/governor.py (99%) diff --git a/README.md b/README.md index a7a86c0..ed5b62a 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,47 @@ -My own version of the Beta project. +#About + +This repository houses the server and client code of a distributed network performance testing framework. Clients receive a daily testing schedule from the central “governor” server, and run NDT tests accordingly. Test data becomes part of MLab’s open data set. + +More about MLab: +http://www.measurementlab.net/ + +More about accessing MLab data: +https://github.com/m-lab/mlab-wikis/blob/master/HowToAccessMLabData.md + +##Original Use + +This framework was originally developed to measure the network performance within Alexandria City Public Schools. This project focused on mapping quantitative data, created by odroid devices serving as clients and placed inside the classrooms, to qualitative descriptions of network conditions based on teacher reports. Analysis of this coupled data was then used to draw conclusions about how connectivity impacts effective use of technology in the classroom. + +##Intended Use + +This testing framework is most effective in a situation that meets the following criteria: +Users are interested measuring end-to-end network performance +There is sufficient bandwidth that the tests run will not affect the results +Users are interested in testing several different locations + +#Quickstart + +##Requirements + +####Server + +*MongoDB +*Static IP or hostname accessible to clients + +####Clients + +*Internet connection + +##Process +To quickly begin creating performance data for your own network follow the steps below: + +1. Clone this repository. +2. Copy “client” folder onto your devices acting as clients. Follow instructions in client/SETUP.md to properly configure the client. +3. Copy “governor” folder onto the server. Follow instructions in governor/SETUP.md to properly configure the governor server. + +##Accessing Data +Clients generate random 16 byte test ID’s associated with each test. These test ID’s are stored in the database on the server mapped to the client ID of the specific client that ran the test. These test ID’s can be used as special flags for querying MLab data. + + + diff --git a/SETUP.md b/SETUP.md deleted file mode 100644 index f8d0df7..0000000 --- a/SETUP.md +++ /dev/null @@ -1,48 +0,0 @@ -## Setup client - -The client is a python script (client.py) intended to run on a small device, receiving its schedule from the governor server. - -### client.cfg - -Client.cfg contains the basic configuration information that client.py will use to request and receive a test schedule from the governor server. It also contains three IDs that define the device privately to the server. -``` -[Addresses] - -partnerID: 1 -groupID: 3 -deviceID: 7 -server_address: 52.91.156.102 -``` - -*partnerID* is an indicator intended to distinguish different users of the governor/client system. - -*groupID* distinguishes different groups of devices, for example at different locations. - -*deviceID* is a unique number for each device deployed by a given partner. - -*server_address* is the world routable IP address where your governor server is running. - -### Setting static IPs - -Put this stanza in /etc/network/interfaces - replacing the IPs with your network info: - -``` -auto eth0 -iface eth0 inet static -address 10.10.1.171 -netmask 255.255.0.0 -gateway 10.10.254.254 -broadcast 10.10.255.255 -dns-nameservers 10.5.0.10 10.5.0.20 -``` - -### Adding script as a startup service - -1) move 'acps-service' to /etc/init.d/ -2) enable it as a startup service: -$ sudo update-rc.d acps-service defaults - -### Connect to the OTI governor server - -ssh -i "oti-measuring-broadband.pem" ubuntu@52.91.156.102 - diff --git a/client.cfg b/client.cfg deleted file mode 100755 index e8703fd..0000000 --- a/client.cfg +++ /dev/null @@ -1,6 +0,0 @@ -[Addresses] - -partnerID: 1 -groupID: 3 -deviceID: 8 -server_address: 52.91.156.102 \ No newline at end of file diff --git a/client/SETUP.md b/client/SETUP.md new file mode 100644 index 0000000..c1bbf2c --- /dev/null +++ b/client/SETUP.md @@ -0,0 +1,85 @@ +#Setup client + +The client is a python script (client.py) intended to run on a small device, receiving its testing schedule from the governor server. + +##client.cfg + +Client.cfg contains the basic configuration information that client.py will use to request and receive a test schedule from the governor server. It also contains three IDs that define the device privately to the server. + +''' +[Addresses] + +partnerID: 1 +groupID: 3 +deviceID: 7 +server_address: 52.91.156.102 +client_path: /home/odroid/ACPS/ +''' + +*partnerID* is an indicator intended to distinguish different users of the governor/client system. +*groupID* distinguishes different groups of devices, for example at different locations. +*deviceID* is a unique number for each device deployed by a given partner. +*server_address* is the world routable IP address where your governor server is running. +*client_path* is the absolute path to the client code + +##Setting Variables in the Configuration File + +The client script depends on absolute paths. This path needs to be defined in the configuration file. If the absolute path to the client script (client.py) is /home/odroid/ACPS/client/client.py, then the client_path variable should be set to /home/odroid/ACPS. + +Make sure the group ID and device ID in the client configuration file (client.cfg) are what you intended for this device. The governor authenticates clients by checking that their client ID (the concatenation of its partner ID, group ID, and device ID) is in its database before assigning it a schedule. + +##Running NDT + +To be able to run NDT, you need to make sure your version of libjansson4 is updated. + +''' +$ sudo apt-get update +$ sudo apt-get install libjansson4 +''' + +##Timezone + +Governor server schedules in relation to the local time. To be sure that the scheduled tests are being run when you expect. Explicitly set the timezone on your client devices to your local time zone. + +''' +$ sudo dpkg-reconfigure tzdata +''' + +##Setting static IPs + +Put this stanza in /etc/network/interfaces - replacing the IPs with your network info: + +''' +auto eth0 +iface eth0 inet static +address 10.10.1.171 +netmask 255.255.0.0 +gateway 10.10.254.254 +broadcast 10.10.255.255 +dns-nameservers 10.5.0.10 10.5.0.20 +''' + +##Adding script as a startup service +1. Make sure the script is executable. + ''' + $ sudo chmod +x 'acps-service' + ''' +2. Move 'acps-service' to /etc/init.d/. While in the client directory: + ''' + $ sudo move 'acps-service' to /etc/init.d/ + ''' +3. Enable it as a startup service. + ''' + $ sudo update-rc.d acps-service defaults + ''' + +##Connect to the OTI governor server +ssh -i "oti-measuring-broadband.pem" ubuntu@52.91.156.102 + + + + + + + + diff --git a/acps-service b/client/acps-service similarity index 100% rename from acps-service rename to client/acps-service diff --git a/client/client.cfg b/client/client.cfg new file mode 100755 index 0000000..55471a8 --- /dev/null +++ b/client/client.cfg @@ -0,0 +1,7 @@ +[Addresses] + +partnerID: 1 +groupID: 3 +deviceID: 8 +server_address: 52.91.156.102 +client_path: /home/odroid/ACPS/ \ No newline at end of file diff --git a/client.py b/client/client.py similarity index 98% rename from client.py rename to client/client.py index c054818..666a1f6 100755 --- a/client.py +++ b/client/client.py @@ -5,8 +5,9 @@ mySchedule= [] governor = sched.scheduler(time.time, time.sleep) config = ConfigParser.RawConfigParser() -clientPath= "/home/odroid/ACPS/" -configFile= clientPath+"client.cfg" +configFile= "client.cfg" + + ############################################ Helpers ############################################################# @@ -200,7 +201,8 @@ def send_testIDs(): talk_to_server(new_message) -config.read(configFile) +config.read(configFile) +clientPath= config.get("Addresses", "client_path") server_address=config.get("Addresses", "server_address") deviceID= config.get("Addresses", "deviceID") #clientID= "partnerID/groupID/deviceID" diff --git a/web100clt b/client/web100clt similarity index 100% rename from web100clt rename to client/web100clt diff --git a/server/SETUP.md b/server/SETUP.md new file mode 100644 index 0000000..2e9d65a --- /dev/null +++ b/server/SETUP.md @@ -0,0 +1,68 @@ +#Setup governor +The governor is a python server (governor.py) intended to schedule tests run on small, remote devices. + +##governor.cfg + +Governor.cfg contains the basic configuration information that the governor server needs to assign schedules. + +''' +[Timing] +runs_per_day: 96 +database_name: ACPS +''' + +*runs_per_day* is the mean of an exponential distribution that is used to schedule about *runs_per_day* tests per day. +*database_name* is the name for the database you specified when setting up mongodb + +##Adding script as a startup service + +The server requires two things to be run on startup. +*Mongodb server in the background +*governor.py + +Instructions for how to run mongodb in the background can be found here: +https://docs.mongodb.org/manual/tutorial/manage-mongodb-processes/ + +To make the governor.py script run on start up: + +1. Write a simple script calling the governor script and printing the output into a log file in the “server” folder. + +''' +$ sudo vi governor_startup_script /etc/init.d/ +''' + +Within the file: + +''' +#!/bin/sh -e +python absolute/path/to/governor.py > absolute/path/to/governor.log +''' + +2. Enable it as a startup service. + +''' +$ sudo update-rc.d acps-service defaults +''' + +##Setting up the database + +By default the governor uses two collections: “devices” and “testIds”. Any device you want to be able to connect with the server should be put into the database. + +Device Schema + +''' +{ + deviceID : ‘1’, + groupID : ‘1’, + partnerID : ‘1’ +} +''' + +TestId Schema + +''' +{ + deviceID : ‘1/1/1’, + testId: ‘91312284cca5bcb38a353ae8c8ec9d4d’ +} +''' \ No newline at end of file diff --git a/governor.cfg b/server/governor.cfg similarity index 75% rename from governor.cfg rename to server/governor.cfg index bbe3cfd..5440255 100755 --- a/governor.cfg +++ b/server/governor.cfg @@ -3,4 +3,5 @@ #rate of testing every 15 minutes runs_per_day: 96 +database_name: ACPS diff --git a/governor.py b/server/governor.py similarity index 99% rename from governor.py rename to server/governor.py index f13a352..c3e29e3 100755 --- a/governor.py +++ b/server/governor.py @@ -21,7 +21,7 @@ #################################### Database Code ##################################################################### client = MongoClient('localhost', 27017) -db = client.ACPS +db = client[config.get("Timing", "database_name")] device_collection = db.devices testID_collection= db.testIds From f4e7eb8073044ce9feecf7e43fd76a8cc78511e5 Mon Sep 17 00:00:00 2001 From: fernandalavalle Date: Thu, 19 Nov 2015 11:53:44 -0500 Subject: [PATCH 2/5] fixed a few style issues --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index ed5b62a..9d7dfba 100644 --- a/README.md +++ b/README.md @@ -25,12 +25,12 @@ Users are interested in testing several different locations ####Server -*MongoDB -*Static IP or hostname accessible to clients +* MongoDB +* Static IP or hostname accessible to clients ####Clients -*Internet connection +* Internet connection ##Process To quickly begin creating performance data for your own network follow the steps below: From 7f4db468fe7c21b1342bb7ba5b80e6e3636cafe4 Mon Sep 17 00:00:00 2001 From: fernandalavalle Date: Thu, 19 Nov 2015 11:58:35 -0500 Subject: [PATCH 3/5] updated server/SETUP.md fixed some style things --- server/SETUP.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/server/SETUP.md b/server/SETUP.md index 2e9d65a..0cff691 100644 --- a/server/SETUP.md +++ b/server/SETUP.md @@ -5,11 +5,11 @@ The governor is a python server (governor.py) intended to schedule tests run on Governor.cfg contains the basic configuration information that the governor server needs to assign schedules. -''' +``` [Timing] runs_per_day: 96 database_name: ACPS -''' +``` *runs_per_day* is the mean of an exponential distribution that is used to schedule about *runs_per_day* tests per day. *database_name* is the name for the database you specified when setting up mongodb @@ -17,8 +17,8 @@ database_name: ACPS ##Adding script as a startup service The server requires two things to be run on startup. -*Mongodb server in the background -*governor.py +* Mongodb server in the background +* governor.py Instructions for how to run mongodb in the background can be found here: https://docs.mongodb.org/manual/tutorial/manage-mongodb-processes/ @@ -27,22 +27,22 @@ To make the governor.py script run on start up: 1. Write a simple script calling the governor script and printing the output into a log file in the “server” folder. -''' +``` $ sudo vi governor_startup_script /etc/init.d/ -''' +``` Within the file: -''' +```python #!/bin/sh -e python absolute/path/to/governor.py > absolute/path/to/governor.log -''' +``` 2. Enable it as a startup service. -''' +``` $ sudo update-rc.d acps-service defaults -''' +``` ##Setting up the database @@ -50,19 +50,19 @@ By default the governor uses two collections: “devices” and “testIds”. A Device Schema -''' +```json { deviceID : ‘1’, groupID : ‘1’, partnerID : ‘1’ } -''' +``` TestId Schema -''' +```json { deviceID : ‘1/1/1’, testId: ‘91312284cca5bcb38a353ae8c8ec9d4d’ } -''' \ No newline at end of file +``` From 4d5d079b2391b090a722bf891c0bb1cafed62970 Mon Sep 17 00:00:00 2001 From: fernandalavalle Date: Thu, 19 Nov 2015 11:59:31 -0500 Subject: [PATCH 4/5] fixed the JSON --- server/SETUP.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/server/SETUP.md b/server/SETUP.md index 0cff691..e9bb8ce 100644 --- a/server/SETUP.md +++ b/server/SETUP.md @@ -50,7 +50,7 @@ By default the governor uses two collections: “devices” and “testIds”. A Device Schema -```json +``` { deviceID : ‘1’, groupID : ‘1’, @@ -60,7 +60,7 @@ Device Schema TestId Schema -```json +``` { deviceID : ‘1/1/1’, testId: ‘91312284cca5bcb38a353ae8c8ec9d4d’ From e9e4f93c5e512686605910e976f3aa492c6dce49 Mon Sep 17 00:00:00 2001 From: fernandalavalle Date: Thu, 19 Nov 2015 12:04:15 -0500 Subject: [PATCH 5/5] updated client/SETUPmd fixed style --- client/SETUP.md | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/client/SETUP.md b/client/SETUP.md index c1bbf2c..cfa935b 100644 --- a/client/SETUP.md +++ b/client/SETUP.md @@ -6,7 +6,7 @@ The client is a python script (client.py) intended to run on a small device, rec Client.cfg contains the basic configuration information that client.py will use to request and receive a test schedule from the governor server. It also contains three IDs that define the device privately to the server. -''' +``` [Addresses] partnerID: 1 @@ -14,7 +14,7 @@ groupID: 3 deviceID: 7 server_address: 52.91.156.102 client_path: /home/odroid/ACPS/ -''' +``` *partnerID* is an indicator intended to distinguish different users of the governor/client system. *groupID* distinguishes different groups of devices, for example at different locations. @@ -32,24 +32,24 @@ Make sure the group ID and device ID in the client configuration file (client.cf To be able to run NDT, you need to make sure your version of libjansson4 is updated. -''' +``` $ sudo apt-get update $ sudo apt-get install libjansson4 -''' +``` ##Timezone Governor server schedules in relation to the local time. To be sure that the scheduled tests are being run when you expect. Explicitly set the timezone on your client devices to your local time zone. -''' +``` $ sudo dpkg-reconfigure tzdata -''' +``` ##Setting static IPs Put this stanza in /etc/network/interfaces - replacing the IPs with your network info: -''' +``` auto eth0 iface eth0 inet static address 10.10.1.171 @@ -57,25 +57,26 @@ netmask 255.255.0.0 gateway 10.10.254.254 broadcast 10.10.255.255 dns-nameservers 10.5.0.10 10.5.0.20 -''' +``` ##Adding script as a startup service 1. Make sure the script is executable. - ''' + ``` $ sudo chmod +x 'acps-service' - ''' + ``` 2. Move 'acps-service' to /etc/init.d/. While in the client directory: - ''' + ``` $ sudo move 'acps-service' to /etc/init.d/ - ''' + ``` 3. Enable it as a startup service. - ''' + ``` $ sudo update-rc.d acps-service defaults - ''' + ``` ##Connect to the OTI governor server +``` ssh -i "oti-measuring-broadband.pem" ubuntu@52.91.156.102 - +```