charity: water – Remote Monitoring
- Ruby 2.1.2
- Postgres 9.3
config/database.yml and update it to
match your database.
.env. The following keys will be added, update them as
FLUID_SURVEYS_HOST=charitywater.fluidsurveys.com:443 FLUID_SURVEYS_USER=<your FluidSurveys email address> FLUID_SURVEYS_PASSWORD=<your FluidSurveys password> FLUID_SURVEYS_LIMIT=200 # Number of survey responses per request. Max: 200. GOOGLE_MAPS_API_KEY=<your Google Maps API key> KAMINARI_PER_PAGE=100 # Number of table rows per page. MAIL_DEFAULT_FROMemail@example.com MAIL_DEFAULT_TOfirstname.lastname@example.org WAZI_APP_ID=<your WAZI app id> WAZI_HOST=<WAZI host address> WAZI_SECRET=<your WAZI secret> WAZI_VERIFY_SSL=true # Set to false in development SECRET_KEY_BASE=<your Rails default secret key base>
To start the app, run:
bundle gem install foreman foreman start
We use the following:
- Foundation is our CSS framework (with Compass). Check out
- Devise provides our authentication.
- Pundit powers our authorization.
To test the app, just run
rake. This will run:
- License Finder, to ensure that we have clearance to use our dependencies.
- Brakeman, our security / vulnerability test.
- RSpec, our primary test suite.
We do test-driven development. This means that we write failing tests first, then we write code to make those tests pass, then we refactor that code.
When developing a new feature, we start with coarse-grain feature tests. These UI-based tests do not handle every possible condition, but serve to drive out the initial code.
Then, we write unit tests. When the feature test requires a new class, such as a new controller, we create the new class and the corresponding unit test. After we bring that class through the red-green-refactor cycle, we return to the feature to continue to the next step of the feature. This, in turn, will drive out a new class or a new method to continue the cycle.
To test integrations with 3rd-parties and all possible cases for critical functions (like authorization), we write integration tests. They generally hit live instances of the services and ensure that the app behaves appropriately. We use VCR to record these external requests and responses to speed up the test suite.
We've put some effort in trying not to hit the database whenever possible. Only feature and model specs do so.
Both staging and production are hosted on Heroku. We use the gem heroku_san to easily deploy the application to Heroku.
Primarily, we deploy after the build has passed in CI. Using Semaphore, navigate to the successful build, click the 'Deploy Manually' link, and select your desired environment.
If you find it necessary to deploy directly to Heroku without going through
Semaphore, something must have gone horribly wrong. Contain your rage, then run
rake <environment> deploy(:force), where
There are other commands provided by
heroku_san that we frequently use. We
rake <environment> console rake <environment> logs:tail
After deploying, if you want to add a new rake task to the scheduled jobs, you can do it through Heroku Scheduler. As of 2014/07/29, these are the scheduled jobs:
rake survey_response_import:both_types[source_observation_v1:source_observation_v02,maintenance_report_v02] rake survey_response_import:remove_deleted
Filtering a View by Program
We have a developed a generic solution for filtering a view by multiple parameters.
For example, you might want to view every project in Ethiopia, or every project that is broken, or every broken project in Ethiopia. Perhaps you want to view the 3rd page of every broken project in Ethiopia.
Our generic solution is a
FilterForm. It wraps the query parameters and it
provides the correct program based on the submitted parameters. Then, a list
ProjectList will use the
FilterForm to figure out the rest of
the filters to apply.
How we show projects on Google Maps
We use ElementalJS to load the map behavior. There's two flavors: a 'Project Map' and a 'Dashboard Map'. The Dashboard Map will load all projects that are visible to the user (based on their access) but has no interaction capability. The Project Map has some additional interactions to support filtering and selecting a project.
A map can be centered based on a project's latitude and longitude or based on a given set of bounds. The projects that are visible to the user and are within the bounds will then be loaded.
When you click on a project on the map, a custom event is triggered. Another behavior then loads the project details in the map sidebar.
Filtering in the map works by grouping the map Markers in a MarkerLayer based on project status. When the status filter is changed, a custom event is triggered. A separate behavior will then tell each MarkerLayer to show or hide their project Markers.
Adding a new version of an existing survey type
- Clone your survey on FluidSurveys into a new, test version.
- Represent the
Structureof the new survey version.
- Add the new
Structureclass to the
- Add the new survey version to the
- Capture business logic changes in the
- Add manual import buttons.
- Add survey version to nightly import tasks.
- Add section title for nightly import email.
- Add new FluidSurveys webhooks.
- Add integration tests.
- Clone your survey on FluidSurveys into a new, test version. This will allow you to submit as many survey responses as you need to test your new integration without actually affecting live data and statistics. Ensure that the new name has the words TEST and LOOKOUT (e.g. TEST for LOOKOUT: Source Observation V.02).
Consider the following steps for both the test and the live versions of the new survey. Of course, you will perform all of these steps by writing your tests first.
- Create a new
FluidSurveys::Structure::<SurveyType><NewVersion>class that subclasses from
FluidSurveys::Structure::Baseto take care of parsing the survey's structure. Note that
Basehas the shared functionality between survey types and versions.
All survey responses from FluidSurveys use a unique key to
represent each question. The app needs to understand which key or keys belong
to each question. We use the
classes to convert each survey's keys back into questions. For example, v1 of
the source observation survey stores its deployment code under the
key, while v02 uses 7 different keys to represent the deployment code. The
SourceObservationV02 classes take care of
determining how to parse the deployment code structure.
The app uses a Ruby symbol to represent each survey type and version. The
FluidSurveys::Structure::Resolverconverts between symbol and the corresponding
Structure. Define a
survey_typeclass method in your new
FluidSurveys::Structure::<SurveyType><NewVersion>with the symbol representing the new survey version (refer to
FluidSurveys::Structure::SourceObservationV1for an example). Then, add your
Add the new symbol to the SurveyResponse model in either
self.maintenance_report_types. These are used throughout the app whenever functionality must differ between survey types.
If the new survey version requires a change of business logic (e.g. changing when a ticket is created), capture that change in the
RemoteMonitoring::PostProcessor::SurveyPolicy. Consider adding a new
PostProcessorto handle any new actions required.
Whenever the app receives a new survey response or a new sensor reading,
several actions must be performed. For example, when repairs were unsuccessful,
a notification email must be sent (among other actions). This action is
represented by the
class RepairsUnsuccessfulEmailSender def process(policy) return unless policy.send_repairs_unsuccessful_email? RemoteMonitoring::JobQueue.enqueue( Email::RepairsUnsuccessfulJob, policy.survey_response.id ) end end class SurveyPolicy # ... other methods ... def send_repairs_unsuccessful_email? from_webhook? && maintenance_report_processable? && try_repairs_again? end end
To be able to manually trigger a full import of the new survey version, add new buttons to the
Import::Surveys#newview for both live and test versions. Ensure that the button for the test version is not shown in production.
When the bulk import is run, a new section will be added for you in the email with the import results of the new survey version. Specify a title for this section in
Add a new FluidSurveys webhook to staging and to production for the new survey version. The app will be notified whenever a new survey response is submitted to FluidSurveys. Refer to the FluidSurveys section for more information. Ensure that a production webhook is not added for the test version.
We hope that any required spec changes will be in obvious locations. Nonetheless, we would like to suggest that you add your new version to the following integration specs:
Adding a new survey type
This one will be trickier. Both the Source Observation survey and the Maintenance Report survey have the same structure, serve similar purposes, and can be handled through the same pipeline. Since, at the time of this writing, we do not know what new information you will be processing, this information will be more generic.
Please consider the actions to add a new survey version. In addition, consider the following actions:
- Add a new entry in the
SurveyResponsemodel with methods for the new survey type.
- Add new methods in the
Webhook::SurveyResponsemodels, to include the new types in the scheduled import jobs on staging and on production.
- Add a new
Import::<NewSurvey>Jobfor the webhook import.
- Add a new
Import::<NewSurveys>Jobfor the nightly bulk import.
- For each of those jobs, add a new importer in
- For each of those jobs, add a new mailer. Usually the bulk sends a full report and the webhook sends a notification about that one event, if the event is relevant.
- Add integration tests.
For further detail, please refer to the implementation of the Source
Observation surveys import (starts at
Pipeline for Importing a Survey Response
Nightly Bulk Import
We use FluidSurveys to collect information about our water points. When field workers visit a water point, they record the water point's conditions by filling out a survey. When FluidSurveys receives a new survey response, it will notify this app via a previously subscribed webhook.
To connect to FluidSurveys, the
FluidSurveys::Client needs the following
FLUID_SURVEYS_HOST=charitywater.fluidsurveys.com:443 FLUID_SURVEYS_USER=<email@example.com> FLUID_SURVEYS_PASSWORD=<yourpassword> FLUID_SURVEYS_LIMIT=200
To list the existing webhooks:
To subscribe to a survey's webhook:
To unsubscribe from a webhook:
Additionally, we have a fallback rake task scheduled as a cron-like job every evening. The task asks FluidSurveys for every survey response for every survey.
To run this task manually:
Use a comma character to separate the source observation types from the
maintenance report types. Use a colon character to separate each type.
rake does not permit whitespace.
Finally, we have another rake task scheduled every evening to remove from our database any survey response that has been deleted from FluidSurveys.
To run this task manually:
Generally, our staging app is subscribed to both live and test versions of each survey type, while our production app is only subscribed to live versions.
The sensor integration has been built off of the spec instead of actual packets
received. The sensors are supposed to hit the
/sensors/receive endpoint. As
of the time of this writing, there are no physical sensors to use
Each sensor will send a weekly message primarily containing the quantity of liters drawn every hour for that week. The sensor will also send an immediate 'red flag' message when it detects an anomaly. It will only send one of these and then resume normal delivery of weekly messages. We receive these messages as JSON from BodyTrace.
GPS with GlobalStar
The app receives regular updates from field vehicles such as drilling rigs. The app logs the response and forwards the decoded payload to WAZI.
GlobalStar sends an XML message to the
/gps/receive endpoint. The primary
element of the XML is the
<payload>0xAF19...</payload> element. There are
documents in Pivotal Tracker to help decode the hexadecimal payload. If you
need these, may God have mercy on your soul.
Using an API key enables you to monitor your application's Maps API usage, and ensures that Google can contact you about your application if necessary If the app generates 25,000 map loads or more each day for 90 days, Google will contact you to ask for payment. However, Google states that apps that in the public's benefit will be allowed access free-of-charge.
To add your Google Maps API key, define the
The app requests project details from WAZI using the
endpoint. A project import can be triggered by an admin through the app. The
app also sends GPS information to the wazi gps endpoint.
- Heroku Postgres is our database.
- Heroku Scheduler handles our nightly tasks.
- New Relic measures server response time, database load, error rate, etc.
- PG Backups takes daily backups of the database with a week's retention. It also takes weekly backups with a month's retention.
- Papertrail stores the application's logs with a searchable interface.
- Redis To Go is used by Resque to handle all of our background jobs.
- SSL ensures we use HTTPS.
- SendGrid sends emails for us.
- Sentry captures and reports application exceptions with insights and trends.
Changing a Project's Status
The sources for these flowcharts are stored in Gliffy.