CC Rails Portal Activity Authoring, Deployment, and Reporting System
Working git, ruby, and rubygems, wget
Simple Getting Started
This example assumes that rvm is installed. This could be a good idea because we use an older version of ruby. You should install Ruby 2.2.6 first:
rvm install 2.2.6
If you use OS X and you see some errors related to SSL, you might need to use following command instead:
rvm install 2.2.6 --with-openssl-dir=`brew --prefix openssl`
git clone firstname.lastname@example.org:concord-consortium/rigse.git portal cd portal echo 'rvm --create use 2.2.6@portal' >.rvmrc rvm --create use 2.2.6@portal gem install bundler -v '1.11.0' bundle install cp config/database.sample.yml config/database.yml (need to fix the mysql password and/or user) cp config/settings.sample.yml config/settings.yml cp config/app_environment_variables.sample.rb config/app_environment_variables.rb rake db:setup rails s
Now open your browser to http://localhost:3000
Install Docker and make sure that docker-compose is installed too (it should be part of the standard Docker installation).
git clone email@example.com:concord-consortium/rigse.git portal cd portal docker-compose up # this will take 15 minutes to download gems
Increase memory available to Docker to 4GiB-5GiB (OSX: Preferences... -> Advanced tab).
Now open your browser to http://0.0.0.0:3000. On OS X this might
take more than 5 minutes to load the first page. Look in the terminal where you ran
docker-compose up to monitor progress.
Visit the Docker docs for how to use your portal running in docker. This also includes: instructions on speeding things up on OS X, using a local dns+proxy system to avoid port conflicts, and setting up ssh for capistrano deploys.
If you get the following error
An error occurred while installing libv8 (126.96.36.199), and Bundler cannot continue. Make sure that `gem install libv8 -v '188.8.131.52'` succeeds before bundling.
To resolve the error install libv8 sepratelly with --with-system-v8
gem install libv8 -v '184.108.40.206' -- --with-system-v8
If you get the following error
An error occurred while installing therubyracer (0.12.1), and Bundler cannot continue. Make sure that `gem install therubyracer -v '0.12.1'` succeeds before bundling.
gem 'therubyracer', "~>0.12.1" entry in the Gemfile to
gem 'therubyracer', "~>0.10.2"
rails s -p 9000 fails due to mysql2 segmentation fault
gems/mysql2-0.3.21/lib/mysql2/mysql2.bundle: [BUG] Segmentation fault
It usually helps to remove mysql2 and install it again
gem uninstall mysql2 bundle install
After getting the server running it's good to confirm that all the tests pass before changing any code.
Prepare a database for use when running the spec tests:
Start SOLR in test environment (it works with cucumber tests too):
RAILS_ENV=test rake sunspot:solr:run
Run the rspec unit tests:
Prepare a database for use when running the cucumber tests:
RAILS_ENV=cucumber rake db:create RAILS_ENV=cucumber rake db:schema:load rake db:test:prepare_cucumber
Run the cucumber integration tests:
All these tests should pass. If you add features make sure and add tests for these new features.
SSO Clients and LARA (authoring) integration
If you want to provide authentication services to LARA, you need to:
- Create a new SSO Client using
- Add the client id and secret to LARA, by editing
Theme support & Rolling your own theme:
We are using the themes_for_rails gem Theme views go in app/themes/(name)/views/ Theme assets go in app/assets/theme/(name)/ Sample config files go in config/themes/(name)/settings.sample.yml
For now the best thing to do is to copy an existing theme. eg:
mkdir ./config/themes/<new_theme_name> # configuration files: cp ./config/themes/<old_theme_name>/settings.sample.yml ./config/themes/<new_theme_name> # view files: cp -r ./themes/<old_theme_name> ./themes/<new_theme_name> # assets: cp -r ./app/assets/themes/<old_theme_name> ./app/assets/themes/<new_theme_name> # finally change the theme setting in your config/settings.yml open config/settings.yml
NCES District and School Tables
When a rails-portal instance is created two tables containing data for schools and districts in the US are created from data supplied by the National Center for Education Statistics (NCES).
NCES maintains a database about US districts and schools called the Common Core of Data
The rake task:
downloads 2006 NCES CCD data files from NCES website and imports data
from these data files into the following models:
Only data from states and provinces identified in the
config/settings.yml for the portal instance are imported.
The NCES district and school models are used to provide data from which districts and schools actively using the portal are be created.
Portal::Nces06District includes about 50 different fields of data
for each district.
Portal::Nces06Schoolincludes about 500 different fields of data
for each school.
PDF documentation for the NCES data schemas
- NCES Common Core of Data Public Elementary/Secondary School Universe Survey: School Year 2006–07, Version 1b
- NCES Common Core of Data Local Education Agency Universe Survey: School Year 2006–07
Cucumber / Capybara
However, if you would like to run Chrome in non-headless mode on your host machine, this is possible by setting an environment variable
HEADLESS=false. You'll need to install
chromedriver on your host machine and start it with the command:
chromedriver --whitelisted-ips. Ensure you have no firewall running on your host machine, or if you do please open port
9515. Also ensure that Chrome is installed on the host machine.
factory_bot allows you to quickly define prototypes for each of your models and ask for instances with properties that are important to the test at hand.
Running the rspec tests
*Running all the rspec tests:
bundle exec rake spec
Running a single file:
bundle exec rake spec SPEC=spec/routing/dataservice/bundle_contents_routing_spec.rb
Running a single directory:
bundle exec rake spec SPEC=spec/routing/dataservice
Running all the controller tests:
bundle exec rake spec SPEC=spec/controllers
Running the feature tests with cucumber
Running all the feature tests: bundle exec rake cucumber
Running all the feature tests using the ci_reporter gem that's used on the hudson CI system: bundle exec rake hudson:cucumber
Running a single feature:
bundle exec cucumber features/student_can_not_see_deactivated_offerings.feature
Running a single feature in non-headless mode:
HEADLESS=false bundle exec cucumber features/student_can_not_see_deactivated_offerings.feature
Using binding.pry with Cucumber tests
Integration tests are difficult to debug without accessing the content in the browser and inspecting the relevant elements. Using debugging tools in the command line or trying to view the problem from a screenshot is not helpful when the problem might be a hidden link or different element type, for example.
pry in non-headless mode in Chrome opens a new Chrome window showing you the state of the page where
pry has paused the test. You can inspect elements in the page at that point in time to more easily identify the problem.
How to use:
Follow the instructions above to set up and start chromedriver.
And I follow "Admin"
Find the corresponding step definition and insert
When /^(?:|I )follow "([^"]*)"$/ do |link| binding.pry first(:link, link).click end
Make sure chromedriver is running and run the test with HEADLESS=false prepended to the path
$ HEADLESS=false bundle exec cucumber features/admin_accesses_special_pages.feature
pry is hit, a new Chrome window will pop up where you can inspect element and use the pry in the command line as usual.
note: Please see documentation regarding running
chromedriver on your host machine above
Understanding the Codebase
Some video walk-throughs
- The Page Elements Model Part I
- Page Elements Model Part II
- HAML, Compass and SASS
- PageElement View partials
- HAML, Compass and SASS
The Page Elements Model Part I:
screencast: The Page Elements Model PartI
Install github version of railroad with aasm patches from ddolar's repo
Generate a graph of the projects models using railroad: railroad -o models.dot -M
Open that file with omnigraffle, or traslate to some other image format using the dot tool.
Page Elements Model Part II:
screencast: Page Elements Model PartII
Using mysql query browser to view schema: Mysql gui-tools
Use the generator to generate page elements eg:
./script/generate element xhtml content:text
PageElement View partials:
screencast: PageElement View partials
Shows the relationship between:
HAML, Compass and SASS:
screencast: HAML, Compass and SASS
Brief introduction to the technologies generally, and how we use them specifically
Updating a staging server.
Once a development branch has been deployed to a development server, tested and found reliable enough to deploy to staging here's how to do it.
Our convention is to create dev, staging, and production branches in the git repository following that use the same name.
For example the xproject family have the following capistrano stages and branches in teh git repository:
In the code below I will assume that we are using the xproject series of stages and branches.
If you don't already have a local branch of staging
git branch --track xproject_staging origin/xproject_staging
Switch to the staging branch and merge from xproject_dev
git co xproject_staging git merge xproject_dev
Push your copy of the staging branch to the gihub repository:
git push origin xproject_staging
Dump the production database to this file
the production server,
download it to the local folder
db/production_data.sql, the cleans up
the production db/ folder.
cap xproject_production db:fetch_remote_db
Push the production database from the local
the staging server, then import the data into the database on staging, then cleanup.
cap xproject_staging db:push_remote_db
Run any migrations on the staging server:
cap xproject_staging deploy:migrate
There may be rake tasks that need to be run to update or fix data in the database.
These should have corresponding capistrano tasks.
Test the staging server: http://xproject.staging.concord.org/
If the authors confirm that there are no blockers then let people know when the update will take place and perform these tasks on the production server.
other Rake tasks:
rigse:make:investigationsThis task simply finds all activities with no parent investigation, and creates a new investigation for that activity. The created investigation has the same name and description as the activity it contains.
We use haml for some templates, see: http://haml.hamptoncatlin.com/
To install this plugin we followed this procedure:
gem install --no-ri haml
haml --rails path/to/rigse_app
We use haml for some templates, see: http://haml.hamptoncatlin.com/
to install this plugin we followed this procedure:
gem install --no-ri haml
haml --rails path/to/rigse_app
Building installers requires that you are running on a mac with a local
installation of Bit Rock
The rake tasks assume that bitrock is in the standard /Applications/
You can override this by setting an environment variable in your shell
which points to the correct path, eg:
Every host should have its own config/installer.yml file. There is a
config/installer.sample.yml file which can help get you started.
The shortname field should be specific to that host. Because of
limitations in bitrock, the shortname can not use
The jnlp_url should point to a jnlp url on the target host. When you
rake build:installer:build_all or
cap installer:create tasks, the jnlp_url must be available.
Installer Rake Tasks:
Most of the installer building happens via rake tasks defined in
lib/tasks/make_installers.rake. a complete list of tasks can
be gotten using:
rake -T installer
Here are the two useful tasks:
rake build:installer:build_all # build all installers rake build:installer:new_release # create a new release specification interactively
Assuming that your installer.yml file is correct, running
rake build:installer:build_all will take care of the rest.
Build_all will automatically clean up, recache jars, and bump version
Installer Capistrano Recipes
There are two cap recipes in config/deploy.rb which take care of creating installers using remote hosts installer.yml files.
cap installer:copy_configcopies the local installer.yml to the remote server. This would be useful if you ran new_release locally, and then wanted to copy those config settings to the remote server.
cap installer:createcreates the installers and updates the remote installer.yml file, and deploys the installer images.
Sample session for building installers:
boot strapping an unconfigured server:
In this session we are assuming that we are working with a host which does not have a local installer.yml file. First we create a new local release. The first rake tasks asks a bunch of questions, which are answered from the point of view of the staging server.
rake build:installer:new_release cap staging installer:copy_config cap staging installer:create
rake build:installer:new_releasewe end up with the
following local installer file:
shortname: RitesStaging version: "200912.00" jnlp_config: [http://rites-investigations.staging.concord.org/investigations/545.jnlp](http://rites-investigations.staging.concord.org/investigations/545.jnlp)
this file gets pushed up to staging. with
cap staging installer:copy_config we only have to do that the first
time we create an
installer on staging. We could just as easily edit config/installer.yml.
cap staging installer:create handles incrementing the version
number, and pushing the new config files and installers onto staging.
creating a fresh installer for a host that has had installers
cap staging installer:create
not much to do.
Authentication, Sessions, and Cookies
User authentication with Devise
Devise is already setup. The routes are setup, along with the mailers and observers. Forgotten password comes setup, so you don't have to mess around setting it up with every project.
Devise uses the pepper parameter within settings.yml to encrypt user passwords. A default pepper is provided in settings.samles.yml You need to change this when deploying to a public server.
Devise is also setup to use user activation. Users which require activation are sent emails automatically.
Uses the Database for Sessions
We use will_paginate in pretty much every project we use.
You don't want your applications to crash and burn so Exception Notifier is already installed to let you know when everything goes to shit.
config/initializers/exception_notifier.rb does the setup. Currently it reads "admin_email" from config/settings.yml and use it as the destination address. The setup can be modified to include multiple email addresses. See the homepage readme of exception notifier.
It seems rails 2.3.3 and 2.3.4 fails to deliver emails when someone passes multiple destination addresses as an array, which exception notifier does. config/initializers/fix_mailer_on_rails_2.3.4.rb fixes the problem.
The code is borrowed from Dmitry Polushkin
On OS X the mysql2 gem usually can't find the mysql client library that it needs to run. The command below fixes that. It assumes your mysql is installed in the default basedir of /usr/local/mysql/lib. And it assumes you are using bundler.
install_name_tool -change libmysqlclient.16.dylib /usr/local/mysql/lib/libmysqlclient.16.dylib \ `bundle show mysql2`/lib/mysql2/mysql2.bundle
For newer versions of rvm and mysql2, you will see an error like this
dlopen(/Users/scytacki/.rvm/gems/ruby-1.9.3-p545/extensions/x86_64-darwin-13/\ 1.9.1/mysql2-0.3.15/mysql2/mysql2.bundle, 9): Library not loaded: libmysqlclient.18.dylib Referenced from: /Users/scytacki/.rvm/gems/ruby-1.9.3-p545/extensions/x86_64-darwin-13/\ 1.9.1/mysql2-0.3.15/mysql2/mysql2.bundle Reason: image not found - /Users/scytacki/.rvm/gems/ruby-1.9.3-p545/extensions/x86_64-darwin-13/\ 1.9.1/mysql2-0.3.15/mysql2/mysql2.bundle
So then to fix a command like this is needed:
install_name_tool -change libmysqlclient.18.dylib /usr/local/mysql/lib/libmysqlclient.18.dylib \ /Users/scytacki/.rvm/gems/ruby-1.9.3-p545/extensions/x86_64-darwin-13/1.9.1/mysql2-0.3.15/mysql2/mysql2.bundle
Rails 3 Asset Pipeline
The portal uses the"Rails 3 Asset
theme specific assets should be placed in
You should read the Rails guide for more information about the asset Pipeline.
themes_for_rails has been configured to play nice with the asset
pipeline. See the initializer:
cap deploy should trigger the rake task assets:precompile to run.
You can also run locally by hand:
bundle exec rake assets:precompile
The assets will be compiled to public/assets which should be ignored by .gitignore
When running in development mode you do not need to pre-compile your assets.
Solr & Sunspot
Sunspot is being used to provide search capabilities.
You will need to create solr cores if you want to update materials or publish. At the least you will need a 'development' solr core. Here is a basic set of directions:
- Make sure there are no solr processes running with
ps auxxwww | grep solr
scp -r firstname.lastname@example.org:/web/portal/shared/solr-template solr
cp -r ./solr/production ./solr/development
vim ./solr/development/core.properties(change name from production to development)
bundle exec rake sunspot:solr:start
bundle exec rake sunspot:solr:reindex(edited)
You could also create a test core by repeating steps 3 & 4.
In development mode you will need to create an index and start sunspot:
bundle exec rake sunspot:solr:start bundle exec rake sunspot:reindex
You can then visit the web interface to the solar server by visiting localhost:8982/solr/admin/. Though I haven't found any good reason to do so.
Rspec testing with sunspot disabled & enabled:
For rspec tests see the helper methods defined in spec/support/solr_spec_helper.rb
For cucumber tests, you can use "Given The materials have been indexed" to update solr indexes after fixture data has been loaded.
Solr delpoyment and index-updating
If you make changes to how Solr does its indexing, you will have to run a cap task to tell it to reindex:
In theory a simple
bundle exec cap <host> solr:reindex should work, but
to be sure use:
bundle exec cap <host> solr:hard_reindex to restart and reindex.
Application Settings & Settings YAML
There is a settings.yml file that contains site-wide stuff. The site name, url and admin email are all used in the Devise mailers, so you don't need to worry about editing them.
Enabling features via environment variables
Certain features of the portal are controlled via environment variables.
PORTAL_FEATURES environment variable can take a string of the form "feature1 feature2" to
include the following features:
geniverse_remote_auth: Remote authentication
allow_cors: Allow CORS requests (see below)
genigames_data: Genigames-related student sata saving
geniverse_wordpress: Geniverse-related Wordpress connection
If CORS is enable, by default it will allow any request from '*.concord.org', to any route, but can be controlled by two additional environment variables:
CORS_ORIGINS="x.example.com y.z.example.org": Sets the allowed CORS origins to a specific whitelist
CORS_RESOURCES="/xyz": Sets the allowed CORS resources to a specific route
Here is a brief list of things which need to be looked into:
- the embeddables should be dryed up with some mixin / super class.
- not all embeddables are using send_update_events, which is causing stale pages.
- transition to unobtrusive JS.
- send_update_events might not do what we want it to do, tests should be written for it.
- password and password_confirmation are set up to be filtered
- there is a default application layout file
- a page title helper has been added
- index.html is already deleted
- rails.png is already deleted
- a few changes have been made to the default views
- a default css file with blank selectors for common rails elements
Archiving a portal
A set of rake tasks is available under the
archive_portal namespace that enable the portal data to be archived. These tasks are:
- archive_portal:extract_and_upload_images - extracts the image binary data from the database and uploads it to S3
- archive_portal:generate_teacher_reports - generates learner details reports for all teachers and uploads them to S3
- archive_portal:generate_runnable_reports - generates learner details reports for all runnables and uploads them to S3
The rake tasks use a config file at /config/archive_portal.yml to specify the S3 bucket parameters to use when extracting images and to use when generating the url to those images in the reports. A /config/archive_portal.sample.yml file exists to be copied and updated with real values.
These tasks will take a long time. Easiest way to run them is to ssh to running server and run them in the background using nohup, e.g.:
nohup bundle exec rake archive_portal:extract_and_upload_images &
You can close your ssh session and the task will be still running. Logs will be saved in
CC Rails Portal is released under the MIT License.