Welcome to the Encyclopedia of Life project. The bulk of the code needed to run www.eol.org is written in Ruby on Rails and is made available to anyone for re-use, repurposing or for improvement. This is both an ambitious project and an ambitious codebase and we are excited to share it with the open source community. The code has been under development since approximately September 2007, but has undergone many revisions and updates from June 2008-December 2008. There is much work to be done, both in adding new features, and in the ongoing process of code refactoring and performance improvements. If you see something you like, share it with your colleagues and friends and reuse it in your own projects. If you see something you don’t like, help us fix it or join the discussion at the “Developers” forum at forum.eol.org, or on GitHub or Google Code.
The full code base is released under the MIT License. Details are available in the “MIT-LICENSE.txt” file at the root of the code folder.
This is a big Rails project. Clearly, some of the installation steps below are big and require multiple steps, but if you are currently a Rails developer, some are already done (like installing Ruby). If you are not a Rails developer, we suggest you first visit www.rubyonrails.org for more information on getting started with Rails and then return to us when you have your feet wet. The www.eol.org codebase probably shouldn’t be the first Rails project you’ve ever seen.
For seasoned Rails developers, you’ll also notice the codebase does some mix and matching — both restful controllers and regular controllers, both ERB and HAML. We like the restful way of doing things and plan to move in that direction. HAML vs ERB is still an open topic for discussion, although we are sticking with ERB until it’s settled.
Hint: start with the TaxonConcept model. Everything stems from there.
To get things up and running, these are the steps you need to take. If you actually run through this process, please update this list with any changes you notice being necessary!
Note that many of these steps require root access on your machine. You have been warned and may need to run them as “sudo” on a Mac/Linux or as an administrator on Windows (there, I acknowledged the existence of Windows).
-
Install Ruby and the Rails framework on your local machine (www.ruby-lang.org and www.rubyonrails.org).
-
Install MySQL on your local machine (www.mysql.com).
-
Get the code from the public repository (repository.eol.org/eol) or GitHub.
Next, continue with the following sections: “INSTALLING REQUIRED GEMS”, “SETTING UP THE DATABASES”, “INSTALLING AND RUNNING MEMCACHED”, and “WATCHING IT COME TO LIFE”
EOL requires Solr (a fast indexing engine) to run properly. Solr is run using Java, and the require JAR file is included with the EOL project’s source code. (TODO - we need to check the legal situtaion with this–we may need to include a LICENCSE for it.) There is a rake task for starting (solr:start) and stopping (solr:stop) the Solr server. Searches will not function without Solr, and at some point, the entire codebase (rendering a taxa page) may require Solr as well.
To re-build your indexes for Solr searching (up to 100 TaxonConcepts), run the command:
rake solr:build
…Note that this command first deletes all existing entries, then adds entires (max: 100) for each TaxonConcept in the development. If you want to build indexes based on the data in your test (or integration, or…) environments, specify the RAILS_ENV:
rake solr:build RAILS_ENV=test
The following gems are required on your local machine for development but NOT deployment (deployment gems should always be frozen into the project):
-
ZenTest
sudo gem install ruby-debug
-
piston
If you are on a Mac, you will also need ruby-growl. … well, if you want to run autotest, you will.
Plugins required to run the site in production should be frozen into the vendor folder as is the version of Rails the app depends on.
To install required gems:
sudo gem install nokogiri rake gems # This could give you some errors... rake gems:build # if there are gems that aren't installed or you got that big error... rake gems:install # if there were gems that are STILL missing... # please manually install gems that are still missing or failed to install.
If you want to add a new gem to the project, please:
-
Add the config.gem line to the config/environment.rb file.
-
Install the gem using “rake gems:install”.
-
Unpack the gem into the vendor directory with “rake gems:unpack”.
-
Be sure to “svn add” the new directories that were created.
-
Setup the config/database.yml
-
Copy “config/database.sample.yml” to “config/database.yml”
-
Create/update the appropriate entries for your development and test environments
-
For development purposes, the
demo,integration, andproductionenvironments may be removed -
For production purposes, set the “master_database” to the master database for the core rails database and the “master_data_database” to the master database for the data database (see more info below in the “READ/WRITE SPLITTING” topic)
-
-
Installing and running memcached
This project doesn’t run without memcached. The reason for this is that no other caching mechanism seems to properly persist Ruby objects. If you really need to run some other caching system, modify your config/environments/development.rb to use what you prefer, but beware of “Stack level too deep” errors, which will occur if object serialization failed. You may need to find all of the “$CACHE” calls in the code and use YAML to serialize and restore the objects being stored.
To install memcached on OSX with MacPorts already on the system is quite easy:
a. Run "sudo port install memcached". This will install the packages required. b. At the end of the output from this, it should recommend you run another command, probably "sudo launchctl load \ -w /Library/LaunchDaemons/org.macports.memcached.plist", but you should just copy/paste and run whatever it suggests, as the path may be different.
That’s it! Memcached is now running and will work for development. Be mindful of this–it may require periodic restarting or flushing (from a Rails console, you can just type “$CACHE.clear”).
If you don’t have MacPorts (or don’t trust it), you will need to find another way to install and launch memcached, and that is beyond the scope of this document (but plenty of tutorials exist online).
Note that you do NOT need to install the “memcache” Ruby gem.
-
Run the following rake commands:
rake eol:db:create:all # Note the "EOL". This keeps bad things from happening. Please use it. rake eol:db:create:all RAILS_ENV=test rake db:migrate rake db:migrate RAILS_ENV=test rake truncate rake scenarios:load NAME=bootstrap #Be sure you installed memcached according to the step 2 # Your "URL" (or an alternative) will be provided to you privately if needed. Most developers who will not run code in production can skip this one. rake eol:site_specific repo=URL
Run the following two commands:
rake solr:start script/server
Go to localhost:3000 and see stuff.
We’re using RSpec for our testing (see the spec/ directory). Run ‘rake spec’ (or ‘rake specdoc’ for specdoc output)
Note that Solr must be running in the test RAILS_ENV before tests will pass:
rake solr:stop RAILS_ENV=test rake solr:start RAILS_ENV=test
Also note that the solr:stop command works even if the server was started in the development environment.
Possible error: “You haven’t loaded the foundation scenario, and tried to build a TaxonConcept with no vetted id.”¶ ↑
If you get this error, you may need to point your code to the correct version of mysqldump.
Create config/environments/local.rb (this will be ignored by revision control), and put this in it:
$MYSQLDUMP_COMPLETE_PATH = '/usr/bin/mysqldump'
(But make sure that the path you use here is really a working version of mysqldump.)
admin user username: admin password: admin
content partner username: test_cp password: test password
user username: test_user2 password: password
curator of Animalia (page #11 ATM) username: test_curator password: password
You can log in as any of the users or content partners with the password “test password”. There are no other consistent users, so either create your own, log in with OpenID, or use the console to find a username.
There are several consistent content partners, and the password is also “test password”. There will be IUCN, catalogueoflife, and test_cp.
Once logged in, every taxon concept should have various images that will only show up based on Vetted/Visibility values: each combination is represented.
Finally, pages/6 (TaxonConcept ID 6) has one additional image which is in “preview” mode for the test_cp Content Partner, and should be visibile only to her.
The EOL code base uses metric_fu-1.0.2 to evaluate the overall code quality.
In order to run the various metrics gems, an environment variable METRICS is required. This variable is observed if and only if the Rails environment RAILS_ENV=test.
It is the team’s opinion that metric-fu, and it’s various dependencies, are system level tools (like ruby-debug) and should not be checked into the code base. A user or a CI server will require the installation of the jscruggs-metric_fu gem which properly install various dependencies such as flay, flog, reek, and roodi.
The following Rake tasks are available when METRICS=true and RAILS_ENV=test:
rake eol:quality:all # Runs metrics on EOL rake eol:quality:flay # Analyze for code duplication rake eol:quality:flog # Analyze for code complexity rake eol:quality:roodi # Check for design issues in: app/*/.rb, lib/*/.rb
An example of how to run them:
RAILS_ENV=test METRICS=true rake eol:quality:all
The site is built to allow for master/slave database read/write splitting for the core rails database and the core “data” database. There are two plugins involved in the use of multiple databases and read/write splitting:
*use_db*:: used to direct some models to a different database
(http://rails.elctech.com/blog/using-and-testing-rails-with-multiple-databases)
*masochism*:: used to split read/writes when using ActiveRecord
(http://www.planetrubyonrails.org/tags/view/masochism)
New abstract class models are created which make connections to the other databases required, and then any models which need to connect to the other databases are subclassed from the new abstract class. In our case, we have two abstract classes representing connections to the data database and the logging database:
- SpeciesSchemaModel - LoggingModel
These extra two databases are referenced in the database.yml in the following way:
- environment_data (e.g. development_data) - environment_logging (e.g. development_logging)
Read/write splitting is accomplished with the masochism plugin by adding two new database connections to the config/database.yml file:
- master_database (the master database connection for the core rails database) - master_data_database (the master database connection for the "data" database)
In addition there are new abstract classes representing a connection to each master database that can be used to run direct SQL queries against the masters:
- MasterDatabase (for the core rails database) - SpeciesSchemaWriter (for the species data database)
The logging database does not require read/write splitting since there is only a single server for this purpose.
To enable read/write splitting via ActiveRecord, include the following in the approriate environment.rb file (e.g. config/environments/production.rb):
config.after_initialize do ActiveReload::ConnectionProxy.setup! ActiveReload::ConnectionProxy.setup_for SpeciesSchemaWriter, SpeciesSchemaModel end
Note that you must also enable class caching for this to work (this is the default in production, but not in development, which is important to note if you wish to test this functionality in development mode):
config.cache_classes = true
Manually crafted SQL queries with SELECT statement will be redirected to slave while all other queries like in the following example will be redirected to master:
SpeciesSchemaModel.connection.execute("DELETE FROM data_objects WHERE id in (#{data_objects})")
You don’t have to worry about master/slave databases in development mode unless you want to test your code against splitting queries. When in development, you could make the master_database and master_data_database must point to the same place as development and development_data respectively. Things should work even if these entries are left out (since the master databases are only connected in a configuration entry in the production environment) but it doesn’t hurt if they are there.
Spots in the code requiring some attention for refactoring, cleanup or further work are marked with a “TODO” comment and sometimes with a level of priority. You can quickly locate all these comments with your IDE, an app like TextMate, or with a rake command:
rake notes:todo
Here are our rake stats as of May 7th:
+----------------------+-------+-------+---------+---------+-----+-------+ | Name | Lines | LOC | Classes | Methods | M/C | LOC/M | +----------------------+-------+-------+---------+---------+-----+-------+ | Controllers | 3848 | 2665 | 34 | 262 | 7 | 8 | | Helpers | 713 | 503 | 2 | 49 | 24 | 8 | | Models | 7120 | 3981 | 104 | 473 | 4 | 6 | | Libraries | 980 | 676 | 9 | 69 | 7 | 7 | | Model specs | 1644 | 637 | 1 | 5 | 5 | 125 | | View specs | 0 | 0 | 0 | 0 | 0 | 0 | | Controller specs | 0 | 0 | 0 | 0 | 0 | 0 | | Helper specs | 10 | 7 | 0 | 0 | 0 | 0 | | Library specs | 0 | 0 | 0 | 0 | 0 | 0 | +----------------------+-------+-------+---------+---------+-----+-------+ | Total | 14315 | 8469 | 150 | 858 | 5 | 7 | +----------------------+-------+-------+---------+---------+-----+-------+ Code LOC: 7825 Test LOC: 644 Code to Test Ratio: 1:0.1
We really need to start improving our tests. To that end, I would like to start using:
rails stats reek flay flog roodi reek saikuro rcov churn
There are lots of configuration settings, and they load in the following order:
1) config/environment.rb 2) config/environments/[RAILS_ENV].rb 3) config/environments/[RAILS_ENV]_eol_org.rb 4) config/environment_eol_org.rb
The site is using the acl_system plugin (brainspl.at/articles/2006/02/20/new-plugin-acl_system).
Basically:
-
Users can be assigned to zero or more roles.
-
Controllers are restricted by role when needed.
So you can protect controllers and actions in the following ways:
-
To indicate a user must be logged in to view particular actions or controllers, just put this at the top of the controller (restricted to certain actions if you wish):
before_filter :check_authentication
This does not perform any checks for roles, just does a simple logged in check and redirects to the login page if needed.
-
To indicate a user must have the rights to a particular controller, just put this at the top of the controller (restricted to certain actions if you wish):
access_control :DEFAULT => 'ROLE NAME GOES HERE'
This checks to be sure the user has access rights to that controller (based on the role you specified) before letting them use actions in the controller. For more advanced control over only certain methods, see the acl_system2 plugin README.
-
A convenience controller/helper method is provided to check if a user is in a particular role called “is_user_in_role?”
For example:
do_this_method_only_for_admins if is_user_in_role?('Administrator')
-
You can also use the built in acl plugin helpers to only show links or other snippets of code in your view based on roles membership.
For example:
<% restrict_to "(Administrator) & !blacklist" do %> admin stuff here <% end %>
The admin section uses the roles setup described above. To create a new admin function:
-
Create a new controller in the “administrator” folder under controllers, e.g. Administrator::NewContollerName
-
Derive that controller from the Admin controller, e.g. class Administrator::NewContollerName < AdminController
-
Create your views in the correct subfolder under “views/administrator”, e.g. “views/administrator/new_controller_name”
-
Decide if you need a new role to access this controller or if an existing role will suffice. Add the new role into the “roles” database if required (you can use the admin interface or a migration).
-
At the top of your controller, restrict access to the role by adding the following line to the top:
access_control :DEFAULT => 'ROLE NAME GOES HERE'
Note that by default, users must also have the generic “administrator” role to access any of the controllers derived from the main admin controller.
-
Add your new controller to the admin navigation menu by editing “views/admin/_navigation.html.erb” See the existing links for restricting the display of menu items to the specific role you have identified in step 5.
-
Make sure you’ve got at least one admin user with access to the correct role, log in and test.
The logging model is intended to be thought of as a data mining system. A separate database is used to store all log data, which must be defined in your config/database.yml file. (See sample file for naming.) Models and operations tend to fall into two categories: dimensions and facts. In short, dimensions represent collected (primary) data. Facts are derived (secondary) caches of information which is more meaningful to the user. Fact table design is highly dependent on the user interface design, because we only need to generate facts if the information will actually be shown. For performance reasons regarding the expected database size, fact tables are also intended to be highly denormalized, non-authoritive sources of information.
Location-based facts require the primary data to go through a geocoding process which requires an external web service. This process is thus performed asynchronously from the main site. Results of IP location lookups are cached and reused whenever possible. While IP location lookups are non-authoritative “best guesses”, they nevertheless provide meaningful information.
In production mode it is CRITICALLY important to understand the automated logging tasks before invoking them to avoid deletion of precious data. To develop logging features, run the following tasks in the given order to populate your logging database with mock data…
rake logging:clear # Deletes all logging-related records. (WARNING: NEVER run in production.) rake logging:dimension:mock THOUSANDS=2 # Creates 2,000 psedo-random mock log entries (a.k.a. primary data). rake logging:geocode:all # Performs geocoding on the primary data, using caches where possible. rake logging:fact:all # Derives secondary data from primary data.
…at this point you should see data in the graph pages of the web application. Alternatively, run the following which does all of the above in one step.…
script/runner script/logging_mock
For cron jobs, you’ll likely want to log all facts for a particular date range:
rake logging:fact:today rake logging:fact:yesterday rake logging:fact:range FROM='01/15/2007' TO='12/19/2008'
Any links to external sites that need to be tracked should use the following two helpers:
external_link_to(text, url) external_link_to(image_tag(image), url)
Both will generate a link (with either the supplied text or the supplied image url) to the supplied URL. The link will be logged in the database, and if the $USE_EXTERNAL_LINK_POPUPS parameter is set to TRUE in the environment.rb file, a javascript pop-up warning window is shown prior to following the link. The following additional parameters can be passed after the URL for both methods:
+:new_window => true or false+:: determines if link appears in new browser window (defaults to true) +:show_only_if_link => true or false+:: determines if image or text is shown if no URL was supplied (defaults to false) +:show_link_icon => true or false+:: determines if the external icon image is shown after the link (defaults to true for text links and false for image links)
For images, the following parameters can also be passed:
+:alt => 'value'+:: alt tag is set with the value passed +:title => 'value'+:: title tag is set with the value passed
Currently no reports are provided for external link tracking, all links are stored in the “external_link_logs” in the logging database for later reporting.
Fragment caching is enabled in the specific environment file (e.g. config/production.rb) and the storage mechanism (i.e. memcached) must be set as well.
For memcached:
config.cache_store = :mem_cache_store, '10.0.0.1:11211', '10.0.0.2:11211'
To enable caching:
config.action_controller.perform_caching = true
All “static” pages coming out of the CMS are fragment cached and the home page cache is cleared each hour (or as set in the $CACHE_CLEAR_IN_HOURS value set in the config/environment.rb file), using language as key to enable multiple fragments. The header and footer navigation of each page is also fragment cached on cleared at the same time interval. When changes are made in the admin interface, these caches are automatically cleared.
Names searches are cached by query type, language and vetted/non-vetted status.
Species pages are cached using the following attributes as keys (since each will cause a different species page to be created). Note that when logged in as an administrator or content partner, the pages are not cached and are generated dynamically each time.
Variables for naming species page fragment caches:
- taxon_id - language - expertise level - vetted or all information - default taxonomic browser - curator for page
Species page caches can be cleared by taxon ID by a CMS Site Administrator by logging into the admin console, and going to “General Site Admin”. Clearing a species page cache automatically clears all of its ancestors as well.
The following URLs can be used to trigger page expiration either manually in the browser or via a web service call. They only work if called from “allowed” IPs (i.e. as specified in configuration) as defined in the application level method “allowed_request” (which returns TRUE or FALSE).
localhost:3000/expire_all # expire all non-species pages localhost:3000/expire_taxon/TAXON_ID # expire specified taxon ID (and it's ancestors) localhost:3000/expire_taxa/?taxa_ids=ID1,ID2,ID3 # will expire a list of taxon IDs (and their unique ancestors) specified in the querystring (or post) parameter "taxa_ids" (separate by commas) localhost:3000/clear_caches # expire all fragment caches (if supported by data store mechanism)
From within the Rails applications, use the following application level methods:
expire_all # expire all non-species pages expire_taxon(taxon_ID) # expire specified taxon id and ancestors (unless :expire_ancestors=>false is set) expire_taxa(taxon_ID_array)# expire specified array of taxon ID and unique ancestors (unless :expire_ancestors=>false is set) clear_all_caches # expire all fragment caches (everything!)
For testing purposes, to install memcached on a Mac see readystate4.com/2008/08/19/installing-memcached-on-os-x-1054-leopard/ If you have a local memcached server installed (Mac/Linux), start it with: “memcached -d -m 24 -p 11211” and stop it with “killall memcached”
This is now using the asset_packager plugin, see details at synthesis.sbecker.net/pages/asset_packager
If you add a javascript include files and you want them included in the page, you must edit the “config/asset_packager.yml” file and place them in the order you wish them to be loaded. When running in development mode, the JS and CSS are included separately each time. When running in production mode, assets are included from packaged entities. A rake task is used to combine and minify CSS and JS referenced. Note that the order the JS files are listed in the config file is the order they are merged together and this order can matter.
You must run this rake task each time the JS/CSS is updated to ensure the latest version is present when running in production mode. The minification process is very sensitive to missing semicolons, extra commas and what not that are dealt with by modern browsers (not IE though…). You have been warned - minification can and will break your JS if you are not careful (watch those semicolons)!
To update/create the combined versions:
rake asset:packager:build_all
In production, this rake command is run as part of the capistrano deploy script.
For testing purposes, you can force the minified/combined version to be referenced in your pages for a particular environment by adding the following line to your “config/environments/development.rb” or “config/environment.rb” file:
Synthesis::AssetPackage.merge_environments = ["development", "production"]
To get attribution for a given taxon concept ID:
-
Get TaxonConcept e.g.
t=TaxonConcept.find(101)
-
Look at hierarchy_entry for that taxon (could be many) e.g.
he_all=t.hierarchy_entries OR he=t.entry (for the default)
-
Look at the associated hierarchy (could be one of many if you get them all) e.g.
h=he_all[0].hierarchy # OR h=he.hierarcy h.label h.agent.full_name h.agent.hompage h.agent.logo_cache_url
-
Look at the associated agents for the hierarchy_entry e.g.
agents=he[0].agents # OR agents=he.agents agents.each {|agent| puts agent.full_name + " " + agent.homepage + " " + agent.logo_cache_url}
To create Google SiteMap files in the correct format, run the following rake task for your requested environment:
rake sitemap:create RAILS_ENV=production[,BASEURL="http://www.eol.org/pages/",BASEURL_SITEMAP="http://www.eol.org/sitemaps/",MAXPERFILE="50000",OUTPUTPREFIX="eol_sitemap",PRIORITY="1",CHANGEFREQ="monthly",LASTMOD="2009-03-01"]
All of the parameters in brackets are optional and have the default shown (except for ‘lastmod’ which defaults to today).
The URLS placed into the site map file are based on ‘BASEURL/XXX’ where XXX is a valid published, trusted taxon concept ID pulled from the specified environment.
We’re not keeping the docs in the svn repo, so you’ll have to generate them on your own. We recommend installing the allison gem, after which you can run this command:
template=`allison --path` rake doc:reapp
…Just remove the template bit, if you don’t care to have the pretty formatting.