BETYdb 4.1 New Database Uniqueness Constraints

New uniqueness constraints for database tables and validations for corresponding models; new RSpec tests; several bug fixes.

This release implements several value and uniqueness constraints on database tables aimed at ensuring that the BETYdb database will not contain multiple rows describing the same citation, site, species, treatment, or other entity. In conjunction with the database changes, new model validations have been added to several of the models so that potentially invalid database insertions or updates are caught at the Rails level before the insertion or update is attempted on the database.

Second, several bugs existed that prevented access to many pages of the site--especially pages for editing database entities. Many of these were introduced by changes to the routing file that eliminated most wildcard routes. These bugs have now been fixed and several new tests have been added to ensure these bugs don't re-appear.

Changes Pertinent to PEcAn Users

Administrators need to do a database migration.

See "Database Changes" below.

Summary of Changes

New Features

New Validations to Prevent Duplicate Rows and Missing Data

Bug Fixes

• Many pages that were previously inaccessible are once again reachable. Prior to this update, attempting to visit these pages yielded the following error:

The page you were looking for doesn't exist.

You may have mistyped the address or the page may have moved.

Affected pages included (for example)

• the Covariate editing page
• the Model editing page
• the Modeltype editing page
• the Species editing page
• the Traits editing page

Steps Needed for Upgrade

Database Changes

Administrators need to do database migrations!

One migration has been added that new value and uniqueness constraints and also adds some convenience functions for implementing these constraints.

The database version for this release is 20150202220519.

Status of RSpec Tests

• All tests continue to pass when run in the default environment and can be run using the command

  bundle exec rspec
  

Every effort has been made to make this command idempotent: You should be able to run the tests multiple times without reloading the test fixtures between runs.

Complete details for running the rspec tests are on the updated Wiki page at https://github.com/PecanProject/bety/wiki/Automated-Tests

BETYdb 4.0.2 Bulk Upload Manager Access

minor release

This is a minor release that fixes a bug in the Bulk Upload feature and gives managers access to the validation portion of the Bulk Upload wizard.

Summary of Changes

Feature Additions

• Users classified as managers (page access level 2) are now able to access the Bulk Upload wizard. They can upload and validate CSV data files but are not permitted to insert the data they contain. This is the first expansion of access to the Bulk upload feature beyond core developers.

Bug Fixes

• A minor bug in the Bulk Upload feature was fixed. Missing citation_year values used to trigger a fatal error, but they no longer do.

Status of RSpec Tests

• All tests continue to pass when run in the default environment and can be run using the command

  bundle exec rspec
  

Complete details for running the rspec tests are on the updated Wiki page at https://github.com/PecanProject/bety/wiki/Automated-Tests

BETYdb 4.0.1

This is a minor release that extends the API search syntax to the traits_and_yeidls_view and fixes some bugs in the Trait and Yield creation and update pages. It was deployed to www.betydb.org on April 30, 2015.

Summary of Changes

Feature Additions

• It is now possible to get query results from the traits_and_yields_view SQL view using multiple columnname=value clauses in the query string appended to the Advanced Search page URLs https://www.betydb.org/search.xml (for results in XML format) and https://www.betydb.org/search.json (for results in JSON format). For example, the URL https://www.betydb.org/search.xml?lat=42.9&lon=-72.3&result_type=traits will find all checked trait data for sites located at 42.9 deg N latitude and 72.3 deg W longitude to which the user has access rights and return it in XML format. Note that matches must be exact! These strict search clauses may be combined with the less-strict searching provided by a clause of the form search=term1+term2+term3... which requires only that each of the terms provided be textually included in the column value of one of the columns sitename, city, scientificname, commonname, author, trait, trait_description, or citation_year.

• A new rake task has been added as a convenience for displaying the database configuration file database.yml. Run

  rake dbconf
  

Bug Fixes

• A long-standing bug in the processing of the New Yield and Editing Yield forms rendered the Julian date fields useless. This has been fixed.
• In addition, the upgrade to Rails 3.2 produced some new problems with the way the standard date fields in the New Yield and Editing Yield forms were processed. Furthermore, when the New Trait and New Yield forms were visited with a row id parameter passed in the URL, the forms would be pre-populated as before, but pressing the Create button would do an update rather than creating a new row as it was supposed to. These problems have been fixed as well.
• Finally, API queries involving an include[]=<table> clause now once again show information for the included table in the query results. Even with this, queries work slightly differently than they did prior to the update to Rails 3.2. Details will be given in the BETYdb Data Access document at https://www.authorea.com/users/5574/articles/7062.

Status of RSpec Tests

• All tests continue to pass when run in the default environment and can be run using the command

  bundle exec rspec
  

Complete details for running the rspec tests are on the updated Wiki page at https://github.com/PecanProject/bety/wiki/Automated-Tests

BETYdb 4.0: enhanced API for rOpenSci, Rails 3.2, Ruby 2.1.5 upgrade

Enhanced API for rOpensci traits package; upgrade to rails 3.2, Ruby 2.1.5.

This release upgrades the Rails version from Rails 3.0.18 to Rails 3.2.21 and changes the RVM-specified version of Ruby from version 1.9.3-p429 to version 2.1.5. In addition to these changes which in and of themselves make for improved security, there are numerous minor changes to the Rails code including security fixes, removal of obsolete code, fixes and improvements to the query API, and phase-out of vendor plugins in favor of gems,

Changes Pertinent to PEcAn Users

Administrators need to do a database migration.

See "Database Changes" below.

Summary of Changes

New Features

Enhanced API for rOpensci traits package

See section 2 of the BETYdb Data Access guide at https://www.authorea.com/users/5574/articles/7062.

API Key generation

Users may now generate their own API keys by going to the Users page and clicking the "key" button. And there is now a new rake task that administrators can use to generate API keys for all current users. See section 2.3 of the BETYdb Data Access guide at https://www.authorea.com/users/5574/articles/7062.

Directory-Specific Session Keys

When multiple copies of BETYdb are deployed on the same host, they now use differently-named session cookies (assuming the basenames of the Rails root directories are different). This mainly affects developers who may visit different copies of BETYdb on the same host simultaneously; up to now, such users will likely have seen baffling interactions between the two copies.

Updated version of selenium-webdriver

The Gem version specification for selenium-webdriver has been changed to a version compatible with the latest versions of Firefox. This mainly affects those running the RSpec tests in debug mode.

Addition of primary keys to join tables

This facilitate the synchronizing of data having to do with many-to-many associations.

Security Updates

Miscellaneous security updates have been implemented, including

• Upgrading of Rails from version 3.0.18 to 3.2.21
• Upgrading the default version of Ruby to version 2.1.5
• Storage of session data in the database instead of in cookies
• Elimination of most non-specific routes

In keeping with recommended practice in Rails 3.2, packages that were formerly kept in vendor plugins are now in Ruby gems.

The change to database storage of session data has the additional benefit of overcoming some restrictions on the size of bulk uploads.

Bug Fixes

• The API for querying and downloading data offers more features than available previously, restoring some features that were lost in the upgrade from Rails 2 to Rails 3. See section 2 of the Data Access Guide: https://www.authorea.com/users/5574/articles/7062
• Cloning of PFTs now works again.
• A bug that prevented editing a PFT if it was associated with some Prior whose paramb column was NULL has now been fixed.

Steps Needed for Upgrade

Database Changes

Administrators need to do database migrations!

One migration has been added that adds id columns to the join tables.

The database version for this release is 20150202215147.

Ruby Upgrade

The officially-supported version of Ruby for use with BETYdb is now Ruby version 2.1.5. It is recommended that you upgrade to some Ruby 2 version since Ruby 1.9.3 is no longer being maintained. (See https://www.ruby-lang.org/en/news/2015/02/23/support-for-ruby-1-9-3-has-ended/ for details.)

Nevertheless, users who can not or do not wish to upgrade will likely be able to continue to use Ruby 1.9.3 without difficulty. It may also be possible to use the latest version of Ruby, Ruby 2.2.1.

To continue using Ruby 1.9.3

If you use RVM or some other Ruby version management system that reads the .ruby-version file, revert the string in that file so that it once again reads ruby-1.9.3-p429. You may wish to add .ruby-version to your .gitignore file so that git doesn't try to change it or flag it as having been modified.

To upgrade to Ruby 2.1.5

Install Ruby 2.1.5 if you haven't done so already. If you use RVM, you may do this by running

rvm install 2.1.5

If you run BETYdb with Apache httpd, change or add the directive

PassengerDefaultRuby /path/to/ruby2/executable

or the

PassengerRuby /path/to/ruby2/executable

directive according to your needs.

Otherwise, if you only use WEBrick, just ensure that the version of Ruby you want to use is first in your path. If you are using RVM, this is automatic (though you may have to leave and then return to the BETYdb Rails root directory if you haven't done so since updating your .ruby-version file; run rvm current to verify that you are using the correct Ruby version and gemset).

Gem Installation, Including Rails Upgrade

Administrators need to run the bundler to install several new Ruby Gems and updated versions of existing ones.

First, change to the root directory of your BETYdb Rails installation. Then, if you are using RVM, run rvm current to ensure you are using the correct Ruby version and gemset. It should say ruby-2.1.5@betydb_rails3 if you upgraded Ruby and ruby-1.9.3-p429@betydb_rails3 if you did not. If you are not using a Ruby version management system, simply run ruby -v to check your version of Ruby.

Once you ensure you are using the correct Ruby version and gemset, run

bundle

If you have trouble installing capybara-webkit and don't need to run the Javascript-based RSpec tests, you may run

bundle --without javascript_testing

instead. If you won't be running RSpec tests at all and want to minimize your Gem installation, run

bundle --without javascript_testing test debug

See https://github.com/PecanProject/bety/wiki/Installing-the-BetyDB-Rails-Application#minimizing-gem-installation for details.

Status of RSpec Tests

• All tests continue to pass when run in the default environment and can be run using the command

  bundle exec rspec
  

Complete details for running the rspec tests are on the updated Wiki page at https://github.com/PecanProject/bety/wiki/Automated-Tests

BETYdb 3.3.2 Foreign Key Constraints

This release primarily implements several foreign-key constraints on the database. It consists of a new migration file (together with an updated version of production_structure.sql), some related changes to the controller code, and an unrelated minor change consisting of an update to a link in the footer and removal of no-longer-needed routing code.

Changes Pertinent to PEcAn Users

Administrators need to do a database migration.

See "Database Changes" below.

Summary of Changes

Database Changes

Administrators need to do database migrations!

The database version for this release is 20141211220550.

Gem Installation

No new gems need be installed.

Additional Notes

• Other than the addition of a migration, the primary change to the Rails code is some error-handling code to gracefully handle foreign-key constraint violations and provide users with helpful error messages. Currently, these message are in most cases very rudimentary but nevertheless may be comprehensible to non-programmers.
• Some unneeded but misleading Rails code was removed. (This is code which looks like it does something but does nothing, so its deletion should not be noticed by users.)
• The target of the Documentation link in the footer is now the BETYdb wiki instead of the PDF version of the Database Description and User's Guide.
• The migration will fail if your copy of the bety database violates any of the new foreign-key constraints. The master database on ebi-forecast, however, has been updated to comply with this constraint, so it is recommended to update your copy of the database to mirror the changes made there.

Status of RSpec Tests

• All tests continue to pass when run in the default environment and can be run using the command

  bundle exec rspec
  

One bulk upload-related test has been marked 'pending', however, until it is reconciled with the new constraints.

Complete details for running the rspec tests are on the updated Wiki page at https://github.com/PecanProject/bety/wiki/Automated-Tests

BETYdb 3.3.1

This release implements changes to the database schema discussed in Github issue #145 and makes a few related changes. It consists of a single migration file together with an updated version of production_structure.sql.

Changes Pertinent to PEcAn Users

Administrators need to do a database migration.

[The BETYdb web interface, however, will, for now, continue to function as before even without doing the pending migration.]

See "Database Changes" below.

Summary of Changes

Database Changes

Administrators need to do database migrations!

The database version for this release is 20141208165401.

• A new table cultivars_pfts has been added so that PFTs consisting of cultivars rather than of species may be formed.
• Uniqueness, foreign-key, and non-null constraints have been added to this new table and to the existing pfts_species table.
• pfts_species and cultivars_pfts have been constrained in such a way that a PFT may be associated with a set of species or with a set of cultivars but not with a set containing both species and cultivars.
• A uniqueness constraint has bee added to pfts to ensure that names are unique within the set of PFTs of a given model type. (PFTs associated with different model types, however, may have the same name.)

Additional Notes

• Other than the addition of addition of a migration, the Rails code is essentially unchanged. In particular, no facility has yet been added in the Rails code to work with PFTs consisting of cultivars.

• The foreign key constraints have been designed in such a way that a PFT may be removed without having to first empty it of species or cultivars. This is as before, except that now, all entries in the join tables pfts_species and cultivars_pfts that refer to the removed PFT are automatically removed as well.

• Species and cultivars, however, may not be removed if they belong to any PFT. They must first be removed from all PFTs containing them before they themselves can be removed.

• The migration will fail if your copy of the bety database violates any of the new uniqueness constraints. In particular, pfts are now required to be unique by (name, modeltype_id). The master database on ebi-forecast has recently been updated to comply with this constraint, so it is recommended to update your copy of the pfts table to mirror the changes made there. A query that can be used to find the offending rows is

  select array_agg(id) as id_list, name, modeltype_id, count(*) from pfts group by name, modeltype_id having count(*) > 1;
  

  Most likely, this will produce only a row or two, each with an array of pft id numbers containing only two or three entries. Thus only two or three rows of the pfts table need be updated which can easily be done manually. (I found it sufficed to update the name in row 83 to 'plants2' and the name in row 95 to 'plants3'. Visit the PFT pages at https://www.betydb.org to see what the name for any particular id number has been changed to in the master database. For example, to see the current name used for the PFT with id=95, visit https://www.betydb.org/pfts/95.)

Status of RSpec Tests

• All tests continue to pass when run in the default environment and can be run using the command

  bundle exec rspec
  

  Complete details for running the rspec tests are on the updated Wiki page at https://github.com/PecanProject/bety/wiki/Automated-Tests

BETYdb 3.3

This incorporates a major new feature--the Bulk Upload Wizard--and also includes bug fixes from the last several sprints. It was deployed to www.betydb.org on December 5, 2014.

Major Changes

Administrators either need to run bundle install to install several new
Ruby Gems or configure bundler to ignore the Gems in the 'debug' group. (The
'debug' group is needed only for running RSpec tests using the Selenium
JavaScript driver.) Also, to avoid the installation of capybara-webkit, the
group to ignore is now called 'javascript_testing', so if you never installed
it and don't want to install it, configure Bundler to ignore the
'javascript_testing' group.

(Note: Unless you have difficulty installing the Gems used for testing, there is
really no compelling reason not to install them.)

To ignore the 'debug' group, either run

bundle install --without debug

or, if you are sure your other Gems are up-to-date, you can just run

bundle config --local without debug

To ignore the 'javascript_testing' group (thus avoiding installation of
capybara-webkit), substitute 'javascript_testing' for 'debug' in either of these
two commands. To ignore both, run

bundle install --without debug javascript_testing

or

bundle config --local without debug:javascript_testing

Or, to ignore ALL gems except those needed in the production environment, ignore all the other groups:

bundle install --without debug javascript_testing test development

or

bundle config --local without debug:javascript_testing:test:development

Note that bundle install --without takes a space-separated list, but bundle config without takes a colon-separated one.

Summary of Changes

New Features

• A wizard for bulk upload of data has been added (currently only available to administrators).
• An easy method has been provided to debug RSpec tests using the Selenium JavaScript driver together with breakpoints (see the updated Wiki page for Automated Tests).
• Distinct pages now have distinct titles, making the site more accessible for those using screen readers and making navigation of the site using the browser history possible.

Minor Improvements

• Redmine issue #2599 ("Different pages of BetyDB should have different
  titles."): Formerly, every page on the site had the same title ("Energy
  Biosciences Institute"). This meant that if one had been clicking around the
  site for awhile and then looked at the browser history, all the recent entries
  in the history list would appear to be exactly the same. Now the default
  title is "BETYdb: " followed by the path/filename portion of the URL of the
  page.
• New validations have been added to the Create and Update pages for some
  tables. For example, it is now checked that yield values are non-negative.
• The rake notes:todo task has been redefined so that it finds not only
  source-code comments beginning with "TODO", but also those beginning with
  "TO-DO" or "to-do".

Bug Fixes

These bugs have been fixed:

• Redmine issue #2406 ("Searching is broken on Ensembles, Inputs, Runs, and
  Workflows pages.") Searches on these pages have now been fixed.
• Redmine issue #2486 ("Baffling trait-creation error.")
• Redmine issue #2604 ("Application error when unsuccessfully updating a yield."

Status of RSpec Tests

• All tests now pass when run in the default environment and can be run using the command

  bundle exec rspec
  

  Complete details for running the rspec tests are on the updated Wiki page at https://github.com/PecanProject/bety/wiki/Automated-Tests

BETYdb 3.2.4

This incorporates some features and bug fixes from the September 29, 2014 sprint. It was deployed to www.betydb.org on October 9, 2014.

Changes Pertinent to PEcAn Users

Administrators need to do database migrations!
See "Database Changes" below.

A constraint was added to the geometry column of the sites table to ensure the geometry is valid.

Release Highlights

• The update.psql.sh script has been updated to incorporate changes to the corresponding pecan script—namely, it now adds a set of default users having all possible permission levels and adds the guestuser account.

Summary of Changes

Database Changes

Administrators need to do database migrations!

The database version for this release is 20141009160121.

• A constraint was added to the geometry column of the sites table to ensure the geometry is valid.

Minor Changes and Improvements

• Contact information on the home page has been updated.
• Instructions for citing BetyDB have changed.

Bug Fixes

• Redundant error messages are no longer displayed on the home page when login fails.

Status of RSpec Tests

• All tests now pass when run in the default environment and can be run using the command

  rspec
  

  Complete details for running the rspec tests are on the updated Wiki page at https://github.com/PecanProject/bety/wiki/Testing

BETYdb 3.2.3

This adds a migration, restoring some defaults for column values that were lost in the migration from MySQL to PostgreSQL. It was deployed to www.betydb.org on September 19, 2014.

Summary of Changes

Database Changes

Administrators may need to do database migrations if they wish to sync their database with the one on ebi-forecast.

The database version for this release is 20140915153555.

• This is a very minor migration: It sets three default values:

1. a default of 0 for the checked column of the traits table
2. a default of 0 for the checked column of the yields table
3. a default of '' for the name column of the users table

This restores defaults that were set in the MySQL version of the database before it was migrated to PostgreSQL.

BETYdb 3.2.2

This adds a migration, adding a new table to the database. It was deployed to www.betydb.org on September 9, 2014.

Changes Pertinent to PEcAn Users

Administrators need to do database migrations!
See "Database Changes" below.

Summary of Changes

Database Changes

Administrators need to do database migrations!

The database version for this release is 20140909212759.

• A new table, trait_covariate_associations has been added. This will be used to facilitate bulk uploads of trait data.

Status of RSpec Tests

• All tests now pass when run in the default environment.
  (Complete details for running the rspec tests are on the updated Wiki page at https://github.com/PecanProject/bety/wiki/Testing)