Welcome to the DRI Hydra Head
This repository contains the Digital Repository of Ireland's Hydra application.
There are a number of other depedencies (see below) but at this point you will need to have a functional Rails stack and at least Java version 7. You can check your Java version by running 'java -version' anything below 1.7x in the output and you should update from java.com/en/download/
For testing and development you will need to bootstrap the project
$ git clone email@example.com:drirepo/dri-app.git $ cd dri-app $ bundle $ cp config/database.yml.sample config/database.yml (and edit as appropriate) $ cp config/fedora.yml.sample config/fedora.yml (and edit as appropriate) $ cp config/jetty.yml.sample config/jetty.yml (and edit as appropriate) $ cp config/redis.yml.sample config/redis.yml (and edit as appropriate) $ cp config/solr.yml.sample config/solr.yml (and edit as appropriate) $ bundle exec rake db:migrate $ bundle exec rake db:seed
Dont forget to prepare the test database with
$ bundle exec rake db:test:prepare
You may also need to edit config/settings.yml and/or create config/settings.local.yml
To start the application for demo purposes
Note: Ensure that a mysql (or mysql replacement) is running. (E.G. MariaDB start alias: alias mariadbstart='cd “/opt/local” ;sudo /opt/local/lib/mariadb/bin/mysqld_safe –datadir=“/opt/local/var/db/mariadb”')
$ cd dri-app $ rake jetty:start $ rake jetty:config $ bundle exec rails server
The db:seed step above will create 4 initial accounts:
firstname.lastname@example.org (admin user)
email@example.com (organisation manager)
firstname.lastname@example.org (collection manager)
email@example.com (public user)
The default password for all 3 accounts is set at install time from the password setting in the settings.yml file. Typically this will be set to “CHANGEME” and it should be updated as soon as possible after installation. To change the password log into the application as each of the default users, navigate to their profile page by clicking on the username at the top of the screen, and clicking on the 'Edit' button.
To run the unit tests
$ cd dri-app $ rake jetty:start $ rake jetty:config $ rake rspec
To run the functional tests
$ cd dri-app $ rake jetty:start $ rake jetty:config $ rake cucumber or bundle exec cucumber
If you are using homebrew on OSX
$ brew install ansible coreutils libtool libksba libxml2 libxslt automake autoconf openssl curl curl-ca-bundle readline lzlib gettext pkg-config libyaml mariadb redis clamav sqlite ffmpeg gource phantomjs fits glib imagemagick ghostscript node
In order to properly ingest videos, please reinstall ffmpeg with all the the supported codecs and encoders available:
$ brew reinstall ffmpeg --with-fdk-aac --with-ffplay --with-freetype --with-frei0r --with-libass --with-libvo-aacenc --with-libvorbis --with-libvpx --with-opencore-amr --with-openjpeg --with-opus --with-rtmpdump --with-schroedinger --with-speex --with-theora --with-tools
The unit tests are written using rspec and the functional tests are written with cucumber.
This project is set up to use guard to automatically run tests as they are changed.
$ cd dri-app $ bundle exec guard
The poltergeist gem is required for testing, as a result phantomjs binary to run poltergeist.
When translations are added to the english locale files in config/locales you should run the rake task localefiles with the parameter file which should be set to the path to the ENGLISH yml file. For example:
$ rake locales:mergefile file=config/locales/dri/en.yml
This will merge any new keys from the English file into any other locale files (with the same naming convention) in that directory. The default value for each key will be the English text.
To merge all files use
$ rake locales:mergeall
This is where the application specific settings are located
Please view and edit this file as needed before deploying the application.
To override settings.yml with local settings please use settings.local.yml.
Configuation of database.yml, fedora.yml and solr.yml
The configuration files for the application
are not configured initially. There are some sample configurations in the config directory. These configuration files must be created before development, testing, deployment. The default sample files can be copied into the correct locations for testing and development. It is suggested that these files are not commited to the repository.
The application includes a queueing system based on Resque and Redis to allow asynchronous processing of background jobs. This introduces the following dependencies:
* Redis (server for the queueing system) * fits (for extracting metadata from a file) * ffmpeg (video file processing tool) * imagemagick (image processing) * Clamav (malware scanning)
For testing, the file processing tools should be installed locally with the hydra-head. See the Dependencies section of this document for more information on installing these and other dependencies.
By default fits should be installed in the directory /opt/local/fits/ (this can be changed by modifying the config.fits_path setting in the file config/initializers/sufia.rb).
By default ffmpeg should be installed in the directory /opt/local/bin (this can be changed by modifying the ffmpeg_path setting in the file config/settings.yml).
For testing purposes Redis can be installed locally and started with the command
$ sudo redis-server /opt/local/etc/redis.conf
The worker queues can be started with the following rake task
$ rake environment resque:work RAILS_ENV=development QUEUE="*" COUNT="2" VERBOSE=1
To view your queues and workers you can visit the path /resque on the web application where resque-web is mounted.
Note that the queuing code is stubbed for the Cucumber tests so you do not need a redis host running in order to run the tests.
To enable virus scanning ClamAV needs to be installed.
$ sudo port install clamav
You will need to create the configuration file freshclam.conf in /opt/local/etc. There is an example file in /opt/local/etc/example-freshclam.conf. Set the directory for the database files and the log
DatabaseDirectory /opt/local/share/clamav UpdateLogFile /opt/local/var/log/clamav/freshclam.log
Update the virus database
$ sudo freshclam
Install the clamav Ruby gem
$ gem install clamav -- --with-cflags=`clamav-config --cflags` --with-ldflags=`clamav-config --libs`
Add the gem to the GemFile
Run bundler. Virus scanning should now take place when an asset is uploaded.
Surrogate files are stored using the S3 interface. To create surrogates you will need access to an S3 server. A Ceph VM with S3 Rados Gateway is available in the drirepo repository. To start up Ceph run
$ vagrant up ceph
When the VM is running, log in with
$ vagrant ssh ceph
Then create a user with no max bucket limit as follows:
$ sudo radosgw-admin user create --uid=johndoe --display-name="John Doe" --firstname.lastname@example.org --max-buckets=0
This will return an access_key and secret_key which must be added to the hydra head's config/settings.yml or config/settings.local.yml file as follows:
S3: server: 10.0.1.201 access_key_id: 4WP751XIFXBH9G97DD91 secret_access_key: v6tX8bZSeN9IRJAcelHLnQFrsYV8ojxgWa7rn0uz
Note that the keys should be pasted in as they appear in the output of the radosgw-admin command, but all '' characters should be stripped.
An alternative, if the Ceph VM is not working for you, is to use the FakeS3 gem. This is included in the development gem group, so should have been installed by bundler.
Create a directory for the server to use to store files, and then start the server with:
$ bundle exec fakes3 -r /path/to/fakes3_root -p 8081 -H localhost
The settings should then be modified to the following:
S3: server: localhost:8081 use_ssl: false access_key_id: anything secret_access_key: anything
Migrating from sqlite3 to mysql
This paragraph will explain the steps requires to migrate the database configuration from sqlite3 to mysql. The first step is to install mysql using homebrew.
$ brew install mysql
Once the installation is completed start your mysql server and log to it.
$ mysql.server start $ mysql -u root -p
In the mysql console create the 3 environment databases required by rails.
mysql> create database dri_development; mysql> grant all privileges on dri_development.* to 'root'@'localhost' identified by 'password'; mysql> flush privileges; mysql> create database dri_test; mysql> grant all privileges on dri_test.* to 'root'@'localhost' identified by 'password'; mysql> flush privileges; mysql> create database dri_production; mysql> grant all privileges on dri_production.* to 'root'@'localhost' identified by 'password'; mysql> flush privileges; mysql> exit
The next step is to change the database configurations in the [yourProjectLocation]/config/database.yml as follows:
development: adapter: mysql database: dri_development username: root password: yourPWD host: localhost test: &test adapter: mysql database: dri_test username: root password: yourPWD host: localhost production: adapter: mysql database: dri_production username: root password: yourPWD host: localhost cucumber: <<: *test
The last step is to install the mysql and mysql2 gems and to setup your new mysql databases by migrating and seeding them.
$ gem install mysql $ gem install mysql2 $ bundle exec rake db:migrate $ bundle exec rake db:seed $ bundle exec rake db:test:prepare
The application uses Google Analytics to store usage data.
All page views are send to Google Analytics using the google-analytics-rails gem. For published objects the object ID, parent collection ID and Depositing Organisation name are also sent.
When an object is viewed, or an asset file is downloaded the server also sends the view and download Events to Google Analytics via the Gabba gem.
The Analytics reports (/analytics) use the Legato gem to query the Google Analytics API.
In order for this to work a number of configuration changes are required at the Google Analytics side.
Configure a Google Analytics account for the site
Create a Google Analytics account for your site and get the Tracking ID number. This should be saved in the environments files as GA.tracker
Create Custom Dimensions
The following custom dimensions are required to be configured in the following order
* Dimension1 'collection' * Dimension2 'organisation' * Dimension3 'object'
This is done via the Admin Menu - Custom Definitions - Custom Dimensions. All three should have hit scope.
Create a project in the Google API console
Go to the Google API Console console.developers.google.com/ and view the Dashboard. You may be prompted to create a project. If not look for the projects list at the top of the screen beside the Google APIs logo. Click on this to see your project list and use the + button to create a new project.
Create credentials for the Google API project
You will need to create a service account credential. Click on the top menu and navigate to IAM & Admin - Service Accounts. Here create a new Service account giving it a role of Project - Viewer.
Save the account and download the key file in p12 format. Save this to the config dir and update the Settings.yml file with the following information:
profile_id should be equal to the part after the p in the url for your analytics page. I.e. your analytics page will have an url of the form
The part after the p is the profile id, in this case 1234567890.
issuer && sub
These should both be the service account ID which you will find in the Service Account section of the Google API Console console.developers.google.com/
This should be the name of the p12 keyfile you downloaded from the Google API Console.
This should match the secret that you were shown on downloading the p12 keyfile.
Other settings should be as follows
disable: false token_credential_uri: 'https://accounts.google.com/o/oauth2/token' audience: 'https://accounts.google.com/o/oauth2/token' scope: 'https://www.googleapis.com/auth/analytics.readonly'