1. Clone the repository
DEPRECATED -- USING ANSIBLE
git clone https://github.com/upenn-libraries/sdbmss.git
2. Create file docker environment file .env
in the root folder of the cloned repository. Define the following environment variables:
This is the password for the public access point for the Jena server, both for making updates and for downloading RDF data file
ADMIN_PASSWORD=<jena_admin_password>
Login for private staging server deployment, if used
RAILS_USER=<rails_staging_username>
RAILS_PASS=<rails_staging_password>
MYSQL database setup. MYSQL_HOST is the name of the docker service defined in docker-compose.yml - everything else is up to you
MYSQL_HOST=db
MYSQL_DATABASE=<mysql_database_name>
MYSQL_ROOT_PASSWORD=<mysql_root_password>
MYSQL_USER=<mysql_username>
MYSQL_PASSWORD=<mysql_password>
RabbitMQ user and password; used for RDF live-update messaging system. Define here, then set the correct values when instantiating the RabitMQ service (done later)
RABBIT_USER=<rabbitmq_username>
RABBIT_PASSWORD=<rabbitmg_password>
Rails email setup. Depends on where the application is running and what mailing service is available
SDBMSS_APP_HOST=<email_app_host>
SDBMSS_SMTP_HOST=<email_smtp_host>
SDBMSS_EMAIL_FROM=<email_sender>
SDBMSS_NOTIFY_EMAIL=<notification_email_sender>
SDBMSS_NOTIFY_EMAIL_PASSWORD=<notification_email_password>
# email address
SDBMSS_EMAIL_EXCEPTIONS_TO=<send_exceptions_to_email>
Generate separate keys (for each) by running rails task secret.
SDBMSS_BLACKLIGHT_SECRET_KEY=<KEY>
SDBMSS_DEVISE_SECRET_KEY=<KEY>
SDBMSS_SECRET_KEY_BASE=<KEY>
SDBMSS_SECRET_TOKEN=<KEY>
URL for SOLR server, using relative location of docker service (using name 'solr' from docker-compose.yml)
NB: The SOLR port here and everywhere else in the code is 8983
. Previous versions
of the code had 8982
in some cases. This was wrong and created a conflict
between Jetty (which was set to 8983
and the docker-compose values, which
had 8982
).
SOLR_URL=http://solr:8983/solr/development
3. Build and Run (First Time)
Start everything up:
docker-compose up --build
4. RabbitMQ First Time Setup
First time we need to create user and grant permissions. Use the same values for USER/PASS as set in your .docker-environment file
docker-compose -f docker-compose-dev.yml rabbitmq /bin/bash
rabbitmqctl add_user <RABBIT_USER> <RABBIT_PASSWORD>
rabbitmqctl set_user_tags <RABBIT_USER> adminstrator
rabbitmqctl set_permissions -p / <RABBIT_USER> ".*" ".*" ".*"
Then restart dependent containers:
docker-compose -f docker-compose-dev.yml restart interface
docker-compose -f docker-compose-dev.yml restart rails
5. First Time Setup: Rails and SOLR
Setup database - perform setup:
docker exec $(docker ps -q -f name=sdbmss_rails) bundle exec rake db:setup
(Optional: Load data from .sql dump)
docker cp sdbm.sql.gz $(docker ps -q -f name=sdbmss_db):/tmp/sdbm.sql.gz
docker exec -it $(docker ps -q -f name=sdbmss_db) bash
cd /tmp
gunzip sdbm.sql.gz
mysql -u <MYSQL_USER> -p <MYSQL_DATABASE> < sdbm.sql
rm sdbm.sql # remove the sql file (it's very big)
exit # exit the MySQL container
docker exec $(docker ps -q -f name=sdbmss_rails) bundle exec rake db:migrate
NOTE: If you are importing from a data file that includes Page objects, the database records will be copied, but not the page files. You will need to move these manually to the appropriate place in the public/static folder (uploads/, tooltips/ or docs/)
docker cp docs $(docker ps -q -f name=sdbmss_rails):/usr/src/app/public/static/
docker cp tooltips $(docker ps -q -f name=sdbmss_rails):/usr/src/app/public/static/
docker cp uploads $(docker ps -q -f name=sdbmss_rails):/usr/src/app/public/static/
Index in Solr:
docker exec $(docker ps -q -f name=sdbmss_rails) bundle exec rake sunspot:reindex
6. Jena First Time Setup
docker exec -t $(docker ps -q -f name=sdbmss_rails) bundle exec rake sparql:test
File should be in .
; gzip it.
gzip test.ttl
Copy file to Jena container and gunzip it
docker cp test.ttl.gz $(docker ps -q -f name=sdbmss_jena):/tmp/
docker exec -t $(docker ps -q -f name=sdbmss_jena) gunzip /tmp/test.ttl.gz
Load the data into Jena
docker exec -t $(docker ps -q -f name=sdbmss_jena) sh -c 'cd /jena-fuseki && ./tdbloader --loc=/fuseki/databases/sdbm /tmp/test.ttl'
Clean up the files.
$ docker exec -t $(docker ps -q -f name=sdbmss_jena) rm /tmp/test.ttl
rm ~/deployments/sdbmss/test.ttl.gz
Create the datset in Jena Fuseki.
Go here and create the sdbm dataset: https://localhost/sparql/manage.html
- Click 'add new data set'
- Enter 'sdbm'
- Select 'Persistent – dataset will persist across Fuseki restarts'
- Click 'create dataset'
Scale the services:
docker-compose -f docker-compose-dev.yml restart jena
docker-compose -f docker-compose-dev.yml restart rabbitmq
docker-compose -f docker-compose-dev.yml restart rails
Run the Jena verify task to confirm that it works. Be sure to hide the debugging output.
docker-compose -f docker-compose-dev.yml exec rails bundle exec rake jena:verify | grep -v DEBUG
NB: You may need to run the command more than once.
sdbm01[~]$ docker-compose -f docker-compose-dev.yml exec rails bundle exec rake jena:verify | grep -v DEBUG
Starting Queue Listening
No more messages in queue.
Remaining responses: 764
$ docker-compose -f docker-compose-dev.yml exec rails bundle exec rake jena:verify | grep -v DEBUG
Starting Queue Listening
Parsed contents: {"id"=>300122, "code"=>"200", "message"=>"OK"}
Jena Update was Successful!
Parsed contents: {"id"=>300211, "code"=>"200", "message"=>"OK"}
Jena Update was Successful!
Parsed contents: {"id"=>300212, "code"=>"200", "message"=>"OK"}
Jena Update was Successful!
Parsed contents: {"id"=>300213, "code"=>"200", "message"=>"OK"}
Jena Update was Successful!
# ... etc.