The Schoenberg Database of Manuscripts. This is the Ruby on Rails / Blacklight project reboot, started in late Nov 2014.
Note that we use 'sdbmss' everywhere in the code because 'sdbm' is the name of a package in Ruby's standard library.
The application is currently structured to be run through Docker using a single docker compose file.
To deploy this application in development or production, consult the sdbm-config README in GitLab.
This illustration represents the current deployment workflow for SDBMSS.
1. Clone the repository
DEPRECATED -- USING ANSIBLE
git clone https://github.com/upenn-libraries/sdbmss.git
2. Create file .docker-environment in the root folder of the cloned repository. Define the following environment variables:
DEPRECATED -- USING ANSIBLE
This is the password for the public access point for the Jena server, both for making updates and for downloading RDF data file
ADMIN_PASSWORD=<jena_admin_password>
Login for private staging server deployment, if used
RAILS_USER=<rails_staging_username>
RAILS_PASS=<rails_staging_password>
MYSQL database setup. MYSQL_HOST is the name of the docker service defined in docker-compose.yml - everything else is up to you
MYSQL_HOST=db
MYSQL_DATABASE=<mysql_database_name>
MYSQL_ROOT_PASSWORD=<mysql_root_password>
MYSQL_USER=<mysql_username>
MYSQL_PASSWORD=<mysql_password>
RabbitMQ user and password; used for RDF live-update messaging system. Define here, then set the correct values when instantiating the RabitMQ service (done later)
RABBIT_USER=<rabbitmq_username>
RABBIT_PASSWORD=<rabbitmg_password>
Rails email setup. Depends on where the application is running and what mailing service is available
SDBMSS_APP_HOST=<email_app_host>
SDBMSS_SMTP_HOST=<email_smtp_host>
SDBMSS_EMAIL_FROM=<email_sender>
SDBMSS_NOTIFY_EMAIL=<notification_email_sender>
SDBMSS_NOTIFY_EMAIL_PASSWORD=<notification_email_password>
# email address
SDBMSS_EMAIL_EXCEPTIONS_TO=<send_exceptions_to_email>
Generate separate keys (for each) by running rails task secret.
SDBMSS_BLACKLIGHT_SECRET_KEY=<KEY>
SDBMSS_DEVISE_SECRET_KEY=<KEY>
SDBMSS_SECRET_KEY_BASE=<KEY>
SDBMSS_SECRET_TOKEN=<KEY>
URL for SOLR server, using relative location of docker service (using name 'solr' from docker-compose.yml)
NB: The SOLR port here and everywhere else in the code is 8983
. Previous versions
of the code had 8982
in some cases. This was wrong and created a conflict
between Jetty (which was set to 8983
and the docker-compose values, which
had 8982
).
SOLR_URL=http://solr:8983/solr/development
3. (Optional) Create a file VERSION in the root folder of the cloned repository.
DEPRECATED -- USING ANSIBLE
This is used to keep track of the version number for the purpose of tagging different docker images of the source code. It is used by the script build.sh. It's entire contents should just be:
0.0.1
4. Build and Run (First Time)
DEPRECATED -- USING ANSIBLE
Run build.sh or the following command:
docker build . -t sdbm:latest
Start everything up:
docker-compose up --build
If you prefer to start it detached, run this instead:
docker-compose up --build --detach && docker-compose start
5. First Time Setup: Rails and SOLR
Before you begin, have the following:
-
Database backup from previous version sdbm.sql.gz
-
Static assets from docs, tooltips, updloads (see:
docker volume inspect sdbmss_sdbm_docs
for location of files on filesystem)Setup database - perform setup:
docker exec $(docker ps -q -f name=sdbmss_rails) bundle exec rake db:setup
(Optional: Load data from .sql dump)
docker cp sdbm.sql.gz $(docker ps -q -f name=sdbmss_db):/tmp/sdbm.sql.gz
docker exec -it $(docker ps -q -f name=sdbmss_db) bash
cd /tmp
gunzip sdbm.sql.gz
mysql -u <MYSQL_USER> -p <MYSQL_DATABASE> < sdbm.sql
rm sdbm.sql # remove the sql file (it's very big)
exit # exit the MySQL container
docker exec $(docker ps -q -f name=sdbmss_rails) bundle exec rake db:migrate
NOTE: If you are importing from a data file that includes Page objects, the database records will be copied, but not the page files. You will need to move these manually to the appropriate place in the public/static folder (uploads/, tooltips/ or docs/)
docker cp docs $(docker ps -q -f name=sdbmss_rails):/usr/src/app/public/static/
docker cp tooltips $(docker ps -q -f name=sdbmss_rails):/usr/src/app/public/static/
docker cp uploads $(docker ps -q -f name=sdbmss_rails):/usr/src/app/public/static/
Index in Solr:
docker exec $(docker ps -q -f name=sdbmss_rails) bundle exec rake sunspot:reindex
8. Jena First Time Setup
Regenerate RDF for dataset:
docker exec -t $(docker ps -q -f name=sdbmss_rails) bundle exec rake sparql:test
# file should be in ~/deployments/sdbms/test.ttl; gzip it
# gzip it
gzip ~/deployments/sdbmss/test.ttl
docker cp ~/deployments/sdbmss/test.ttl.gz $(docker ps -q -f name=sdbmss_jena):/tmp/
docker exec -it $(docker ps -q -f name=sdbmss_jena) bash
cd /tmp
gunzip test.ttl.gz
cd /jena-fuseki
./tdbloader --loc=/fuseki/databases/sdbm /tmp/test.ttl
# delete the test.ttl
rm /tmp/test.ttl
# exit and delete the test.ttl.gz
exit
rm ~/deployments/sdbms/test.ttl.gz
From front-end interface (https:///sparql), create new persistent dataset "sdbm", then scale the services:
docker service scale sdbmss_jena=0
docker service scale sdbmss_jena=1
docker service scale sdbmss_rabbitmq=0
docker service scale sdbmss_rabbitmq=1
docker service scale sdbmss_rails=0
docker service scale sdbmss_rails=1
Run the Jena verify task to confirm that it works:
$ /bin/bash -l -c 'docker exec -t $(docker ps -q -f name=sdbmss_rails) bundle exec rake jena:verify'
Starting Queue Listening
Parsed contents: {"id"=>300122, "code"=>"200", "message"=>"OK"}
Jena Update was Successful!
Parsed contents: {"id"=>300211, "code"=>"200", "message"=>"OK"}
Jena Update was Successful!
Parsed contents: {"id"=>300212, "code"=>"200", "message"=>"OK"}
Jena Update was Successful!
Parsed contents: {"id"=>300213, "code"=>"200", "message"=>"OK"}
Jena Update was Successful!
# ... etc.
9. Logging
DEPRECATED?
Logging is set in docker-compose.yml, and may need to be altered depending on the logging drivers available on the parent device. For the current setup (JournalCtl), the logs may be accessed as follows:
Entire logs:
journalctl -u docker
Follow logs
journalctl -f -u docker
Additionally, specific containers can be specified by name:
journalctl -u docker CONTAINER_NAME=sdbmss_rails_1
journalctl -f -u docker CONTAINER_NAME=sdbmss_rails_1 @follow