Skip to content

Latest commit

 

History

History
1015 lines (644 loc) · 43.2 KB

Provisioning-Islandora-on-Ubuntu.md

File metadata and controls

1015 lines (644 loc) · 43.2 KB

Introduction

Within this repository, there will be a stand alone bash script designed to install an all-in-one Islandora stack upon any type of platform e.g Virtual Machine, bare metal server etc.

This document is for the evaluation and testing of Islandora but with the secondary purpose of explaining all involved installation steps for wider Islandora adoption and educational transparency.

Document Maintainer

This document is being submitted to the Islandora DevOps Interest Group to demonstrate a base Islandora install as performed and agreed upon by the Islandora Community. Any feedback should be directed at the Islandora DevOps Interest Group at https://github.com/islandora-interest-groups/Islandora-DevOps-Interest-Group.

Disclaimer

This documentation is provided by the Islandora DevOps Interest Group "as is" and "with all faults." Islandora DevOps makes no representations or warranties of any kind concerning the safety, suitability, inaccuracies, typographical errors, or other harmful components resulting from the use of this documentation or the software installed and referenced in the documentation. There are inherent dangers in the use of any software, and the user is solely responsible for determining whether this software product and this documentation is compatible with their equipment and other software installed on their equipment. Users are also solely responsible for the protection of their equipment and backup of their data, and Islandora DevOps will not be liable for any damages one may suffer in connection with using or modifying this documentation or referenced software.

Index

Server

This document is outlining a single (all-in-one) Islandora stack installation using Ubuntu 14.04 LTS system.

For general purpose repository sites, we recommend at minimum 2-4 CPU cores and 8GB-10GB of RAM.

This would be scaled up according to user-load, the type of derivative files being generated by the system (ie. creating web friendly version of archival quality images or video are quite processor intensive) and many other factors.

Environment

Create an islandora-install.properties file that should be removed or secured later:

cat ~/islandora-install.properties

    #!/bin/bash
    DB_SERVER="localhost"
    DB_ROOT_PASSWORD="password"            # Set MySQL root password -> keep it simple -> run mysql_secure_installation after install to secure mySQL
    DRUPAL_DB_NAME="drupal7"               # Drupal database name
    DRUPAL_DB_USER="drupal"                # Drupal username for settings.php file
    DRUPAL_DB_PASS="password"              # Drupal password for settings.php file
    DRUPAL_ADMIN_USER="admin"              # Drupal admin username to log into Drupal Site
    DRUPAL_ADMIN_PASS="password"           # Drupal admin password to log into Drupal Site
    DRUPAL_SITE_NAME="Islandora Install"   # Drupal site name - displayed on the web site
    FEDORA_VERSION="3.8.1"                 # Fedora version to install:  3.5 or 3.6.2 or 3.7.0 or 3.7.1
    FEDORA_DB_NAME="fedora3"               # Name of fedora MySQL database -> fedora3 is recommended
    FEDORA_DB_USER="fedoraAdmin"           # Fedora db username -> for fedora.fcfg file
    FEDORA_DB_PASS="password"              # Fedora db password -> for fedora.fcfg file
    FEDORA_ADMIN_USER="fedoraAdmin"        # Username for http://localhost:8080/fedora/admin
    FEDORA_ADMIN_PASS="password"           # Password for http://localhost:8080/fedora/admin
    FEDORA_USER="fedora"                   # User that tomcat runs as
    FEDORA_HOME="/usr/local/fedora"        # Fedora home dir
    CATALINA_HOME="/usr/share/tomcat"    # Tomcat location
    TOMCAT_VERSION="7.0.55"
# NOTE adjust memory so that it is no larger than half of total system memory. Depending on stack deployment this can be adjusted further. We don’t recommend running stack on system with less than 4GB of ram. Recommend adding -XX:+UseParallelOldGC for multi cpu systems

    JAVA_OPTS="-Xms3072m -Xmx3072m -server -Djavax.net.ssl.trustStore=/usr/local/fedora/server/truststore -Djavax.net.ssl.trustStorePassword=tomcat -Dsolr.solr.home=/usr/local/fedora/solr -Dkakadu.home=/opt/adore-djatoka/bin/Linux-x86-64 -Djava.library.path=/opt/adore-djatoka/lib/Linux-x86-64 -DLD_LIBRARY_PATH=/opt/adore-djatoka/lib/Linux-x86-64"
    JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre  
    # Moved this down below java install please change to match java version
    TOMCAT_BASE="http://localhost:8080"
    ISLANDORA_BASE="$TOMCAT_BASE/fedora"
    SOLR_BASE="$TOMCAT_BASE/solr"
    ISLANDORA_BRANCH="7.x"
    TUQUE_BRANCH="1.x"
    ERROR_LEVEL="2"
    FEDORA_GSEARCH_URL="http://downloads.sourceforge.net/fedora-commons/fedoragsearch-2.8.zip"
    FEDORA_GSEARCH_NAME="fedoragsearch-2.8"
    SOLR_VERSION="4.2.0"
    NUMBER_OF_GSEARCH_THREADS="2"          #do one gsearch thread per cpu might want to skip multithreading if you only have 1 cpu
    GSEARCH_UPDATER_NAMES="FgsUpdaters FgsUpdater1"
    DRUPAL_FILTER_URL="https://github.com/Islandora/islandora_drupal_filter/releases/download/v7.1.3/fcrepo-drupalauthfilter-3.8.1.jar"
    SOLR_URL="http://archive.apache.org/dist/lucene/solr/4.2.0/solr-4.2.0.tgz"
    SOLR_NAME="solr-4.2.0"
    SOLR_DEFAULT_CORE_PATH="collection1"

    Q1="CREATE DATABASE IF NOT EXISTS $DRUPAL_DB_NAME CHARACTER SET utf8 COLLATE utf8_bin;"
    Q2="GRANT ALL ON $DRUPAL_DB_NAME.* TO $DRUPAL_DB_USER@localhost IDENTIFIED BY '$DRUPAL_DB_PASS';"
    Q3="CREATE DATABASE IF NOT EXISTS $FEDORA_DB_NAME CHARACTER SET utf8 COLLATE utf8_bin;"
    Q4="GRANT ALL ON $FEDORA_DB_NAME.* TO $FEDORA_DB_USER@localhost IDENTIFIED BY '$FEDORA_DB_PASS';"
    Q5="FLUSH PRIVILEGES;"
    SQL="${Q1}${Q2}${Q3}${Q4}${Q5}"

    APACHE_DEFAULT_SITE="000-default"
    APC_CONFIG_FILE="/etc/php5/mods-available/apcu.ini"
    PROXY_DJATOKA_CONFIG="/etc/apache2/conf-available/proxy_djatoka.conf"
    APACHE_USER="www-data"
    APACHE_SERVICE="apache2"
    OS_DEFAULT_DOCUMENTROOT="/var/www"
    SCHEDULE_TOMCAT_SERVICE_COMMAND="update-rc.d tomcat start 95 2 3 4 5 . stop 15 0 1 6 ."
    SCHEDULE_OPENOFFICE_SERVICE_COMMAND="update-rc.d openoffice start 92 2 3 4 5 . stop 30 0 1 6 ."
    CRON_SPOOL_DIR="/var/spool/cron/crontabs"
    MONIT_CONF_DIR="/etc/monit/conf.d"
    MONIT_CONFIG_FILE="/etc/monit/monitrc"
    #Resolvable hostname.
    DNS_HOSTNAME="yourhostname.local"

Source environment (Please note: this must be redone if you close your shell mid-install):

chmod +x ~/islandora-install.properties

. ~/islandora-install.properties

Operating System

  1. Ensure latest core os updates: apt-get update && apt-get upgrade

  2. Add repo to easily obtain oracle java installer: apt-get -y install python-software-properties && add-apt-repository -y ppa:webupd8team/java && apt-get -y update

  3. Add optional multiverse repos in /etc/apt/sources.list

  4. Uncomment each line in this file that ends in multiverse.

Software Dependencies

Software Dependencies Installed by Apt-Get Package Manager

Accept license agreement and enter root password when asked.

apt-get -y install oracle-java8-installer libjpeg-dev libpng12-dev libtiff4-dev php5 php5-cli php5-curl php5-dev php5-gd php5-ldap php5-mysql php5-xsl php-apc php-soap php-xml-htmlsax3 php-xml-parser php-xml-rpc php-xml-rpc2 php-xml-rss php-xml-serializer php5-imagick php5-mcrypt php-xml* mysql-server vim curl apache2 rsync wget imagemagick ant libimage-exiftool-perl unzip lame autoconf build-essential checkinstall git libass-dev libfaac-dev libgpac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev librtmp-dev libtheora-dev libtool libvorbis-dev pkg-config texi2html zlib1g-dev ffmpeg2theora poppler-utils python-pip libreoffice libreoffice-writer libreoffice-calc libreoffice-impress libreoffice-draw bibutils ufraw links monit tesseract-ocr tesseract-ocr-eng tesseract-ocr-fra tesseract-ocr-spa tesseract-ocr-ita tesseract-ocr-por tesseract-ocr-hin tesseract-ocr-deu tesseract-ocr-jpn tesseract-ocr-rus

Software Dependencies Compiled from Source

ghostscript

Notes: ubuntu 14.04 installs ghostscript 9.10 which is currently failing our test sets and prevents tiffs from being generated from pdfs. Recommend installing from source to get correct version.

wget http://downloads.ghostscript.com/public/old-gs-releases/ghostscript-9.05.tar.gz

tar xvzf ghostscript-9.05.tar.gz

cd ghostscript-9.05
  
./configure
 
make && make install

`gs --version` #should return result.

ffmpeg

mkdir ~/ffmpeg-source
cd ~/ffmpeg-source

yasm

wget http://www.tortall.net/projects/yasm/releases/yasm-1.2.0.tar.gz

tar xzvf yasm-1.2.0.tar.gz && rm -rf yasm-1.2.0.tar.gz

cd yasm-1.2.0

./configure

make

checkinstall --pkgname=yasm --pkgversion="1.2.0" --backup=no --deldoc=yes --fstrans=no --default

x264

cd ~/ffmpeg-source

git clone --depth 1 git://git.videolan.org/x264.git

cd x264

./configure --enable-static --enable-shared

make

checkinstall --pkgname=x264 --pkgversion="3:$(./version.sh | awk -F'[" ]' '/POINT/{print $4"+git"$5}')" --backup=no --deldoc=yes  --fstrans=no --default

ldconfig

cd ~/ffmpeg-source

aac

git clone --depth 1 git://github.com/mstorsjo/fdk-aac.git

cd fdk-aac

autoreconf -fiv

./configure --disable-shared

make

checkinstall --pkgname=fdk-aac --pkgversion="$(date +%Y%m%d%H%M)-git" --backup=no --deldoc=yes --fstrans=no --default

libvpx

cd ~/ffmpeg-source

git clone https://chromium.googlesource.com/webm/libvpx.git

cd libvpx

git checkout 8366a6e4ba95e6d5af040815d2afbb4bfe628d3f

./configure --disable-examples --disable-unit-tests

make

checkinstall --pkgname=libvpx --pkgversion="1:$(date +%Y%m%d%H%M)-git" --backup=no --deldoc=yes --fstrans=no --default

opus

cd ~/ffmpeg-source

git clone --depth 1 git://git.xiph.org/opus.git

cd opus

./autogen.sh

./configure --disable-shared

make

checkinstall --pkgname=libopus --pkgversion="$(date +%Y%m%d%H%M)-git" --backup=no --deldoc=yes --fstrans=no --default

ffmpeg

cd ~/ffmpeg-source

wget http://www.ffmpeg.org/releases/ffmpeg-1.1.1.tar.gz

tar xf ffmpeg-1.1.1.tar.gz && rm -rf ffmpeg-1.1.1.tar.gz  

cd ffmpeg-1.1.1

Had to do this to remove strange characters that cause build errors.

sed -i 's/×/x/' doc/filters.texi

sed -i 's/×/x/' doc/ffmpeg.texi

./configure --enable-gpl --enable-libass --enable-libfaac --enable-libfdk-aac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-version3 --enable-libopus

make

checkinstall --pkgname=ffmpeg --pkgversion="7:$(date +%Y%m%d%H%M)-git" --backup=no --deldoc=yes --fstrans=no --default

make install

hash -r

The following should tell apt to not upgrade ffmpeg (we need version 1.1.1):

apt-mark hold ffmpeg

Cleanup ffmpeg source code:

ffmpeg -version && cd ~ && rm -rf ~/ffmpeg-source

Software Dependencies installed by Binaries

ffmpeg2theora

cd ~ && wget http://v2v.cc/~j/ffmpeg2theora/ffmpeg2theora-0.29.linux64.bin && chmod a+x ffmpeg2theora-0.29.linux64.bin && install -m 755 ffmpeg2theora-0.29.linux64.bin /usr/bin/ffmpeg2theora && rm -rf ffmpeg2theora-0.29.linux64.bin

java and tomcat

Now that java is installed, we can set JAVA_HOME and also add it to ~/islandora-install.properties

echo $( dirname $( dirname $( readlink -e /usr/bin/java ) ) )

This should match the oracle java not openjdk.

Update ~/islandora-install.properties to match proper java version.

For example:

JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre

And source again if necessary:

. ~/islandora-install.properties

Note that you need to ensure that your update-alternatives point to the right java and javac.

update-alternatives --config java

update-alternatives --config javac

Create Fedora/Tomcat user which is used to run Tomcat.

FEDORA_USER_TEST=`getent passwd $FEDORA_USER`

useradd -d $FEDORA_HOME -s /bin/false $FEDORA_USER
#using mkdir since skel dirs cause Fedora installer to prompt to overwrite.
mkdir /usr/local/fedora

Update Tomcat environment variables and JAVA OPTS

echo -e 'export FEDORA_HOME=/usr/local/fedora\nexport CATALINA_HOME=/usr/local/fedora/tomcat\nexport CATALINA_PID="$CATALINA_HOME/catalina.pid"\nexport JAVA_OPTS="-Xms1024m -Xmx1024m -XX:MaxPermSize=512m -XX:+CMSClassUnloadingEnabled -Djavax.net.ssl.trustStore=/usr/local/fedora/server/truststore -Djavax.net.ssl.trustStorePassword=tomcat"\nexport JAVA_HOME=/usr/lib/jvm/java-7-oracle/jre\nexport FEDORA_USER=fedora' > /etc/profile.d/fedora.sh

sed -i "s|JAVA_OPTS=\"-Xms1024m -Xmx1024m -XX:MaxPermSize=512m -XX:+CMSClassUnloadingEnabled -Djavax.net.ssl.trustStore=/usr/local/fedora/server/truststore -Djavax.net.ssl.trustStorePassword=tomcat\"|JAVA_OPTS=\"$JAVA_OPTS\"|g" /etc/profile.d/fedora.sh
sed -i "s|FEDORA_HOME=\"/usr/local/fedora\"|FEDORA_HOME=\"$FEDORA_HOME\"|g" /etc/profile.d/fedora.sh
sed -i "s|CATALINA_HOME=\"/usr/local/fedora/tomcat\"|CATALINA_HOME=\"$CATALINA_HOME\"|g" /etc/profile.d/fedora.sh
sed -i "s|FEDORA_USER=fedora|FEDORA_USER=$FEDORA_USER|g" /etc/profile.d/fedora.sh
sed -i "s|JAVA_HOME=/usr/lib/jvm/java-7-oracle/jre|JAVA_HOME=$JAVA_HOME|g" /etc/profile.d/fedora.sh

Note: make sure you check /etc/profile.d/fedora.sh to ensure everything has been generated properly. If it didn’t you may have closed your shell you may need to resource islandora-install.properties.

For example:

export FEDORA_HOME=/usr/local/fedora
export CATALINA_HOME=/usr/share/tomcat
export CATALINA_PID="$CATALINA_HOME/catalina.pid"
export JAVA_OPTS="-Xms3993m -Xmx3993m -Djavax.net.ssl.trustStore=/usr/local/fedora/server/truststore -Djavax.net.ssl.trustStorePassword=tomcat"
export JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre

Create tomcat init.d startup script:

cd /etc/init.d

wget --no-check-certificate https://raw.github.com/discoverygarden/Fedora-Init-Script/master/tomcat

ln -s tomcat fedora

chmod a+x tomcat 

$SCHEDULE_TOMCAT_SERVICE_COMMAND

cd ~
wget http://archive.apache.org/dist/tomcat/tomcat-7/v$TOMCAT_VERSION/bin/apache-tomcat-$TOMCAT_VERSION.tar.gz
tar xf apache-tomcat-$TOMCAT_VERSION.tar.gz
mv apache-tomcat-$TOMCAT_VERSION /usr/share
ln -s /usr/share/apache-tomcat-$TOMCAT_VERSION /usr/share/tomcat

chown -R $FEDORA_USER:$FEDORA_USER /usr/share/apache-tomcat-$TOMCAT_VERSION

Fits

cd /opt

wget http://projects.iq.harvard.edu/files/fits/files/fits-0.10.1.zip

unzip -o fits-0.10.1.zip && rm -rf fits-0.10.1.zip && ln -s fits-0.10.1 fits && chmod a+x /opt/fits/fits.sh

adore-djatoka

cd /opt

wget http://sourceforge.net/projects/djatoka/files/djatoka/1.1/adore-djatoka-1.1.tar.gz/download -O adore-djatoka-1.1.tar.gz

tar xf adore-djatoka-1.1.tar.gz && rm -rf adore-djatoka-1.1.tar.gz

ln -s adore-djatoka-1.1 adore-djatoka

Make kakadu available on the command line:

ln -s /opt/adore-djatoka/bin/Linux-x86-64/kdu_expand /usr/bin/kdu_expand

ln -s /opt/adore-djatoka/bin/Linux-x86-64/kdu_compress /usr/bin/kdu_compress

echo "/opt/adore-djatoka/lib/Linux-x86-64" > /etc/ld.so.conf.d/kakadu.conf

ldconfig

Drush

Specify drush commit due to issues with newer drush versions and automated tests used by QA:

cd /opt

git clone https://github.com/drush-ops/drush.git && cd drush && git checkout b9e6c8c00da0fbf1227869cdf915b0c6cea466cc

ln -s /opt/drush/drush /usr/bin/drush

Configuration

OpenOffice

cd /etc/init.d && wget --no-check-certificate https://raw.github.com/discoverygarden/openoffice-init-script/master/openoffice && chmod a+x openoffice

useradd -m -d /home/openoffice openoffice

service openoffice start

$SCHEDULE_OPENOFFICE_SERVICE_COMMAND

Monit

Keep openoffice running as a service with Monit as it has been known to crash.

echo -e "check process openoffice\n        matching \"/usr/lib/libreoffice/program/soffice.bin\"\n        start program = \"/etc/init.d/openoffice start\"\n        stop program = \"/etc/init.d/openoffice stop\"\n        if failed host 127.0.0.1 port 8100 then restart\n        if 5 restarts within 5 cycles then timeout" > $MONIT_CONF_DIR/openoffice.conf

sed -i 's|# set httpd| set httpd|g' $MONIT_CONFIG_FILE

sed -i 's|#    use address|    use address|g' $MONIT_CONFIG_FILE

sed -i 's|#    allow localhost|    allow localhost|g' $MONIT_CONFIG_FILE

sed -i 's|    allow admin:monit|#    allow admin:monit|g' $MONIT_CONFIG_FILE  

sed -i 's|    allow @monit |#    allow @monit |g' $MONIT_CONFIG_FILE

sed -i 's|    allow @users readonly |#    allow @users readonly |g' $MONIT_CONFIG_FILE  

service monit restart

Apache and PHP

Setup apache vhost:

vi /etc/apache2/sites-available/000-default.conf

<VirtualHost *:80>
        ServerAdmin webmaster@localhost

        DocumentRoot /var/www/drupal7
        <Directory />
                Options FollowSymLinks
                AllowOverride None
        </Directory>
        <Directory /var/www/drupal7/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride all
                Order allow,deny
                allow from all
        </Directory>

        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
        <Directory "/usr/lib/cgi-bin">
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
        </Directory>

        ErrorLog ${APACHE_LOG_DIR}/error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn

        CustomLog ${APACHE_LOG_DIR}/access.log combined

    Alias /doc/ "/usr/share/doc/"
    <Directory "/usr/share/doc/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>

        ProxyPass /adore-djatoka http://localhost:8080/adore-djatoka
        ProxyPassReverse /adore-djatoka http://localhost:8080/adore-djatoka
    
</VirtualHost>
a2enmod rewrite  

a2enmod proxy

a2enmod proxy_http  

pecl install uploadprogress 

sed -i '949iextension=uploadprogress.so' /etc/php5/apache2/php.ini 

sed -i "s|memory_limit = 128M|memory_limit = 512M|g" /etc/php5/apache2/php.ini

sed -i "s|post_max_size = 8M|post_max_size = 2048M|g" /etc/php5/apache2/php.ini

sed -i "s|upload_max_filesize = 2M|upload_max_filesize = 2048M|g" /etc/php5/apache2/php.ini

echo "apc.shm_size = 64M" >> $APC_CONFIG_FILE

Setup MySQL Databases and Server

Run mysql_secure_installation to ensure security:

MYSQL=`which mysql`  

$MYSQL -uroot -p$DB_ROOT_PASSWORD -e "$SQL"  

Add a mysql backup script:

if [ ! -d /root/bin ]; then
  mkdir -p /root/bin
  fi  
  echo -e '#!/bin/bash\n#\n# Dumps all MySQL databases and removes backups older than 1 week\n# With optional scp to another host\n\nBackupFolder=/dbbackups\nMySQLUser=root\nMySQLPass=PASSWORD\n# These next few are just used for scp, which is optional if DO_SCP != yes\nDO_SCP=no\nREMOVE_LOCAL_COPY=no\nFOLDER=dbBACK/mysql-newdb\nBACKUP_USER=backupuser\nBACKUP_HOST=backuppass\n\nDATE=$(date +%Y%m%d)\nOLDDATE=$(date +%Y%m%d -d " 7 days ago")\n\nif [ ! -d "$BackupFolder" ]; then\n        mkdir -p "$BackupFolder"\n        if [ $? -gt 0 ]; then\n                echo "ERROR: Could not create $BackupFolder"\n                exit 1\n        fi\nfi\n\nfor dbname in $(mysql --user="$MySQLUser" --password="$MySQLPass" -Bse "show databases"); do\n  rm -f "$BackupFolder/$dbname-$OLDDATE.bz2"\n  DumpName="$BackupFolder/$dbname-$DATE.bz2"\n  mysqldump --user="$MySQLUser" --password="$MySQLPass" --opt "$dbname" | bzip2 -9 >"$DumpName"\n  if [ $? -gt 0 ]; then\n        echo "Backup of $dbname failed! (mysqldump)"\n  else\n        if [ "$DO_SCP" == "yes" ]; then\n            scp -B -o StrictHostKeyChecking=no "$DumpName" "$BACKUP_USER"@"$BACKUP_HOST":"$FOLDER/"\n            if [ $? -gt 0 ]; then\n                echo "Backup of $dbname failed! (scp)"\n            fi\n            if [ "$REMOVE_LOCAL_COPY" == "yes" ]; then\n                rm -f "$DumpName"\n            fi\n        fi\n  fi\ndone\n' > /root/bin/mysqlBackup.sh && chmod a+x /root/bin/mysqlBackup.sh

/root/bin/mysqlBackup.sh

sed -i "s|PASSWORD|$DB_ROOT_PASSWORD|g" /root/bin/mysqlBackup.sh

Update crontab to schedule mysql backup:

crontab -e

Add:

15 1 * * * /root/bin/mysqlBackup.sh

mkdir -p /dbbackups

Dumps will be stored in /dbbackups

Install Fedora Commons

Fedora Commons Base Install

Create install properties file


cd ~ 

echo -e 'keystore.file=included\nri.enabled=true\nmessaging.enabled=true\napia.auth.required=false\ndatabase.jdbcDriverClass=com.mysql.jdbc.Driver\ntomcat.ssl.port=8443\nssl.available=true\ndatabase.jdbcURL=jdbc\:mysql\://localhost/fedora3?useUnicode\=true&amp;characterEncoding\=UTF-8&amp;autoReconnect\=true\nmessaging.uri=vm\:(broker\:(tcp\://localhost\:61616))\ndatabase.password=islandora\ndatabase.mysql.driver=included\ndatabase.username=fedoraAdmin\nfesl.authz.enabled=false\ntomcat.shutdown.port=8055\ndeploy.local.services=true\nxacml.enabled=true\ndatabase.mysql.jdbcDriverClass=com.mysql.jdbc.Driver\ntomcat.http.port=8080\nfedora.serverHost=localhost\ndatabase=mysql\ndatabase.driver=included\nfedora.serverContext=fedora\nllstore.type=akubra-fs\ntomcat.home=/usr/local/fedora/tomcat\nfesl.authn.enabled=true\ndatabase.mysql.jdbcURL=jdbc\:mysql\://localhost/fedora3?useUnicode\=true&amp;characterEncoding\=UTF-8&amp;autoReconnect\=true\nfedora.home=/usr/local/fedora\ninstall.type=custom\nservlet.engine=included\napim.ssl.required=false\nfedora.admin.pass=islandora\napia.ssl.required=false' > ~/install.properties

sed -i "s|localhost/fedora3?|localhost/$FEDORA_DB_NAME?|g" ~/install.properties

sed -i "s|database.password=islandora|database.password=$FEDORA_DB_PASS|g" ~/install.properties 

sed -i "s|database.username=fedoraAdmin|database.username=$FEDORA_DB_USER|g" ~/install.properties  

sed -i "s|fedora.home=/usr/local/fedora|fedora.home=$FEDORA_HOME|g" ~/install.properties 

sed -i "s|tomcat.home=/usr/local/fedora/tomcat|tomcat.home=$CATALINA_HOME|g" ~/install.properties

sed -i "s|fedora.admin.pass=islandora|fedora.admin.pass=$FEDORA_ADMIN_PASS|g" ~/install.properties

sed -i "s|servlet.engine=included|servlet.engine=existingTomcat|g"  ~/install.properties

cd ~  

wget http://downloads.sourceforge.net/fedora-commons/fcrepo-installer-$FEDORA_VERSION.jar  

java -jar fcrepo-installer-$FEDORA_VERSION.jar install.properties  

rm -rf fcrepo-installer-$FEDORA_VERSION.jar install.properties  

sed -i "s|changeme|islandora|g" $FEDORA_HOME/server/config/fedora.fcfg

#copy over server.xml with the one provided with fedora cp $CATALINA_HOME/conf/server.xml $CATALINA_HOME/conf/server.bak cp /usr/local/fedora/install/server.xml $CATALINA_HOME/conf/server.xml #Note remove maxSpareThreads property in /usr/share/tomcat/conf/server.xml as it no longer does anything.

Tweak some Fedora settings:

sed -i "s|security.fesl.authN.jaas.apia.enabled=false|security.fesl.authN.jaas.apia.enabled=true|g" $FEDORA_HOME/server/config/spring/web/web.properties 

unzip -o $CATALINA_HOME/webapps/fedora.war -d $CATALINA_HOME/webapps/fedora

chown -R $FEDORA_USER:$FEDORA_USER $FEDORA_HOME

chown -R $FEDORA_USER:$FEDORA_USER /usr/share/apache-tomcat-$TOMCAT_VERSION

#Needed to be compatible with basic-solr-configs
ln -s /usr/share/tomcat /usr/local/fedora/tomcat

Start and stop tomcat so fedora creates some dirs:

service tomcat deploy

XACML Settings

Adjust xacml policies:

rm -rf $FEDORA_HOME/data/fedora-xacml-policies/repository-policies/default/deny-purge-* 

rm -rf $FEDORA_HOME/data/fedora-xacml-policies/repository-policies/default/deny-inactive-or-deleted-objects-or-datastreams-if-not-administrator.xml  

rm -rf $FEDORA_HOME/data/fedora-xacml-policies/repository-policies/default/deny-policy-management-if-not-administrator.xml

Note: This file should either be removed or tweaked if you wish to access fedoraAdmin. Keep in mind firewall rules would need to be updated as well.

rm -rf $FEDORA_HOME/data/fedora-xacml-policies/repository-policies/default/deny-apim-if-not-localhost.xml  

cd ~  

git clone https://github.com/Islandora/islandora-xacml-policies && cd islandora-xacml-policies 

mkdir $FEDORA_HOME/data/fedora-xacml-policies/repository-policies/islandora_policies && cp *.xml $FEDORA_HOME/data/fedora-xacml-policies/repository-policies/islandora_policies && rm -rf ~/islandora-xacml-policies

GSearch and Solr

Grab fedoragsearch and solr - do basic config:

cd ~  

wget $FEDORA_GSEARCH_URL

unzip -o $FEDORA_GSEARCH_NAME.zip 

cp $FEDORA_GSEARCH_NAME/fedoragsearch.war $CATALINA_HOME/webapps/ && unzip -o $FEDORA_GSEARCH_NAME/fedoragsearch.war -d $CATALINA_HOME/webapps/fedoragsearch && rm -rf $FEDORA_GSEARCH_NAME*

wget $SOLR_URL

tar -xf $SOLR_NAME.tgz

cp -r $SOLR_NAME/example/solr $FEDORA_HOME/solr

cp $SOLR_NAME/example/webapps/solr.war $CATALINA_HOME/webapps/ && unzip -o $SOLR_NAME/example/webapps/solr.war -d $CATALINA_HOME/webapps/solr 

mkdir $FEDORA_HOME/solr/$SOLR_DEFAULT_CORE_PATH/data

chown -R $FEDORA_USER:$FEDORA_USER $FEDORA_HOME

cd $CATALINA_HOME/webapps/fedoragsearch/FgsConfig

ant generateIndexingXslt:

ant -f fgsconfig-basic.xml -Dlocal.FEDORA_HOME=$FEDORA_HOME -DgsearchUser=$FEDORA_ADMIN_USER -DgsearchPass=$FEDORA_ADMIN_PASS -DfinalConfigPath=$CATALINA_HOME/webapps/fedoragsearch/WEB-INF/classes -DlogFilePath=$FEDORA_HOME/server/logs -DfedoraUser=$FEDORA_ADMIN_USER -DfedoraPass=$FEDORA_ADMIN_PASS -DobjectStoreBase=$FEDORA_HOME/data/objectStore -DindexDir=$FEDORA_HOME/solr/$SOLR_DEFAULT_CORE_PATH/data/index -DindexingDocXslt=foxmlToSolr -propertyfile fgsconfig-basic-for-islandora.properties
mv $FEDORA_HOME/solr/$SOLR_DEFAULT_CORE_PATH/conf/schema.xml $FEDORA_HOME/solr/$SOLR_DEFAULT_CORE_PATH/conf/schema.xml.bak  

mv $FEDORA_HOME/solr/$SOLR_DEFAULT_CORE_PATH/conf/solrconfig.xml $FEDORA_HOME/solr/$SOLR_DEFAULT_CORE_PATH/conf/solrconfig.xml.bak  

mv $CATALINA_HOME/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/foxmlToSolr.xslt $CATALINA_HOME/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/foxmlToSolr.xslt.bak 
Solr Configuration
cd ~

git clone --recursive git://github.com/discoverygarden/basic-solr-config.git 

cd basic-solr-config 

git checkout 4.x

mv ~/basic-solr-config/conf/* $FEDORA_HOME/solr/$SOLR_DEFAULT_CORE_PATH/conf  

mv ~/basic-solr-config/islandora_transforms $CATALINA_HOME/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex

mv ~/basic-solr-config/foxmlToSolr.xslt $CATALINA_HOME/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/foxmlToSolr.xslt  

cp ~/basic-solr-config/index.properties $CATALINA_HOME/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/index.properties

cd ~ 

rm -rf ~/basic-solr-config  

rm -rf ~/$SOLR_NAME* 

cd $CATALINA_HOME/webapps/fedoragsearch/WEB-INF/lib  

wget https://github.com/discoverygarden/dgi_gsearch_extensions/releases/download/v0.1.1/gsearch_extensions-0.1.1-jar-with-dependencies.jar -O gsearch_extensions-0.1.1-jar-with-dependencies.jar

Adore-Djatoka

Install djatoka war:

cp /opt/adore-djatoka/dist/adore-djatoka.war $CATALINA_HOME/webapps && unzip -o /opt/adore-djatoka/dist/adore-djatoka.war -d $CATALINA_HOME/webapps/adore-djatoka

Setup Logging

Please note logging still needs some TLC log4j and logrotate clash with some files:

cd ~ 

git clone https://github.com/discoverygarden/islandora_log_config.git

cd islandora_log_config

cp islandora_logrotate /etc/logrotate.d/  

cp log4j.xml /usr/local/fedora/tomcat/webapps/fedoragsearch/WEB-INF/classes/log4j.xml 

cp logging.properties /usr/local/fedora/tomcat/conf/logging.properties

cp log4j.properties /usr/local/fedora/tomcat/webapps/adore-djatoka/WEB-INF/classes/log4j.properties

cp logback.xml /usr/local/fedora/server/config/logback.xml

chown -R $FEDORA_USER:$FEDORA_USER $FEDORA_HOME 

Drupal Filter <a id=drupal-filter">

Setup Drupal filter:

cd $CATALINA_HOME/webapps/fedora/WEB-INF/lib 

wget --no-check-certificate $DRUPAL_FILTER_URL 

echo -e 'fedora-auth\n{\n\torg.fcrepo.server.security.jaas.auth.module.XmlUsersFileModule required\n\tdebug=true; \n\tca.upei.roblib.fedora.servletfilter.DrupalAuthModule required\n\tdebug=true; \n};\n\nfedora-auth-xmlusersfile\n{\n\torg.fcrepo.server.security.jaas.auth.module.XmlUsersFileModule required\n\tdebug=true;\n};\n\nfedora-auth-ldap-bind\n{\n\torg.fcrepo.server.security.jaas.auth.module.LdapModule required\n\thost.url="ldap://dev01.muradora.org"\n\tauth.type="simple"\n\tbind.mode="bind"\n\tbind.filter="uid={0},ou=people,dc=muradora,dc=org"\n\tdebug=true;\n};\n\nfedora-auth-ldap-bind-search-bind\n{\n\torg.fcrepo.server.security.jaas.auth.module.LdapModule required\n\thost.url="ldap://dev01.muradora.org"\n\tauth.type="simple"\n\tbind.mode="bind-search-bind"\n\tbind.user="uid=binduser,ou=people,dc=muradora,dc=org"\n\tbind.pass="murabind"\n\tsearch.base="ou=people,dc=muradora,dc=org"\n\tsearch.filter="(uid={0})"\n\tattrs.fetch="cn,sn,mail,displayName,carLicense"\n\tdebug=true;\n};\n\nfedora-auth-ldap-bind-search-compare\n{\n\torg.fcrepo.server.security.jaas.auth.module.LdapModule required\n\thost.url="ldap://dev01.muradora.org"\n\tauth.type="simple"\n\tbind.mode="bind-search-compare"\n\tbind.user="uid=binduser,ou=people,dc=muradora,dc=org"\n\tbind.pass="murabind"\n\tsearch.base="ou=people,dc=muradora,dc=org"\n\tsearch.filter="(uid={0})"\n\tattrs.fetch="cn,sn,mail,displayName,carLicense"\n\tdebug=true;\n};' > $FEDORA_HOME/server/config/jaas.conf 

cd $FEDORA_HOME/server/config  

wget https://raw.github.com/Islandora/islandora_drupal_filter/master/filter-drupal.xml 

sed -i "s|DB_SERVER|$DB_SERVER|g" $FEDORA_HOME/server/config/filter-drupal.xml

sed -i "s|DRUPAL_DB_NAME|$DRUPAL_DB_NAME|g" $FEDORA_HOME/server/config/filter-drupal.xml

sed -i "s|DRUPAL_DB_USER|$DRUPAL_DB_USER|g" $FEDORA_HOME/server/config/filter-drupal.xml 

sed -i "s|DRUPAL_DB_PASS|$DRUPAL_DB_PASS|g" $FEDORA_HOME/server/config/filter-drupal.xml

Start fedora fully configured:

chown -R $FEDORA_USER:$FEDORA_USER $FEDORA_HOME

service tomcat start

Install Drupal

CLUI Config

Note: Some libraries, modules, contrib are optional but are included to support additional theming. e.g. features, strongarm, node_export.

cd $OS_DEFAULT_DOCUMENTROOT
drush dl drupal  

mv drupal-7* drupal7

cd drupal7

mkdir sites/default/files  

mkdir sites/all/{modules,themes,libraries}

cp sites/default/default.settings.php sites/default/settings.php

cd sites/all/modules  

Islandora Modules

Git clone the following modules:

modslist.sh helper script to handle the git repos to install modules:

#!/bin/bash
cd /var/www/drupal7/sites/all/modules

git clone https://github.com/Islandora/islandora.git

git clone https://github.com/Islandora/islandora_scholar.git

git clone https://github.com/discoverygarden/google_analytics_reports.git

git clone https://github.com/discoverygarden/islandora_ga_reports.git

git clone https://github.com/Islandora/islandora_solr_search.git

git clone https://github.com/Islandora/islandora_solr_views.git

git clone https://github.com/Islandora/islandora_solution_pack_collection.git

git clone https://github.com/Islandora/objective_forms.git

git clone https://github.com/Islandora/islandora_xml_forms.git

git clone https://github.com/Islandora/php_lib.git

git clone https://github.com/Islandora/islandora_importer.git

git clone https://github.com/Islandora/islandora_bookmark.git

git clone https://github.com/Islandora/islandora_oai.git

git clone https://github.com/Islandora/islandora_solution_pack_audio.git

git clone https://github.com/Islandora/islandora_solution_pack_book.git

git clone https://github.com/Islandora/islandora_solution_pack_image.git

git clone https://github.com/Islandora/islandora_solution_pack_large_image.git

git clone https://github.com/Islandora/islandora_solution_pack_pdf.git

git clone https://github.com/Islandora/islandora_solution_pack_video.git

git clone https://github.com/Islandora/islandora_paged_content.git

git clone https://github.com/Islandora/islandora_internet_archive_bookreader.git

git clone https://github.com/Islandora/islandora_ocr.git

git clone https://github.com/Islandora/islandora_openseadragon.git

git clone https://github.com/Islandora/islandora_videojs.git

git clone https://github.com/Islandora/islandora_fits.git

git clone https://github.com/Islandora/islandora_simple_workflow.git

git clone https://github.com/Islandora/islandora_book_batch.git

git clone https://github.com/Islandora/islandora_batch.git

git clone https://github.com/Islandora/islandora_ip_embargo.git

git clone https://github.com/Islandora/islandora_solution_pack_compound.git

git clone https://github.com/Islandora/islandora_solution_pack_newspaper.git

git clone https://github.com/Islandora/islandora_xacml_editor.git

git clone https://github.com/Islandora/islandora_marcxml.git

git clone https://github.com/discoverygarden/islandora_featured_collection.git

git clone https://github.com/Islandora/islandora_solr_metadata.git

git clone https://github.com/discoverygarden/solrmetadataconfigs.git

git clone https://github.com/discoverygarden/islandora_solution_pack_document.git

git clone https://github.com/discoverygarden/islandora_jodconverter.git

git clone https://github.com/discoverygarden/islandora_plupload.git

git clone https://github.com/discoverygarden/islandora_solution_pack_entities.git

git clone https://github.com/Islandora-Labs/islandora_binary_object.git

git clone https://github.com/Islandora/islandora_checksum.git

git clone https://github.com/Islandora/islandora_checksum_checker.git

git clone https://github.com/Islandora/islandora_premis.git

git clone https://github.com/Islandora/islandora_bagit.git

Dependency if bagit is to be used:

/usr/bin/pear install Archive_Tar

Libraries

cd /var/www/drupal7/sites/all/libraries/

git clone -b $TUQUE_BRANCH git://github.com/Islandora/tuque.git

git clone  https://github.com/Islandora/internet_archive_bookreader.git bookreader

wget http://openseadragon.github.io/releases/openseadragon-bin-0.9.129.zip  && unzip openseadragon-bin-0.9.129.zip && rm -rf openseadragon-bin-0.9.129.zip && mv openseadragon-bin-0.9.129 openseadragon

wget https://github.com/moxiecode/plupload/archive/v1.5.8.zip -O v1.5.8.zip && unzip -o v1.5.8.zip && rm -rf v1.5.8.zip && mv plupload-1.5.8 plupload

wget http://sourceforge.net/projects/jodconverter/files/JODConverter/2.2.2/jodconverter-2.2.2.zip/download -O jodconverter-2.2.2.zip && unzip -o jodconverter-2.2.2.zip && rm -rf jodconverter-2.2.2.zip

mkdir jquery.cycle && cd jquery.cycle && wget http://malsup.github.com/jquery.cycle.all.js  

drush dl imagemagick libraries views ctools oauth chart google_analytics views_slideshow views_responsive_grid strongarm features designkit conditional_styles socialmedia widgets features_extra uuid node_export block_class ldap entity colorbox rules xmlsitemap css_injector

Drupal site install

Please note that you should consider making the drupal directory permissions more secure. These permissions will allow you to install modules through the drupal web interface however if this functionality is not required recommend locking down permissions using something such as https://github.com/discoverygarden/secure_drupal_file after the install.

chown -R $APACHE_USER:$APACHE_USER /var/www/drupal7 

drush -y site-install standard --account-name=$DRUPAL_ADMIN_USER --account-pass=$DRUPAL_ADMIN_PASS --db-url=mysql://$DRUPAL_DB_USER:$DRUPAL_DB_PASS@localhost/$DRUPAL_DB_NAME
Secure settings.php

chmod 440 /var/www/drupal7/sites/default/settings.php

Drush Enables and Configuration
drush -y en block color comment contextual dashboard dblog field field_sql_storage field_ui file filter help image list menu node number options overlay path rdf shortcut system taxonomy text toolbar user bartik seven imagemagick libraries views update ctools oauth_common oauth_common_providerui system_charts chart_views chart googleanalytics views_responsive_grid strongarm features designkit conditional_styles fe_block uuid node_export node_export_features widgets socialmedia block_class colorbox rules entity_token css_injector

drush -y colorbox-plugin

drush -y dis overlay

drush vset islandora_base_url "$ISLANDORA_BASE"

drush vset islandora_solr_url "$SOLR_BASE"

drush -y --user=1 en islandora islandora_audio islandora_basic_collection islandora_basic_image islandora_fits islandora_importer islandora_openseadragon islandora_simple_workflow islandora_video islandora_videojs islandora_pdf  islandora_paged_content islandora_ocr islandora_internet_archive_bookreader islandora_large_image islandora_book islandora_batch islandora_book_batch xml_form_api xml_form_elements xml_schema_api objective_forms php_lib islandora_solr islandora_solr_config islandora_solr_views islandora_ga_reports islandora_scholar islandora_oai google_analytics_reports islandora_importer xml_form_builder xml_forms islandora_bibliography islandora_scholar_embargo islandora_google_scholar islandora_marcxml islandora_xacml_editor islandora_xacml_api zip_importer pmid_importer ris_importer islandora_bookmark doi_importer endnotexml_importer citation_exporter bartik seven imagemagick libraries views views_ui ctools csl citeproc oauth_common oauth_common_providerui system_charts chart_views chart googleanalytics islandora_compound_object islandora_ip_embargo islandora_newspaper views_slideshow views_slideshow_cycle islandora_featured_collection islandora_solr_metadata islandora_document islandora_jodconverter islandora_entities islandora_entities_csv_import islandora_binary_object

drush -y videojs-plugin

drush php-eval "variable_set('islandora_large_image_viewers', array('name' => array('none' => 'none', 'islandora_openseadragon' => 'islandora_openseadragon'),'default' => 'islandora_openseadragon'));"

drush php-eval "variable_set('islandora_video_viewers', array('name' => array('none' => 'none', 'islandora_videojs' => 'islandora_videojs'),'default' => 'islandora_videojs'));"

drush php-eval "variable_set('islandora_audio_viewers', array('name' => array('none' => 'none', 'islandora_videojs' => 'islandora_videojs'),'default' => 'islandora_videojs'));"

drush php-eval "variable_set('islandora_book_viewers', array('name' => array('none' => 'none', 'islandora_internet_archive_bookreader' => 'islandora_internet_archive_bookreader'), 'default' => 'islandora_internet_archive_bookreader'));"

drush php-eval "variable_set('islandora_book_page_viewers', array('name' => array('none' => 'none', 'islandora_openseadragon' => 'islandora_openseadragon'), 'default' => 'islandora_openseadragon'));"

drush php-eval "variable_set('islandora_newspaper_page_viewers', array('name' => array('none' => 'none', 'islandora_openseadragon' => 'islandora_openseadragon'),'default' => 'islandora_openseadragon'))"

drush php-eval "variable_set('islandora_newspaper_issue_viewers', array('name' => array('none' => 'none', 'islandora_internet_archive_bookreader' => 'islandora_internet_archive_bookreader'),'default' => 'islandora_internet_archive_bookreader'))"

drush vset islandora_pdf_create_fulltext "1"

drush vset islandora_pdf_path_to_pdftotext `which pdftotext`  

drush vset islandora_metadata_display "islandora_solr_metadata"

drush vset islandora_fits_executable_path "/opt/fits/fits.sh" 

drush vset islandora_book_tesseract `which tesseract`  

drush vset islandora_ocr_tesseract `which tesseract`  

drush vset islandora_batch_java `which java`  

drush php-eval "variable_set('islandora_ocr_tesseract_enabled_languages', array('deu-frak' => deu-frak, 'eng' => eng, 'fra' => fra, 'ita' => ita, 'jpn' => jpn, 'por' => por, 'spa' => spa, 'dan-frak' => 0, 'deu' => 0, 'hin' => 0, 'ita_old' => 0, 'rus' => 0, 'slk-frak' => 0, 'spa_old' => 0));"

drush vset islandora_lame_url `which lame`

drush vset islandora_video_ffmpeg_path `which ffmpeg`  

drush vset islandora_video_ffmpeg2theora_path `which ffmpeg2theora`  

drush vset islandora_paged_content_gs `which gs`  

drush vset imagemagick_convert `which convert`

drush vset islandora_document_create_fulltext "1"  

drush vset islandora_document_path_to_pdftotext `which pdftotext`

drush vset site_name "Stock Islandora and Fedora"

drush vset image_toolkit "imagemagick"

drush vset error_level $ERROR_LEVEL

drush php-eval "variable_set('oai2_date_field', 'fgs_lastModifiedDate_dt')"

Note: This should be a publicly resolvable URL or viewers will not work for people who cannot resolve the name. You also should ensure that the /etc/hosts file is pointing the name at localhost.

drush vset islandora_paged_content_djatoka_url $DNS_HOSTNAME

drush vset user_register 0 

drush -y updb

drush -y cc all

service $APACHE_SERVICE restart

drush cc all

Follow-up Notes

More tesseract languages can be found here: https://code.google.com/p/tesseract-ocr/downloads/list

Recommend locking down Drupal permissons on Production e.g. run something like https://github.com/discoverygarden/secure_drupal_file

Keep server firewalled in production! Don't expose any ports to the Internet asides from 80/443. Port 8080 should be kept locked down to localhost only.

Visit https://github.com/discoverygarden/basic-solr-config/wiki/performance-tuning-for-multithreaded-solr-ingest if interested in multi fgsupdaters. Be warned this can drastically increase HEAP pressure if not careful.