-
Notifications
You must be signed in to change notification settings - Fork 64
Installation instructions for Kitodo.Production 2.3.1
Kitodo is software. You need a computer to run it. Because Kitodo deeply interferes with the operating system and bends many things, even during operation, it is recommended to dedicate an entire server to Kitodo. This is also because Kitodo requires quite a lot of system resources.
This guide describes how to install Kitodo on an Ubuntu Linux system. You can deviate from these suggestions in any direction.
We assume that you have already prepared an empty Ubuntu Linux server. If not, do so now. Take the latest long-term support version (typically marked with the letters LTS), because your digitization projects will surely take a while.
For most of the following steps you need elevated (administrator) privileges. You can run the sudo su command to get them. Enter your password. You can also execute the individual commands with sudo, but you will notice this is a fizzy job.
You need to install many additional packages. Do this:
apt-get update
apt-get install mysql-server tomcat8 slapd ldap-utils samba apache2 libapache2-mod-jk libnss-ldap
Some of the packages require further settings and the setup will ask for these during the installation. Answer the questions wisely, it is very difficult to repeat this step if things go wrong. You need the following settings:
For MySQL you only need an administrator password.
| Setting | Type | Value |
|---|---|---|
| No configuration | Boolean | False |
| Domain | String | Here we need the so-called fully qualified domain name (FQDN). To do this, enter the hostname -f command. In your mind, replace each period with a , and prefix each segment with dc=.Example: The command reports: kitprod.subnet.myorganization.orgThe FQDN is: dc=kitprod,dc=subnet,dc=myorganization,dc=orgWe refer to this string as #FQDN# below. |
| Organization | String | Here you can enter something about how the server calls itself to the outside. (This is essentially meaningless.) For example: My organization Kitodo Production LDAP Server |
| Password | Password | You enter an administrator password for the LDAP server. You will need to store this password later in plain text in a configuration file, so do not choose a valuable password. If in doubt, use kitodo. |
| Backend | Select | MDB |
| Purge database | Boolean | False |
| Move old database | Boolean | True |
| Allow LDAP v.2 | Boolean | False |
| Setting | Type | Value |
|---|---|---|
| DB root login | Boolean | False |
| PAM password | Select | MD5 |
| Move to debconf | Boolean | True |
| LDAP server | String | ldapi:/// |
| Base DN | String |
ou=users,#FQDN# (as explained above in the LDAP settings)Example: ou=users,dc=kitprod,dc=subnet,dc=myorganization,dc=org
|
| Override | Boolean | True |
| LDAP version | Select | 3 |
| DB login | Boolean | False |
Digitization produces many data. We recommend that you save this data on a separate hard drive. The hard disk can also be in a network. Configure the hard disk in file /etc/fstab and make sure that you mounted it in the operating system. With a physical hard drive, you should now consider whether you want to install logical volume management (LVM). Then later it will be easier to enlarge the hard disk if you run out of space. With a virtual server, it is better to manage the size via the virtualization solution.
In this documentation, we assume that you have provided the hard drive for the data under path /mnt/data.
Kitodo is an open source software. You can get it free of charge from the Internet and use it free of license fees. It is necessary to stick to the license conditions.
You can find the latest release of Kitodo on the Github homepage. To download it, use the following command:
wget -O kitodo-production.war https://github.com/kitodo/kitodo-production/releases/download/kitodo-production-2.3.1/kitodo-production-2.3.1.war
If you want to use the Active MQ interface of Kitodo, you also need the ActiveMQ software. You can download it in the same way:
wget -O apache-activemq-5.15.11-bin.tar.gz http://www.apache.org/dyn/closer.cgi?filename=/activemq/5.15.11/apache-activemq-5.15.11-bin.tar.gz&action=download
Kitodo stores some processing data in a database. But that is relatively small, that means neither pictures nor the descriptive data (“metadata”) is stored there. This means that it is not worth running the database on a separate system. Before you can let the application run, you have to set up this database. It works like this:
Make sure the MySQL server is running. You get the information whether it is already running with the command: service mysql status If not, start it with the command: service mysql start
Log on to the MySQL server with the command: mysql -u root -p It will ask you for the password. Name the password that you have previously defined for MySQL Administration. Issue the following commands to create a database, create a user, and allow it to write on the database:
CREATE DATABASE kitodo;
CREATE USER 'kitodo'@'localhost' IDENTIFIED BY 'kitodo';
GRANT ALL ON kitodo.* TO 'kitodo'@'localhost';
FLUSH PRIVILEGES;
You exit MySQL dialog mode with: exit;
If you understand a little bit of what you did there, you saw that we use kitodo as a password. The web application is preconfigured in this way. You can also choose a different password here. In this case, you will also have to adapt the hibernate.cfg.xml file in the /var/lib/tomcat8/webapps/kitodo-production/WEB-INF/classes folder later. The password must be entered in plain text in this file, so it is little security gain to change this.
The Kitodo.Production database content can now move into the database. It is in the file that we have already downloaded. The move-in takes place with the following commands:
unzip -p kitodo-production.war database-setup/schema.sql | mysql -D kitodo -u kitodo -p
unzip -p kitodo-production.war database-setup/default.sql | mysql -D kitodo -u kitodo -p
If the question is about the password, this is kitodo, except you changed it. If this results in an error, then the unzip program is not yet installed on the computer. In this case, issue this command to install it: apt-get install unzip
We have found that problems arise in Samba drive share access when usernames in Kitodo contain uppercase letters. Kitodo exemplary users in the database have a capital letter in the middle (aka. camel case) and that causes problems. That is why you should make it lowercase and this is how it works: You log in to MySQL again with the mysql -D kitodo -u kitodo -p command. If the question is about the password, that is kitodo. Then issue the following command to find out the character set name:
SELECT character_set_name FROM information_schema.COLUMNS WHERE table_schema = "kitodo"
AND table_name = "benutzer" AND column_name = "login";
Then issue this command and use in place of #…# what came out of the previous command:
UPDATE benutzer SET login = LOWER(login) WHERE LOWER(login) != login COLLATE #…#;
Another problem with the database is spaces in the names of projects, which can also lead to incomprehensible crashes and should be remoevd. It works like this:
UPDATE projekte SET titel = REPLACE(titel, ' ', '_');
You get out of there with: exit; The database is now ready.
The LDAP server has to run for the next step: This works just like MySQL (or any services). You get the information whether it is already on duty with the command: service slapd status If not, run it with the command: service slapd start
It works like this:
zcat /usr/share/doc/samba/examples/LDAP/samba.ldif.gz | ldapadd -Q -Y EXTERNAL -H ldapi:///
The LDAP installation produced something called root distinguished name (or RDN, for short). We need it many times below and have to find it out. We can get it from the output of the command: slapcat. That gives us the entire LDAP address book. An empty line separates the individual entries, you can see that. We need the entry in which the line is: objectClass: top. And there what is in the dn: field. That is the magical value, which we will refer to as #RDN# below.
The concept of LDAP, as used in Kitodo, assumes that our organization consists of organizational units, and one of them is the users of your Kitodo.Production server. You can think of an organizational unit as a subfolder in which the users are then placed.
The easiest way is to create a small text file, e.g. users.ldif with the following content:
dn: ou=users,#RDN#
objectClass: organizationalUnit
ou: users
The RDN that we have determined above is to be used here. We then place the file in LDAP with the following command. The root DN is also used here. *** stands for the LDAP administrator password, which you specified above:
ldapadd -x -D cn=admin,#RDN# -w *** < users.ldif
The different users in Linux are assigned to different groups. One of these groups identifies all human users and is called users. The user groups are represented by numbers and we now have to find out the number of the user group for users. You can do this with the command:
grep '^users:' /etc/group | cut -d: -f3
We refer to this number as #usergroupnumber# below.
Linux is designed from the ground up so that multiple users can work on the same computer (even simultaneously). That’s why you have to log in at the beginning before you can do anything on the computer. The various users are identified by numbers inside the Linux operating system. Numbers below 1000 stand for "system users" (e.g. printers, or the email system) and numbers from 1000 upwards for the human users of the system. We now have to find out which user has the largest user number. The next free user number is logically one larger.
Basic Linux user management is very simple, the existing users are simply in file /etc/passwd. The file is a table, with the columns separated by colons. The login name is in the first column, there is usually only an x in the second column, and the user number is in the third column. We are looking for the largest user number and add one to it, so we have the #nextfreeusernumber#, which we will also need in the next step.
The NextFreeUnixId is an LDAP entry for the currently next free user number (which we have just found out). We have to install it there now. For this we create a file again, for example nextFreeUnixId.ldif with the following content:
dn: cn=NextFreeUnixId,#RDN#
objectClass: inetOrgPerson
sn: NextFreeUnixId
objectClass: posixAccount
cn: NextFreeUnixId
uid: nextfreeunixid
uidNumber: #nextfreeusernumber#
gidNumber: #usergroupnumber#
homeDirectory: /dev/null
The corresponding values that we have previously determined must be entered in the #…# areas. We also put this file in LDAP, as before:
ldapadd -x -D cn=admin,#RDN# -w *** < nextFreeUnixId.ldif
The standard settings of Tomcat are in file /etc/default/tomcat8 and we have to edit them. You can do that with an editor of your choice. You should create the situation that JAVA_OPTS contains the following:
"-Djava.awt.headless=true -Xmx4G -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC"
That means the server cannot open windows, give more memory to it, more memory to load the program, too, and better garbage collection. The user experience will benefit significantly from this.
The Tomcat server has to run for the next step: (You already know that:) You get the information whether it is already running with the command: service tomcat8 status If not, start it with the command: service tomcat8 start
We now copy the WAR file into the Tomcat application directory, so the web application is started:
cp kitodo-production.war /var/lib/tomcat8/webapps/
The program needs many folders for everything. We are going to create them now:
mkdir -p /usr/local/kitodo
mkdir -p /etc/kitodo/config
mkdir /etc/kitodo/messages
mkdir /mnt/data/metadata
mkdir /usr/local/kitodo/webdav
mkdir /usr/local/kitodo/xslt
mkdir /usr/local/kitodo/success
mkdir /usr/local/kitodo/error
mkdir -p /mnt/data/var/hotfolder
Kitodo needs many files that were unpacked when the web application was first started and are now lying around in the application directory. We will now move and copy them to the right places:
cd /var/lib/tomcat8/webapps/
mv WEB-INF/classes/goobi_digitalCollections.xml /etc/kitodo/config
mv WEB-INF/classes/goobi_metadataDisplayRules.xml /etc/kitodo/config
mv WEB-INF/classes/goobi_opac.xml /etc/kitodo/config
mv WEB-INF/classes/goobi_processProperties.xml /etc/kitodo/config
mv WEB-INF/classes/goobi_projects.xml /etc/kitodo/config
mv WEB-INF/classes/goobi_webapi.xml /etc/kitodo/config
mv WEB-INF/classes/kitodo_mods_opac.xml /etc/kitodo/config
mv WEB-INF/classes/modules.xml /etc/kitodo/config
mv rulesets /etc/kitodo
mv scripts /usr/local/kitodo
mv WEB-INF/classes/*.xsl /etc/kitodo/xslt
cp WEB-INF/classes/messages/messages_*.properties /etc/kitodo/messages
The Tomcat must be able to read in many directories and also be able to write in some and run some scripts. For WebDAV, Apache must be able to write in the webdav directory:
chown tomcat8 /etc/kitodo/config /etc/kitodo/messages /etc/kitodo/rulesets
chown tomcat8 /usr/local/kitodo/scripts /etc/kitodo/xslt /usr/local/kitodo/success
chown tomcat8 /usr/local/kitodo/error /mnt/data/metadata /mnt/data/var/hotfolder
chown www-data /usr/local/kitodo/webdav
chmod u+x /usr/local/kitodo/scripts/*.sh
chmod u+rx /etc/kitodo/config /etc/kitodo/messages /etc/kitodo/rulesets
chmod u+rx /usr/local/kitodo/scripts /etc/kitodo/xslt /usr/local/kitodo/success
chmod u+rx /usr/local/kitodo/error
chmod u+rwx /mnt/data/metadata /mnt/data/var/zedExporter/new /mnt/data/var/hotfolder
Two scripts have to be edited to execute some commands with sudo, see the following examples. Add the block of commands below the comment to automatically configures WebDAV access for new users. You can do this with an editor of your choice:
script_createDirUserHome.sh
User="$1"
Home="$2"
sudo /bin/mkdir "$Home"
sudo /bin/chmod g+w "$Home"
sudo /bin/chown $User "$Home"
sudo /bin/chgrp tomcat "$Home"
# Write conf file for WebDAV access
THIS=$(readlink -f "$0")
SCRIPTPATH=$(dirname "$THIS")
REPLACE='s/XXXUSERXXX/'"$1"'/g'
sed "$REPLACE" $SCRIPTPATH/davuser.conf.tmpl > /tmp/webdav-"$1".conf
sudo /bin/mv /tmp/webdav-"$1".conf /etc/apache2/conf-enabled/
sudo /usr/sbin/service apache2 gracefulscript_createSymLink.sh
Source="$1"
Target="$2"
User="$3"
/bin/ln -vs "$Source" "$Target"
sudo /bin/chown -vR "$User" "$Source"For WebDAV we need a user group in which both Tomcat and Apache are members:
groupadd webdav
usermod -a -G webdav tomcat8
usermod -a -G webdav www-data
In folder /usr/local/kitodo/scripts we have to create the following file:
davuser.conf.tmpl
<IfModule mod_dav_fs.c>
DavLockDB /usr/local/kitodo/webdav/DavLockDB
</IfModule>
<IfModule mod_dav.c>
LimitXMLRequestBody 10240
LimitRequestBody 0
BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully
BrowserMatch "^Microsoft-WebDAV-MiniRedir" redirect-carefully
Alias /webdav/XXXUSERXXX "/home/XXXUSERXXX"
<Directory /home/XXXUSERXXX>
Dav on
AddDefaultCharset utf-8
Options Indexes FollowSymLinks
AuthType Basic
AuthName "XXXUSERXXX WebDAV realm"
AuthBasicProvider ldap
AuthLDAPUrl "ldap://localhost/ou=users,#RDN#"
BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully
BrowserMatch "^Microsoft-WebDAV-MiniRedir" redirect-carefully
<LimitExcept OPTIONS>
Require user XXXUSERXXX
</LimitExcept>
Order allow,deny
Allow from all
</Directory>
</IfModule>
With the command sudoedit /etc/sudoers.d/kitodo we create a file with the following content:
User_Alias TOMCAT_USER = tomcat8
Cmnd_Alias CHOWN_METADATA = /bin/chown * /mnt/data/metadata/*
Cmnd_Alias CHGRP_USERS = /bin/chgrp webdav /home/*
Cmnd_Alias CHMOD_USERS = /bin/chmod g+w /home/*
Cmnd_Alias CHOWN_USERS = /bin/chown * /home/*
Cmnd_Alias MKDIR_USERS = /bin/mkdir /home/*
Cmnd_Alias MV_WEBDAV = /bin/mv /tmp/* /etc/apache2/conf-enabled/
Cmnd_Alias RESTART_APACHE = /usr/sbin/service apache2 graceful
TOMCAT_USER ALL=NOPASSWD: CHOWN_METADATA, CHGRP_USERS, CHMOD_USERS, CHOWN_USERS, MKDIR_USERS, MV_WEBDAV, RESTART_APACHE
If you want to use the Active MQ API you need the active message queue. The package is so inherently broken on Ubuntu that we install it separately. We have already downloaded the program above.
Under Linux it is recommended to let the various servers run in the system with their own user IDs. We are now creating a user for the Active MQ server:
groupadd -r activemq
useradd -g activemq -d /usr/local/activemq activemq
We create a folder for the server home and unpack the downloaded server there:
mkdir /usr/local/activemq
tar -zxf apache-activemq-5.15.11-bin.tar.gz -C /usr/local/activemq --strip-components=1 apache-activemq-5.15.11
In order to shut down the Active MQ server, it must be able to speak to itself. We allow this in the configuration file /usr/local/activemq/conf/activemq.xml by setting the <managementContext createConnector="false"> to true.
In section <policyEntries> of the same file, we add the following:
<policyEntry topic="KitodoProduction.ResultMessages.Topic">
<subscriptionRecoveryPolicy>
<timedSubscriptionRecoveryPolicy recoverDuration="604800000" />
</subscriptionRecoveryPolicy>
</policyEntry>The number is the duration of a week in milliseconds, it is the time how long log messages are kept around on the Active MQ server before they are forgotten.
Everything in the Active MQ directory must belong to the Active MQ user for this to work. Issue the following command:
chown -R activemq:activemq /usr/local/activemq
So that the operating system treats the server as a service, create a file /etc/systemd/system/activemq.service with the following content:
[Unit]
Description=Active MQ Server
After=network.target
[Service]
Type=forking
WorkingDirectory=/usr/local/activemq/bin
ExecStart=/usr/local/activemq/bin/activemq start
ExecStop=/usr/local/activemq/bin/activemq stop
Restart=on-abort
User=activemq
Group=activemq
[Install]
WantedBy=multi-user.targetLoad the file into the operating system with the systemctl --system daemon-reload command. Issue command systemctl enable activemq so that the server is sent to work each time the system is started.
The configuration file of the web application is /var/lib/tomcat8/webapps/WEB-INF/classes/goobi_config.properties. A whole set of values must be adjusted here:
KonfigurationVerzeichnis=/etc/kitodo/config/
RegelsaetzeVerzeichnis=/etc/kitodo/rulesets/
xsltFolder=/etc/kitodo/xslt/
MetadatenVerzeichnis=/mnt/data/metadata/
dir_Users=/home/
tempfolder=/tmp/
localMessages=/etc/kitodo/messages/
debugFolder=/var/log/tomcat8/
pluginFolder=/var/lib/tomcat8/webapps/kitodo-production/plugins/
swapPath=/tmp/
script_createDirUserHome=/usr/local/kitodo/scripts/script_createDirUserHome.sh
script_createDirMeta=/usr/local/kitodo/scripts/script_createDirMeta.sh
script_createSymLink=/usr/local/kitodo/scripts/script_createSymLink.sh
script_deleteSymLink=/usr/local/kitodo/scripts/script_deleteSymLink.sh
ldap_nextFreeUnixId=cn\=NextFreeUnixId,#RDN#
ldap_adminLogin=cn\=Admin,#RDN#
ldap_adminPassword=***
ldap_readonly=false
activeMQ.hostURL=failover:(tcp://localhost:61616?closeAsync=false)
activeMQ.results.topic=KitodoProduction.ResultMessages.Topic
activeMQ.results.timeToLive=604800000
activeMQ.createNewProcess.queue=KitodoProduction.CreateNewProcesses.Queue
activeMQ.finaliseStep.queue=KitodoProduction.FinaliseStep.QueueThe configuration file for logging is /var/lib/tomcat8/webapps/WEB-INF/classes /log4j.properties. Change the path to the log file (log4j.appender.rolling.File) to: /var/log/tomcat8/kitodo.log
The web application is delivered in English and German. If you speak another language and you want the web application in your other language, translate the file messages_en.properties from folder /var/lib/tomcat8/webapps/WEB-INF/classes /messages. Enter the file name messeges_ with the ISO 2-letter language identifier. Now put your language file in folder /var/lib/tomcat8/webapps/WEB-INF/classes/messages and in folder /etc/kitodo/messages/. Yes, this is strange, it takes in both folders, otherwise it won’t work. Then you need to edit the file to configure for faces, /var/lib/tomcat8/webapps/WEB-INF/faces-config.xml: In section <locale-config> add the following with 2-letter language identifier for your language:
<supported-locale>la</supported-locale>
For that the hard disk does not fill up, the log has to be turned over every now and then. This is done by the operating system, if file /etc/logrotate.d/kitodo exists with this content:
/var/log/tomcat8/kitodo.log {
copytruncate
daily
rotate 7
ifempty
compress
compresscmd /usr/bin/gzip
uncompresscmd /usr/bin/gunzip
missingok
nomail
}
The configuration file for this is /etc/samba/smb.conf. Here, we need to use #RDN# again. You already know that. Set the following (commend-out the rest):
[global]
workgroup = KITODO
unix extensions = no
wide links = yes
load printers = no
security = user
invalid users = root
encrypt passwords = yes
passdb backend = ldapsam:ldap://127.0.0.1
ldap suffix = #RDN#
ldap user suffix = ou=users
ldap group suffix = ou=groups
ldap admin dn = cn=admin,#RDN#
ldap ssl = no
ldap passwd sync = yes
ldap delete dn = no
[homes]
comment = Home Directories
path = /home/%U
read only = no
browseable = no
valid users = %S
guest ok = no
inherit permissions = yesSo that Samba can address LDAP, with this command you set the LDAP password in Samba:
smbpasswd -w ***
*** is the password for LDAP, as you chose at the beginning.
AJP is for the Apache speaks to the Jakarta module, with Tomcat as the server behind Jakarta module. To allow the AJP connector, you have to edit file: /etc/tomcat8/server.xml. <Connector port="8009" is in the file and this is commented out. Remove the comment mark to turn it on.
In /etc/apache2/sites-available/000-default.conf, in section <VirtualHost>, add the following for Kidodo.Production. And the same in /etc/apache2/sites-available/default-ssl.conf:
JkMount /kitodo-production ajp13_worker
JkMount /kitodo-production/* ajp13_worker
<Location /kitodo-production>
Order allow,deny
Allow from all
</Location>
You must allow encrypted pages (SSL) with command: a2ensite default-ssl
Then we need a few modules to turn it on:
a2enmod ssl
a2enmod dav
a2enmod dav_fs
a2enmod ldap
a2enmod authnz_ldap
Then we have to go a few more steps in the web application in the web browser.
But for that we have to get that working. This means that we have now restarted the various services, we have configured so that everything fits:
service slapd restart
service activemq restart
service tomcat8 restart
service apache2 restart
systemctl restart smbd.service nmbd.service
We need Localsid right away. Find with command: net getlocalsid
Open web browser window, and then visit address from:
http://server-name/kitodo-production/
Log in with user admin and password kitodo.
Click on LDAP groups in the menu and for table entry on edit (note with pen, far right). Then set the following values. Here you have to add some values (marked #…#). For sambaAcctFlags, that are nine spaces in the clammy, sambaPassword History is 72 0s, and sambaLogonHours is 40 Fs.
| Row | Value |
|---|---|
| Title | Your organization name |
| User DN | sn={login},ou=users,#RDN# |
| gidNumber | #usergroupnumber# |
| objectClasses | inetOrgPerson,posixAccount,shadowAccount,sambaSamAccount |
| sambaSID | #localsid#-{uidnumber*2+1000} |
| sn | {login} |
| uid | {login} |
| description | LDAP group |
| displayName | {full user name} |
| gecos | {login} |
| loginShell | /bin/false |
| sambaAcctFlags | [UX ] |
| sambaLogonScript | _{login}.bat |
| sambaPrimaryGroupSID | #localsid#-100 |
| sambaPwdMustChange | 2147483647 |
| sambaPasswordHistory | 000000000000000000000000000000000000000000000000000000000000000000000000 |
| sambaLogonHours | FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF |
| sambaKickoffTime | 0 |
When that is done, click on "save" below.
Important, leave the web browser window open. Now with an editor, change file /var/lib/tomcat8/webapps/WEB-INF/classes/goobi_config.properties. Set ldap_use=true and save. Now LDAP is used. Still leave the web browser window open.
Now click on Users in the menu, and on Admin, test on edit (the note symbol). Here, next to the LDAP group is a small piece of paper with a key symbol. Click this to add the user to LDAP. Admin can now log in again and you can close the web browser window.