libra2koha - Tools for migrating data from the Libra ILS to Koha.
DATA FROM LIBRA
After you get your data exported from Libra, you get a bunch of .txt files. Most of these follow this pattern:
Something.txt - data from the table Something.
Somethingspec.txt - specification of the columns in the Something table.
Files that do not follow this pattern:
exportCat.txt - bibliographic records in some variataion of mnemonic/line mode MARC format
exportCatMatch.txt - mapping between IdCat (a simple, numeric identifier used to connect e.g. items to records (via Items.IdCat)) and the concatenated contents of 001 and 003. So if you know the 001 and 003 of a record you can concatenate that and look it up in this file, and get the numeric identifier that you can use to look up items connected to the record.
TYPES OF DATA
libra2koha currently handles the following types of data:
Bibliographic records and items
Borrowers (partially, work in progress)
When the data dumped from Libra are going to be used, they are loaded into a MySQL databse, before being extracted and massaged. The scripts are currently hardcoded to use a database on localhost with the following credentials:
databasename = libra2koha
username = libra2koha
password = pass
If you have MySQL installed, you should be able to create the necessary database and user with the following commands:
$ mysql -u root -p mysql> create database libra2koha; mysql> grant all privileges on libra2koha.* to 'libra2koha'@'localhost' identified by 'pass'; mysql> flush privileges;
The scripts in this repository rely on some resources not included in the repo itself:
A Linux environment
The tools have all been successfully run on Ubuntu 14.04.
This is a collection of tools createed by Libriotech. It can be obtained thusly:
git clone https://github.com/Libriotech/LibrioTools.git
The main script in this repository is libra2koha.sh. In theory it is the only script you should need to run directly. It will run all the other scripts and transformations required to do the migration.
The main idea is that you run libra2koha.sh twice. The first time it will complain about some missing config and mapping files, but also produce stubs for those same files. After you have filled in the stub files you run libra2koha.sh again, and the migration produces a set of output files that can be loaded into Koha.
Before you run libra2koha.sh for the first time you need to have two directories set up:
A config directory, which should be empty initially
The directory that contains the data dump from Libra
When you have these set up, you can run libra2koha.sh thusly:
libra2koha.sh /path/to/config /path/to/data
As mentioned earlier, this will complain about missing files, but it will also generate stubs for these files in /path/to/config. When this is done, you have to go through all the stub files and fill in the blanks. This will typically also involve filling Koha with corresponding settings, like values for the LOC and CCODE authorized values.
The format of the generated files will look something like this:
# Generated from /path/to/data/utf8/Departments.txt # 2015-10-14 11:31:40 1: '' # Something 2: '' # Something else 3: '' # Another one
On the left are actual values/IDs from Libra. These should not be changed. On the right are textual explanations of what the values/IDs represent. Your job is to fill in the empty quotes with something that makes sense to Koha. What this is depends on the file:
config.yaml - This is the main config file, and does not follow the pattern described above. Look at the comments and names of variables to figure out if any changes are necessary.
branchcodes.yaml - All the libraries/branches. Define all the libraries/branches you need in Koha, under Administration > Libraries and groups. Fill in the "Code" for each library in the blank quotes.
loc.yaml - Authorized values of the "LOC" category. Fill in the "Authorized value" of each value into the blank quotes.
ccode.yaml - Authorized values of the "CCODE" category. Fill in the "Authorized value" of each value into the blank quotes.
patroncategories.yaml - Go to Administration > Patron categories in Koha and create the categories you need. Fill in the "Code" for each category in the blank quotes.
(There is no checking that the blanks in the stub files have actually been filled in, so make sure you do fill them in before you run libra2koha.sh for the second time. Otherwise, who knows what might happen?)
Itemtypes are often one of the trickiest parts of a migration, and this is certainly true of a Libra to Koha migration. The main reason for this is that Libra stores item type information as codes in given positions of fields 000, 006, 007 and 008. Sometimes it has also proven necessary to use information from 500 and 852 in order to decide on the item type for a record.
Koha on the other hand stores item type information on the item level, in 952$y.
libra2koha has put the logic for figuring out item types into lib/Itemtypes.pm and more specifically, the get_itemtype() subroutine. Given a MARC record, this sub will spit out the item type of that record.
To figure out what item types you need to define in Koha, you should run the itemtypes.pl helper script against the .marcxml file that is created during the first run:
perl itemtypes.pl -i /path/to/data/bib/raw-records.marcxml
This should give you a list of item type codes and their frequencies, from which you should be able to create the necessary list of item types in Koha, under Administration > Item types.
When all the configs and mappings are in place, you should run libra2koha.sh in exactly the same way as earlier:
libra2koha.sh /path/to/config /path/to/data
This should produce a bunch of output files in /path/to/data/out
In thoery, your data should now be fully migrated and ready to be loaded into Koha. In practice you might run into problems that mean you have to tweak the configs and mappings and run libra2koha.sh multiple times to get the desired results. You might even have to tweak the scripts before you are perfectly happy. Your mileage WILL wary!
GETTING THE DATA INTO KOHA
The data needs to be loaded into Koha in the right order:
records.marcxml should go first, and be processed with the bulkmarcimport.pl tool (unless you have just a few records, in which case you can use the Stage/Manage tools in the web UI):
sudo koha-shell -c "/usr/share/koha/bin/migration_tools/bulkmarcimport.pl -b -file records.marcxml -v -commit 100 -m MARCXML -d" <instancename>
.sql files can be loaded with the koha-mysql tool:
sudo koha-mysql <instancename> < borrowers.sql sudo koha-mysql <instancename> < issues.sql
Make sure you finnish it all off with a full reindex:
sudo koha-rebuild-zebra -f -v <instancename>
All the provided Perl files have embedded documentation in the form of POD. To read this documentation, run the files through
These are the scripts that play an active part in the migration:
perldoc records.pl perldoc borrowers.pl perldoc issues.pl
These are used for minor tasks during the migration:
perldoc analyze00x.pl perldoc create_tables.pl perldoc fix_borrowers.pl perldoc itemtypes.pl perldoc table2config.pl
Magnus Enger, Libriotech
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.