Skip to content
Tools for migrating data from the Libra ILS to Koha. WIP!
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
config_sample
lib
.gitignore
README.pod
analyze00x.pl
borrowers.pl
borrowers.tt
create_tables.pl
create_tables.tt
fix_borrowers.pl
issues.pl
issues.tt
itemtypes.pl
libra2koha.sh
records.pl
table2config.pl

README.pod

NAME

libra2koha - Tools for migrating data from the Libra ILS to Koha.

DATA FROM LIBRA

After you get your data exported from Libra, you get a bunch of .txt files. Most of these follow this pattern:

  • Something.txt - data from the table Something.

  • Somethingspec.txt - specification of the columns in the Something table.

Files that do not follow this pattern:

  • exportCat.txt - bibliographic records in some variataion of mnemonic/line mode MARC format

  • exportCatMatch.txt - mapping between IdCat (a simple, numeric identifier used to connect e.g. items to records (via Items.IdCat)) and the concatenated contents of 001 and 003. So if you know the 001 and 003 of a record you can concatenate that and look it up in this file, and get the numeric identifier that you can use to look up items connected to the record.

TYPES OF DATA

libra2koha currently handles the following types of data:

  • Bibliographic records and items

  • Borrowers (partially, work in progress)

  • Active issues/loans

DEPENDENCIES

Database

When the data dumped from Libra are going to be used, they are loaded into a MySQL databse, before being extracted and massaged. The scripts are currently hardcoded to use a database on localhost with the following credentials:

  • databasename = libra2koha

  • username = libra2koha

  • password = pass

If you have MySQL installed, you should be able to create the necessary database and user with the following commands:

$ mysql -u root -p
mysql> create database libra2koha;
mysql> grant all privileges on libra2koha.* to 'libra2koha'@'localhost' identified by 'pass';
mysql> flush privileges;

Other stuff

The scripts in this repository rely on some resources not included in the repo itself:

A Linux environment

The tools have all been successfully run on Ubuntu 14.04.

LibrioTools

This is a collection of tools createed by Libriotech. It can be obtained thusly:

git clone https://github.com/Libriotech/LibrioTools.git

WORKFLOW

The main script in this repository is libra2koha.sh. In theory it is the only script you should need to run directly. It will run all the other scripts and transformations required to do the migration.

The main idea is that you run libra2koha.sh twice. The first time it will complain about some missing config and mapping files, but also produce stubs for those same files. After you have filled in the stub files you run libra2koha.sh again, and the migration produces a set of output files that can be loaded into Koha.

First run

Before you run libra2koha.sh for the first time you need to have two directories set up:

  • A config directory, which should be empty initially

  • The directory that contains the data dump from Libra

When you have these set up, you can run libra2koha.sh thusly:

libra2koha.sh /path/to/config /path/to/data

As mentioned earlier, this will complain about missing files, but it will also generate stubs for these files in /path/to/config. When this is done, you have to go through all the stub files and fill in the blanks. This will typically also involve filling Koha with corresponding settings, like values for the LOC and CCODE authorized values.

The format of the generated files will look something like this:

# Generated from /path/to/data/utf8/Departments.txt
# 2015-10-14 11:31:40

1: '' # Something 
2: '' # Something else
3: '' # Another one

On the left are actual values/IDs from Libra. These should not be changed. On the right are textual explanations of what the values/IDs represent. Your job is to fill in the empty quotes with something that makes sense to Koha. What this is depends on the file:

  • config.yaml - This is the main config file, and does not follow the pattern described above. Look at the comments and names of variables to figure out if any changes are necessary.

  • branchcodes.yaml - All the libraries/branches. Define all the libraries/branches you need in Koha, under Administration > Libraries and groups. Fill in the "Code" for each library in the blank quotes.

  • loc.yaml - Authorized values of the "LOC" category. Fill in the "Authorized value" of each value into the blank quotes.

  • ccode.yaml - Authorized values of the "CCODE" category. Fill in the "Authorized value" of each value into the blank quotes.

  • patroncategories.yaml - Go to Administration > Patron categories in Koha and create the categories you need. Fill in the "Code" for each category in the blank quotes.

(There is no checking that the blanks in the stub files have actually been filled in, so make sure you do fill them in before you run libra2koha.sh for the second time. Otherwise, who knows what might happen?)

Itemtypes

Itemtypes are often one of the trickiest parts of a migration, and this is certainly true of a Libra to Koha migration. The main reason for this is that Libra stores item type information as codes in given positions of fields 000, 006, 007 and 008. Sometimes it has also proven necessary to use information from 500 and 852 in order to decide on the item type for a record.

Koha on the other hand stores item type information on the item level, in 952$y.

libra2koha has put the logic for figuring out item types into lib/Itemtypes.pm and more specifically, the get_itemtype() subroutine. Given a MARC record, this sub will spit out the item type of that record.

To figure out what item types you need to define in Koha, you should run the itemtypes.pl helper script against the .marcxml file that is created during the first run:

perl itemtypes.pl -i /path/to/data/bib/raw-records.marcxml

This should give you a list of item type codes and their frequencies, from which you should be able to create the necessary list of item types in Koha, under Administration > Item types.

Second run

When all the configs and mappings are in place, you should run libra2koha.sh in exactly the same way as earlier:

libra2koha.sh /path/to/config /path/to/data

This should produce a bunch of output files in /path/to/data/out

Caveat

In thoery, your data should now be fully migrated and ready to be loaded into Koha. In practice you might run into problems that mean you have to tweak the configs and mappings and run libra2koha.sh multiple times to get the desired results. You might even have to tweak the scripts before you are perfectly happy. Your mileage WILL wary!

GETTING THE DATA INTO KOHA

The data needs to be loaded into Koha in the right order:

records.marcxml should go first, and be processed with the bulkmarcimport.pl tool (unless you have just a few records, in which case you can use the Stage/Manage tools in the web UI):

sudo koha-shell -c "/usr/share/koha/bin/migration_tools/bulkmarcimport.pl -b -file records.marcxml -v -commit 100 -m MARCXML -d" <instancename>

.sql files can be loaded with the koha-mysql tool:

sudo koha-mysql <instancename> < borrowers.sql
sudo koha-mysql <instancename> < issues.sql

Make sure you finnish it all off with a full reindex:

sudo koha-rebuild-zebra -f -v <instancename>

MORE DOCUMENTATION

All the provided Perl files have embedded documentation in the form of POD. To read this documentation, run the files through perldoc:

Main scripts

These are the scripts that play an active part in the migration:

perldoc records.pl
perldoc borrowers.pl
perldoc issues.pl

Helper scripts

These are used for minor tasks during the migration:

perldoc analyze00x.pl
perldoc create_tables.pl
perldoc fix_borrowers.pl
perldoc itemtypes.pl
perldoc table2config.pl

AUTHOR

Magnus Enger, Libriotech

LICENSE

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

You can’t perform that action at this time.