---------------------------------

 Source library system->Koha Migration Toolbox
 Copyright 2018 NatLibFi
 Copyright 2010 ByWater Solutions
 Copyright 2019 Hypernova Oy

---------------------------------

The files in this repository are free software: you can redistribute 
them and/or modify them under the terms of the GNU General Public 
License as published by the Free Software Foundation, either version 
3 of the License, or (at your option) any later version.

These files are distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

---------------------------------

1) Introduction

    This ETL-program is written by the National Library of Finland and
    Hypernova Oy based on the Voyager scripts by ByWater Solutions.
    These scripts are for use in migration and support of the Koha integrated
    library system. We give them to the community, in the hope that others find
    them useful. Most should be fairly self-explanatory; if you have specific
    questions, please direct them to the committers.

1.1) Credits

    To the National Library of Finland and Ere Maijala for providing the
    excellent USEMARCON FinMARC to MARC21 conversion package.

2) Submitting patches

    No one suggests that these scripts are the be-all,
    end-all of ILS migration tools; if you have enhancements that you are
    willing to share, please send them, and we will include them!
    A pull-request would be the most-welcome way to do that.

3) Supported source library systems

   See config/main.yaml sourceSystemType description

4) Installation instructions

This program is intended to be ran on two separate servers:
- the source library system ("sourceSystem") database server ("sourceDB-server")
- the Koha application server

a) Overview of the program flow:

- A small script and the relevant config is put to the sourceDB-server to
  extract all data
- Koha-server is configured to access the sourceDB-server via ssh and pulls a DB
  dump when needed
- Koha is configured with the desired configurations
- Transformation phase mapping tables and configurations match the configured
  Koha and sourceSystem instances.

- When the migrate-command is given:
-- Koha-server instructs the sourceDB to hand over the data
-- Data is pulled to Koha-server and unzipped
-- Data is transformed using mapping tables and program configurations
-- Transformed Koha-objects are stored to files
-- All configurations in the Koha Staff Client are backed up, and the Koha DB is
   cleared
-- Those files are read by bulk*Import.pl -scripts and relevant business objects
   are written to Koha via the Koha Perl API.
-- ElasticSearch reindexing is triggered.

b) Commands to Run

0. Run INSTALL.sh

NB! If installing module 'Carp::Always::Color' fails, change line

cpanm -L extlib --installdeps .

to

cpanm -L extlib --force --installdeps .

and re-run the script.

0. Install the sourceDB extraction script

- Configure ssh-connection to the sourceDB-server from the Koha-server.
- Copy the "extractor/<sourceSystem>Extractor" directory to the sourceDB-server
  and configure the scripts there as per instructions in
  extractor/<sourceSystem>Extractor/README.
- Configure the extraction automation script. Set the script used to pull the
  sourceSystem data dump to the main configuration at
    ~/MMT-<sourceSystem>/config/main.yaml
  For example implementations take a look at
    extract/<sourceSystem>ToKoha_*

1. Configure mapping tables.

- edit ~/MMT-<sourceSystem>/config/translationTables/<sourceSystem>/

2. Configure Koha

2.1 Follow the Koha-manual's implementation checklist closely

2.2 ExtendedPatronAttributes

- KITT2 needs 'statistic', which is repeatable, non-unique,
  authorized_value-category should be defined and used based on the
  patron_stat.yaml mapping table.
- Hetula needs 'SSN', which is non-repeatable, unique anonymized SSN-key
  reference to the real SSN-value

3. Extract sourceSystem data and download to the Koha-server, unzip to
   <sourceSystem>Export-directory

  see the example pipeline script deployed during INSTALL.sh

3.1. Anonymization

  It is possible to anonymize the Patron data in sourceSystem, to make it's
  handling less sensitive.

  See extractor/<sourceSystem>Extractor/extract.pl --help
  and adjust your pipeline to match

4. transform.pl to transform the data

  see migrate.pl and migrate.sh

  It is possible to run multiple ETL-pipelines by setting the
  MMT_HOME-environment variable.

5. Load to Koha

  see the example pipeline script deployed during INSTALL.sh

5.1 Preserving legacy database IDs

  see loader/KohaLoder/load.sh
  and adjust your pipeline to match