Skip to content
Libris XL
Branch: develop
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
gradle/wrapper Update Gradle to 4.5.1 and adapt subprojects Feb 7, 2018
librisxl-tools Another tiny script adjustment Mar 21, 2019

Libris XL


The project consists of:

  • Applications
    • apix_export/ Exports data from Libris XL back to Voyager (the old system).
    • harvesters/ An OAIPMH harvester. Servlet web application.
    • importers/ Java application to load or reindex data into the system.
    • oaipmh/ Servlet web application. OAIPMH service for Libris XL
    • rest/ A servlet web application. The REST and other HTTP APIs
  • Tools
    • librisxl-tools/ Configuration and scripts used for setup, maintenance and operations.

Related external repositories:

  • The applications above depend on the Whelk Core repository.

  • Core metadata to be loaded is managed in the definitions repository.

  • Also see LXLViewer, our application for viewing and editing the datasets through the REST API.


  1. Gradle

    No setup required. Just use the checked-in gradle wrapper to automatically get the specified version of Gradle and Groovy.

  2. Elasticsearch

    For OS X:

    $ brew install elasticsearch

    For Debian, follow instructions on first, then:

    apt-get install elasticsearch

    For Windows, download and install:

    NOTE: You will also need to set in /etc/elasticsearch/elasticsearch.yml to something unique on the network. This name is later specified when you configure the system. Don't forget to restart Elasticsearch after the change.

    For Elasticsearch version 2.2 and greater, you must also install the delete-by-query plugin. This functionality was removed in ElasticSearch 2.0 and needs to be added as a plugin:

    $ /path/to/elasticsearch/bin/plugin install delete-by-query

    NOTE: You will need to reinstall the plugin whenever you upgrade ElasticSearch.

  3. PostgreSQL (version 9.4 or later)

    # OS X
    $ brew install postgresql
    # Debian
    $ apt-get install postgresql postgresql-client

    Windows: Download and install


Configuring secrets

Use librisxl/ as a starting point:

$ cp
$ vim

Setting up PostgreSQL

  1. Ensure PostgreSQL is started


    $ postgres -D /usr/local/var/postgres
  2. Create database

    # You might need to become the postgres user (e.g. sudo -u postgres bash) first
    $ createdb whelk_dev

    (Optionally, create a database user)

    $ psql whelk_dev
    psql (9.5.4)
    Type "help" for help.
    whelk=# CREATE SCHEMA whelk_dev;
    whelk=# CREATE USER whelk PASSWORD 'whelk';
    whelk=# GRANT ALL ON SCHEMA whelk_dev TO whelk;
    whelk=# GRANT ALL ON ALL TABLES IN SCHEMA whelk_dev TO whelk;
    whelk=# \q

Setting up Elasticsearch

TODO: This is now generated! This step can probably be omitted. (See the devops repo for details.)

Create index and mappings:

$ curl -XPOST http://localhost:9200/whelk_dev -d@librisxl-tools/elasticsearch/libris_config.json

NOTE: Windows users can install curl by:

$ choco install curl

Import test data

Check out the Devops repository, which is private (ask a team member for access and put it in the same directory as the 'librisxl' repo):

For *NIX:

$ cd devops
$ pip install -r requirements.txt
$ fab conf.xl_local app.whelk.reload_example_data:force=True


To start the CRUD part of the whelk, run the following commands:


$ cd $LIBRISXL/rest
$ export JAVA_OPTS="-Dfile.encoding=utf-8"
$ ../gradlew appRun


$ cd $LIBRISXL/rest
$ setx JAVA_OPTS "-Dfile.encoding=utf-8"
$ ../gradlew.bat appRun

The system is then available on http://localhost:8180.


Everything you would want to do should be covered by the devops repo. This section is mostly kept as a reminder of alternate (less preferred) ways.

Development Workflow

If you need to work locally (e.g. in this or the "definitions" repo) and perform specific tests, you can use this workflow:

  1. Create and push a branch for your work.
  2. Set the branch in the conf.xl_local config in the devops repo.
  3. Use the regular tasks to e.g. reload data.

New Elasticsearch config

If a new index is to be set up, and unless you run locally in a pristine setup, or use the recommended devops-method for loading data you need to PUT the config to the index, like:

$ curl -XPUT http://localhost:9200/indexname_versionnumber \
    -d @librisxl-tools/elasticsearch/libris_config.json

Create an alias for your index

$ curl -XPOST http://localhost:9200/_aliases \
    -d  '{"actions":[{"add":{"index":"indexname_versionnumber","alias":"indexname"}}]}'

(To replace an existing setup with entirely new configuration, you need to delete the index curl -XDELETE http://localhost:9200/<indexname>/ and read all data again (even locally).)

Format updates

If the MARC conversion process has been updated and needs to be run anew, the only option is to reload the data from vcopy using the importers application.


Produce a stats file (here for bib) by running:

$ cd importers && ../gradlew build
$ RECTYPE=bib && time java -jar build/libs/vcopyImporter.jar vcopyjsondump $RECTYPE | grep '^{' | pypy ../librisxl-tools/scripts/ $RECTYPE /tmp/usage-stats-$RECTYPE.json
You can’t perform that action at this time.