Transformation, web frontend, and API 2.0 for the hbz catalog as LOD
Java HTML CSS Shell Other
Switch branches/tags
Nothing to show
Clone or download
Failed to load latest commit information.
.settings Store dc(t):subject as a list Nov 17, 2016
src Merge branch '855-missingAltLabelsConferences' of… Jul 3, 2018
web Merge branch '855-missingAltLabelsConferences' of… Jul 3, 2018
.classpath Use main resources folder as source folder in Eclipse config Oct 24, 2017
.gitignore Merge branch 'master' of Feb 7, 2018
.project Add project structure, dependencies, and documentation Jan 28, 2016
LICENSE initial commit Oct 15, 2015
README.textile Add missing detail in setup instructions Nov 21, 2017
getNwbibSubjectLocationsAsWikidataEntities.txt ETL wikidata geo data to lobid geo elasticsearch Sep 18, 2017 Use jitpack to handle lobid-rdf-to-json library Mar 1, 2016
settings.xml Add project structure, dependencies, and documentation Jan 28, 2016 Add repo name and branch name to git pull Mar 10, 2016



Transform MAB-XML to JSON for Elasticsearch indexing with Metafacture, serve API and UI with Play Framework.

This repo replaces the lobid-resources part of

For information about the Lobid architecture and development process, see


Prerequisites: Java 8, Maven 3; verify with mvn -version

Create and change into a folder where you want to store the projects:

  • mkdir ~/git ; cd ~/git

Build the hbz metafacture-core fork:

  • git clone
  • cd metafacture-core
  • mvn clean install -DskipTests
  • cd ..

Build lobid-resources:

  • git clone
  • cd lobid-resources
  • mvn clean install

Build the web application:

  • cd web
  • wget
  • unzip
  • ./activator-1.3.10-minimal/bin/activator test

See the .travis.yml file for details on the CI config used by Travis.

Eclipse setup

Replace test with other Play commands, e.g. "eclipse with-source=true" (generate Eclipse project config files, then import as existing project in Eclipse), ~ run (run in test mode, recompiles changed files on save, use this to keep your Eclipse project in sync while working, make sure to enable automatic workspace refresh in Eclipse: Preferences > General > Workspace > Refresh using native hooks or polling).


Use "start 8000" to run in production background mode on port 8000 (hit Ctrl+D to exit logs). To restart a production instance running in the background, you can use the included script (configured to use port 8000). For more information, see the Play documentation.

Example of getting the data

In the online test the data is indexed to a living elasticsearch instance.
This instance is only reachable within our internally network, thus this test
must be executed manually. Then elasticsearch can be looked up like this:

For querying it you can use the elasticsearch query DSL, like:

The result shows the data which also produces.

Developer instructions

This section explains how to make a successful build after changing the transformations,
how to update the JSON-LD and its context, and how to index the data.

Changing transformations

After changing the morph the build must be executed:

mvn clean install

Two possible outcomes:

  • BUILD SUCCESS: the tested resources don’t reflect the changes.
    In this case you should add an Aleph-MabXml resource to hbz01XmlClobs.tar.bz2 that would reflect your changes. Do like this to add the resource HT018895767:
    cd src/test/resources; rm -rf hbz01XmlClobs; tar xfj hbz01XmlClobs; xmllint --format > hbhbz01XmlClobs/HT018895767; tar cfj hbz01XmlClobs.tar.bz2 hbz01XmlClobs; cd -; mvn clean install; git add src/test/resources/jsonld/ ; git add src/test/resources/
  • BUILD FAILURE: the newly generated data isn’t equal to the test resources.
    This is a good thing because you wanted the change.

Doing mvn test -DgenerateTestData=true the test data is generated and also updated in the filesystem.
These new data will now act as the template for sucessful tests. So, if you would rebuild now, the build will pass successfully.
You just must approve the new outcome by committing it.

Now you must approve the new outcome.
Let’s see what has changed:

git status

Let’s make a diff on the changes, e.g. all JSON-LD documents:

git diff src/test/resources/jsonld/

If you are satisfied with the changes, go ahead and add and commit them:

git add src/test/resources/jsonld/; git commit

Do this respectivly for all other test files (Ntriples …).
If you’ve added and commited everything, check again if all is ok:

mvn clean install

This should result in BUILD SUCCESS. Push your changes. You’re done :)

Propagate the context.json to lobid-resources-web

The generated context.jsonld is automatically written to the proper directory
so that it is automatically deployed when the web application is deployed.

When the small test set is indexed by using deploy your branch in
the staging directory of the web application. The context for the resources is adapted
to use the “”-domain and thus the staging-context.jsonld will resolve using the one in that directory.

Elasticsearch index

This is about the building of the production index as well as the small test index.


All automation is configured at one central crontab entry: hduser@weywot1 .
This is your starting point for tracing what scripts are triggered on what server
at which time. All is logged, see the crontab entries resp. the scripts.

Like with the old (and still productive) lobid index, a weekly fulldump
index is build on Saturday/Sunday. This is the base which will then be complemented by the daily incremental updates.
Have a look at both source data, base and daily updates.
The base-file must be referenced by a symbolic link residing in the nfs: “/files/open_data/closed/hbzvk/”. It is this symbolic link which is used by indexing processes.
Building the full index takes around 12 hours. The finished index is aliased
to resources-staging. This is manually to be tested. If it’s ok, the
index is switched by simply renaming this alias to resources . Both aliases are
reflected in the lookup-URI (e.g. vs resp. internal ES URI and ).

The three latest indices will be retained, also every index with no -staging
suffix on the alias. Other indices which start with the same string (e.g. resources) will be
removed by the index program after the indexing is finished – the hard disk
is not of endless size ;) .

Note: in production, always make sure that the JSON-LD context is the proper one.

Small test

This will download ~5k aleph xml clobs (if not already residing in your filesystem)
and index them into elasticsearch, aliased test-resources (not interfering with the
productive indexes, thus not removing any productive index).

cd src/test/resources; bash

Data updates

How to manually invoke data updates for both API 1.x data and API 2.0 data.

Data source for updates and baselines:

These are created with weywot1:/home/hduser/git/hbz-aleph-dumping/bin/

For all commands below, add this to redirect output to file and follow that file:

> log/ 2>&1 & tail -f log/

Data 1.x, at hduser@weywot1:/home/hduser/git/lodmill/lodmill-rd/doc/scripts/hbz01$

Single update:

bash -x master /files/open_data/open/DE-605/mabxml/DE-605-aleph-update-marcxchange-20160411-20160412.tar.gz lobid-resources-staging NOALIAS quaoar exact

Updates listed in file:

bash -x master dummy_ignore lobid-resources-staging NOALIAS quaoar exact doc/scripts/hbz01/toBeUpdateFilesXmlClobs_afterBasedump.txt

New baseline index:

bash -x master /files/open_data/closed/hbzvk/ lobid-resources-201605090945 "-staging" quaoar create doc/scripts/hbz01/toBeUpdateFilesXmlClobs_afterBasedump.txt

Data 2.0, at hduser@gaia:/opt/hadoop/git/lobid-resources$

Single update:

bash -x master /files/open_data/open/DE-605/mabxml/DE-605-aleph-update-marcxchange-20160411-20160412.tar.gz resources-staging NOALIAS lobid-gaia exact

Updates listed in file:

bash -x master dummy_ignore resources-staging NOALIAS lobid-gaia exact toBeUpdateFilesXmlClobs_afterBasedump.txt

New baseline index:

bash -x master /files/open_data/closed/hbzvk/ resources-201605090945 NOALIAS lobid-gaia create


Eclipse Public License: