The FLAT project aims to develop an easy to use and maintainable archive setup for language resources with Component Metadata. Its based on Fedora Commons and Islandora. It should meet the technical requirements for a CLARIN B centre, the CoreTrustSeal and those from the organizations, the Max Planck Institute for Psycholinguistics and the Meertens Institute, that cooperate in The Language Archive.
Currently the setup of this project consists of a series of docker setups:
A FLAT base image that
- installs Fedora Commons and Islandora
- provides tools and scripts to import CMD records into Fedora
- adds support for rendering of CMD records in Islandora
A FLAT Blazegraph image builds on the base image and
- replaces the builtin Mulgara triplestore by Blazegraph
A FLAT Islandora SOLR image builds on the base image and
- installs Islandora's SOLR modules
A FLAT Islandora OAI image builds on the base image and the Islandora SOLR image
- installs Islandora's OAI module
- installs the FLAT OAI Islandora module, which adds support for providing CMD (and OLAC) records via OAI-PMH
A FLAT search image builds on the base image and the Islandora SOLR image, and
- installs generic search for Fedora Commons
- provides tools and scripts to configure the index proces for a specific set of CMD records and profiles
A FLAT SWORD image builds on the base image and
- installs a SWORD v2 API to receive bags
A FLAT DoorKeeper image builds on the base image and
- installs the DoorKeeper, which guards the repository and checks new or updated resources and metadata
- installs the DoorKeeper API to process bags
A FLAT example setup image builds on the doorkeeper image and
- adds a comic book collection,
- a comic book SIP, and
- related users
An experimental FLAT deposit UI image builds on the base image and the SWORD image, and
- installs a module that provides an UI for users to deposit data
An experimental FLAT Shibbolet image builds on the base image and
- installs Shibboleth
- installs Drupal's Shibboleth modules
An experimental FLAT solution packs image builds on the base image and
- installs Islandora solution packs
- provides scripts to trigger the addition of derived datastreams like thumbnails
A deprecated FLAT proai image builds on the base image and
- installs and configures the proai Fedora extension
The FLAT base image is required, but the other ones can be added to it as needed (but might depend on eachother).
Additionally there are two docker setups specific for IMDI and CMDIfied IMDI:
A FLAT IMDI conversion image builds on the base image and
- provides tools and scripts to convert from IMDI to CMDI
A FLAT IMDI search image builds on the search image and
- provides the mapping to configure the index proces for CMDIfied IMDI records and profiles
Building a FLAT docker image
This description assumes you're using a recent native Docker (17 or higher).
CMDI records can vary a lot. Based on the VLO configuration a mapping to Dublin Core is determined. You might want to tweak that to your specific needs (see the FLAT search image configuration section). If you have IMDI records extra conversion and configuration is needed (see section below).
NOTE: simple passwords are included in the setup, they should not be take along to a production environment!
NOTE for Windows users: It is important that the line endings of the files stay as they are in the git repository. By default, a git clone on a Windows client will translate line endings into DOS/Windows line endings (carriage return + line feed). You can turn this off globally using
git config --global core.autocrlf input (or "false" instead of "input", depending on whether you need to preserve DOS/Windows line endings for other repositories you commit to). You can also specify this upon cloning the repository, e.g.
git clone https://github.com/TLA-FLAT/FLAT.git -c core.autocrlf=input
The following commands show how to build a setup that supports FLAT base plus facetted search and the DoorKeeper:
cd docker #start with the FLAT base docker build --squash -t flat flat/ #add Blazegraph docker build --squash -t flat add-blazegraph-to-flat/ #add Fedora gsearch + SOLR docker build --squash -t flat add-gsearch-to-flat/ #add Islandora SOLR module docker build --squash -t flat add-islandora-solr-to-flat/ #add Islandora OAI module docker build --squash -t flat add-islandora-oai-to-flat/ #add SWORD API docker build --squash -t flat add-sword-to-flat/ #add the DoorKeeper docker build --squash -t flat add-doorkeeper-to-flat/ #add the example setup docker build --squash -t flat add-example-setup-to-flat/
Running a FLAT docker image
Now the FLAT docker image can be run:
docker run -p 80:80 -it flat
In the container shell run:
#run all the steps to batch import the example comic book collection do.sh #and add the example SIP #- packup the SIP directory flat-create-sip.sh /app/flat/test/test-sip #- upload the SIP via SWORD flat-sword-upload.sh test-sip.zip test #- check if the SWORD upload went fine curl -u flat:sword http://localhost/flat/easy-deposit/statement/test | xmllint --format - #- trigger the DoorKeeper run for the SIP wget --method=PUT http://localhost/flat/doorkeeper/test #- inspect the result wget http://localhost/flat/doorkeeper/test #- inspect the developers log tail -f deposit/bags/test/bag-test-sip/data/test-sip/logs/devel.log
Now visit FLAT in your browser.
Building a FLAT docker image if you have IMDI records
If you have IMDI records you can add the conversion to CMDI and the configuration for CMDIfied IMDI search.
cd docker #start with the FLAT base docker build --squash -t flat flat/ #add Blazegraph docker build --squash -t flat add-blazegraph-to-flat/ #add IMDI conversion docker build --squash -t flat add-imdi-conversion-to-flat/ #add Fedora gsearch + SOLR docker build --squash -t flat add-gsearch-to-flat/ #add Islandora SOLR module docker build --squash -t flat add-islandora-solr-to-flat/ #add Islandora OAI module docker build --squash -t flat add-islandora-oai-to-flat/ #add configuration for CMDIfied IMDI search docker build --squash -t flat add-imdi-gsearch-to-flat/ #add SWORD API docker build --squash -t flat add-sword-to-flat/ #add the DoorKeeper docker build --squash -t flat add-doorkeeper-to-flat/
Running a FLAT docker image if you have IMDI records
Now the FLAT docker image can be run:
docker run -p 80:80 -v ./some-directory:/lat -it flat
Run the various
do-*.sh scripts in their natural order. And visit FLAT in your browser.
- PROBLEM: Starting and stopping the Tomcat application server can take longer than expected, as it depends on the power or activity of the host.
- SOLUTION: Increase the
- SOLUTION: Increase the
- PROBLEM: During the compilation of the MediaShelf fedora-client the test sometimes runs into a locking problem.
- SOLUTION: Just restart the build, the test will most likely succeed this time.
Publications, Presentations & Demonstrations
- M. Windhouwer. Fedora Commons in the CLARIN Infrastructure. Presentation at CLARIN-PLUS workshop: "Facilitating the Creation of National Consortia - Repositories". Prague, Czech Republic, February 9 - 10, 2017.
- P. Trilsbeek, M. Windhouwer. FLAT: A CLARIN-compatible repository solution based on Fedora Commons. At the CLARIN Annual Conference. Aix-en-Provence, France, 26-28 October, 2016.
- M. Windhouwer, M. Kemps-Snijders, P. Trilsbeek, A. Moreira, B. van der Veen, G. Silva, D. von Rhein. FLAT: constructing a CLARIN compatible home for language resources. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), European Language Resources Association (ELRA), Portorož, Slovenia, May 23 - 28, 2016. (Local updated version)
FLAT was once upon a time known as EasyLAT, so occassionally documentation and code might still use that name.