Skip to content

Introduction

Jump to bottom Edit New page
Patrick Hochstenbach edited this page May 8, 2014 · 52 revisions

Catmandu provides a suite of Perl modules to ease the import, storage, retrieval, export and transformation of metadata records. Combine Catmandu modules with web application frameworks such as PSGI/Plack document stores such as MongoDB and full text indexes as Solr to create a rapid development environment for digital library services such as institutional repositories and search engines.

###Where do we use it?

In the LibreCat project it is our goal to provide in open source a set of programming components to build up digital libraries services suited to your local needs. Here is an example of the projects we are working on:

  • LibreCat-Catalog : a next generation institutional repository (in development).
  • LibreCat-Citation : a CSL based citation list (in development).
  • LibreCat-Search : an ElasticSearch based front-end for institutional repositories. GitHub
  • LibreCat-Grim : a Solr/(IIPImage)[http://iipimage.sourceforge.net/] based image database.
  • LibreCat-Archive : a Fedora Commons based digital repository.
  • LibreCat-Imaging : a MediaMosa based digitization workflow engine.

We have more than 60 Catmandu projects available at GitHub LibreCat.

###Why do we use it?

####Extract, Transform and Load

Create a search engine, one of your first tasks will to import data from various sources, map the fields to a common data model and post it to a full-text search engine. Perl modules such as WebService::Solr or ElasticSearch provide easy access to your favorite document stores, but you keep writing a lot of boilerplate code to create the connections, massaging the incoming data into the correct format, validating and uploading and indexing the data in the database. Next morning you are asked to provide a fast dump of records into an Excel worksheet. After some fixes are applied you are asked to upload it into your database. Again you hit Emacs or Vi and provide an ad-hoc script. In our LibreCat group we saw this workflow over and over. We tried to abstract this problem to a set of Perl tools which can work with library data such as MARC, Dublin Core, EndNote protocols such as OAI-PMH, SRU and repositories such as DSpace and Fedora. In data warehouses these processes are called ETL, Extract, Transform, Load. Many tools currenty exist for ETL processing but none adress typical library data models and services.

Clone this wiki locally