Skip to content

Commit

Permalink
Merge pull request #347 from LibreCat/docker
Browse files Browse the repository at this point in the history
 Move minimal docker image to base directory and build from source
  • Loading branch information
nics committed May 22, 2018
2 parents 956ae92 + 77ad777 commit 4c8a32a
Show file tree
Hide file tree
Showing 6 changed files with 90 additions and 119 deletions.
4 changes: 4 additions & 0 deletions .dockerignore
@@ -0,0 +1,4 @@
.git/*
local/*
blib/*
cover_db/*
30 changes: 30 additions & 0 deletions Dockerfile
@@ -0,0 +1,30 @@
# Minimal Dockerfile of a base image with Catmandu core on Debian stretch
FROM debian:stretch-slim

MAINTAINER LibreCat community <librecat-dev@lists.uni-bielefeld.de>

ADD docker/apt.txt .
# Perl packages used by Catmandu (if available as Debian package) and cpanm
RUN apt-get update && apt-get install -y --no-install-recommends \
$(grep -vE "^\s*#" apt.txt | tr "\n" " ") cpanminus \
&& rm -rf /var/lib/apt/lists/*

ADD . /tmp/catmandu

WORKDIR /tmp/catmandu

# install from source
RUN cpanm -n -q --installdeps --skip-satisfied .
RUN perl Build.PL && ./Build && ./Build install

# cleanup sources
WORKDIR /
RUN rm -rf /tmp/catmandu

# make user feel home
RUN adduser --home /home/catmandu --uid 1000 --disabled-password --gecos "" catmandu
WORKDIR /home/catmandu
USER catmandu

# Default command
CMD ["bash"]
40 changes: 0 additions & 40 deletions docker/Dockerfile

This file was deleted.

36 changes: 36 additions & 0 deletions docker/apt.txt
@@ -0,0 +1,36 @@
# Catmandu dependencies as Debian packages

# This list is used to build Docker images. See also
# https://salsa.debian.org/perl-team/modules/packages/libcatmandu-perl

libany-uri-escape-perl
libasa-perl
libcgi-expand-perl
libclone-perl
libconfig-onion-perl
libcpanel-json-xs-perl
libdata-compare-perl
libdata-util-perl
libdata-uuid-perl
libhash-merge-simple-perl
libio-handle-util-perl
liblist-moreutils-perl
liblog-any-perl
libmime-types-perl
libmodule-info-perl
libmoo-perl
libmoox-aliases-perl
libnamespace-clean-perl
libparser-mgc-perl
libpath-iterator-rule-perl
libpath-tiny-perl
libsub-exporter-perl
libtext-csv-perl
libtext-hogan-perl
libthrowable-perl
libtry-tiny-byclass-perl
liburi-perl
liburi-template-perl
libwww-perl
libyaml-libyaml-perl
build-essential
28 changes: 0 additions & 28 deletions docker/docker-cpanfile

This file was deleted.

71 changes: 20 additions & 51 deletions lib/Catmandu/Introduction.pod
Expand Up @@ -6,18 +6,13 @@ Catmandu::Introduction - An Introduction to Catmandu data processing toolkit

# STATUS

[![Build Status](https://travis-ci.org/LibreCat/Catmandu.svg?branch=master)](https://travis-ci.org/LibreCat/Catmandu)
[![Coverage](https://coveralls.io/repos/LibreCat/Catmandu/badge.svg?branch=master)](https://coveralls.io/r/LibreCat/Catmandu)
[![CPANTS kwalitee](http://cpants.cpanauthors.org/dist/Catmandu.png)](http://cpants.cpanauthors.org/dist/Catmandu)
[![Build Status](https://travis-ci.org/LibreCat/Catmandu.svg?branch=master)](https://travis-ci.org/LibreCat/Catmandu) [![Coverage](https://coveralls.io/repos/LibreCat/Catmandu/badge.svg?branch=master)](https://coveralls.io/r/LibreCat/Catmandu) [![CPANTS kwalitee](http://cpants.cpanauthors.org/dist/Catmandu.png)](http://cpants.cpanauthors.org/dist/Catmandu)

=end markdown

=head1 Introduction

Catmandu is a data processing toolkit developed as part of the L<LibreCat|http://librecat.org> project.
Catmandu provides the command line client L<catmandu> and a suite of tools to ease the import, storage,
retrieval, export and transformation of data. For instance, to transform a CSV file into JSON use the
command:
Catmandu is a data processing toolkit developed as part of the L<LibreCat|http://librecat.org> project. Catmandu provides the command line client L<catmandu> and a suite of tools to ease the import, storage, retrieval, export and transformation of data. For instance, to transform a CSV file into JSON use the command:

$ catmandu convert JSON to CSV < data.json

Expand All @@ -35,34 +30,27 @@ With Catmandu one can import OAI-PMH records in your application (requires L<Cat

and export records into formats such as JSON, YAML, CSV, XLS, RDF and many more.

Catmandu also provides a small scripting language to manipulate data, extract parts of your dataset and
transform records. For instance, rename fields with the 'L<move_field|Catmandu::Fix::move_field>' Fix command:
Catmandu also provides the small scripting language "Fix" to manipulate data, extract parts of your dataset and transform records. For instance, rename fields with the L<move_field|Catmandu::Fix::move_field> Fix command:

$ catmandu convert JSON --fix 'move_field(title,my_title)' < data.json

In the example above, we renamed all the 'title' fields in the dataset into the 'my_title' field.
In the example above, we renamed all the C<title> fields in the dataset into the C<my_title> field.

One can also work on deeply nested data. E.g. create a deeply nested data structure with the
'move_field' Fix command:
One can also work on deeply nested data. E.g. create a deeply nested data structure with the C<move_field> Fix command:

$ catmandu convert JSON --fix 'move_field(title,my.deeply.nested.title)' < data.json

In this example we moved the field 'title' into the field 'my', which contains a (sub)field 'deeply',
which contains a (sub)field 'nested'.
In this example we moved the field C<title> into the field C<my>, which contains a (sub)field C<deeply>, which contains a (sub)field C<nested>.

Catmandu was originally created by librarians for librarians. We process a lot of metadata especially
library metadata in formats such as MARC, MAB2 and MODS. With the following command we can extract
data from a marc record and to store it into the title field (requires L<Catmandu::MARC>):
Catmandu was originally created by librarians for librarians. We process a lot of metadata especially library metadata in formats such as MARC, MAB2 and MODS. With the following command we can extract data from a marc record and to store it into the title field (requires L<Catmandu::MARC>):

$ catmandu convert MARC --fix 'marc_map(245,title)' < data.mrc

Or, in case only the 245a subfield is needed write:

$ catmandu convert MARC --fix 'marc_map(245a,title)' < data.mrc

When processing data a lot of Fix commands could be required. It wouldn't be very practical to
type them all on the command line. By creating a Fix script which contains all the fix commands complicated
data transformations can be created. For instance, if the file C<myfixes.txt> contains:
When processing data a lot of Fix commands could be required. It wouldn't be very practical to type them all on the command line. By creating a Fix script which contains all the fix commands complicated data transformations can be created. For instance, if the file C<myfixes.txt> contains:

marc_map(245a,title)
marc_map(100a,author.$append)
Expand All @@ -74,8 +62,7 @@ then they can be executed on a MARC file using this command:

$ catmandu convert MARC --fix myfixes.txt < data.mrc

Fixes can also be turned into executable scripts by adding a bash 'shebang' line at the top. E.g.
to harvest records from an OAI repository write this fix file:
Fixes can also be turned into executable scripts by adding a bash 'shebang' line at the top. E.g. to harvest records from an OAI repository write this fix file:

#!/usr/bin/env catmandu run
do importer(OAI,url:"http://lib.ugent.be/oai")
Expand All @@ -87,8 +74,7 @@ Run this (on Linux) by setting the executable bit:
$ chmod 755 myfix.fix
$ ./myfix.fix

To experiment with the Fix language you can also run the catmandu Fix interpreter in an
interactive mode:
To experiment with the Fix language you can also run the catmandu Fix interpreter in an interactive mode:

$ catmandu run
Catmandu 0.95 interactive mode
Expand All @@ -99,29 +85,23 @@ interactive mode:
...
fix >

Catmandu contains many powerful fixes. Visit L<http://librecat.org/assets/catmandu_cheat_sheet.pdf> to get
an overview what is possible.
Catmandu contains many powerful fixes. Visit L<http://librecat.org/assets/catmandu_cheat_sheet.pdf> to get an overview what is possible.

=head1 Documentation

For more information read our L<documentation pages|http://librecat.org/Catmandu/>
and L<blog|https://librecatproject.wordpress.com/>
for a complete introduction and update into all Catmandu features.
For more information read our L<documentation pages|http://librecat.org/Catmandu/> and L<blog|https://librecatproject.wordpress.com/> for a complete introduction and update into all Catmandu features.

In the winter of 2014 an Advent calendar tutorial was created to provide a day by
day introduction into the UNIX command line and Catmandu:
In the winter of 2014 an Advent calendar tutorial was created to provide a day by day introduction into the UNIX command line and Catmandu:

L<https://librecatproject.wordpress.com/2014/12/01/day-1-getting-catmandu/>

If you need extra training, our developers regulary host workshops at library
conferences and events: L<http://librecat.org/events.html>
If you need extra training, our developers regulary host workshops at library conferences and events: L<http://librecat.org/events.html>

=head1 Installation

There are several ways to get a working version of Catmandu on your computer.
For a quick and demo installation visit our L<blog|https://librecatproject.wordpress.com/get-catmandu/>
where a VirtualBox image is available containing all the Catmandu modules, including
ElasticSearch and MongoDB.

For a quick and demo installation visit our L<blog|https://librecatproject.wordpress.com/get-catmandu/> where a VirtualBox image is available containing all the Catmandu modules, including ElasticSearch and MongoDB. A similarly easy method is using L<Docker|https://www.docker.com/>: At L<librecat/catmandu|https://hub.docker.com/r/librecat/catmandu/> a Docker image is provided with version tags for each release, C<latest> for the latest release and C<dev> for the latest development version. The image can be tried online in a Jupyter notebook L<via binder|https://mybinder.org/v2/gh/LibreCat/catmandu-notebook/master>.

On our L<website|http://librecat.org/Catmandu/#installation> we provide installation instructions for:

Expand All @@ -141,32 +121,21 @@ On our L<website|http://librecat.org/Catmandu/#installation> we provide installa

=back

and even a generic installation using L<Docker|https://www.docker.com/>: L<https://hub.docker.com/r/librecat/catmandu/>.

=head1 Open Source

Catmandu software published at https://github.com/LibreCat/Catmandu is free software without warranty, liabilities
or support; you can redistribute it and/or modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 or any later version. Every contributor is free
to state her/his copyright.
Catmandu software published at https://github.com/LibreCat/Catmandu is free software without warranty, liabilities or support; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 or any later version. Every contributor is free to state her/his copyright.

=head1 Developers & Support

Catmandu has a very active international developer community. We welcome all feedback, bug reports and
feature enhancement.
Catmandu has a very active international developer community. We welcome all feedback, bug reports and feature enhancement.

Join our mailing list to receive more information: C<< librecat-dev@librecat.org >>

Are a developer and want to contribute to the project? Feel free to submit pull requests or create new
Catmandu packages!
Are a developer and want to contribute to the project? Feel free to submit pull requests or create new Catmandu packages!

=head1 Kudos

Catmandu is created in a cooperation with many developers world wide. Without them this project isn't possible.
We would like to thank our core maintainer: Nicolas Steenlant and all contributors: Christian Pietsch ,
Dave Sherohman , Friedrich Summann , Jakob Voss , Johann Rolschewski , Jorgen Eriksson , Magnus Enger ,
Maria Hedberg , Mathias Loesch , Najko Jahn , Nicolas Franck , Patrick Hochstenbach , Petra Kohorst ,
Snorri Briem , Upasana Shukla and Vitali Peil
Catmandu is created in a cooperation with many developers world wide. Without them this project isn't possible. We would like to thank our core maintainer: Nicolas Steenlant and all contributors: Christian Pietsch, Dave Sherohman, Friedrich Summann, Jakob Voss, Johann Rolschewski, Jorgen Eriksson, Magnus Enger, Maria Hedberg, Mathias Loesch, Najko Jahn, Nicolas Franck, Patrick Hochstenbach, Petra Kohorst, Snorri Briem, Upasana Shukla and Vitali Peil.

=head1 SEE ALSO

Expand Down

0 comments on commit 4c8a32a

Please sign in to comment.