ezPAARSE can ingest your (proxy) log files and show how users access suscribed electronic ressources.
JavaScript HTML Shell CSS Makefile
Latest commit 73420a0 Mar 28, 2017 @nojhamster nojhamster Version 2.10.0
Permalink
Failed to load latest commit information.
bin source nvm instead of appending to PATH Sep 7, 2016
doc implement ezPAARSE-Filter-Status Mar 28, 2017
lib implement ezPAARSE-Filter-Status Mar 28, 2017
logs add blank file to be sure logs folder exists Dec 19, 2012
misc Macro Excel - 64 Bits Dec 15, 2016
public locales slight rephrasing Mar 9, 2017
routes fix user management Nov 18, 2016
test implement ezPAARSE-Filter-Status Mar 28, 2017
views alphabetically sort sections of report categories Mar 7, 2017
.bithoundrc implement bithound Aug 10, 2016
.bowerrc add bowerrc and bower.json Feb 26, 2014
.codeclimate.yml to play with https://codeclimate.com/github/ezpaarse-project/ezpaarse May 11, 2015
.dockerignore dockerfile refactoring Aug 30, 2016
.eslintignore update modules Sep 20, 2016
.eslintrc update modules Sep 20, 2016
.foreverignore to remove a forever warning Apr 10, 2013
.gitignore ignore .vscode Feb 21, 2017
.jshintignore setup for jshint Dec 19, 2012
.jshintrc allow latedef of functions Jan 29, 2016
.jsonhintrc validates JSON files Jan 28, 2015
.npmrc use npm to bump version Aug 30, 2016
.travis.yml try to fix travis Sep 27, 2016
Dockerfile upgrade mongo and debian in the docker stuff because old version not … Dec 29, 2016
LICENCE.txt adding license files Aug 28, 2013
LICENSE.txt adding license files Aug 28, 2013
Makefile use npm to bump version Aug 30, 2016
README.md ezpaarse tweeter badge Nov 10, 2016
app.js put feedback/castor status routes before admin routes Sep 12, 2016
bower.json update bower deps Sep 21, 2016
config.json add log_id to default output Mar 28, 2017
docker-compose.yml upgrade mongo and debian in the docker stuff because old version not … Dec 29, 2016
npm-shrinkwrap.json Version 2.10.0 Mar 28, 2017
package.json Version 2.10.0 Mar 28, 2017
statuscodes.json improve emergency stop Feb 14, 2017

README.md

ezPAARSE

Documentation Status Build Status Dependencies Status bitHound Overall Score Docker stars Docker Pulls Ezpaarse tweeter

ezPAARSE is an open-source software that can ingest your (proxy) log files and show how users access subscribed electronic resources. It filters, extracts and enriches the consultation events that were spotted and produces a CSV file following COUNTER codes of practice. This document describes how to install and run ezPAARSE on your computer.

Moreover, have a look to the ezpaarse demo, it will show you a nice user interface where you can register and test to process your own proxy logs.

Built-in proxies supported log formats are: ezproxy, bibliopam, and squid

Table of content

Recommended system requirements

  • a linux box or VM (eg: Ubuntu)
  • 50Gb disk space (to be adjusted, depending on the quantity and size of logfiles to be simultaneously processed)
  • 2 cores of CPU
  • 2 to 4 Gb of RAM space

Prerequisites

The tools you need to let ezPAARSE run are:

  • Linux OS: See the prerequisites for those OSes
  • Standard Linux tools: bash, make, grep, sed ...
  • curl (used by nvm)
  • git >= 1.7.10 (required to clone ezpaarse from github and to keep your ezpaarse copy up to date)

ezPAARSE then comes with all the elements it needs to run. When the prerequesites are met, you can launch the make command (see below) that will run all installation steps.

Installation quickstart

If you are a Windows user, you can install ezPAARSE on your computer as a docker image. Please refer to the docker section below.

To install the latest stable version of ezPAARSE on a Unix-type system, open a terminal and type:

git clone https://github.com/ezpaarse-project/ezpaarse.git
cd ezpaarse
git checkout `git describe --tags --abbrev=0`
make

If you want to install the version in development (unstable), open a terminal and type:

git clone https://github.com/ezpaarse-project/ezpaarse.git
cd ezpaarse
make

Test the installation

This step allows you to validate that your install is working.

make start
make test

Usage

Anonymised example logfiles are made available in the repositories of ezPAARSE.

You need to make sure that ezPAARSE is started. To do so, type the following command:

make start

If you are not computer-savvy, the easiest way to work with ezPAARSE is to use its HTML form, accessible from your favorite webbrowser and open the following URL: http://localhost:59599/

If you are computer-savvy, you can use an HTTP client (like curl) to send a logfile (for this example, we will use ./test/dataset/sd.2012-11-30.300.log) to ezPAARSE's Web service and get a CSV stream of consultation events as a response.

curl -X POST http://127.0.0.1:59599 \
             -v --proxy "" --no-buffer \
             --data-binary @./test/dataset/sd.2012-11-30.300.log

Or you can use the command ./bin/loginjector ezPAARSE provides you with to send the logfile to the web service in a simpler way:

. ./bin/env
cat ./test/dataset/sd.2012-11-30.300.log | ./bin/loginjector

You can also see quick countings on your data if you add the command ./bin/csvtotalizer at the end of the command line. Doing so, you will get an overview of the consultation events extracted from your logs by ezPAARSE:

. ./bin/env
cat ./test/dataset/sd.2012-11-30.300.log | ./bin/loginjector | ./bin/csvtotalizer

To stop ezPAARSE, you have to type the following command:

make stop

Go further

To go further, you can consult the full documentation

Advanced parameters

The default ezPAARSE parameters can be found in the config.json file. All these parameters can be changed. A good practice is to define a new file called config.local.json containing just the parameters you need to override.

For example, to change the ezPAARSE listening port (59599 by default), you can override the EZPAARSE_NODEJS_PORT by defining a new config.local.json file this way:

{
  "EZPAARSE_NODEJS_PORT": 45000
}

Use with docker

ezPAARSE is available as a docker image.

You need:

Then, you can run the dockerized ezpaarse this way:

mkdir ezpaarse/
wget https://raw.githubusercontent.com/ezpaarse-project/ezpaarse/master/docker-compose.yml
docker-compose up -d

Then ezpaarse is available at this URL: http://127.0.0.1:59599

To have a look to the ezpaarse system logs, you can run: docker logs -f ezpaarse