ETL With a Shell
Perl Perl 6 HTML CSS
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin
lib/ETL
site
t
.gitignore
.mailmap
.travis.yml
CHANGES
CONTRIBUTING.md
LICENSE
Makefile.PL
README.mkdn
cpanfile
dist.ini
site.yml
weaver.ini

README.mkdn

NAME

ETL::Yertl - ETL with a Shell

VERSION

version 0.037

STATUS

Coverage Status

SYNOPSIS

### On a shell...
# Convert file to Yertl's format
$ yfrom csv file.csv >work.yml
$ yfrom json file.json >work.yml

# Mask document
$ ymask 'field/inner' work.yml >masked.yml

# Convert file to output format
$ yto csv work.yml
$ yto json work.yml

# Parse HTTP logs into documents
$ ygrok '%{LOG.HTTP_COMMON}' httpd.log

# Read data from a database
$ ysql db_name 'SELECT * FROM employee'

# Write data to a database
$ ysql db_name 'INSERT INTO employee ( id, name ) VALUES ( $.id, $.name )'

### In Perl...
use ETL::Yertl;

# XXX: To do: Perl API

DESCRIPTION

ETL::Yertl is an ETL (Extract, Transform, Load) for shells. It is designed to accept data from multiple formats (CSV, JSON), manipulate them using simple tools, and then convert them to an output format.

Yertl will have tools for:

  • Extracting data from databases (MySQL, Postgres, MongoDB)
  • Loading data into databases
  • Extracting data from web services
  • Writing data to web services
  • Distributing data through messaging APIs (ZeroMQ)

SEE ALSO

Yertl Tools

  • yfrom

    Convert incoming data (CSV, JSON) to Yertl documents.

  • yto

    Convert Yertl documents into another format (CSV, JSON).

  • ygrok

    Parse lines of text into Yertl documents.

  • ysql

    Read/write documents from SQL databases.

  • ymask

    Filter documents with a mask, letting only matching fields through.

  • yq

    A powerful mini-language for munging and filtering.

Other Tools

Here are some other tools that can be used with Yertl

  • recs (App::RecordStream)

    A set of tools for manipulating JSON (constrast with Yertl's YAML). For interoperability, set the YERTL_FORMAT environment variable to "json".

  • jq

    A filter for JSON documents. The inspiration for yq. For interoperability, set the YERTL_FORMAT environment variable to "json".

  • jt

    JSON Transformer. Allows multiple ways of manipulating JSON, including JSONPath. For interoperability, set the YERTL_FORMAT environment variable to "json".

AUTHOR

Doug Bell preaction@cpan.org

CONTRIBUTOR

Luke Triantafyllidis ltriant@cpan.org

COPYRIGHT AND LICENSE

This software is copyright (c) 2016 by Doug Bell.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.