Hopefully, it will be an awesome Framework to do ETL with Ruby. It should only work with Ruby 1.9.
This README is work in progress
You should not trust it, until I clean it up.
For better examples and documentation, refer to the features folder.
TODO: Everything I want it to be, below:
Define the Itiel project structure
I have in mind something like this:
/itiel_example/ |~config/ | `-sources.yml |~lib/ | `-transformation_class1.rb |~jobs/ | `-main_job.rb |~tasks/ | `-itiel.rake |-Gemfile |-Gemfile.lock
Take care of defining and running the jobs. It's where you put all the code like the example above.
A command line bin to generate the basic structure
itiel new project_name.
Picture yourself migrating form SQL to Mongo
Joins, Scripts and lots more transformations
Some on my mind:
Itiel::Joins::DatabaseTable Itiel::Lookups::DatabaseTable Itiel::Loads::FTP Itiel::Loads::HTTPRequest Itiel::Loads::MongoDBDocument Itiel::Script::ExecuteInSystem Itiel::Extracts::HTTP Itiel::Extracts::MongoDBDocument Itiel::Transforms::CustomSort Itiel::Transforms::MapValues Itiel::Transforms::IfNil
Just ideas, I have nothing on how to implement most of them
Filters and conditional stream routing
If the column X has a value of Y, send true output to this step, send false output to this other steps
Slowly Changing Dimentions
Can't have ETL without these
So you can TDD your ETL
It would be extra nice if all the steps logged statistics
- Processing X rows of Y
- Time spent
- Transformation Name
- Configurable log level
Creates a graphical representation of your ETL flow
Not really, Ruby 1.9 is enough. I don't have plans to make it work on 1.8
About the Author
Crowd Interactive is a Ruby and Rails consultancy firm powered by a team of enthusiast engineers who love programming. We turn your ideas into web applications, and we like challenging projects. We also have a lot of experience in retail, so it doesn't matter if your idea is about something you'd like to sell, we can surely help you.