Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Noting the problems I have discovered with this design.

  • Loading branch information...
commit 795cf99429f00b0bf49af99a6aeb6ea191f027aa 1 parent 232d702
@JEG2 authored
Showing with 45 additions and 0 deletions.
  1. +8 −0 README
  2. +29 −0 lib/mungr/dsl.rb
  3. +8 −0 lib/mungr/script.rb
View
8 README
@@ -1,3 +1,11 @@
= Mungr
Mungr is a Ruby DSL that makes it easy to read one or more inputs, munge the data around, and write the results to one or more outputs.
+
+Update: The goal is to build a nice DSL, and a command-line program that wraps that, for expressing inputs, outputs, and munging operations. However, the current implementation is flawed.
+
+A munger needs to be less tied to the readers and writers. Currently a munger cannot send more data down to the writers than it receives from readers and each invocation requires a pull from the readers. This makes it impossible to do something simple like read a list of files from a directory but generate lines of input from each of those files.
+
+Another issue is that the roles are very locked in. Reading stock information from a CSV file could handle the CSV conversion as a reader while reading from a URL would require a munge level conversion. This raises the barrier of entry for adding new capabilities to the system.
+
+This needs to be fixed implementation wise and I believe the right approach is to base the system on something like Ruby 1.9's Enumerator or Fiber and just define a minimal interface for pipelining them to each other.
View
29 lib/mungr/dsl.rb
@@ -0,0 +1,29 @@
+# encoding: UTF-8
+
+require "mungr/job"
+
+module Mungr
+ module DSL
+ def mungr_job
+ @mungr_job ||= Job.new
+ end
+
+ def reader(&init)
+ mungr_job << Reader.new(&init)
+ end
+
+ def munger(&init)
+ mungr_job << Munge.new(&init)
+ end
+
+ def munge(&transform)
+ munger do |m|
+ m.munge { |_, *data| transform[*data] }
+ end
+ end
+
+ def writer(&init)
+ mungr_job << Writer.new(&init)
+ end
+ end
+end
View
8 lib/mungr/script.rb
@@ -0,0 +1,8 @@
+# encoding: UTF-8
+
+require "mungr/dsl"
+
+include Mungr::DSL
+at_exit do
+ mungr_job.build.run
+end
Please sign in to comment.
Something went wrong with that request. Please try again.