Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 107 lines (68 sloc) 3.634 kB
947156b Big cleanup of the examples/ directory
Philip (flip) Kromer authored
1 h2. Wukong v2.0.0
2
3 h4. Important changes
4
5 * Passing options to streamers is now deprecated. Use @Settings@ instead.
6
7 * Streamer by default has a periodic monitor that logs (to STDERR by default) every 10_000 lines or 30 seconds
8
9 * Examples cleaned up, should all run
10
11 h4. Simplified syntax
12
13 * you can now pass Script.new an *instance* of Streamer to use as mapper or reducer
14 * Adding an experimental sugar:
15
16 <pre>
17 #!/usr/bin/env ruby
18 require 'wukong/script'
19
20 LineStreamer.map do |line|
21 emit line.reverse
22 end.run
23 </pre>
24
25 Note that you can now tweet a wukong script.
26
27 * It's now recommended that at the top of a wukong script you say
28 <pre>
29 require 'wukong/script'
30 </pre>
31 Among other benefits, this lets you refer to wukong streamers without prefix.
32
8d749b1 CHANGELOG updated: new emr work
Philip (flip) Kromer authored
33 h2. Wukong v1.5.4
34
35 * EMR support now works very well
36
4a61609 Version bump
Philip (flip) Kromer authored
37 h2. Wukong v1.5.3
38
39 * A couple of bugfixes. Sorry about that.
40 * Documentation fixes
41
b795bf9 Changelog upd. for new version
Philip (flip) Kromer authored
42 h2. Wukong v1.5.0
43
44 h4. Elastic Map-Reduce
45
46 Use --run=emr to launch a job onto the Amazon Elastic MapReduce cloud.
47
48 * copies the script to s3, as foo-mapper.rb and foo-reducer.rb (removing the need for the --map flag)
49 * copies the wukong libs up as a .tar/bz2, and extracts it into the cwd
50 * combines settings from commandline and yaml config, etc to configure and launch job
51
52 It's still **way** shaky and I don't think anything but the sample app will run. That sample app runs, tho.
53
54 h4. Greatly simplified script launching.
2e220c9 Reworked Script to have a much more coherent and compelling plot
Philip (flip) Kromer authored
55
56 Incompatible changes to option handling and script launching:
57 * Script doesn't use extra_options any more. You should relocate them to the initializer or to configliere.
58 * there is no more default_mapper or default_reducer
59
0b64632 Resolved trivial conflict in CHANGELOG
Philip (flip) Kromer authored
60 h2. Wukong v.14.12 2010-08-31
48e1ca4 minor update to the tyrant store
Philip (flip) Kromer authored
61
62 * Improvements to the pig conversion methods
63 * @hdp-rm@ respects the -skipTrash method
64
2e220c9 Reworked Script to have a much more coherent and compelling plot
Philip (flip) Kromer authored
65
66 h2. Wukong v1.4.11 2010-07-30
6a857c7 version bump
Philip (flip) Kromer authored
67
68 * added the @max_(maps|reduces)_per_(node|cluster)@ jobconfs.
69 * added jobconfs for io_job_mb and friends.
70 * added a loadable module to convert output data to pig bags and tuples
71 * pulled in several methods from active_support, incl. Enumerable#sum
72 * Scripts to find percentile rank of elements in a dataset
73 * We are starting to move wukong to a model where streaming is from a generic
74 source into a generic sink. Several stores have been landed in the code, but
75 many are in a half- or un-baked state. Please ignore this for the moment.
76
f927134 bump version -- CHANGELOG
Philip (flip) Kromer authored
77 h2. Wukong v1.4.8 2010-06-05
78
79 * made scripts inject a helpful job name using mapred.job.name
80 * Hash.compact_blank! and HashLike.compact_blank! -- eliminate all key-values whoes value is blank?
81
0ac5854 version bump
Philip (flip) Kromer authored
82 h2. Wukong v1.4.8 2010-05-17
83
84 * Bug in passing commandline args down to map and reduce child processes
85
b087c05 making gem version
Philip (flip) Kromer authored
86 h2. Wukong v1.4.7 2010-03-05
87
88 Lots more examples:
89 * examples/stats/avg_value_frequency.rb does an Average Value Frequency histogram
90 * examples/server_logs has a quite useful apache log file parser
91 * Made the base streamer use each_record, opening the door for alternative record injection (eg Datamapper!)
92 * wukong/streamer/counting_reducer.rb is an um reducer and it counts things.
93
94 h2. Wukong v1.4.6 2010-01-26
95
96 * A HELLA AWESOME working example from retail web analytics by @lenbust
97
7bed0a9 By request, in local mode, you can use '-' to mean STDIN or STDOUT fo…
Philip (flip) Kromer authored
98 h2. Wukong v1.4.5 2010-01-18
99
100 * In @--run=local@ mode, you can use '-' alone as a filename to indicate STDIN / STDOUT as input/output respectively.
101 * Minor tweaks to contrib/jeans
102
aa40eac gemspec needs dependency on configliere; light cleanup on contrib script
Philip (flip) Kromer authored
103 h2. Wukong v1.4.4 2010-01-15
104
105 * Moved settings management & command line handling over to "Configliere":http://github.com/mrflip/configliere (
106 * Added "example script and notes":http://github.com/fredrikmollerstrand/hackblag/blob/master/wukong/notes.markdown from Fredrik Möllerstrand (@lenbust)
Something went wrong with that request. Please try again.