From 4732462f2bee6c48819295202a44bd48446ee44e Mon Sep 17 00:00:00 2001 From: Matt Bossenbroek Date: Mon, 19 Jan 2015 12:17:23 -0800 Subject: [PATCH] Update version --- README.md | 64 +++++++++++++++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index fc2e86b..d7c69f1 100644 --- a/README.md +++ b/README.md @@ -27,13 +27,13 @@ _Note: PigPen is **not** a Clojure wrapper for writing Pig scripts you can hand With Leiningen: ``` clj -[com.netflix.pigpen/pigpen "0.2.12"] +[com.netflix.pigpen/pigpen "0.2.13"] ``` With Gradle: ``` groovy -compile "com.netflix.pigpen:pigpen:0.2.12" +compile "com.netflix.pigpen:pigpen:0.2.13" ``` With Maven: @@ -42,7 +42,7 @@ With Maven: com.netflix.pigpen pigpen - 0.2.12 + 0.2.13 ``` @@ -50,31 +50,43 @@ _Note: PigPen requires Clojure 1.5.1 or greater_ ## Parquet -To use the parquet loader, add this to your dependencies: +To use the parquet loader (alpha), add this to your dependencies: ``` clj -[com.netflix.pigpen/pigpen-parquet "0.2.12"] +[com.netflix.pigpen/pigpen-parquet "0.2.13"] ``` And because of the weird way Pig handles dependencies, you'll also need to add antlr and log4j for local mode to work properly: ``` clj :dependencies [[org.clojure/clojure "1.6.0"] - [com.netflix.pigpen/pigpen "0.2.12"] - [com.netflix.pigpen/pigpen-parquet "0.2.12"]] + [com.netflix.pigpen/pigpen "0.2.13"] + [com.netflix.pigpen/pigpen-parquet "0.2.13"]] :profiles {:dev {:dependencies [[org.apache.pig/pig "0.11.1"] [org.apache.hadoop/hadoop-core "1.1.2"] [org.antlr/antlr "3.5.2"] [log4j/log4j "1.2.17"]]}} ``` +## Avro + +To use the avro loader (alpha), add this to your dependencies: + +``` clj +[com.netflix.pigpen/pigpen-avro "0.2.13"] +``` + # Release Notes - * 0.2.12 - * Added load-csv, which allows for quoting per RFC 4180 - * 0.2.11 + * 0.2.13 - 1/19/15 + * Added `load-avro` in the pigpen-avro project: http://avro.apache.org/ + * Fixed the nRepl configuration; use `gradlew nRepl` to start an nRepl + * Exclude nested relations from closures + * 0.2.12 - 12/16/14 + * Added `load-csv`, which allows for quoting per RFC 4180 + * 0.2.11 - 10/24/14 * Fixed a bug (feature?) introduced by new rx version. Also upgraded to rc7. This would have only affected local mode where the data being read was faster than the code consuming it. - * 0.2.10 + * 0.2.10 - 9/21/14 * Removed load-pig and store-pig. The pig data format is very bad and should not be used. If you used these and want them back, email pigpen-support@googlegroups.com and we'll put it into a separate jar. The jars required for this feature were causing conflicts elsewhere. * Upgraded the following dependencies: * org.clojure/clojure 1.5.1 -> 1.6.0 - this was also changed to a provided dependency, so you should be able to use any version greater than 1.5.1 @@ -85,46 +97,46 @@ And because of the weird way Pig handles dependencies, you'll also need to add a * instaparse 1.2.14 - no longer needed * io.reactivex/rxjava 0.9.2 -> 1.0.0-rc.1 * Fixed the rx limit bug. `pigpen.local/*max-load-records*` is no longer required. - * 0.2.9 + * 0.2.9 - 9/16/14 * Fix a local-mode bug in `pigpen.fold/avg` where some collections would produce a NPE. * Change fake pig delimiter to \n instead of \0. Allows for \0 to exist in input data. * Remove 1000 record limit for local-mode. This was originally introduced to mitigate an rx bug. Until #61 is fixed, bind `pigpen.local/*max-load-records*` to the maximum number of records you want to read locally when reading large files. This now defaults to `nil` (no limit). * Fix a local dispatch bug that would prevent loading folders locally - * 0.2.8 + * 0.2.8 - 7/31/14 * Fix a bug in `load-tsv` and `load-lazy` - * 0.2.7 *** Don't use *** + * 0.2.7 - 7/31/14 *** Don't use *** * Fix `load-lazy` and speed up both `load-tsv` and `load-lazy` * Convert to multi-project build * Added pigpen-parquet with initial support for loading the Parquet format: https://github.com/apache/incubator-parquet-mr - * 0.2.6 + * 0.2.6 - 6/17/14 * Minor optimization for local mode. The creation of a UDF was occurring for every value processed, causing it to run out of perm-gen space when processing large collections locally. * Fix `(pig/return [])` * Fix `(pig/dump (pig/reduce + (pig/return [])))` * Fix `Long`s in scripts that are larger than an Integer * Memoize local UDF instances per use of `pig/dump` * The jar location in the generated script is now configurable. Use the `:pigpen-jar-location` option with `pig/generate-script` or `pig/write-script`. - * 0.2.5 + * 0.2.5 - 4/9/14 * Remove `dump&show` and `dump&show+` in favor of `pigpen.oven/bake`. Call `bake` once and pass to as many outputs as you want. This is a breaking change, but I didn't increment the version because `dump&show` was just a tool to be used in the REPL. No scripts should break because of this change. * Remove `dymp-async`. It appeared to be broken and was a bad idea from the start. * Fix self-joins. This was a rare issue as a self join (with the same key) just duplicates data in a very expensive way. * Clean up functional tests * Fix `pigpen.oven/clean`. When it was pruning the graph, it was also removing REGISTER commands. - * 0.2.4 + * 0.2.4 - 4/2/14 * Fix arity checking bug (affected varargs fns) * Fix cases where an Algebraic fold function was falling back to the Accumulator interface, which was not supported. This affected using `cogroup` with `fold` over multiple relations. * Fix debug mode (broken in 0.1.5) * Change UDF initialization to not rely on memoization (caused stale data in REPL) * Enable AOT. Improves cluster perf * Add `:partition-by` option to `distinct` - * 0.2.3 + * 0.2.3 - 3/27/14 * Added `load-json`, `store-json`, `load-string`, `store-string` * Added `filter-by`, and `remove-by` - * 0.2.2 + * 0.2.2 - 3/25/14 * Fixed bug in `pigpen.fold/vec`. This would also cause `fold/map` and `fold/filter` to not work when run in the cluster. - * 0.2.1 + * 0.2.1 - 3/24/14 * Fixed bug when using `for` to generate scripts * Fixed local mode bug with `map` followed by `reduce` or `fold` - * 0.2.0 + * 0.2.0 - 3/3/14 * Added pigpen.fold - Note: this includes a breaking change in the join and cogroup syntax as follows: ``` clj @@ -141,17 +153,17 @@ And because of the weird way Pig handles dependencies, you'll also need to add a Each of the select clauses must now be wrapped in a vector - there is no longer a varargs overload to either of these forms. Within each of the select clauses, :on is now a keyword instead of a symbol, but a symbol will still work if used. If `optional` or `required` were used, they must be updated to `:type :optional` and `:type :required`, respectively. - * 0.1.5 + * 0.1.5 - 2/17/14 * Performance improvements * Implemented Pig's Accumulator interface * Tuned nippy * Reduced number of times data is serialized - * 0.1.4 + * 0.1.4 - 1/31/14 * Fix sort bug in local mode - * 0.1.3 + * 0.1.3 - 1/30/14 * Change Pig & Hadoop to be transitive dependencies * Add support for consuming user code via closure - * 0.1.2 + * 0.1.2 - 1/3/14 * Upgrade instaparse to 1.2.14 - * 0.1.1 + * 0.1.1 - 1/3/14 * Initial Release