Add parquet-scrooge sources

[isnotinvain]

This adds parquet-scrooge sources, and does some cleanup to prevent duplication between the various parquet sources.

[johnynek]

We are planning to drop 2.9 support after 0.12. Why do the extra work?

On Monday, September 22, 2014, Alex Levenson notifications@github.com
wrote:

This adds parquet-scrooge sources, and does some cleanup to prevent
duplication between the various parquet sources.

This can't be merged until parquet publishes a scala 2.9 artifact for

parquet-scrooge.

You can merge this Pull Request by running

git pull https://github.com/twitter/scalding alexlevenson/parquet-scrooge

Or view, comment on, or merge it at:

#1064
Commit Summary

• Add parquet-scrooge sources

File Changes

• M project/Build.scala
  https://github.com/twitter/scalding/pull/1064/files#diff-0 (12)
• A scalding-parquet-scrooge/README.md
  https://github.com/twitter/scalding/pull/1064/files#diff-1 (3)
• A
  scalding-parquet-scrooge/src/main/scala/com/twitter/scalding/parquet/scrooge/ParquetScrooge.scala
  https://github.com/twitter/scalding/pull/1064/files#diff-2 (30)
• A
  scalding-parquet-scrooge/src/test/scala/com/twitter/scalding/parquet/scrooge/ParquetScroogeTests.scala
  https://github.com/twitter/scalding/pull/1064/files#diff-3 (68)
• M scalding-parquet/README.md
  https://github.com/twitter/scalding/pull/1064/files#diff-4 (6)
• M
  scalding-parquet/src/main/scala/com/twitter/scalding/parquet/thrift/ParquetThrift.scala
  https://github.com/twitter/scalding/pull/1064/files#diff-5 (19)
• M
  scalding-parquet/src/test/scala/com/twitter/scalding/parquet/ParquetSourcesTests.scala
  https://github.com/twitter/scalding/pull/1064/files#diff-6 (87)

Patch Links:

• https://github.com/twitter/scalding/pull/1064.patch
• https://github.com/twitter/scalding/pull/1064.diff

—
Reply to this email directly or view it on GitHub
#1064.

Oscar Boykin :: @posco :: http://twitter.com/posco

[isnotinvain]

@johnynek well, at that point, we'll need a 2.11 artifact. So same problem whether it's 2.9/2.10 or 2.10/2.11 -- parquet publishes 2.10 only currently.

Using a %% in:
"com.twitter" %% "parquet-scrooge" % "1.6.0rc2"
Is not going to work with anything other than 2.10 currently.

[isnotinvain]

Refactored to just support 2.10

[ianoc]

Why delta from the other projects handling of this? I'm a big fan of being consistent.

https://github.com/twitter/bijection/blob/develop/project/Build.scala#L13

[johnynek]

I missed that. This is a non-exhaustive match that will fail when we try to use 2.11. I guess we'll see it then and have to fix it.

[isnotinvain]

Yeah the one we looked at in chill was not exhaustive either:
https://github.com/twitter/chill/blob/0.4.0/project/Build.scala#L151
(though it is now, we were looking at an older revision).

I just assumed it was not exhaustive on purpose (throw an exception for unrecognized scala version)
The reason I changed this to return the version instead of true / false was I figured it'd be more helpful for modules to have the version to make decisions on. Though I guess if we plan to always be in 2 versions supported at a time world, then mapping to true/false is effectively the same.

I can send an PR to make this match the other projects though.