[FLINK-1980] allowing users to decorate input streams

[sekruse]

• add a decorateInputStream() method as hook in FileInputFormat
• provide a InputStreamFSInputWrapper to conveniently wrap InputStreams
• base existing .deflate file support on these changes
• add a test to verify the decoration

[rmetzger]

Thank you for this pull request. I'll review it now.

The first thing I saw is that the build has failed:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.728 s
[INFO] Finished at: 2015-05-07T12:08:03+00:00
[INFO] Final Memory: 25M/448M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (default) on project flink-parent: Too many files with unapproved license: 1 See RAT report in: /home/travis/build/apache/flink/target/rat.txt -> [Help 1]

https://travis-ci.org/apache/flink/jobs/61608003

You can also do a local mvn clean install -DskipTests to run the rat plugin.

[rmetzger]

Missing license header, that's also why the build fails ;)

[rmetzger]

The change looks good, its not completely consistent with our coding guidelines. But a few runs of "mvn install -DskipTests" will resolve this ;)
(also, the indentation is done using spaces, we use tabs).

[sekruse]

Alright, I incorporated all the feedback. The build seems to be passing now.

[rmetzger]

+1 to merge.

[StephanEwen]

Looks good. One comment, though:

• The position tracking in the stream wrapper is a nice idea, but since the skip() method is anyways only a hint (it may skip less if it wants) and seeking works only in one direction, why not skip seek support completely? Also simplifies the code and makes it even more lightweight.

As a followup: Should we add more built-in decompressors for other file endings, like *.gz ?

[rmetzger]

I think Sebastian has already filed a JIRA for adding gz read support.

[sekruse]

I wanted to integrate the gz support with the decorateStream method, therefore I was waiting for this PR to be merged.
I can of course skip the seek, but I thought that forward seeking would be useful as it enables splittable file formats.

[StephanEwen]

That is a good point. It is a bit tricky, though, since the skip method does not guarantee efficient skipping. It also reserves the right to skip less (and return the number of bytes skipped, iirc).

[sekruse]

I see your point and double-checked it with the Java doc. I will adjust the seek method accordingly.

[rmetzger]

I think the PR is good to merge.

[StephanEwen]

Looks good.

Will merge this in my next batch...