Skip to content

[FLINK-7766] [FLINK-7767] [file system sink] Cleanups in the streaming FS sinks#4780

Merged
asfgit merged 2 commits intoapache:masterfrom
StephanEwen:fs_cleanups
Oct 6, 2017
Merged

[FLINK-7766] [FLINK-7767] [file system sink] Cleanups in the streaming FS sinks#4780
asfgit merged 2 commits intoapache:masterfrom
StephanEwen:fs_cleanups

Conversation

@StephanEwen
Copy link
Copy Markdown
Contributor

This build on top of #4776 - only the last two commits are relevant.

What is the purpose of the change

This change gets rid of some legacy code and improves Hadoop configuration access.

Brief change log

  • Drop obsolete reflective hflush() calls. This was done reflectively before for Hadoop 1 compatibility.
    Since Hadoop 1 is no longer supported, this is obsolete now.
  • Avoid loading Hadoop conf dynamically at runtime. That way we guarantee consistent FS config use.

Verifying this change

This change is already covered by the existing tests for the BucketingSink and RollingSink.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

This was done reflectively before for Hadoop 1 compatibility.
Since Hadoop 1 is no longer supported, this is obsolete now.
@aljoscha
Copy link
Copy Markdown
Contributor

aljoscha commented Oct 6, 2017

This LGTM! 👍 I looked at the last two commits here and also the predecessor commits from #4776

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants