Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-9684][historyserver] HistoryServer should initialize FileSystem before using it #6225

Closed
wants to merge 1 commit into from

Conversation

Ethanlm
Copy link
Contributor

@Ethanlm Ethanlm commented Jun 28, 2018

What is the purpose of the change

Details explained in https://issues.apache.org/jira/browse/FLINK-9684 . This pull request makes HistoryServer initialize FileSystem before using it. That way HistoryServerArchiveFetcher will be able to talk to secure hdfs correctly.

Brief change log

  • HistoryServer initializes FileSystem before using it

Verifying this change

This change added tests and can be verified as follows:

  • Manually verified the change by running a historyserver with kerberos-secured hdfs cluster.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

@Ethanlm Ethanlm changed the title [FLINK-9684][historyserver] HistoryServer should initialize FileSystem before using it' [FLINK-9684][historyserver] HistoryServer should initialize FileSystem before using it Jun 28, 2018
@Ethanlm
Copy link
Contributor Author

Ethanlm commented Jun 28, 2018

Just realized fs.hdfs.hadoopconf is deprecated and we want to use HADOOP_CONF_DIR instead.
https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java#L652-L658

Not sure why we are doing it. But if HADOOP_CONF_DIR is set explicitly, HistoryServer should work fine.

However, I believe we still need to initialize FileSystem before using it.

@Ethanlm Ethanlm closed this Jun 28, 2018
@Ethanlm Ethanlm reopened this Jun 28, 2018
Copy link
Contributor

@tillrohrmann tillrohrmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for opening the PR @Ethanlm. Changes look good to me. Merging this PR.

asfgit pushed a commit that referenced this pull request Jun 30, 2018
@asfgit asfgit closed this in c280898 Jun 30, 2018
sampathBhat pushed a commit to sampathBhat/flink that referenced this pull request Jul 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants