This repository has been archived by the owner on Apr 5, 2022. It is now read-only.
[Sprint: 50] XD-3079 Hadoop kerberos config not used in jobs #1657
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- In XD-3079 user complained that kerberos tickets are not renewed. Due to missing kerberos config, these jobs doesn't even work with kerberized hdfs if job is only one accessing hdfs. Because hadoop sec config is a system wide setting in hadoop classes, jobs can access kerberized hdfs if i.e. hdfs sink first did a login. - Having this proper sec config should allow jobs to work beyond kerberos tgt expire and renew times. - Adding same sec config from hdfs sink to hdfsmongodb, hdfsjdbc, jdbchdfs, ftphdfs and filepollhdfs jobs. - In hdfs/hdfs-dataset sinks we support security settings in both hadoop.properties and servers.yml, so property-placeholder is needed solely for hadoop.properties. - In tests remove property-placeholder bean in favor of registering needed properties in context for everything to work with property-placeholder beans in job xml's.
sabbyanandan
changed the title
|
Merged via commit e2deca6 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
not renewed. Due to missing kerberos config, these jobs
doesn't even work with kerberized hdfs if job is only
one accessing hdfs. Because hadoop sec config is a system
wide setting in hadoop classes, jobs can access kerberized
hdfs if i.e. hdfs sink first did a login.
beyond kerberos tgt expire and renew times.
hdfsjdbc, jdbchdfs, ftphdfs and filepollhdfs jobs.
hadoop.properties and servers.yml, so property-placeholder
is needed solely for hadoop.properties.
needed properties in context for everything to
work with property-placeholder beans in job xml's.