[SPARK-7110] [CORE] when use saveAsNewAPIHadoopFile, sometimes it throws "Delegation Token can be issued only with kerberos or web authentication"

[srowen]

Call addCredentials in saveAsNewAPIHadoopDataset too to propagate credential for new-API calls.

@tgravescs -- to your comment, new NewAPIHadoopJob(hadoopConf) does merge the credentials but I don't see they were set in hadoopConf before? or did I miss your point entirely?

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #32384 has started for PR 6052 at commit 0c6e680.

[SparkQA]

Test build #32384 has finished for PR 6052 at commit 0c6e680.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32384/
Test PASSed.

[tgravescs]

See the constructors:

=> first calls this one:
public Job(Configuration conf) throws IOException {
this(new JobConf(conf));
} => calls below constructor

Job(JobConf conf) throws IOException {
super(conf, null);
// propagate existing user credentials to job
this.credentials.mergeAll(this.ugi.getCredentials());
this.cluster = null;
}

It merges what is in the current ugi credentials into the Job. The Job this.credentials is just pointing to the confs credentials ( this.credentials = this.conf.getCredentials();) so when we call the job.getConfiguration it should return the conf populated with the credentials.

This should be exactly the same as what you have coded up. Well except for its doing the merge twice now.

I would really like this person to respond with the exact conditions this happened and stack trace. Have you reproduced it?

[srowen]

Got it, I will probably close this shortly, but let's see if we can finish that conversation.