Skip to content

Commit

Permalink
ZEPPELIN-3870. Configuration in spark-defaults.conf doesn't take effect
Browse files Browse the repository at this point in the history
### What is this PR for?

By default, spark.executor.memory is not set in spark's interpreter setting, if I set it in spark-defaults.xml, then it won't take effect. Because spark.executor.memory in spark's interpreter setting will override that value in spark-defaults.xml. This doesn't make sense, we should only override property in spark-default.xml when you have non-empty value.

### What type of PR is it?
[Bug Fix]

### Todos
* [ ] - Task

### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-3870

### How should this be tested?
* CI pass

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: Jeff Zhang <zjffdu@apache.org>

Closes #3238 from zjffdu/ZEPPELIN-3870 and squashes the following commits:

b6c0017 [Jeff Zhang] ZEPPELIN-3870. Configuration in spark-defaults.conf doesn't take effect when it is empty in interpreter setting
  • Loading branch information
zjffdu authored and jeffzhang.zjf committed Nov 26, 2018
1 parent ae7318f commit acd75b1
Showing 1 changed file with 3 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@

import com.google.gson.Gson;
import com.google.gson.reflect.TypeToken;
import org.apache.commons.lang3.StringUtils;
import org.apache.thrift.TException;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.protocol.TProtocol;
Expand Down Expand Up @@ -343,9 +344,7 @@ private void setSystemProperty(Properties properties) {
for (Object key : properties.keySet()) {
if (!RemoteInterpreterUtils.isEnvString((String) key)) {
String value = properties.getProperty((String) key);
if (value == null || value.isEmpty()) {
System.clearProperty((String) key);
} else {
if (!StringUtils.isBlank(value)) {
System.setProperty((String) key, properties.getProperty((String) key));
}
}
Expand Down Expand Up @@ -470,7 +469,7 @@ public RemoteInterpreterResult interpret(String sessionId, String className, Str
context.getGui(),
context.getNoteGui());
}

class InterpretJobListener implements JobListener {

@Override
Expand Down

0 comments on commit acd75b1

Please sign in to comment.