Skip to content

[Backport] Fix the Injector creation in HadoopTask (#13138) #13432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kfaraz merged 1 commit into from
Nov 24, 2022
Merged

[Backport] Fix the Injector creation in HadoopTask (#13138) #13432

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions indexing-service/src/main/java/org/apache/druid/indexing/common/task/HadoopIndexTask.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,7 @@ private TaskStatus runInternal(TaskToolbox toolbox) throws Exception
try {
registerResourceCloserOnAbnormalExit(config -> killHadoopJob());
String hadoopJobIdFile = getHadoopJobIdFileName();
logExtensionsConfig();
final ClassLoader loader = buildClassLoader(toolbox);
boolean determineIntervals = spec.getDataSchema().getGranularitySpec().inputIntervals().isEmpty();

Expand Down
11 changes: 10 additions & 1 deletion indexing-service/src/main/java/org/apache/druid/indexing/common/task/HadoopTask.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ public abstract class HadoopTask extends AbstractBatchIndexTask
{
private static final Logger log = new Logger(HadoopTask.class);

static final Injector INJECTOR = new StartupInjectorBuilder().withExtensions().build();
static final Injector INJECTOR = new StartupInjectorBuilder().forServer().build();
private static final ExtensionsLoader EXTENSIONS_LOADER = ExtensionsLoader.instance(INJECTOR);

private final List<String> hadoopDependencyCoordinates;
Expand Down Expand Up @@ -207,6 +207,15 @@ public static ClassLoader buildClassLoader(final List<String> hadoopDependencyCo
return classLoader;
}

/**
* This method logs the {@link ExtensionsConfig} that was used to fetch the hadoop dependencies and build the classpath
* for the jobs
*/
protected static void logExtensionsConfig()
{
log.info("HadoopTask started with the following config:\n%s", EXTENSIONS_LOADER.config().toString());
}

/**
* This method tries to isolate class loading during a Function call
*
Expand Down