SAMZA-2498: [Job coordinator isolation] Classloader isolation for SamzaApplication.describe execution in job coordinator #1337
Conversation
| * If {@code canApplyFrameworkIsolationIfEnabled} is true and isolation is enabled (in the config), then this will | ||
| * build the {@link SamzaApplication} using a special isolating classloader. Otherwise, this will use the current |
There was a problem hiding this comment.
why do we need two levels (this boolean & the config) to determine the behavior? Can we not infer from the config and then use the appropriate loader?
There was a problem hiding this comment.
Currently, we only support using this classloader on the job coordinator. So we can't use this new flow on the processing containers nor RemoteApplicationRunner yet. Once we support split deployment on the processing containers and remove the need from RemoteApplicationRunner, then we can remove this flag. I will update the doc with that clarification.
| // almost everything should come from the parent classloader | ||
| .addDelegatePreferredClassPredicate(new GlobMatcher("*")) | ||
| // blacklisting the SamzaApplication class from the parent so that it gets loaded by this classloader | ||
| .addBlacklistedClassPredicate(new GlobMatcher(samzaApplicationClassName)) |
There was a problem hiding this comment.
Is it possible that by only blacklisting samzaApplicationClass, we end up class loading some of the dependencies of user code through parent class loader assuming both parent and application class loader are capable of classing loading that class?
There was a problem hiding this comment.
Ah yes, that is a good catch. I need to think through what the impact of that is.
|
This approach does not properly satisfy the isolation requirements. I will open up a separate PR with a different solution. |
Feature: https://cwiki.apache.org/confluence/display/SAMZA/SEP-24%3A+Cluster-based+Job+Coordinator+Dependency+Isolation#SEP-24:Cluster-basedJobCoordinatorDependencyIsolation-HandlingSamzaApplication.describe
Changes:
IsolatingClassLoaderFactory.buildSamzaApplicationClassLoaderto construct a classloader for loading theSamzaApplicationimplementation for an app.buildSamzaApplicationClassLoaderto load theSamzaApplicationimplementation in the job coordinator.Tests:
Deployed
wikipedia-feedapplication (uses low-level API, has custom input descriptor, uses Kafka output descriptor) andwikipedia-parserapplication (uses low-level API, Kafka descriptors) in split deployment mode with some debug logs, and verified the following:TaskApplicationDescriptorImplandKafkaSystemDescriptorcame from the framework infrastructure classloader.WikipediaSystemDescriptor(custom descriptor) came from the application classloader.WikipediaFeedTaskApplication(impl ofSamzaApplication) came from the newly built application describe classloader.TaskFactoryinWikipediaFeedTaskApplicationcame from the newly built application describe classloader, and it implemented theStreamTaskFactoryinterface which was loaded by the framework API classloader.WikipediaSystemFactory)Deployed
wikipedia-parserapplication (uses low-level API, Kafka descriptors) and verified the classloaders using debug logs.API/usage changes: None