Task:
Adds a factory method to create an AtumAgent instance backed by an explicitly provided Typesafe Config, enabling multiple agents with different dispatcher setups within the same JVM.
Context:
The above PRs updates the deployment pipeline to extract, package, and publish the external TypeSafe config for the data-measurement Spark plugin and ensure it is shipped to Spark via --files:
This supports the “external first, classpath fallback” strategy within AUL: the plugin first attempts to load application.conf distributed through Spark --files, and falls back to the classpath config if no external file is available.
Essentially, we want to externalize all configuration files. So, it is easier for the Ops teams and Dev teams to make changes to the configuration files without a need to always release a new version of Atum-Services or Unify. But since Atum-Services is in the codebase of Unify, a small change is needed on Atum-Services to be able to externalize these config files.
import com.typesafe.config.ConfigFactory
import za.co.absa.atum.agent.AtumAgent
val config = ConfigFactory.load()
val customAgent = AtumAgent.fromConfig(config)
Task:
Adds a factory method to create an
AtumAgentinstance backed by an explicitly providedTypesafe Config, enabling multiple agents with different dispatcher setups within the same JVM.Context:
The above PRs updates the deployment pipeline to extract, package, and publish the external TypeSafe config for the data-measurement Spark plugin and ensure it is shipped to Spark via
--files:This supports the “external first, classpath fallback” strategy within AUL: the plugin first attempts to load
application.confdistributed through Spark--files, and falls back to the classpath config if no external file is available.Essentially, we want to externalize all configuration files. So, it is easier for the Ops teams and Dev teams to make changes to the configuration files without a need to always release a new version of Atum-Services or Unify. But since Atum-Services is in the codebase of Unify, a small change is needed on Atum-Services to be able to externalize these config files.