Make server_urls dynamically reloadable

[wolframhaussig]

closes #723

PropertyFileConfigurationSource now reloads changes done to the config file(currently every minute).

Checklist

• Implement code
• Add tests
• Update documentation
• Update CHANGELOG.md

[felixbarny]

Hi and thanks for the PR!

The agent already supports reloading the configuration file:

apm-agent-java/apm-agent-core/src/main/java/co/elastic/apm/agent/impl/ElasticApmTracerBuilder.java

Line 119 in 35f13a9

configurationRegistry.scheduleReloadAtRate(30, TimeUnit.SECONDS);

So refreshing the values for server_urls is not the problem here. It's rather that after initialization, it's not read from the configuration anymore. One of the reasons is that we shuffle the list in order to achieve load balancing. Another reason is that we have explicitly set the dynamic flag to false which signals users (this is also gets rendered in the configuration docs) that this is an option which is not reloadable at runtime:

apm-agent-java/apm-agent-core/src/main/java/co/elastic/apm/agent/report/ReporterConfiguration.java

Lines 57 to 72 in 35f13a9

	private final ConfigurationOption<List<URL>> serverUrl = ConfigurationOption.urlsOption()
	.key("server_urls")
	.aliasKeys("server_url")
	.configurationCategory(REPORTER_CATEGORY)
	.label("The URLs for your APM Servers")
	.description("The URLs must be fully qualified, including protocol (http or https) and port.\n" +
	"\n" +
	"Fails over to the next APM Server URL in the event of connection errors.\n" +
	"Achieves load-balancing by shuffling the list of configured URLs.\n" +
	"When multiple agents are active, they'll tend towards spreading evenly across the set of servers due to randomization.\n" +
	"\n" +
	"If outgoing HTTP traffic has to go through a proxy," +
	"you can use the Java system properties `http.proxyHost` and `http.proxyPort` to set that up.\n" +
	"See also [Java's proxy documentation](https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html) for more information.")
	.dynamic(false)
	.buildWithDefault(Collections.singletonList(UrlValueConverter.INSTANCE.convert("http://localhost:8200")));

In a PR which is currently in review, I have also refactored a lot in that area and created an ApmServerClient.

So it makes sense to wait for that PR to land until we tackle the challenge of reading the updated configuration. We can either do this inside of the ApmServerClient or we can swap out the references to the client for all users with a new instance of the client when the configuration changes. I haven't really made up my mind yet on which option is better. Probably the first is a bit easier. The configuration option allows us to add a change listener via org.stagemonitor.configuration.ConfigurationOption#addChangeListener.

[felixbarny]

sorry for not providing more detail earlier 🙈