Do not update metadata cache on metadata call

[richardlarocque]

The call to metadata can include a list of topics. If it does, the
metadata we receive will include metadata only for the specified topics.

We don't want to replace our cache of all-topic metadata with a
topic-specific response. We should leave the cache as-is.

A slightly more optimal way to do this would be to update the metadata
cache only if the metadata request did not specify any topics, but that
seems a bit more effort than it's worth.

[sourcelevel-bot]

Hello, @richardlarocque! This is your first Pull Request that will be reviewed by SourceLevel, an automatic Code Review service. It will leave comments on this diff with potential issues and style violations found in the code as you push new commits. You can also see all the issues found on this Pull Request on its review page. Please check our documentation for more information.

[richardlarocque]

I tried to run the tests on this PR locally, as described in CONTRIBUTING.md. They didn't quite work, but they didn't work for me on master either, so this does not seem to be a regression specific to this PR.

---

The idea behind this PR is as follows.

When we try to produce a message, the client will end up having to update its metadata if it doesn't currently have metadata for the given topic (

kafka_ex/lib/kafka_ex/server.ex

Lines 437 to 459 in 9255704

	nil ->
	{retrieved_corr_id, _} =
	retrieve_metadata(
	state.brokers,
	state.correlation_id,
	config_sync_timeout(),
	produce_request.topic,
	state.api_versions
	)
	state =
	update_metadata(%{state | correlation_id: retrieved_corr_id})
	{
	MetadataResponse.broker_for_topic(
	state.metadata,
	state.brokers,
	produce_request.topic,
	produce_request.partition
	),
	state,
	retrieved_corr_id
	}

). That logic seems to assume (reasonably) that the contents of the metadata state variable are an all-topic metadata response.

However, an explicit request for topic-specific metadata could, prior to the change presented here, replace the all-topic metadata maintained by the periodic timer with a topic-specific metadata response. That would be bad because the next produce request would then be more likely to have to go re-fetch all-topic metadata.

That would be inefficient. Prior to #395, this bug would result in sending a lot of traffic at the "lead" broker, potentially overwhelming it with all-topic metadata requests. I suspect that PR partitally mitigates the effect since the metadata requests would be more evenly spread out and you could horizontally-scale your way past the problem, but still, the inefficiency would remain.

[joshuawscott]

This seems reasonable to me. Thanks for the PR!

[jbruggem]

@joshuawscott what's our policy on approvals ? it seems that 2 is enough for such a simple change.

[joshuawscott]

oh yeah, 2 is plenty.

[yingyingtang-brex]

do we need to update the other server implementation with this change, e.g. here?