Fix: no DeadLetters when publishing to a Topic with no subscribers

[eaba]

The summary of the PubSub design is this, as it relates to #5352, if there are no subscribers, wait on the Deadline and if no new subscriber within that window terminate the Topic actor.

It is the waiting on the Deadline that is the actual issue.

[eaba]

This relies on the Topic ActorRef termination and if we want it to publish to deadletters when there are zero subscribers, we will need to change the current design:

akka.net/src/contrib/cluster/Akka.Cluster.Tools/PublishSubscribe/DistributedPubSubMediator.cs

Lines 497 to 529 in dbe4999

	private void IgnoreOrSendToDeadLetters(object message)
	{
	if (_settings.SendToDeadLettersWhenNoSubscribers)
	Context.System.DeadLetters.Tell(new DeadLetter(message, Sender, Context.Self));
	}
	private void PublishMessage(string path, object message, bool allButSelf = false)
	{
	IEnumerable<IActorRef> Refs()
	{
	foreach (var entry in _registry)
	{
	var address = entry.Key;
	var bucket = entry.Value;
	if (!(allButSelf && address == _cluster.SelfAddress) && bucket.Content.TryGetValue(path, out var valueHolder))
	{
	if (valueHolder != null && !valueHolder.Ref.IsNobody())
	yield return valueHolder.Ref;
	}
	}
	}
	var counter = 0;
	foreach (var r in Refs())
	{
	if (r == null) continue;
	r.Forward(message);
	counter++;
	}
	if (counter == 0) IgnoreOrSendToDeadLetters(message);
	}

[Aaronontheweb]

Left some comments on the structure of the test

[eaba]

@Aaronontheweb improved the test with the latest commit

[Aaronontheweb]

Left some comments on the changes included in here

[Aaronontheweb]

Reproduction - as @eaba pointed out this spec would fail before because the TTL value kept the topic actor alive for 120s. The real bug was that the topic actor never published any DeadLetter instances if it was alive but had no subscribers.

[Aaronontheweb]

Propagate akka.cluster.pub-sub.send-to-dead-letters-when-no-subscribers to the topic actors, who are the ones who actually need to implement this.

[Aaronontheweb]

Created an envelope NoSubscribersDeadLetter in order to help make it clear inside the user's logging system why this DeadLetter was published. Created it as a readonly struct because it will immediately be captured inside a DeadLetter and therefore we can save on an allocation here in the event of a large number of deadletters being published all at once.

[Aaronontheweb]

Updated constructor to receive new value. These are all internal classes so no breaking changes here.

[Aaronontheweb]

The bugfix: @eaba pointed out that this actor never was programmed to publish to deadletters when the subscriber count is zero, so we do this after the foreach loop (arbitrary) if we're configured for it. This causes the reproduction spec to now pass.