Remove failed stale producer from the connection

[rdhabalia]

Motivation

As described in #4138, broker tries to clean up stale failed-producer from the connection however, while cleaning up producer-future, it tries to remove newly created producer-future rather old-failed producer because of that broker still gives below error

17:22:00.700 [pulsar-io-21-26] WARN  org.apache.pulsar.broker.service.ServerCnx - [/1.1.1.1:1111][453] Producer with id persistent://prop/cluster/ns/topic is already present on the connection

[rdhabalia]

rerun java8 tests

[sijie]

retest this please

[merlimat]

@rdhabalia The check for the same future on delete was there to prevent deleting the wrong future, eg. after a timeout from client that resend the create-producer command. The tests were actually simulating this condition.

[rdhabalia]

@merlimat
No, actually check was added as part of #4145 and #4138 to fix the same issue (Producer/Consumer is already connected) but that PR was trying to remove wrong producer/consumer which is not present into the cache so, underlying issue was not fixed.

I modified the test because test was wrong and having exact same issue which we are trying to fix: test-sequence was

1. try to create consumer (which will not be created immediately because test has delayed topic creation logic)
2. close the consumer: which fails the pending-consumer future and doesn't clean up from the cache.
3. so, any subsequent consumer request always receives error: Consumer with id is already connected. Test was not checking failure reason of consumer : consumer is already connected else we could have caught it long time back.
   Because broker was never cleaning failed future and because of the consumer will never be able to connect.

Same thing was happening for producers and we are keep seeing this issue when broker unloads the bundle to different broker and same bundle again comes back to the old broker. the entire issue was described into #4138. So, we tried to fix in #4138 but tried to remove producer-future for current request which is definitely not in the cache and therefore, client keeps seeing error Producer is already connected and we have to manually restart the broker to fix this issue.

[rdhabalia]

rerun integration tests

[sijie]

@rdhabalia you might need to rebase to latest master to fix the integration tests issues.

[rdhabalia]

retest this please

[wolfstudy]

Change the Milestone to 2.4.2, because of conflict.