Do not cache failed futures for async security policies indefinitely.

[mateusazis]

Currently, if caching is enabled (as is often the case) and AsyncSecurityPolicy returns a failed future, then this future is cached forever, without giving the SecurityPolicy implementation a chance to be retried. Going forward, new invocations will trigger new security checks if the last one could not be completed successfuly.

Part of #10566.

[jdcormie]

Can you share more context on this PR? I remember we discussed not having more than one async security policy check against the same uid in-flight at a time. But is there a reason to not cache negative outcomes too? How should we think about this change in the context of the SecurityPolicy javadoc that says "As long as any given UID has active processes, this method should return the same value for that UID. In order words, policy changes which occur while a transport instance is active, will have no effect on that transport instance." Thanks!

[mateusazis]

Sure. Before this PR, any ListenableFuture<Status> provided by a AsyncSecurityPolicy was cached.

This PR addresses the case when the ListenableFuture ends up failing. This piece of code ends the call with status INTERNAL. By a mistake, though, we're caching that ListenableFuture of INTERNAL indefinitely and replying INTERNAL to all subsequent requests. Instead, we should cache only "legitimate" statuses to let the AsyncSecurityPolicy recover.

Please see the new unit tests in this CL; they are good examples of the behavior I'm trying to enable. Without the caching changes from this PR, these tests would fail because all requests would result in status INTERNAL.

In summary, as of this PR, we should have:

• cancelled futures: NOT cached* (NEW)
• failed futures: NOT cached* (NEW)
• successful futures for any status: cached (existing behavior)

---

*: it's actually cached up until the next request, since I couldn't find a good way to remove the listenable future from the cache upon failure. Up to suggestions though.

[markb74]

Thanks, looks good.