Fix call chain reentrancy

[andhesky]

Current behavior is to only allow call chain re-entrancy on A->B->A or A->A. This adds a new option for allowing A->B->C->A. Also flow the reentrancy context via RequestContext so that it can be viewed and serialized by the application layer.

• Refactor re-entrancy tests to use common code
• Change bool to enum in config and introduce new option CallChainReentrancyStrategy.EntireChain

Passes call chain tests. Sending an early version for feedback on the approach.

Remaining questions:

• Do we want to reconcile the code for Single Call and full call chain?
• Do we want to eliminate the option for single call (probably needs perf testing to determine impact)
• Still need to figure out logging in tests

[sergeybykov]

I perf tested it, and do not see any measurable negative impact.

[sergeybykov]

@dotnet-bot test functional

[andhesky]

Thanks for the info. I will move forward with this design. Ben mentioned removing the enum and just keeping it as the default behavior. This will change the behavior, but since there is no perf impact and we consider the current behavior a bug I will take his suggestion.

[benjaminpetit]

While the on-wire compatibility shouldn't be an issue between silos from 2.1.x and silos that will implement this logic, there might be issues during the deployment since we are changing the logic here.

But I don't think we have any way to address that...

[andhesky]

We could have release notes that say enable deadlock detection if you want it to work during upgrade.

[benjaminpetit]

Forgot to publish my reviews... dammit GitHub UI

[ReubenBond]

@dotnet-bot test functional

[andhesky]

@benjaminpetit, I think I addressed all your comments. Let me know if there are any other changes you would like to see.

[benjaminpetit]

LGTM. I'll kick off a backward compat testing