fix(mcp): protect connection on non-fatal client side timeout error

[dbschmigelski]

Description

This PR enhances MCP client resilience by implementing error filtering to prevent connection collapse from recoverable client-side errors. The changes address "unknown request id" errors that occur when responses arrive after client-side timeouts, which previously caused unnecessary connection termination.

The implementation adds a non-fatal error pattern system in the MCP client's message handler. When errors match predefined patterns like "unknown request id", they are logged and ignored rather than terminating the connection. This maintains connection stability while still allowing genuine server errors to propagate normally.

Also included is a new md file to help demystify the decisions made regarding the MCPClient. As part of the this PR review, I encourage the review of the _MCP_CLIENT_ARCHITECTURE.md file.

After this PR, we need to improve our mocking posture. The test_mcp_client is testing private methods seemingly because testing from a public entry point is too difficult at the moment. We have integration test coverage. But we need to make sure our unit tests have the coverage we expect.

Related Issues

#1221
#922
#1169

Documentation PR

N/A

Type of Change

Bug fix

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli

• I ran hatch run prepare

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!