Skip to content

fix(csharp): apply CloudFetch timeout and remove redundant HttpClient#112

Merged
eric-wang-1990 merged 1 commit intomainfrom
stack/e.wang/fix-cloudfetch-timeout
Jan 6, 2026
Merged

fix(csharp): apply CloudFetch timeout and remove redundant HttpClient#112
eric-wang-1990 merged 1 commit intomainfrom
stack/e.wang/fix-cloudfetch-timeout

Conversation

@eric-wang-1990
Copy link
Collaborator

@eric-wang-1990 eric-wang-1990 commented Jan 5, 2026

What's Changed

This PR addresses two issues in the CloudFetch implementation and one fix in the proxy test infrastructure:

1. Apply CloudFetch Timeout Configuration

Problem: The adbc.databricks.cloudfetch.timeout_minutes configuration parameter was being read but never applied to the HttpClient. The HttpClient used its default 100-second timeout instead of the configured value (default: 5 minutes).

Solution: Added timeout configuration in CloudFetchDownloader constructor:

_httpClient.Timeout = TimeSpan.FromMinutes(config.TimeoutMinutes);

Impact: The CloudFetch HTTP client now correctly respects the user-configured timeout value, allowing for proper timeout behavior in CloudFetch downloads.

Files Changed:

  • csharp/src/Reader/CloudFetch/CloudFetchDownloader.cs

2. Remove Redundant HttpClient from CloudFetchDownloadManager

Problem: CloudFetchDownloadManager accepted an HttpClient parameter but never used it for any operations. It only disposed of it, which is incorrect since it doesn't own the resource. The actual HTTP operations are performed by CloudFetchDownloader.

Solution: Removed the redundant httpClient parameter from:

  • CloudFetchDownloadManager constructor and field
  • CloudFetchReaderFactory.CreateThriftReader() call
  • CloudFetchReaderFactory.CreateStatementExecutionReader() call

Impact: Cleaner code architecture with proper resource ownership. The HttpClient is now owned solely by CloudFetchDownloader, which actually uses it for HTTP operations.

Files Changed:

  • csharp/src/Reader/CloudFetch/CloudFetchDownloadManager.cs
  • csharp/src/Reader/CloudFetch/CloudFetchReaderFactory.cs

3. Fix Proxy Timing for Test Reliability

Problem: The mitmproxy addon was disabling failure scenarios AFTER the delay completed, causing race conditions. When the client timed out and retried, the scenario was still enabled, triggering the delay again.

Solution: Modified mitmproxy_addon.py to disable the scenario immediately when triggered, before the delay starts:

self._disable_scenario(scenario_name)  # Disable BEFORE delay
await asyncio.sleep(duration_seconds)

Impact: CloudFetch timeout tests now behave correctly - only the first request is delayed, and retry attempts succeed immediately.

Files Changed:

  • test-infrastructure/proxy-server/mitmproxy_addon.py

Testing

All 5 CloudFetch proxy tests now pass:

  • CloudFetchConnectionReset_RetriesWithExponentialBackoff
  • CloudFetchExpiredLink_RefreshesLinkViaFetchResults
  • NormalCloudFetch_SucceedsWithoutFailureScenarios
  • CloudFetchTimeout_RetriesWithExponentialBackoff
  • CloudFetch403_RefreshesLinkViaFetchResults

@eric-wang-1990 eric-wang-1990 force-pushed the stack/e.wang/fix-cloudfetch-timeout branch from 277c10f to 276ccfa Compare January 5, 2026 07:35
@eric-wang-1990 eric-wang-1990 marked this pull request as ready for review January 5, 2026 07:38
@eric-wang-1990 eric-wang-1990 changed the title fix(csharp): apply CloudFetch timeout and remove redundant HttpClient chore: apply CloudFetch timeout and remove redundant HttpClient Jan 5, 2026
@eric-wang-1990 eric-wang-1990 changed the title chore: apply CloudFetch timeout and remove redundant HttpClient fix(chsarp): apply CloudFetch timeout and remove redundant HttpClient Jan 5, 2026
@eric-wang-1990 eric-wang-1990 changed the title fix(chsarp): apply CloudFetch timeout and remove redundant HttpClient fix(csharp): apply CloudFetch timeout and remove redundant HttpClient Jan 5, 2026
eric-wang-1990 added a commit that referenced this pull request Jan 6, 2026
## 🥞 Stacked PR
Use this
[link](https://github.com/adbc-drivers/databricks/pull/111/files) to
review incremental changes.
-
[**stack/e.wang/add-thrift-call-verification**](#111)
[[Files
changed](https://github.com/adbc-drivers/databricks/pull/111/files)]
-
[stack/e.wang/fix-cloudfetch-timeout](#112)
[[Files
changed](https://github.com/adbc-drivers/databricks/pull/112/files/fbd12dc00750ec405c2130138ce59f4d1db9c46d..277c10fc5ba030ae3f3f18d4026c38186fc8941f)]

---------
## What's Changed

This PR adds comprehensive call verification to CloudFetch proxy tests
using dynamic baseline measurement.

### Changes

- **Dynamic baseline verification**: Tests now establish baselines by
running queries without failure scenarios, then compare actual vs
expected call counts
- **CloudFetch failure scenarios**: Added verification for:
- cloudfetch_expired_link: Verifies FetchResults is called again to
refresh expired links
- cloudfetch_403: Verifies FetchResults is called again after 403
Forbidden
- cloudfetch_timeout: Verifies retry behavior with configurable timeout
- cloudfetch_connection_reset: Verifies retry behavior on connection
reset
- **Helper methods**: Added CreateProxiedConnectionWithParameters for
tests requiring custom configuration
- **Test infrastructure**: Uses ProxyControlClient to count Thrift
method calls and cloud downloads

### Testing

The tests validate that the driver correctly:
1. Refreshes CloudFetch download links by calling FetchResults again
when links expire or return 403
2. Retries CloudFetch downloads with exponential backoff on timeout or
connection reset
3. Uses dynamic baselines to account for different result set sizes and
prefetch behavior

Note: The CloudFetchTimeout test currently fails because the Thrift
driver does not apply the configured CloudFetch timeout to the
HttpClient. This will be addressed in a follow-up PR.

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit addresses two issues in the CloudFetch implementation:

1. **Apply CloudFetch timeout configuration**
   - CloudFetchConfiguration.TimeoutMinutes was being read but never applied
   - CloudFetch downloads used the default HttpClient timeout (100s) instead of configured timeout
   - Fix: Apply timeout in CloudFetchDownloader constructor following OAuthClientCredentialsProvider pattern
   - Impact: CloudFetch downloads now respect configured timeout (default: 5 minutes)
   - Fixes CloudFetchTimeout test that was failing because timeouts never occurred

2. **Remove redundant HttpClient from CloudFetchDownloadManager**
   - CloudFetchDownloadManager received an HttpClient parameter but never used it for operations
   - Only used it for disposal, which is incorrect as it doesn't own the resource
   - The actual HTTP operations are performed by CloudFetchDownloader which has its own HttpClient
   - Fix: Remove httpClient parameter from CloudFetchDownloadManager constructor and field
   - HttpClient lifecycle is now solely managed by its actual owner

Changes:
- CloudFetchDownloader.cs: Apply config.TimeoutMinutes to _httpClient
- CloudFetchDownloadManager.cs: Remove unused _httpClient field and parameter
- CloudFetchReaderFactory.cs: Remove httpClient argument in both CreateForThrift methods

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@eric-wang-1990 eric-wang-1990 force-pushed the stack/e.wang/fix-cloudfetch-timeout branch from 276ccfa to 89b8b4e Compare January 6, 2026 17:27
@eric-wang-1990 eric-wang-1990 changed the title fix(csharp): apply CloudFetch timeout and remove redundant HttpClient fix(csharp/src): apply CloudFetch timeout and remove redundant HttpClient Jan 6, 2026
@eric-wang-1990 eric-wang-1990 changed the title fix(csharp/src): apply CloudFetch timeout and remove redundant HttpClient fix(csharp): apply CloudFetch timeout and remove redundant HttpClient Jan 6, 2026
@eric-wang-1990 eric-wang-1990 changed the title fix(csharp): apply CloudFetch timeout and remove redundant HttpClient fix: apply CloudFetch timeout and remove redundant HttpClient Jan 6, 2026
@eric-wang-1990 eric-wang-1990 changed the title fix: apply CloudFetch timeout and remove redundant HttpClient feat(chsarp): apply CloudFetch timeout and remove redundant HttpClient Jan 6, 2026
@eric-wang-1990 eric-wang-1990 changed the title feat(chsarp): apply CloudFetch timeout and remove redundant HttpClient feat(csharp): apply CloudFetch timeout and remove redundant HttpClient Jan 6, 2026
@eric-wang-1990 eric-wang-1990 changed the title feat(csharp): apply CloudFetch timeout and remove redundant HttpClient fix(csharp): apply CloudFetch timeout and remove redundant HttpClient Jan 6, 2026
@eric-wang-1990 eric-wang-1990 changed the title fix(csharp): apply CloudFetch timeout and remove redundant HttpClient feature(csharp): apply CloudFetch timeout and remove redundant HttpClient Jan 6, 2026
@eric-wang-1990 eric-wang-1990 changed the title feature(csharp): apply CloudFetch timeout and remove redundant HttpClient fix: apply CloudFetch timeout and remove redundant HttpClient Jan 6, 2026
@eric-wang-1990 eric-wang-1990 force-pushed the stack/e.wang/fix-cloudfetch-timeout branch from 9c90ff8 to 89b8b4e Compare January 6, 2026 18:34
@eric-wang-1990 eric-wang-1990 changed the title fix: apply CloudFetch timeout and remove redundant HttpClient fix(csharp): apply CloudFetch timeout and remove redundant HttpClient Jan 6, 2026
@eric-wang-1990 eric-wang-1990 merged commit f0afca8 into main Jan 6, 2026
31 of 44 checks passed
@eric-wang-1990 eric-wang-1990 deleted the stack/e.wang/fix-cloudfetch-timeout branch January 6, 2026 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants