feat(webdav): add bearer_token_command for dynamic token acquisition #10917

GreenHatHG · 2025-11-22T07:48:52Z

Code-related

Related Issues

Support for refreshing access tokens for authentification of WebDAV remote #10689

Description

This PR introduces dynamic Bearer Token management for WebDAV remotes via the new bearer_token_command configuration option. Unlike static token configurations, this feature enables DVC to:

Execute external commands to fetch fresh tokens (e.g., OAuth device flow, cloud auth helpers)
Automatically refresh expired tokens during operations
Securely persist tokens exclusively to local DVC config (.dvc/config.local)
Maintain full backward compatibility with existing static token configurations

Key Enhancements

Dynamic Token Workflow

[remote "my-webdav"]
    url = webdavs://example.com/data
    bearer_token_command = "gcloud auth print-access-token"  # Example command

DVC executes this command on-demand to acquire tokens, eliminating manual token rotation.

Secure Token Lifecycle
- Tokens are never stored globally – persisted only to local config
- Thread-safe refresh mechanism prevents concurrent token fetch storms
- Automatic cleanup of expired tokens from config
Fallback Compatibility
Existing static tokens (via token = xyz config) continue working unchanged. This feature only activates when bearer_token_command is set.

Issue Resolution

This PR directly addresses the use case described in issue #10689, where users of short-lived OIDC tokens (like those from oidc-agent that expire every 5 minutes) currently need to manually update tokens in .dvc/config.local. With this implementation, users can now configure:

[remote "my-remote"]
    url = webdavs://example.com/data
    bearer_token_command = "oidc-token my-oidc-config"

DVC will automatically execute this command whenever a token is needed, eliminating the manual token rotation workflow. This matches the pattern used by tools like rclone that was referenced in the issue discussion.

Testing Instructions

Testing Environment Setup

WebDAV Server (Docker):

docker run --rm -p 9000:8080 -v $(pwd)/data:/data rclone/rclone:latest serve webdav /data --addr :8080

Token Fetch Script (get_token.sh):

DVC_TOKEN=$(curl -s http://localhost:8080/token)
echo "$DVC_TOKEN"

Proxy Server (proxy.py)

Short-Lived Tokens: Generates time-bound tokens (default 5s expiry) via /token endpoint, mimicking OIDC's short-lived access tokens that require frequent renewal.
401-Driven Refresh Cycle: Returns 401 Unauthorized on expired/invalid tokens, triggering DVC's bearer_token_command to automatically fetch new tokens – exactly replicating OIDC's refresh flow.

import logging
import os
import secrets
import time
import requests
from flask import Flask, request, Response, stream_with_context

# Configure logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

app = Flask(__name__)

# Environment configuration
TOKEN_EXPIRY = int(os.getenv('TOKEN_EXPIRY_SECONDS', '5'))
UPSTREAM_URL = os.getenv('UPSTREAM_URL', 'http://127.0.0.1:9000')

valid_tokens = {}

def generate_token():
    """Generate a new time-limited token"""
    timestamp = int(time.time())
    token = f"tk_{timestamp}_{secrets.token_hex(4)}"
    valid_tokens[token] = timestamp
    logger.info(f"Generated token: {token}")
    return token

def validate_token(token):
    """Validate token existence and expiration"""
    # 1. Fixed token (for debugging/backdoor access)
    if token == "valid_token_123":
        return True

    # 2. Dynamic token validation
    if token in valid_tokens:
        if time.time() - valid_tokens[token] < TOKEN_EXPIRY:
            return True
        else:
            # Auto-clean expired tokens
            logger.info(f"Token expired: {token}")
            del valid_tokens[token]
            return False
    
    logger.warning(f"Token not found: {token}")
    return False

# --------------------------
#   1) Token issuance endpoint
# --------------------------
@app.route('/token', methods=['GET', 'POST'])
def issue_token():
    """Endpoint to obtain a new authentication token"""
    token = generate_token()
    # Return as plain text to match shell script consumption
    return token, 200

# --------------------------
#   2) Authentication middleware
# --------------------------
@app.before_request
def auth():
    """Global authentication handler"""
    # Skip authentication for token endpoint
    if request.path == '/token':
        return None

    auth_header = request.headers.get("Authorization")
    
    # Validate header format
    if not auth_header or not auth_header.startswith("Bearer "):
        logger.warning(f"Missing or malformed Authorization header from {request.remote_addr}")
        return Response("Missing valid Authorization header", 401)

    token = auth_header[7:]  # Remove "Bearer " prefix
    
    # CRITICAL: Return 401 on failure to trigger client token refresh
    if not validate_token(token):
        logger.warning(f"Invalid or expired token from {request.remote_addr}")
        return Response("Token expired or invalid", 401)

# --------------------------
#   3) WebDAV reverse proxy
# --------------------------
@app.route('/', defaults={'path': ''}, methods=["GET", "PUT", "PROPFIND",
                                            "MKCOL", "DELETE", "COPY",
                                            "MOVE", "OPTIONS", "HEAD"])
@app.route('/<path:path>', methods=["GET", "PUT", "PROPFIND",
                                   "MKCOL", "DELETE", "COPY",
                                   "MOVE", "OPTIONS", "HEAD"])
def proxy(path):
    """Reverse proxy for WebDAV operations with streaming support"""
    target_url = f"{UPSTREAM_URL}/{path}"
    
    # Filter hop-by-hop headers to prevent protocol conflicts
    excluded_headers = [
        'content-encoding', 'content-length', 
        'transfer-encoding', 'connection', 'host'
    ]
    
    # Prepare headers for upstream request
    upstream_headers = {
        k: v for k, v in request.headers.items()
        if k.lower() not in excluded_headers
    }

    try:
        # CRITICAL FIX: Use streaming for file uploads
        upstream_data = request.stream if request.method in ['PUT', 'POST'] else None

        # Forward request to upstream WebDAV server
        resp = requests.request(
            method=request.method,
            url=target_url,
            headers=upstream_headers,
            data=upstream_data,
            stream=True,  # Enable streaming mode
            allow_redirects=False
        )

        # Prepare response headers for client
        downstream_headers = [
            (name, value) for name, value in resp.raw.headers.items()
            if name.lower() not in excluded_headers
        ]

        # CRITICAL FIX: Stream large file downloads
        return Response(
            stream_with_context(resp.iter_content(chunk_size=8192)),
            status=resp.status_code,
            headers=downstream_headers
        )

    except requests.exceptions.RequestException as e:
        logger.error(f"Upstream connection error: {e}")
        return Response(f"Upstream Proxy Error: {e}", 502)

if __name__ == "__main__":
    # Disable ASCII conversion for non-English filenames
    app.config['JSON_AS_ASCII'] = False
    # Enable threading for concurrent WebDAV operations
    app.run(host="0.0.0.0", port=8080, threaded=True)

Validation Tests Performed

Run these before DVC operations (see Testing Environment Setup): Start WebDAV Server (Docker), Run Proxy Server (python proxy.py)

Short Token Expiry (5s default):

mkdir test1 && cd test1
dvc init --no-scm
dvc remote add -d test-webdav webdav://localhost:8080/
dvc remote modify test-webdav bearer_token_command "sh /home/jooooody/webdav-test/py/get_token.sh"  -v

touch test.txt
echo "$(date) | $(head /dev/urandom -c 32 | base64)" > test.txt
dvc add test.txt
dvc push -v  # Initial push
echo "$(date) | $(head /dev/urandom -c 32 | base64)" >> test.txt
dvc push -v  # Token refresh during push
rm test.txt
dvc pull -v  # Token refresh during pull

Ouput example:

╰─$ dvc push -v                                                                                                   130 ↵
2025-11-22 15:15:55,318 DEBUG: v3.63.1.dev25+gfa5c33ed2.d20251116, CPython 3.12.12 on Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.42
2025-11-22 15:15:55,318 DEBUG: command: /home/jooooody/projects/pycharm_dvc/venv/bin/dvc push -v
2025-11-22 15:15:55,790 DEBUG: Bearer token command provided, using BearerAuthClient, command: sh /home/jooooody/webdav-test/py/get_token.sh
Collecting                                                                                    |0.00 [00:00,    ?entry/s]
2025-11-22 15:15:55,926 DEBUG: Preparing to transfer data from '/home/jooooody/webdav-test/dvc/.dvc/cache/files/md5' to 'http://localhost:8080/dvc/files/md5'
2025-11-22 15:15:55,926 DEBUG: Preparing to collect status from 'files/md5'
2025-11-22 15:15:55,927 DEBUG: Collecting status from 'files/md5'
2025-11-22 15:15:55,928 DEBUG: Querying 1 oids via object_exists
2025-11-22 15:15:55,949 DEBUG: [ThreadPoolExecutor-0_0] [BearerAuthClient] Received 401. Attempting recovery.
2025-11-22 15:15:55,950 DEBUG: [ThreadPoolExecutor-0_0] [BearerAuthClient] Refreshing token via command...
2025-11-22 15:15:55,999 DEBUG: Saved token for remote 'test-webdav'
2025-11-22 15:15:56,001 DEBUG: Writing '/home/jooooody/webdav-test/dvc/.dvc/config.local'.
2025-11-22 15:15:56,012 DEBUG: [ThreadPoolExecutor-0_0] [BearerAuthClient] Token refreshed successfully.
Pushing
Everything is up to date.
2025-11-22 15:15:56,026 DEBUG: Analytics is enabled.
2025-11-22 15:15:56,089 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmps_s184o4', '-v']
2025-11-22 15:15:56,099 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmps_s184o4', '-v'] with pid 103183

Long Token Expiry:

Same workflow with TOKEN_EXPIRY_SECONDS=300 (5 minutes)
Verify that the token will not be refreshed in the short term

Output example:

╰─$ dvc push -v
2025-11-22 15:16:13,624 DEBUG: v3.63.1.dev25+gfa5c33ed2.d20251116, CPython 3.12.12 on Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.42
2025-11-22 15:16:13,625 DEBUG: command: /home/jooooody/projects/pycharm_dvc/venv/bin/dvc push -v
2025-11-22 15:16:14,074 DEBUG: Bearer token command provided, using BearerAuthClient, command: sh /home/jooooody/webdav-test/py/get_token.sh
Collecting                                                                                    |0.00 [00:00,    ?entry/s]
2025-11-22 15:16:14,214 DEBUG: Preparing to transfer data from '/home/jooooody/webdav-test/dvc/.dvc/cache/files/md5' to 'http://localhost:8080/dvc/files/md5'
2025-11-22 15:16:14,215 DEBUG: Preparing to collect status from 'files/md5'
2025-11-22 15:16:14,215 DEBUG: Collecting status from 'files/md5'
2025-11-22 15:16:14,217 DEBUG: Querying 1 oids via object_exists
Pushing
Everything is up to date.
2025-11-22 15:16:14,247 DEBUG: Analytics is enabled.
2025-11-22 15:16:14,309 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmpe7wyiogz', '-v']
2025-11-22 15:16:14,318 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmpe7wyiogz', '-v'] with pid 103404

…eval

codecov · 2025-11-22T07:53:07Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.00%. Comparing base (2431ec6) to head (9ff8562).
⚠️ Report is 154 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10917      +/-   ##
==========================================
+ Coverage   90.68%   91.00%   +0.31%     
==========================================
  Files         504      504              
  Lines       39795    40936    +1141     
  Branches     3141     3241     +100     
==========================================
+ Hits        36087    37252    +1165     
- Misses       3042     3047       +5     
+ Partials      666      637      -29

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

skshetry · 2025-11-28T14:04:40Z

@GreenHatHG, do you know where rclone stores the token?

GreenHatHG · 2025-11-28T17:07:18Z

@GreenHatHG, do you know where rclone stores the token?

The rclone WebDAV I set up requires no authentication. A Python script is used to simulate WebDAV token functionality, allowing token retrieval via API. Detailed steps are included in the test tutorial—please refer to it.

skshetry · 2025-11-28T17:20:37Z

@GreenHatHG, do you know where rclone stores the token?

The rclone WebDAV I set up requires no authentication. A Python script is used to simulate WebDAV token functionality, allowing token retrieval via API. Detailed steps are included in the test tutorial—please refer to it.

Sorry, I am not asking about the setup. I was asking about how rclone stores the token, as you said rclone works similarly above.

The biggest issue that I see with this PR is saving/reading the token from the config, so I am wondering what others do. Making FileSystem responsible in dvc for this is not a good idea, so we'll need to figure out a different mechanism.

GreenHatHG · 2025-11-30T04:38:49Z

@GreenHatHG, do you know where rclone stores the token?

The rclone WebDAV I set up requires no authentication. A Python script is used to simulate WebDAV token functionality, allowing token retrieval via API. Detailed steps are included in the test tutorial—please refer to it.

Sorry, I am not asking about the setup. I was asking about how rclone stores the token, as you said rclone works similarly above.

The biggest issue that I see with this PR is saving/reading the token from the config, so I am wondering what others do. Making FileSystem responsible in dvc for this is not a good idea, so we'll need to figure out a different mechanism.

I’ve looked into the Rclone source code and verified the behavior experimentally. Rclone does not persist the token generated by bearer_token_command to the config file. It keeps the token in memory only for the duration of the process.

Source Code Analysis

// backend/webdav/webdav.go
// fetch the bearer token and set it if successful
func (f *Fs) fetchAndSetBearerToken() error {
	_, err, _ := f.authSingleflight.Do("bearerToken", func() (interface{}, error) {
		if len(f.opt.BearerTokenCommand) == 0 {
			return nil, nil
		}
		token, err := f.fetchBearerToken(f.opt.BearerTokenCommand)
		if err != nil {
			return nil, err
		}
		f.setBearerToken(token)
		return nil, nil
	})
	return err
}

// backend/webdav/webdav.go

func (f *Fs) setBearerToken(token string) {
	f.opt.BearerToken = token
	f.srv.SetHeader("Authorization", "Bearer "+token)
}

In backend/webdav/webdav.go, the function setBearerToken handles the token update. This updates the in-memory Options struct (f.opt) and the HTTP client headers (f.srv). Crucially, there is no call to fs.ConfigFileSet (Rclone's method for writing to disk) anywhere in this workflow.

Experimental Verification

I verified this behavior by setting up a test environment where I could intercept the token generation requests.

Setup

webdav server:

docker run --rm -p 9000:8080 -v $(pwd)/data:/data rclone/rclone:latest serve webdav /data --addr :8080

proxy.py: A Python script intercepting traffic to serve as the endpoint and generate tokens. It has been mentioned above.

rclone.conf:

[mywebdav]
type = webdav
url = http://localhost:8080/
vendor = other
bearer_token_command = sh /home/jooooody/webdav-test/py/get_token.sh

Execution

I ran rclone lsd twice in quick succession.

[30/11/25 11:47:14] ╭─jooooody@DESKTOP-35U5SPV ~/webdav-test/rclone
╰─$ rclone --config ./rclone.conf lsd mywebdav:
          -1 2025-11-22 12:57:13        -1 dvc
[30/11/25 11:47:28] ╭─jooooody@DESKTOP-35U5SPV ~/webdav-test/rclone
╰─$ rclone --config ./rclone.conf lsd mywebdav:
          -1 2025-11-22 12:57:13        -1 dvc

Result

The proxy logs showed that rclone requested a new token for every single execution.

2025-11-30 11:47:28,566 - INFO - Generated token: tk_1764474448_fdf4485f
2025-11-30 11:47:28,567 - INFO - 127.0.0.1 - - [30/Nov/2025 11:47:28] "GET /token HTTP/1.1" 200 -
2025-11-30 11:47:28,578 - INFO - 127.0.0.1 - - [30/Nov/2025 11:47:28] "PROPFIND / HTTP/1.1" 207 -
2025-11-30 11:47:35,277 - INFO - Generated token: tk_1764474455_29864977
2025-11-30 11:47:35,277 - INFO - 127.0.0.1 - - [30/Nov/2025 11:47:35] "GET /token HTTP/1.1" 200 -
2025-11-30 11:47:35,285 - INFO - 127.0.0.1 - - [30/Nov/2025 11:47:35] "PROPFIND / HTTP/1.1" 207 -

If Rclone were persisting the token, the second run would have reused the previous token (The token expiration time is set to 50 seconds.).

My Implementation for DVC

In my current PR, I have added bearer_token_command support, but I implemented it differently: I persist the token to the local config file. This avoids executing the shell command repeatedly; it only calls the command again if the stored token has expired (401).

I am unsure if writing to the config file by default is the best approach. How would you prefer we handle this?

Option 1: Follow Rclone (Memory Only)
Do not store the token. Execute the command every time DVC runs.

Pros: Clean; doesn't modify user config files; no risk of stale tokens on disk.
Cons: Performance overhead if the token command is slow; Frequent command execution might hit rate limits of the external token provider.

Option 2: Persist by Default (Current PR behavior)
Store the token in the local config and reuse it until it expires.

Pros: Efficient; minimizes calls to the external command.
Cons: "Pollutes" the config file with ephemeral tokens;

Option 3: User Configurable
Default to memory-only (Option 1), but add a flag (e.g., --cache-token) for users who want persistence.

feat(webdav): add bearer_token_command config for dynamic token retri…

75dc395

…eval

github-project-automation bot added this to DVC Nov 22, 2025

github-project-automation bot moved this to Backlog in DVC Nov 22, 2025

GreenHatHG mentioned this pull request Nov 22, 2025

feat(webdav): add bearer_token_command for dynamic token acquisition treeverse/dvc-webdav#95

Open

Merge branch 'main' into feat/webdav-dynamic-token-command

9ff8562

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(webdav): add bearer_token_command for dynamic token acquisition #10917

feat(webdav): add bearer_token_command for dynamic token acquisition #10917

GreenHatHG commented Nov 22, 2025 •

edited

Loading

Uh oh!

codecov bot commented Nov 22, 2025 •

edited

Loading

Uh oh!

skshetry commented Nov 28, 2025

Uh oh!

GreenHatHG commented Nov 28, 2025

Uh oh!

skshetry commented Nov 28, 2025 •

edited

Loading

Uh oh!

GreenHatHG commented Nov 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(webdav): add bearer_token_command for dynamic token acquisition #10917

Are you sure you want to change the base?

feat(webdav): add bearer_token_command for dynamic token acquisition #10917

Conversation

GreenHatHG commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code-related

Related Issues

Description

Key Enhancements

Issue Resolution

Testing Instructions

Testing Environment Setup

Validation Tests Performed

Uh oh!

codecov bot commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

skshetry commented Nov 28, 2025

Uh oh!

GreenHatHG commented Nov 28, 2025

Uh oh!

skshetry commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GreenHatHG commented Nov 30, 2025

Source Code Analysis

Experimental Verification

Setup

Execution

Result

My Implementation for DVC

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

GreenHatHG commented Nov 22, 2025 •

edited

Loading

codecov bot commented Nov 22, 2025 •

edited

Loading

skshetry commented Nov 28, 2025 •

edited

Loading