Bug
R2Client default refetch_interval is 3600 seconds, but R2 credentials expire at exactly 60 minutes. This causes streaming to fail at the 1-hour mark.
S3Client already uses 3300 seconds as the default. R2Client should match this.
To Reproduce
- Use
StreamingDataset with R2 storage
- Run for longer than 60 minutes
- Job fails at exactly 1 hour with auth errors
Expected behavior
Default refetch_interval for R2Client should be 3300s to match S3Client and ensure credentials refresh before expiration.
Environment
- litdata: 0.2.59
- Python: 3.12.11
- OS: Linux