Skip to content

R2Client credential refetch interval causes auth failure at 60 minutes #776

@vlad-heidi

Description

@vlad-heidi

Bug

R2Client default refetch_interval is 3600 seconds, but R2 credentials expire at exactly 60 minutes. This causes streaming to fail at the 1-hour mark.

S3Client already uses 3300 seconds as the default. R2Client should match this.

To Reproduce

  1. Use StreamingDataset with R2 storage
  2. Run for longer than 60 minutes
  3. Job fails at exactly 1 hour with auth errors

Expected behavior

Default refetch_interval for R2Client should be 3300s to match S3Client and ensure credentials refresh before expiration.

Environment

  • litdata: 0.2.59
  • Python: 3.12.11
  • OS: Linux

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions