Skip to content

Conversation

@rdheekonda
Copy link
Contributor

@rdheekonda rdheekonda commented Jul 31, 2025

Refactored the S3 credential management system to resolve race conditions and improve reliability during long-running operations.

Key Changes:

  • Introduced dedicated CredentialManager class for centralized s3 credential lifecycle management
  • Implemented automatic retry mechanism with exponential backoff for credential-related failures
  • Extended credential refresh buffer from 5 to 15 minutes to prevent edge-case expirations
  • Replaced decorator-based refresh pattern with operation-wrapped retry logic

Added:

  • CredentialManager class with automatic refresh detection and retry logic
  • Error-specific retry handling for ExpiredToken, InvalidAccessKeyId, and SignatureDoesNotMatch
  • Comprehensive logging for credential refresh events and failure modes

Changed:

-Migrated ArtifactStorage and RunSpan to use centralized credential manager

  • Updated buffer window from 300s to 900s for more reliable refresh timing
  • Simplified filesystem credential passing through dedicated manager interface

Removed:

  • storage_utils.py decorator-based refresh mechanism
  • Manual credential refresh methods in main Dreadnode class
  • Direct S3FileSystem parameter passing in span constructors

Testing: Validated with 1.40 hour continuous operation - credential refreshes occurred seamlessly without interruption to ongoing uploads and operations.


Generated Summary:

  • Refactored ArtifactStorage to utilize CredentialManager for handling S3 credentials, improving management of authentication.
  • Removed the with_credential_refresh decorator and corresponding methods for credential refreshing, simplifying error handling.
  • Introduced the CredentialManager class to manage S3 credentials, incorporating automatic refresh logic.
  • Adjusted the FS_CREDENTIAL_REFRESH_BUFFER from 5 to 15 minutes to allow a longer window before refreshing credentials.
  • Updated methods in ArtifactStorage and RunSpan to use CredentialManager, including operations for storing files and batch uploads.
  • Removed the storage_utils.py file as its functionality is now integrated into the CredentialManager.
  • Bumped project version to 1.13.3 to reflect changes.

This summary was generated with ❤️ by rigging

@dreadnode-renovate-bot dreadnode-renovate-bot bot added the area/python Changes to Python package configuration and dependencies label Jul 31, 2025
@dreadnode-renovate-bot dreadnode-renovate-bot bot added area/docs Changes to documentation and guides type/docs Documentation updates and improvements labels Jul 31, 2025
@rdheekonda rdheekonda merged commit 95f7faa into main Jul 31, 2025
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Changes to documentation and guides area/python Changes to Python package configuration and dependencies type/docs Documentation updates and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants