- 
                Notifications
    You must be signed in to change notification settings 
- Fork 79
          Add GCP support for directory resolution inresolve_dir
          #659
        
          New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
| 
 | 
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 5685611 | Triggered | Generic High Entropy Secret | 122125f | tests/streaming/test_resolver.py | View secret | 
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
resolve_dir
      | Codecov Report❌ Patch coverage is  Additional details and impacted files@@         Coverage Diff         @@
##           main   #659   +/-   ##
===================================
  Coverage    83%    83%           
===================================
  Files        50     50           
  Lines      7097   7136   +39     
===================================
+ Hits       5912   5946   +34     
- Misses     1185   1190    +5     🚀 New features to boost your workflow:
 | 
| Initial testing for one of the existing folders in  ⚡ ~ ls /teamspace/gcs_folders/                                                                                                                                        
pipelines-gcp-lightning-public-prod
⚡ ~ uv pip install -q git+https://github.com/bhimrazy/litdata@upd/resolve_dir                                                                                         
⚡ ~ python3 -c "from litdata.streaming.resolver import _resolve_dir; path=_resolve_dir('/teamspace/gcs_folders/pipelines-gcp-lightning-public-prod'); print(path.url)"
gs://lai-gcp-lightning-public-prod/projects/01jsn2s83c8m1ftedf6p223ra6/gcs_folders/pipelines-gcp-lightning-public-prod/
⚡ ~ gsutil ls gs://lai-gcp-lightning-public-prod/projects/01jsn2s83c8m1ftedf6p223ra6/gcs_folders                                                                      
gs://lai-gcp-lightning-public-prod/projects/01jsn2s83c8m1ftedf6p223ra6/gcs_folders/pipelines-gcp-lightning-public-prod/
⚡ ~  | 
| LitData test on Lightning Studio's   cc: @tchaton | 
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Super neat !

What does this PR do ?
Introduce support for GCP cloud providers and enhance directory resolution for GCS connections and folders. Improve error handling for invalid directory paths.