Skip to content

Conversation

@bhimrazy
Copy link
Collaborator

@bhimrazy bhimrazy commented Jul 24, 2025

What does this PR do ?

Introduce support for GCP cloud providers and enhance directory resolution for GCS connections and folders. Improve error handling for invalid directory paths.

Follow up to #652

@gitguardian
Copy link

gitguardian bot commented Jul 24, 2025

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
5685611 Triggered Generic High Entropy Secret 122125f tests/streaming/test_resolver.py View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@bhimrazy bhimrazy changed the title Add GCP support for directory resolution in streaming Add GCP support for directory resolution inresolve_dir Jul 24, 2025
@bhimrazy bhimrazy self-assigned this Jul 24, 2025
@codecov
Copy link

codecov bot commented Jul 24, 2025

Codecov Report

❌ Patch coverage is 82.92683% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 83%. Comparing base (268f3c7) to head (7774fdf).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@         Coverage Diff         @@
##           main   #659   +/-   ##
===================================
  Coverage    83%    83%           
===================================
  Files        50     50           
  Lines      7097   7136   +39     
===================================
+ Hits       5912   5946   +34     
- Misses     1185   1190    +5     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bhimrazy
Copy link
Collaborator Author

Initial testing for one of the existing folders in gcs_folder under the General teamspace of LightningAI.

~ ls /teamspace/gcs_folders/                                                                                                                                        
pipelines-gcp-lightning-public-prod~ uv pip install -q git+https://github.com/bhimrazy/litdata@upd/resolve_dir~ python3 -c "from litdata.streaming.resolver import _resolve_dir; path=_resolve_dir('/teamspace/gcs_folders/pipelines-gcp-lightning-public-prod'); print(path.url)"
gs://lai-gcp-lightning-public-prod/projects/01jsn2s83c8m1ftedf6p223ra6/gcs_folders/pipelines-gcp-lightning-public-prod/~ gsutil ls gs://lai-gcp-lightning-public-prod/projects/01jsn2s83c8m1ftedf6p223ra6/gcs_folders                                                                      
gs://lai-gcp-lightning-public-prod/projects/01jsn2s83c8m1ftedf6p223ra6/gcs_folders/pipelines-gcp-lightning-public-prod/~ 
Screenshot

image

@bhimrazy
Copy link
Collaborator Author

bhimrazy commented Jul 28, 2025

LitData test on Lightning Studio's gcs_folders.

image cc: @tchaton

Copy link
Collaborator

@tchaton tchaton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super neat !

@tchaton tchaton merged commit 68b74f3 into Lightning-AI:main Jul 28, 2025
48 checks passed
@bhimrazy bhimrazy deleted the upd/resolve_dir branch August 4, 2025 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants