Skip to content

feat(storage): support glob patterns for cloud source inputs#160

Merged
malon64 merged 3 commits intomainfrom
feature/storage-hardening
Feb 24, 2026
Merged

feat(storage): support glob patterns for cloud source inputs#160
malon64 merged 3 commits intomainfrom
feature/storage-hardening

Conversation

@malon64
Copy link
Copy Markdown
Owner

@malon64 malon64 commented Feb 24, 2026

Summary

  • support glob patterns in cloud source paths (S3/GCS/ADLS) by deriving a safe listing prefix from the literal path before the first glob metachar
  • list only under the derived prefix and then glob-filter object keys while preserving adapter suffix filtering and stable ordering
  • reject cloud glob patterns that do not include a non-empty literal prefix to avoid full bucket/container listings

Details

  • keeps existing non-glob cloud prefix behavior unchanged
  • keeps local input glob behavior unchanged
  • reuses the resolved-inputs pipeline so dry-run and run stay aligned

Tests

  • wildcard suffix: sales_*.csv
  • nested path segment wildcard: data/*/sales.csv
  • invalid/no-prefix patterns return clear errors
  • stable ordering after filtering

Validation

  • cargo fmt --all
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test --all

Issue

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4506477116

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/floe-core/src/io/storage/ops/inputs.rs Outdated
@malon64 malon64 merged commit efbd7d7 into main Feb 24, 2026
3 checks passed
@malon64 malon64 deleted the feature/storage-hardening branch February 24, 2026 13:57
@malon64 malon64 mentioned this pull request Feb 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support glob patterns for cloud source inputs

1 participant