Skip to content

✨feat(source-s3): adjust file record message protocol - #57498

Merged
Aldo Gonzalez (aldogonzalez8) merged 22 commits into
masterfrom
aldogonzalez8/source-s3/adjust-file-record-message-protocol
May 5, 2025
Merged

Aldo Gonzalez (aldogonzalez8) merged 22 commits into
masterfrom
aldogonzalez8/source-s3/adjust-file-record-message-protocol

Conversation

@aldogonzalez8

@aldogonzalez8 Aldo Gonzalez (aldogonzalez8) commented Apr 7, 2025 •

Copy link
Copy Markdown
Contributor

What

This PR updates the source-s3 connector to comply with the latest file transfer record protocol.

Resolves: https://github.com/airbytehq/airbyte-internal-issues/issues/12341

How

  • Updated the source-s3 connector's implementation to align with the latest protocol specifications.
  • Modified the release information accordingly.

Review guide

1.airbyte-integrations/connectors/source-s3/source_s3/v4/stream_reader.py: returns file reference and file metadata.

User Impact

  • Internal protocol alignment that ensures consistent file handling across file-based connectors

Can this PR be safely reverted and rolled back?

  • YES 💚
  • NO ❌

@vercel

vercel Bot commented Apr 7, 2025 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback May 5, 2025 3:14pm

@aldogonzalez8

Aldo Gonzalez (aldogonzalez8) commented Apr 16, 2025 •

Copy link
Copy Markdown
Contributor Author

/approve-regression-tests

Check job output.

✅ Approving regression tests

@aldogonzalez8

Aldo Gonzalez (aldogonzalez8) commented Apr 16, 2025 •

Copy link
Copy Markdown
Contributor Author

/approve-regression-tests

Check job output.

✅ Approving regression tests

@aldogonzalez8

Aldo Gonzalez (aldogonzalez8) commented Apr 19, 2025 •

Copy link
Copy Markdown
Contributor Author

/approve-regression-tests

Check job output.

✅ Approving regression tests

Returns:
str: The properly formatted S3 URI
"""
file_path = file.uri.split("#")[0] if isinstance(file, RemoteFileInsideArchive) else file.uri

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be better to use the re and search for the certain pattern here?


logger.debug(f"try to open {file.uri}")
try:
s3_uri = self._construct_s3_uri(file)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for using the construction here

@aldogonzalez8

Copy link
Copy Markdown
Contributor Author

/approve-regression-tests

@aldogonzalez8

Aldo Gonzalez (aldogonzalez8) commented Apr 24, 2025 •

Copy link
Copy Markdown
Contributor Author

/approve-regression-tests

Check job output.

✅ Approving regression tests

@aldogonzalez8

Aldo Gonzalez (aldogonzalez8) commented Apr 25, 2025 •

Copy link
Copy Markdown
Contributor Author

/approve-regression-tests

Check job output.

✅ Approving regression tests

@aldogonzalez8

Aldo Gonzalez (aldogonzalez8) commented Apr 30, 2025 •

Copy link
Copy Markdown
Contributor Author

/approve-regression-tests

Check job output.

✅ Approving regression tests

@aldogonzalez8

Aldo Gonzalez (aldogonzalez8) commented May 5, 2025 •

Copy link
Copy Markdown
Contributor Author

/approve-regression-tests

Check job output.

✅ Approving regression tests

@aldogonzalez8
Aldo Gonzalez (aldogonzalez8) deleted the aldogonzalez8/source-s3/adjust-file-record-message-protocol branch May 5, 2025 16:00
letiescanciano added a commit that referenced this pull request May 6, 2025
…ectors-in-sidebar

* master: (3385 commits)
  fix(Source-Harvest): missing schema for `time_team`, `users`, `time_entries`, `company`, & `project_assignments` streams.  (#59215)
  chore: add `poe_tasks.toml` for all java connectors (#59670)
  ci: add Dockerfile base image definitions, along with sample `.sh` build scripts (#59212)
  Correct Dest SF readme. (#59658)
  ✨ TikTok Marketing Source: By country daily/hourly report streams (#55681)
  source-vitally contribution from aidanlister (#58062)
  Upgrade Gradle to 8.14. (#59662)
  source-awin-advertiser contribution from ryanmcg2203 (#58632)
  🐛 Source asana: add new stream portfolio_items (#59224)
  docs: add pre 1.0 semver exception to breaking changes
  ✨ Source Brevo: fix pagination on contact stream (#59652)
  Load CDK: Pre-Speed Cleanup: Remove CheckpointManager interface/abstract class (#59218)
  Load CDK: Pre-Speed Cleanup: Remove Sync/Stream manager interfaces (#59217)
  Load CDK: Non-functional Cleanup: Remove Launcher Interface/Secondary (#59213)
  fix(live-tests): use first 8 symbols of connection id instead of hashing (#59653)
  ✨feat(source-s3): adjust file record message protocol (#57498)
  feat(source-zendesk-support): new article attachments with file transfer support  (#56419)
  source-mixmax contribution from cbuckle1 (#59160)
  ✨feat(source-google-drive): adjust file record message protocol (#57496)
  Typo fix (#59209)
  ...

This branch was successfully deployed

1 active deployment
Preview — a0074a44 Deployed May 5, 2025 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants