🎉 Source S3: Reduce image size and add acceptance test by flash1293 · Pull Request #31654 · airbytehq/airbyte

Joe Reuter (flash1293) · 2023-10-20T12:11:15Z

Adding the experimental document type parser bloated the image size unnecessarily. This PR reduces the required dependencies.

It also adds an acceptance test to verify the parser works correctly for regular PDFs and OCR-only PDFs (e.g. scans)

⚠️ This PR means that the hi_res parsing strategy can't be applied anymore - it will default to fast and fall back to ocr_only if necessary. ⚠️

vercel · 2023-10-20T12:11:22Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 25, 2023 10:27am

github-actions · 2023-10-20T12:11:38Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

airbyte-oss-build-runner · 2023-10-20T12:30:05Z

source-s3 test report (commit `a971df111d`) - ❌

⏲️ Total pipeline duration: 15mn10s

Step	Result
Build source-s3 docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌
Check our base image is used	✅
Code format checks	✅
Validate metadata for source-s3	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

airbyte-oss-build-runner · 2023-10-20T13:10:18Z

source-s3 test report (commit `e76c488b42`) - ❌

⏲️ Total pipeline duration: 14mn35s

Step	Result
Build source-s3 docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌
Check our base image is used	✅
Code format checks	✅
Validate metadata for source-s3	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

airbyte-oss-build-runner · 2023-10-20T13:22:28Z

source-s3 test report (commit `eb6941d301`) - ✅

⏲️ Total pipeline duration: 06mn19s

Step	Result
Build source-s3 docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	✅
Validate metadata for source-s3	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

Joe Reuter (flash1293) · 2023-10-20T23:36:56Z

One other thing I need to check here is whether we should pull in pytesseract

…e-and-test

airbyte-oss-build-runner · 2023-10-23T11:26:46Z

source-s3 test report (commit `f628f5831e`) - ❌

⏲️ Total pipeline duration: 14mn29s

Step	Result
Build source-s3 docker image for platform(s) linux/amd64	✅
Unit tests	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	❌
Validate metadata for source-s3	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

airbyte-oss-build-runner · 2023-10-23T11:48:46Z

source-s3 test report (commit `254aa91d6d`) - ✅

⏲️ Total pipeline duration: 14mn38s

Step	Result
Build source-s3 docker image for platform(s) linux/amd64	✅
Unit tests	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	✅
Validate metadata for source-s3	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

Augustin (alafanechere)

👍 Just a suggestion to make the build_customization.py module more readable / documented.

Augustin (alafanechere) · 2023-10-23T13:45:41Z


-
-async def post_connector_install(connector_container: Container) -> Container:
+async def setup_nltk(connector_container: Container) -> Container:


These two new functions do not need to be async. (there's no await call inside them).

The post_connector_install has to remain async because we want to allow awaits inside them if necessary (which is not your case)

airbyte-oss-build-runner · 2023-10-23T14:08:09Z

source-s3 test report (commit `efa0727cb8`) - ✅

⏲️ Total pipeline duration: 13mn32s

Step	Result
Build source-s3 docker image for platform(s) linux/amd64	✅
Unit tests	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	✅
Validate metadata for source-s3	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

Joe Reuter (flash1293) · 2023-10-23T14:12:25Z

Augustin (@alafanechere) thanks, fixed!

airbyte-oss-build-runner · 2023-10-23T14:27:39Z

source-s3 test report (commit `1eadd5df96`) - ✅

⏲️ Total pipeline duration: 13mn40s

Step	Result
Build source-s3 docker image for platform(s) linux/amd64	✅
Unit tests	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	✅
Validate metadata for source-s3	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

Aaron ("AJ") Steers (aaronsteers)

LGTM!

…e-and-test

airbyte-oss-build-runner · 2023-10-25T10:43:52Z

source-s3 test report (commit `f601bef3d7`) - ✅

⏲️ Total pipeline duration: 16mn06s

Step	Result
Build source-s3 docker image for platform(s) linux/amd64	✅
Unit tests	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	✅
Validate metadata for source-s3	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

reduce image size and add acceptance test

3bf54c5

Octavia Squidington III (octavia-squidington-iii) added the area/connectors Connector related issues label Oct 20, 2023

Octavia Squidington III (octavia-squidington-iii) added the connectors/source/s3 label Oct 20, 2023

prepare release

a971df1

Octavia Squidington III (octavia-squidington-iii) added the area/documentation Improvements or additions to documentation label Oct 20, 2023

Joe Reuter (flash1293) marked this pull request as ready for review October 20, 2023 12:12

vercel Bot deployed to Preview October 20, 2023 12:13 View deployment

Octavia Squidington IV (octavia-squidington-iv) requested a review from a team October 20, 2023 12:13

Baz (bazarnov) changed the title ~~S3: Reduce image size and add acceptance test~~ 🎉 Source S3: Reduce image size and add acceptance test Oct 20, 2023

fix test

e76c488

fix test

eb6941d

Baz (bazarnov) approved these changes Oct 20, 2023

View reviewed changes

Joe Reuter added 2 commits October 23, 2023 13:09

make OCR work

13c4d90

Merge remote-tracking branch 'origin/master' into flash1293/s3-improv…

f628f58

…e-and-test

Joe Reuter (flash1293) requested a review from a team October 23, 2023 11:10

vercel Bot deployed to Preview October 23, 2023 11:12 View deployment

Automated Commit - Formatting Changes

254aa91

Augustin (alafanechere) approved these changes Oct 23, 2023

View reviewed changes

Comment thread airbyte-integrations/connectors/source-s3/build_customization.py Outdated

Comment thread airbyte-integrations/connectors/source-s3/source_s3/v4/unstructured_parser.py Outdated

review comments

4cbcf93

Augustin (alafanechere) suggested changes Oct 23, 2023

View reviewed changes

Automated Commit - Formatting Changes

efa0727

review comments

1eadd5d

Joe Reuter (flash1293) requested a review from Augustin (alafanechere) October 23, 2023 14:12

Augustin (alafanechere) approved these changes Oct 24, 2023

View reviewed changes

Aaron ("AJ") Steers (aaronsteers) approved these changes Oct 25, 2023

View reviewed changes

Merge remote-tracking branch 'origin/master' into flash1293/s3-improv…

f601bef

…e-and-test

vercel Bot deployed to Preview October 25, 2023 10:27 View deployment

Joe Reuter (flash1293) enabled auto-merge (squash) October 25, 2023 10:31

Joe Reuter (flash1293) merged commit 68e99ce into master Oct 25, 2023

Joe Reuter (flash1293) deleted the flash1293/s3-improve-and-test branch October 25, 2023 10:44



		async def post_connector_install(connector_container: Container) -> Container:
		async def setup_nltk(connector_container: Container) -> Container:

Uh oh!

Conversation

Joe Reuter (flash1293) commented Oct 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel Bot commented Oct 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 20, 2023 • edited by flash1293 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

Uh oh!

airbyte-oss-build-runner commented Oct 20, 2023

source-s3 test report (commit a971df111d) - ❌

Uh oh!

airbyte-oss-build-runner commented Oct 20, 2023

source-s3 test report (commit e76c488b42) - ❌

Uh oh!

airbyte-oss-build-runner commented Oct 20, 2023

source-s3 test report (commit eb6941d301) - ✅

Uh oh!

Joe Reuter (flash1293) commented Oct 20, 2023

Uh oh!

airbyte-oss-build-runner commented Oct 23, 2023

source-s3 test report (commit f628f5831e) - ❌

Uh oh!

airbyte-oss-build-runner commented Oct 23, 2023

source-s3 test report (commit 254aa91d6d) - ✅

Uh oh!

Augustin (alafanechere) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Augustin (alafanechere) Oct 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

airbyte-oss-build-runner commented Oct 23, 2023

source-s3 test report (commit efa0727cb8) - ✅

Uh oh!

Joe Reuter (flash1293) commented Oct 23, 2023

Uh oh!

airbyte-oss-build-runner commented Oct 23, 2023

source-s3 test report (commit 1eadd5df96) - ✅

Uh oh!

Aaron ("AJ") Steers (aaronsteers) left a comment

Choose a reason for hiding this comment

Uh oh!

airbyte-oss-build-runner commented Oct 25, 2023

source-s3 test report (commit f601bef3d7) - ✅

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Joe Reuter (flash1293) commented Oct 20, 2023 •

edited

Loading

vercel Bot commented Oct 20, 2023 •

edited

Loading

github-actions Bot commented Oct 20, 2023 •

edited by flash1293

Loading

source-s3 test report (commit `a971df111d`) - ❌

source-s3 test report (commit `e76c488b42`) - ❌

source-s3 test report (commit `eb6941d301`) - ✅

source-s3 test report (commit `f628f5831e`) - ❌

source-s3 test report (commit `254aa91d6d`) - ✅

Augustin (alafanechere) Oct 23, 2023 •

edited

Loading

source-s3 test report (commit `efa0727cb8`) - ✅

source-s3 test report (commit `1eadd5df96`) - ✅

source-s3 test report (commit `f601bef3d7`) - ✅