Skip to content

Destination BigQuery - remove file_buffer_count - #32234

Merged
Evan Tahler (evantahler) merged 12 commits into
masterfrom
evan/remove-file_buffer_count
Nov 9, 2023
Merged

Evan Tahler (evantahler) merged 12 commits into
masterfrom
evan/remove-file_buffer_count

Conversation

@evantahler

@evantahler Evan Tahler (evantahler) commented Nov 7, 2023 •

Copy link
Copy Markdown
Contributor

The config option file_buffer_count doesn't matter now that these destinations have been async'd. Let's remove it!

Closes #32275

@vercel

vercel Bot commented Nov 7, 2023 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Nov 8, 2023 11:32pm

@github-actions

github-actions Bot commented Nov 7, 2023 •

Copy link
Copy Markdown
Contributor

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

@evantahler

Copy link
Copy Markdown
Contributor Author

Cynthia Yin (@cynthiaxyin) I ran a number of the failing redshift tests locally and they seem to be OK... is this flaky test behavior expected for destination-redshift?

Note: to run the DATs locally, you'll need to change useLocalCdk = true in redshift's gradlefile and then tell InteliJ to "reload all gradle projects"

@cynthiaxyin

Copy link
Copy Markdown
Contributor

Evan Tahler (@evantahler) Hmm I'm not sure I haven't run destination-redshift tests recently.

@evantahler

Copy link
Copy Markdown
Contributor Author

Ryan Br... (@tryangul) or Benoit Moriceau (@benmoriceau) - can you confirm that getSerializedMessageConsumer() should not be used by Redshift any more? Things are compiling, and most of the tests pass, but the flakiness of the tests is giving me pause

@tryangul

Copy link
Copy Markdown
Contributor

Evan Tahler (@evantahler) Last I was in this code getSerializedMessageConsumer was the code that setup the async destination and was a critical piece. Is there a new a new path to trigger this? I'm spelunking around right now, but it's been a while since I've been in here.

@evantahler Evan Tahler (evantahler) changed the title Destination BigQuery and Redshift - remove file_buffer_count Destination BigQuery - remove file_buffer_count Nov 8, 2023
@evantahler

Copy link
Copy Markdown
Contributor Author

Ok, I'm scoping this PR down to be just bigquery changes, as those tests are passing

This branch was successfully deployed

1 active deployment
Preview — 235a7e52 Deployed Nov 8, 2023 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Destination BigQuery - remove file_buffer_count

5 participants