Remove entry and allocated memory for empty buffers. #31194
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
.../core/src/main/java/io/airbyte/cdk/integrations/destination_async/buffers/BufferDequeue.java
Outdated
Show resolved
Hide resolved
tryangul
force-pushed
the
rbroughan/clean-up-empty-buffer-memory
branch
12 times, most recently
from
e21617f
to
d78caf0
Compare
octavia-squidington-iii
added
area/connectors
|
@benmoriceau should be ready for re-review |
|
| Step | Result |
|---|---|
| Build connector tar | ✅ |
| Build destination-redshift docker image for platform(s) linux/x86_64 | ✅ |
| Java Connector Unit Tests | ✅ |
| Build airbyte/normalization-redshift:dev | ✅ |
| Java Connector Integration Tests | ❌ |
| Validate metadata for destination-redshift | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-redshift test
tryangul
force-pushed
the
rbroughan/clean-up-empty-buffer-memory
branch
from
480a1a9
to
deca106
Compare
|
| Step | Result |
|---|---|
| Build connector tar | ❌ |
| Validate metadata for destination-snowflake | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-snowflake test
|
| Step | Result |
|---|---|
| Build connector tar | ❌ |
| Validate metadata for destination-redshift | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-redshift test
|
| Step | Result |
|---|---|
| Build connector tar | ❌ |
| Validate metadata for destination-bigquery | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery test
|
| Step | Result |
|---|---|
| Build connector tar | ❌ |
| Validate metadata for destination-redshift | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-redshift test
|
| Step | Result |
|---|---|
| Build connector tar | ❌ |
| Validate metadata for destination-snowflake | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-snowflake test
tryangul
force-pushed
the
rbroughan/clean-up-empty-buffer-memory
branch
from
164a818
to
1a7fad8
Compare
|
| Step | Result |
|---|---|
| Build connector tar | ❌ |
| Validate metadata for destination-snowflake | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-snowflake test
|
| Step | Result |
|---|---|
| Build connector tar | ❌ |
| Validate metadata for destination-bigquery | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery test
|
| Step | Result |
|---|---|
| Build connector tar | ❌ |
| Validate metadata for destination-redshift | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-redshift test
|
| Step | Result |
|---|---|
| Build connector tar | ❌ |
| Validate metadata for destination-snowflake | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-snowflake test
|
| Step | Result |
|---|---|
| Build connector tar | ❌ |
| Validate metadata for destination-bigquery | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery test
|
| Step | Result |
|---|---|
| Build connector tar | ❌ |
| Validate metadata for destination-redshift | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-redshift test
|
/publish-java-cdk
|
tryangul
force-pushed
the
rbroughan/clean-up-empty-buffer-memory
branch
from
7566b0f
to
4a96e8e
Compare
tryangul
force-pushed
the
rbroughan/clean-up-empty-buffer-memory
branch
from
4a96e8e
to
f52bb50
Compare
|
| Step | Result |
|---|---|
| Build connector tar | ✅ |
| Build destination-bigquery docker image for platform(s) linux/x86_64 | ✅ |
| Java Connector Unit Tests | ✅ |
| Java Connector Integration Tests | ✅ |
| Validate metadata for destination-bigquery | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery test
|
| Step | Result |
|---|---|
| Build connector tar | ✅ |
| Build destination-snowflake docker image for platform(s) linux/x86_64 | ✅ |
| Java Connector Unit Tests | ✅ |
| Java Connector Integration Tests | ✅ |
| Validate metadata for destination-snowflake | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-snowflake test
|
| Step | Result |
|---|---|
| Build connector tar | ✅ |
| Build destination-redshift docker image for platform(s) linux/x86_64 | ✅ |
| Java Connector Unit Tests | ✅ |
| Build airbyte/normalization-redshift:dev | ✅ |
| Java Connector Integration Tests | ✅ |
| Validate metadata for destination-redshift | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-redshift test
|
| Step | Result |
|---|---|
| Build connector tar | ✅ |
| Build destination-bigquery docker image for platform(s) linux/x86_64 | ✅ |
| Java Connector Unit Tests | ✅ |
| Java Connector Integration Tests | ✅ |
| Validate metadata for destination-bigquery | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery test
|
| Step | Result |
|---|---|
| Build connector tar | ✅ |
| Build destination-snowflake docker image for platform(s) linux/x86_64 | ✅ |
| Java Connector Unit Tests | ✅ |
| Java Connector Integration Tests | ✅ |
| Validate metadata for destination-snowflake | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-snowflake test
Co-authored-by: tryangul <tryangul@users.noreply.github.com>
What
Fixes memory leak due to empty buffers holding onto allocated memory. Fix for this issue.
How
If a buffer for a stream descriptor ever empties, we shrink the queue and release any lingering memory not covered by the pending batch release.
Notes
We are shrinking the queue and not removing it because 1) enough code assumed the queue was always present that it presented a bigger change. 2) if there is some race on the map (there shouldn't be) removing the entry could lead to dropped data.
🚨 User Impact 🚨
This fixes an ongoing issue. If users use an async connector on tables with lots of streams, they will want to upgrade.