Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bmoric/async memory bump #30775

Merged
merged 16 commits into from Sep 27, 2023
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Expand Up @@ -38,7 +38,7 @@ public class GlobalMemoryManager {
// In cases where a queue is rapidly expanding, a larger block size allows less allocation calls. On
// the flip size, a smaller block size allows more granular memory management. Since this overhead
// is minimal for now, err on a smaller block sizes.
public static final long BLOCK_SIZE_BYTES = 10 * 1024 * 1024; // 10MB
public static final long BLOCK_SIZE_BYTES = 30 * 1024 * 1024; // 30MB
private final long maxMemoryBytes;

private final AtomicLong currentMemoryBytes = new AtomicLong(0);
Expand All @@ -61,7 +61,7 @@ public long getCurrentMemoryBytes() {
* @return the size of the allocated block, in bytes
*/
public synchronized long requestMemory() {
// todo(davin): what happens if the incoming record is larger than 10MB?
// todo(davin): what happens if the incoming record is larger than 30MB?
if (currentMemoryBytes.get() >= maxMemoryBytes) {
return 0L;
}
Expand Down
Expand Up @@ -25,5 +25,5 @@ ENV AIRBYTE_NORMALIZATION_INTEGRATION bigquery

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=2.0.17
LABEL io.airbyte.version=2.0.18
LABEL io.airbyte.name=airbyte/destination-bigquery
Expand Up @@ -2,7 +2,7 @@ data:
connectorSubtype: database
connectorType: destination
definitionId: 22f6c74f-5699-40ff-833c-4a879ea40133
dockerImageTag: 2.0.17
dockerImageTag: 2.0.18
dockerRepository: airbyte/destination-bigquery
githubIssueLabel: destination-bigquery
icon: bigquery.svg
Expand Down
Expand Up @@ -29,5 +29,5 @@ RUN tar xf ${APPLICATION}.tar --strip-components=1

ENV ENABLE_SENTRY true

LABEL io.airbyte.version=3.1.13
LABEL io.airbyte.version=3.1.14
LABEL io.airbyte.name=airbyte/destination-snowflake
Expand Up @@ -2,7 +2,7 @@ data:
connectorSubtype: database
connectorType: destination
definitionId: 424892c4-daac-4491-b35d-c6688ba547ba
dockerImageTag: 3.1.13
dockerImageTag: 3.1.14
dockerRepository: airbyte/destination-snowflake
githubIssueLabel: destination-snowflake
icon: snowflake.svg
Expand Down
Expand Up @@ -219,7 +219,7 @@
"type": "array"
},
"custom_fields": {
"properties": {},
"properties": {},
"additionalProperties": true,
"type": ["null", "object"]
},
Expand Down
2 changes: 1 addition & 1 deletion deps.toml
Expand Up @@ -3,7 +3,7 @@ airbyte-protocol = "0.3.6"
commons_io = "2.7"
testcontainers = "1.19.0"
datadog-version = "0.111.0"
fasterxml_version = "2.14.0"
fasterxml_version = "2.15.2"
flyway = "7.14.0"
glassfish_version = "2.31"
hikaricp = "5.0.1"
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/bigquery.md
Expand Up @@ -133,6 +133,7 @@ Now that you have set up the BigQuery destination connector, check out the follo

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-----------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2.0.18 | 2023-09-26 | [\#30775](https://github.com/airbytehq/airbyte/pull/30775 | Increase async block size |
| 2.0.17 | 2023-09-26 | [\#30696](https://github.com/airbytehq/airbyte/pull/30696) | Attempt unsafe typing operations with an exception clause |
| 2.0.16 | 2023-09-22 | [\#30697](https://github.com/airbytehq/airbyte/pull/30697) | Improve resiliency to unclean exit during schema change |
| 2.0.15 | 2023-09-21 | [\#30640](https://github.com/airbytehq/airbyte/pull/30640) | Handle streams with identical name and namespace |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/snowflake.md
Expand Up @@ -271,6 +271,7 @@ Otherwise, make sure to grant the role the required permissions in the desired n

| Version | Date | Pull Request | Subject |
|:----------------|:-----------|:-----------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 3.1.14 | 2023-09-26 | [\#30775](https://github.com/airbytehq/airbyte/pull/30775 | Increase async block size |
| 3.1.13 | 2023-09-19 | [\#30599](https://github.com/airbytehq/airbyte/pull/30599) | Support concurrent syncs with identical stream name but different namespace |
| 3.1.12 | 2023-09-21 | [\#30671](https://github.com/airbytehq/airbyte/pull/30671) | Reduce async buffer size |
| 3.1.11 | 2023-09-19 | [\#30592](https://github.com/airbytehq/airbyte/pull/30592) | Internal code changes |
Expand Down