Skip to content

Source Postgres : Changes to max fetch row size - #22623

Merged
Akash Kulkarni (akashkulk) merged 11 commits into
masterfrom
logging
Feb 24, 2023
Merged

Akash Kulkarni (akashkulk) merged 11 commits into
masterfrom
logging

Conversation

@akashkulk

Copy link
Copy Markdown
Contributor
  • Add frequent logging for fetch size
  • Increase MAX_ROW_FETCH_SIZE to 1,000,000,000. Previously, we capped this @ 2 million

Context : https://airbytehq-team.slack.com/archives/C048E3BK1KM/p1675745575268449

- Add frequent logging for fetch size
- Increase MAX_ROW_FETCH_SIZE to 1,000,000,000
@akashkulk
Akash Kulkarni (akashkulk) marked this pull request as ready for review February 9, 2023 01:21
@akashkulk
Akash Kulkarni (akashkulk) temporarily deployed to more-secrets February 9, 2023 01:22 — with GitHub Actions Inactive
@akashkulk
Akash Kulkarni (akashkulk) temporarily deployed to more-secrets February 9, 2023 01:22 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Feb 9, 2023 •

Copy link
Copy Markdown
Contributor

Airbyte Code Coverage

File Coverage [100%] 🍏
AdaptiveStreamingQueryConfig.java 100% 🍏
Total Project Coverage 24.66% ❌

@akashkulk
Akash Kulkarni (akashkulk) temporarily deployed to more-secrets February 9, 2023 22:26 — with GitHub Actions Inactive
@akashkulk
Akash Kulkarni (akashkulk) temporarily deployed to more-secrets February 9, 2023 22:26 — with GitHub Actions Inactive
@akashkulk
Akash Kulkarni (akashkulk) temporarily deployed to more-secrets February 10, 2023 17:31 — with GitHub Actions Inactive
@akashkulk
Akash Kulkarni (akashkulk) temporarily deployed to more-secrets February 10, 2023 17:31 — with GitHub Actions Inactive
@akashkulk
Akash Kulkarni (akashkulk) temporarily deployed to more-secrets February 10, 2023 20:05 — with GitHub Actions Inactive
@akashkulk
Akash Kulkarni (akashkulk) temporarily deployed to more-secrets February 10, 2023 20:05 — with GitHub Actions Inactive
@akashkulk

Akash Kulkarni (akashkulk) commented Feb 10, 2023 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4147302669
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4147302669
🐛 https://gradle.com/s/ki5hu2nznx2ng

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs1]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs1]
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_incremental.py:22: `future_state` has a bypass reason, skipping.
============== 3 failed, 63 passed, 6 skipped in 99.51s (0:01:39) ==============

@akashkulk

Akash Kulkarni (akashkulk) commented Feb 15, 2023 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4186747570
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4186747570
🐛 https://gradle.com/s/xroz3kkrvv26s

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs1]
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_incremental.py:22: `future_state` has a bypass reason, skipping.
============= 2 failed, 64 passed, 6 skipped in 421.77s (0:07:01) ==============

@akashkulk
Akash Kulkarni (akashkulk) temporarily deployed to more-secrets February 15, 2023 18:01 — with GitHub Actions Inactive
@akashkulk
Akash Kulkarni (akashkulk) temporarily deployed to more-secrets February 15, 2023 18:02 — with GitHub Actions Inactive
@akashkulk

Akash Kulkarni (akashkulk) commented Feb 16, 2023 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4195966766
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4195966766
🐛 https://gradle.com/s/dyem5nqpravl4

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs1]
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_incremental.py:22: `future_state` has a bypass reason, skipping.
============= 2 failed, 64 passed, 6 skipped in 415.64s (0:06:55) ==============

@akashkulk
Akash Kulkarni (akashkulk) temporarily deployed to more-secrets February 16, 2023 16:20 — with GitHub Actions Inactive
@akashkulk
Akash Kulkarni (akashkulk) temporarily deployed to more-secrets February 16, 2023 16:20 — with GitHub Actions Inactive
@akashkulk
Akash Kulkarni (akashkulk) requested a review from a team as a code owner February 23, 2023 20:45
@octavia-squidington-iii Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Feb 23, 2023
@akashkulk

Akash Kulkarni (akashkulk) commented Feb 23, 2023 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4256697759
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4256697759
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
=================== 62 passed, 5 skipped in 86.20s (0:01:26) ===================

@github-actions

github-actions Bot commented Feb 23, 2023 •

Copy link
Copy Markdown
Contributor

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

  • Run integration tests
  • Bump connector or module version
  • Add changelog
  • Publish the new version

✅ Sources (3)

Connector Version Changelog Publish
source-alloydb 1.0.47 ✅ ✅
source-alloydb-strict-encrypt 1.0.47 🔵
(ignored)
🔵
(ignored)
source-postgres-strict-encrypt 1.0.48 🔵
(ignored)
🔵
(ignored)
  • See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector Version Changelog Publish
  • See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category Status Actionable Item
Version ❌
mismatch
The version of the connector is different from its normal variant. Please bump the version of the connector.
⚠
doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog ⚠
doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
❌
changelog missing
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish ⚠
not in seed
The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
❌
diff seed version
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version.

@akashkulk

Akash Kulkarni (akashkulk) commented Feb 23, 2023 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres-strict-encrypt

🕑 connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/4256717625
❌ connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/4256717625
🐛

Build Passed

Test summary info:

All Passed

@akashkulk

Akash Kulkarni (akashkulk) commented Feb 23, 2023 •

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/source-postgres run-tests=false

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/4257268906


Connector Did it publish? Were definitions generated?
connectors/source-postgres ✅ ✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@akashkulk

Akash Kulkarni (akashkulk) commented Feb 23, 2023 •

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/source-postgres-strict-encrypt run-tests=false

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4257269541


Connector Did it publish? Were definitions generated?
connectors/source-postgres-strict-encrypt ✅ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants