You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
When running with log replication, if the extractor encounters array data, it will open multiple connections to Postgres in order to figure out the array data's schema. When there are large tables that contain array fields, the number of connections being opened to Postgres will cause new connections to be denied. This results in a connection timeout error.
To Reproduce
Steps to reproduce the behavior:
Setup the extractor to run against a large table containing array fields
Perform a migration in Postgres on this large table (causing a large number of records to be dumped into the transaction log)
Run the extractor using log replication
Monitor the increase in connections being opened
Hit connection timeout at Postgres starts rejecting new connections
Timeout error: name=tap_postgres level=CRITICAL message=connection to server at "XXXXXX" (XXXXXX), port 5432 failed: timeout expired
Spikes in connections count when running Meltano. Behaviour is shown being turned on and off by including / excluding the table containing the array field. Session counts drop after the timeout error is hit.
Expected behavior
There should be options to allow for the array fields to be handled as string or parsed as JSON. I would be hassle to have to write a potentially have to write a UDF in a downstream platform to parse the string but this is better than a process that fails and also degrades the production database or the read replica database.
Your environment
Version of tap: 2.1.0
meltano:v3.3.0
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered:
Describe the bug
When running with log replication, if the extractor encounters array data, it will open multiple connections to Postgres in order to figure out the array data's schema. When there are large tables that contain array fields, the number of connections being opened to Postgres will cause new connections to be denied. This results in a connection timeout error.
To Reproduce
Steps to reproduce the behavior:
Timeout error:
name=tap_postgres level=CRITICAL message=connection to server at "XXXXXX" (XXXXXX), port 5432 failed: timeout expired
Spikes in connections count when running Meltano. Behaviour is shown being turned on and off by including / excluding the table containing the array field. Session counts drop after the timeout error is hit.
![Screenshot 2024-04-22 at 16 04 11](https://private-user-images.githubusercontent.com/42941066/324510908-292fc12d-ed1e-494a-872b-b9dd1b7ffc28.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE0NTI3ODksIm5iZiI6MTcyMTQ1MjQ4OSwicGF0aCI6Ii80Mjk0MTA2Ni8zMjQ1MTA5MDgtMjkyZmMxMmQtZWQxZS00OTRhLTg3MmItYjlkZDFiN2ZmYzI4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzIwVDA1MTQ0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTVjNmMzZTY5MjhhYjk4MTE4OTExNzEwZjllZTNlNmJhOWQ0ODljNjZlYTdmNjdiZGRlNTg4MjI5OWNjODdkNDcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.YmO6AgSVLSHsIEdy1gprmU3tN8cC62rmKwsG9dDtd8w)
Expected behavior
There should be options to allow for the array fields to be handled as string or parsed as JSON. I would be hassle to have to write a potentially have to write a UDF in a downstream platform to parse the string but this is better than a process that fails and also degrades the production database or the read replica database.
Your environment
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: