🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handle wide rows by using byte-based record buffering

[Sherif A. Nada (sherifnada)]

What

JDBC Destinations, when running in INSERT mode, typically write records as follows:

1. Accumulate records in a buffer that can grow up to 10_000 records in size
2. Flush buffer to memory when it's full
3. Flush buffer once the sync is complete

This approach is problematic because it does not take into account how large those records are, which means wide rows can be problematic (10k 5mb rows has very different memory implications than 10k 4kb rows). This is especially exacerbated because when flushing records to the destination, the JDBC drivers needs to hold a SQL string containing all the rows in memory (i.e: INSERT xyz INTO table where xyz is the content of all the rows), so if the queue is about to write N bytes, we actually need to have a multiple of N bytes capacity in memory (somewhere between 2 and 3 as far as I can tell looking under the JDBC hood).

Recently, a user was syncing very wide rows from a Postgres DB which caused this issue to appear. So this PR changes the buffering strategy to count bytes instead of number of records.

I've confirmed this fix works for the user in question.

How

There are two changes being made here:

1. keep track of the buffer size in bytes and flush when the size has reached the maximum alloted. This requires an extra serialization of the input records. This is ineffecient but didn't seem to be a huge cost when I was profiling. We definitely need more rigorous measurement of the CPU overhead of this (and we can probably even remove the need for it) but I think we should do that separately, as this fix unblocks using Snowflake destination for this category of data sources. We can ship a more CPU efficient solution as an incremental improvement. I've set the default buffer size to 256Mib. This way even if we hit a 3x multiple, we are using less than 1Gib of memory.
2. When writing records, cap all SQL INSERT statements to at most 10k records because some destinations like Snowflake have limitations on how many records can be written at a time (snowflake limits to 16k).

Recommended reading order

1. BufferedStreamConsumer.java
2. The rest is just adjusting everything to use the new approach

Pre-merge Checklist

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here
• After the new connector version is published, connector version bumped in the seed directory as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

[Sherif A. Nada (sherifnada)]

/test connector=destination-snowflake

🕑 destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1432867826
❌ destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1432867826
🐛 https://gradle.com/s/tpzamgexz3aho

[Sherif A. Nada (sherifnada)]

/test connector=destination-redshift

🕑 destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1432868024
❌ destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1432868024
🐛 https://gradle.com/s/nqciyaqdr4o7m

[Sherif A. Nada (sherifnada)]

/test connector=destination-postgres

🕑 destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1432868262
❌ destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1432868262
🐛 https://gradle.com/s/xkyrnd3fder56

[Sherif A. Nada (sherifnada)]

/test connector=destination-mysql

🕑 destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1432868483
❌ destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1432868483
🐛 https://gradle.com/s/pb3hu46aedkcy

[Sherif A. Nada (sherifnada)]

/test connector=destination-postgres

🕑 destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1436924952
✅ destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1436924952
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 468    287    39%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1160    472    59%

[Sherif A. Nada (sherifnada)]

/publish connector=connectors/destination-mssql

🕑 connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/1437738459
✅ connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/1437738459

[Sherif A. Nada (sherifnada)]

/publish connector=connectors/destination-mssql-strict-encrypt

🕑 connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1437738541
✅ connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1437738541

[Sherif A. Nada (sherifnada)]

/publish connector=connectors/destination-mysql

🕑 connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1437738884
✅ connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1437738884