feat: Configurable batch size and max wait limit for targets

[edgarrmondragon]

Feature scope

Targets (data type handling, batching, SQL object generation, etc.)

Description

Prior art

Many targets in the Singer ecosystem (target-snowflake is one) support the following settings or some variant of them:

Property	Type	Description
batch_size_rows	Integer	(Default: 100000) Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Snowflake.
batch_wait_limit_seconds	Integer	(Default: None) Maximum time to wait for batch to reach batch_size_rows.

Current SDK support

The Singer SDK currently only allows target developers to configure the maximum size of a batch via an attribute:

sdk/singer_sdk/sinks/core.py

Line 45 in 43f7323

MAX_SIZE_DEFAULT = 10000

By making this configurable, users would have better control of the resources in a constrained environment (i.e. low memory).

Proposal

• Add a new pair of built-in settings batch_size_rows and batch_wait_limit_seconds
• Update Sink.is_full to use batch_size_rows and fall back to Sink.MAX_SIZE_DEFAULT if batch_size_rows is not configured
• Track the start time of each batch and implement Sink.is_too_old to check against batch_wait_limit_seconds
• Update the TargetBase class to make use of the new method that checks the batch TTL

Related

• Target sink - optimization strategies for when to flush batches #134
• feat: Track memory usage in targets and drain record batch if a treshold is passed #1627

[tayloramurphy]

Requested by multiple people in slack. We may want to do this as part of v1