Skip to content

ref(bigquery): Improve BigQuery encoding and logs#878

Draft
iambriccardo wants to merge 6 commits into
mainfrom
remove-duplicated-logs
Draft

ref(bigquery): Improve BigQuery encoding and logs#878
iambriccardo wants to merge 6 commits into
mainfrom
remove-duplicated-logs

Conversation

@iambriccardo

@iambriccardo iambriccardo commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

This PR removes duplicated and overly verbose logging, and reworks BigQuery row encoding for correctness and efficiency.

Logging

  • Downgrades per-step info! logs during pipeline shutdown, replication slot lookup/deletion, and table sync worker startup/state persistence to debug!, since a higher-level summary is already logged elsewhere.
  • Removes now-redundant logs that duplicated information already captured by metrics (table copy completion) or by the surrounding log statements.

BigQuery row encoding

  • BigQueryTableRow now stores only its already-encoded Protocol Buffer bytes (Vec<u8>) instead of keeping both the source cells and a separate encoded-bytes cache alive. Encoding happens once, up front, in try_from_tagged_cells, and encoded_len/encode_raw just read the already-encoded buffer. Previously, gcp-bigquery-client calling encoded_len() for batch budgeting and then encode_to_vec() (which calls encoded_len() again before encode_raw()) caused every row's date/numeric/UUID/JSON fields to be formatted up to 3 times.
  • Removes CellNonOptional/ArrayCellNonOptional from etl::data, which existed solely to give BigQuery a type-level guarantee that arrays contain no NULL elements. BigQuery is now the only destination that cared about this, so the check moved to a runtime validation (validate_array_cell_for_bigquery) that rejects NULL array elements with NullValuesNotSupportedInArrayInDestination, rather than duplicating Cell/ArrayCell in a parallel non-nullable enum.

@iambriccardo iambriccardo changed the title ref(logs): Remove duplicated and verbose logs ref(logs): Improve BigQuery encoding and logs Jul 6, 2026
@iambriccardo iambriccardo changed the title ref(logs): Improve BigQuery encoding and logs ref(bigquery): Improve BigQuery encoding and logs Jul 6, 2026
@coveralls

Copy link
Copy Markdown

Coverage Status

coverage: 73.092% (-0.03%) from 73.123% — remove-duplicated-logs into main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants