Summary pipeline: TypeError: Integer exceeds 64-bit range (while running WriteRequestsToBigQuery...)

[rviscomi]

TypeError: Integer exceeds 64-bit range [while running 'WriteRequestsToBigQuery/WriteToBigQuery/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn)-ptransform-63']

Error details

This started happening yesterday and the pipeline has since stopped writing data to BQ.

[rviscomi]

I have a hunch that there are wildly high Cache-Control: max-age values.

data-pipeline/modules/transformation.py

Lines 314 to 318 in c51fa1e

	try:
	exp_age = int(re.findall(r"\d+", cc)[0])
	except Exception:
	# TODO compare results from old and new pipeline for these errors
	logging.warning(f"Unable to parse max-age, cc:{cc}", exc_info=True)

In the April dataset, the largest max-age value was 28 digits:

# 28
SELECT
  MAX(LENGTH(REGEXP_EXTRACT(resp_cache_control, r'(?i)max-age\s*=\s*(\d+)'))) AS digits
FROM
  `httparchive.summary_requests.2022_04_01_mobile`

2^64 is 20 digits, so this greatly exceeds the limit.

Python doesn't have a max int value, so it doesn't throw an exception. But BigQuery does have a max int value, so it throws the "exceeds 64-bit range" error.

There may be other user-defined fields like this in the wild that exceed 64 bits that need special handling in our application code.