Decat BigQuery INVALID_ARGUMENT

[troyraen]

View the error in GCP Console

"error": {
    "code": 400,
    "message": "Invalid JSON payload received. Unexpected token.\n15757751465, \"mag\": NaN, \"magerr\": -0.83\n                    ^",
    "errors": [
      {
        "message": "Invalid JSON payload received. Unexpected token.\n15757751465, \"mag\": NaN, \"magerr\": -0.83\n                    ^",
        "domain": "global",
        "reason": "parseError"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }

It appears that alerts are coming in with mag=NaN. Need to log some things (objectId's) and download alerts with problems.

This is producing a lot of errors in the bq_sink Dataflow job, and is preventing it from draining properly.

I have tried (without success)

• changing the schema to allow a null mag
• changing the BQ upload retry strategy to RETRY_NEVER

[troyraen]

NaN is not valid JSON. In the future, we should probably encode these as Null, but see here for several options, and look into what BigQuery allows.

Decat is over, and we don't use Dataflow anymore. Closing the issue.