Athena queries fail when json parsing error occurs

[sruti]

Athena queries were failing with the following error:
HIVE_CURSOR_ERROR: Row is not a valid JSON Object - JSONException: Unterminated string at 198 [character 199 line 1]

The offending file was cut off mid-line and looked like this:

Temporary fix was to replace the file in s3 with the line removed.

Possible permanent fixes:

• Set up alerts for Athena query failures
• Modify lambda function to detect json errors before gzipping

[jflasher]

Would be great to alarm on this! I just looked and CloudWatch does not provide any native alarms for Athena. I think we may be able to modify Fluffy to watch out for this? It could probably look something like https://github.com/openaq/openaq-api/pull/424/files#diff-3467d50aec9f3331a916f24fa115bdacR288 where we check over the recent executions and see if anything is failing. Once we allow user submitted queries for Athena, this may make less sense though, since we don't really care if users are creating queries that fail, we only care about the system ones. There may also just be a way to use the API to loop over /locations and see when the last updated was, that might be easiest.

[majesticio]

@russbiggs is this still a bug? I have not used Athena in this context yet

[russbiggs]

This has been replaced by the new ingest process i believe