Bug 1624908 Add sendFailure subfields for health ping

[jklukas]

Checklist for reviewer:

• Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
• Scan the PR and verify that no changes (particularly to .circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logs
• If the PR comes from a fork, trigger the integration CI test by pushing this revision as discussed in the README and review the report posted in the comments.

For glean changes:

• Update include/glean/CHANGELOG.md

[dataops-ci-bot]

Integration report for "Bug 1624908 Add sendFailure subfields for health ping"

Report for upstream
Report for latest commit

3f11d8b-7007956.diff

No content detected.

bq_schema_3f11d8b-7007956.diff

Click to expand!

diff integration/3f11d8b/telemetry.health.4.bq integration/7007956/telemetry.health.4.bq
107a108
>             "description": "Never populated; this field was added unintentionally and is retained only for compatibility",
110a112,126
>           },
>           {
>             "mode": "NULLABLE",
>             "name": "e_terminated",
>             "type": "INT64"
>           },
>           {
>             "mode": "NULLABLE",
>             "name": "e_unreachable",
>             "type": "INT64"
>           },
>           {
>             "mode": "NULLABLE",
>             "name": "timeout",
>             "type": "INT64"

[dataops-ci-bot]

Integration report for "Add pingDiscardedForSize subfields"

Report for upstream
Report for latest commit

3f11d8b-a4e091e.diff

No content detected.

bq_schema_3f11d8b-a4e091e.diff

Click to expand!

diff integration/3f11d8b/telemetry.health.4.bq integration/a4e091e/telemetry.health.4.bq
90a91
>             "description": "Never populated; this field was added unintentionally and is retained only for compatibility",
93a95,114
>           },
>           {
>             "mode": "NULLABLE",
>             "name": "crash",
>             "type": "INT64"
>           },
>           {
>             "mode": "NULLABLE",
>             "name": "main",
>             "type": "INT64"
>           },
>           {
>             "mode": "NULLABLE",
>             "name": "prio",
>             "type": "INT64"
>           },
>           {
>             "mode": "NULLABLE",
>             "name": "sync",
>             "type": "INT64"
108a130,135
>             "name": "abort",
>             "type": "INT64"
>           },
>           {
>             "description": "Never populated; this field was added unintentionally and is retained only for compatibility",
>             "mode": "NULLABLE",
110a138,162
>           },
>           {
>             "mode": "NULLABLE",
>             "name": "e_channel_open",
>             "type": "INT64"
>           },
>           {
>             "mode": "NULLABLE",
>             "name": "e_terminated",
>             "type": "INT64"
>           },
>           {
>             "mode": "NULLABLE",
>             "name": "e_unreachable",
>             "type": "INT64"
>           },
>           {
>             "mode": "NULLABLE",
>             "name": "timeout",
>             "type": "INT64"
>           },
>           {
>             "mode": "NULLABLE",
>             "name": "undefined",
>             "type": "INT64"

[jklukas]

@acmiyaguchi From the bq schema diff, it looks like this field with name "<unknown>" just gets dropped and is absent from the bq schema. Does that sound like intended behavior?

I realize that generating a sane bq-compatible field name for "<unknown>" is a bit of a stretch, but I was expecting that this might be normalized to _unknown_.

[acmiyaguchi]

No, there is currently no behavior for normalizing an empty string as a key into BigQuery. This is certainly something that may be added, perhaps mapping it to _ or __unknown__. However, this key would conflict with other JSON keys like ., _, or __. I think the right behavior would be to panic or drop the fields in the case where the transpiler detects these name clashes.

Would this be handled deterministically in the decoder?

>> x = {}; x[""]=1; x["."]=2; x["_"]=3;
3
>> x
Object { "": 1, ".": 2, _: 3 }

[acmiyaguchi]

I'm actually not sure why <unknown> is converted into "", that's unintended behavior for sure.

[jklukas]

Redash was converting "<unknown>" to "". The BQ console shows the value as "". So I would expect the schema generator to be seeing "<unknown>" as the input name and potentially normalizing to _unknown_.

[jklukas]

Yeesh, having this discussion is difficult due to literally typing <unknown> getting interpreted as an html tag and hidden.

[acmiyaguchi]

I was able to reproduce this locally in a shell, outside of any rendering done by javascript/browser.

% echo '{"properties": {"payload": {"properties": {"<unknown>": {"type": "string"}, "foo": {"type": "string"}}}}}' | jq                               
{
  "properties": {
    "payload": {
      "properties": {
        "<unknown>": {
          "type": "string"
        },
        "foo": {
          "type": "string"
        }
      }
    }
  }

 % echo '{"properties": {"payload": {"properties": {"<unknown>": {"type": "string"}, "foo": {"type": "string"}}}}}' | jsonschema-transpiler -t bigquery               
[
  {
    "fields": [
      {
        "mode": "NULLABLE",
        "name": "foo",
        "type": "STRING"
      }
    ],
    "mode": "NULLABLE",
    "name": "payload",
    "type": "RECORD"
  }
]

[jklukas]

We include the list of observed keys as seen in query: https://sql.telemetry.mozilla.org/queries/72242/source

[jklukas]

We include the observed keys as seen in query: https://sql.telemetry.mozilla.org/queries/72243/source

[jklukas]

Closing in favor of mozilla/bigquery-etl#1173