Missing API Validation Layer - Results in lost data

[kevin-kline]

Description:

• API validation is needed to ensure response data from APIs are consistent with what we expect
• Every entity we request from an API gets converted from a response to a Go struct prior to saving in our DB
  • Ex. Commits from GitHub
• If the responses from the API change in any way from what we expect to get back, then we risk losing data
• This can happen silently because we don’t have validation to ensure the data matches what we expect
• API responses can vary, or be changed intentionally by the provider, so we must be prepared for this
• We have discovered this problem exists in our system in the GitHub plugin, so it may exist in others as well. If not now, it could in the future.

Example (tested, proven):

- In the GitHub plugin, only some of our response data contained author data and committer data
- This is because some users are not verified on GitHub
- Some commits were not being saved because of the failed conversion
- We expected certain data, but the responses were inconsistent leading to lost data
- Without validation we have no way to know this is happening, since this error occurs at runtime

Screenshots:

1. Commit w/ author
   

2. Commit w/out author
   

3. Proof of why author / committer is not found (no valid user account on GitHub)
   

4. Problem in the code
   

Possible Solutions:

• Add validation layer
• Maybe this package: https://pkg.go.dev/gopkg.in/go-playground/validator.v8#section-readme

@joncodo Please comment / edit with your understanding if needed

[Startrekzky]

root-reason: no error swallowing so the bug doesn't expose in development and QA stage