store compliance run info

[rickmarry]

Signed-off-by: Rick Marry rmarry@chef.io

🔩 Description: What code changed, and why?

this pr implements the requirement described in issue: #5860
Store the run information in ES index while Compliance report ingestion

⛓️ Related Resources

#5860

👍 Definition of Done

👟 How to Build and Test the Change

in hab:
rebuild components/compliance-service/
start_all_services

Once the services are all running, run the integration tests
compliance_run_info_integration

If you would like to see what data looks like in elasticsearch, you may run chef_load to load up some node runs like so:
chef_load_compliance_scans -D1 -N2 -M2 -T4

Then issue the following curl to see the run_info that was created:
curl --request GET 'localhost:10141/comp-1-run-info/_search' | jq

which will produce similar to the following response:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 23,
    "max_score": 1,
    "hits": [
      {
        "_index": "comp-1-run-info",
        "_type": "_doc",
        "_id": "bb93e1b2-36d6-439e-ac70-cccccccccc08",
        "_score": 1,
        "_source": {
          "first_run": "2021-10-14T00:00:10Z",
          "last_run": "2021-10-14T00:00:10Z",
          "node_uuid": "bb93e1b2-36d6-439e-ac70-cccccccccc08"
        }
      },
      {
        "_index": "comp-1-run-info",
        "_type": "_doc",
        "_id": "7541037b-212a-4890-92ca-f6dc3e2c8469",
        "_score": 1,
        "_source": {
          "first_run": "2021-10-11T04:17:29Z",
          "last_run": "2021-10-11T04:17:29Z",
          "node_uuid": "7541037b-212a-4890-92ca-f6dc3e2c8469"
        }
      },
      {
        "_index": "comp-1-run-info",
        "_type": "_doc",
        "_id": "38f3adc2-5ea8-4836-a343-43a6232d82a0",
        "_score": 1,
        "_source": {
          "first_run": "2021-10-14T04:22:33Z",
          "last_run": "2021-10-14T04:22:33Z",
          "node_uuid": "38f3adc2-5ea8-4836-a343-43a6232d82a0"
        }
      },
    .
    .
    .

✅ Checklist

All PRs must tick these:

• I have read the CONTRIBUTING document.
• All commits signed-off for the Developer Certification of Origin.

With occasional exceptions, all PRs from Progress employees must tick these:

• Is the code clear? (complicated code or lots of comments--subdivide and use well-named methods, meaningful variable names, etc.)
• Consistency checked? (user notifications, user prompts, visual patterns, code patterns, variable names)
• Repeated code blocks eliminated? (adapt and reuse existing components, blocks, functions, etc.)
• Spelling, grammar, typos checked? (at a minimum use make spell in any component directory)
• Code well-formatted? (indents, line breaks, etc. improve rather than hinder readability)

All PRs from Progress employees should tick these if appropriate:

• Tests added/updated? (all new code needs new tests)
• Docs added/updated? (all customer-facing changes)

Please add a note next to any checkbox above if you are NOT ticking it.

📷 Screenshots, if applicable

[Dmaddu]

This statement may fail if update is empty. And it is not guaranteed that the Do method always returns a non-empty response. In the error case, the method will return an empty response.

[rickmarry]

thanks.. I've taken out that update var.. I was only using it for debug before

[kalroy]

Can msg.Report or msg.Shared be nil?

[rickmarry]

msg itself can but not msg.Report or msg.Shared as they're not pointers. msg being nil is unlikely as as we just create an instance of the struct and start sending it thru the pipeline

[kalroy]

We should handle nil scenarios. Default can be the current time.

[rickmarry]

Good catch.. currently if we were to get an inspec report that has no end_time(I've never heard of this happening btw but that doesn't mean it has not), it would become an empty string upon ingestion which in turn becomes the following date in our reports: 0001-01-01T00:00:00Z.. if it were to actually happen, it it would wind up in an es index named with that date as part of the index name.. it would never show up in a subsequent search either. In a way, this may be good as it flags and consolidates into one index, node runs that misbehave in this way

I like the idea of handling such bad data but maybe we should do so by logging/rejecting the incoming report as it's not really valid without an end_time.

This would require us to put this error handling into the report ingestion pipeline as well as in this run info ingestion. Probably big enough to warrant it's own issue/pr

Should we do this now? or get this in as is, since it matches the exact end_time handling of report summary and detail ingestion.

[kalroy]

We can add issues to handle these corner cases

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

0.0% Coverage
7.1% Duplication

[kalroy]

We can add issues to handle these corner cases