[7.x] Add logging.ecs and update beats framework to 553c92eeff2d

[simitt]

Motivation/summary

Updates to latest beats changes in 7.x branch, major changes that are pulled in:

• change how apm-server.yml file is built
  This PR only adds minimal required changes to be compatible with newly introduced way of building yml configuration files. Since libbeat now has a dedicated template file per section, we could change our process and start pulling in templates from libbeat again instead of manually syncing them, but this is out of scope for this update.
• introduce logging.ecs option to enable logging with minimal fieldset for ECS

Checklist

- [ ] I have signed the Contributor License Agreement.

• My code follows the style guidelines of this project (run make check-full for static code checks and linting)
• I have rebased my changes on top of the latest 7.x branch

• I have made corresponding changes to the documentation
  - [ ] I have added tests that prove my fix is effective or that my feature works (tested in libbeat)
• New and existing unit tests pass locally with my changes

- [ ] I have updated CHANGELOG.asciidoc (part of beats changelog)

How to test these changes

Start APM Server with logging.json=true and logging.ecs=true and see

• @timestamp, log.level, message and ecs.version added to every log line.
• If EnableCaller is set log.origin.file and log.origin.line are added.
• error.message is logged when error is given

this can be tested for

• logging.to_files: true
• logging.to_syslog: true
• logging.to_stderr: true
• log output to console ./beat -e

Related issues

#3155

[graphaelli]

oh come on ;)

[apmmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request [7.x] Add logging.ecs and update beats framework to 553c92eeff2d  #3749 updated]

• Start Time: 2020-05-06T07:04:25.356+0000

• Duration: 57 min 37 sec (3396675)

• Commit: 69ff52b

Test stats 🧪

Test	Results
Failed	0
Passed	3008
Skipped	142
Total	3150

Steps errors

Expand to view the steps failures

• Name: Test Sync

  • Description: ./script/jenkins/sync.sh

  • Result: FAILURE

  • Duration: 3 min 40 sec<

  • Start Time: 2020-05-06T07:14:47.132+0000

[graphaelli]

some suggestions on wording

[axw]

With logging.ecs: false:

{"level":"info","timestamp":"2020-05-14T06:01:03.400Z","logger":"request","caller":"middleware/log_middleware.go:97","message":"not modified","request_id":"99b0ca4b-bcb7-4de5-a897-57d6e5949e90","method":"GET","URL":"/config/v1/agents?service.name=apm-server","content_length":0,"remote_address":"pipe","user-agent":"elasticapm-go/1.7.2 go/go1.13.9","response_code":304}

With logging.ecs: true:

{"log.level":"info","@timestamp":"2020-05-14T06:02:25.710Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":97},"message":"request ok","request_id":"bba74925-9625-4205-95bd-434cdcc16ea1","method":"GET","URL":"/","content_length":0,"remote_address":"127.0.0.1","user-agent":"curl/7.29.0","response_code":200,"ecs.version":"1.5.0"}

The ECS fields look good to me. We might want to update the field names we add in apm-server to be ECS-compliant, but that's out of scope here. I've opened an enhancement for that: #3796