Intake v2 support

[roncohen]

This is an issue to track intake v2 support in the APM Server

Blockers:

• Refactor processors and move remaining models into "models". #1137 Refactor processors
• Separate span #1141 Separate span
• Refactor Payloads as a preparation for Intake V2 #1239 Refactor payloads
• Implement Intake V2 basic functionality #1238, Implementation of elementary v2 functionality, including:
  • v2 endpoints
  • processing of ndjson objects
  • allowing spans to be sent separatly
  • add timestamp and transaction.id to spans
  • allow for same compression methods as v1
  • allow for transactions, spans, errors, metadata
• v2Handler: use 'application/json' content-type for error responses #1353 Set the right content type
• Create a summary response for Intake V2 #1236 Create a summary response for Intake V2
• Move non-web related processing into processor, add "publish" package #1324 Move non-web related processing into processor, add "publish" package
• Add server-level integration tests to test endpoints
• Add integration tests for various errors
• Handling errors in Intake V2 #1240 Handling errors in Intake V2
• How to handle partial transactions in intake v2? #1241 How to handle partial transactions
• How to handle spans starting after transaction is already finished.  #1242 How to handle spans starting after transaction is already finished.
• intake v2 agent simulation utility #1254 intake v2 client utility
• Update json spec for v2 Intake related to distributed tracing #1276 Update json spec for V2
• [v2] adapt application metrics #1355 adapt application metrics
• Design API endpoints for v2 #1336 design intake v2 endpoints
• [v2] Rename "metric" entity to "metricset" #1356 rename 'metric' to 'metricset'
• [v2] Add benchmark tests #1376 Add benchmark tests
• Require and change description for error.id #1384 Require error.id and change format.
• Intake v2: Finalize processor/package_tests style tests #1288 Finalize processor/package_tests style tests
• v2: Investigate "full queue" behavior #1298 Investigate "full queue" behavior
• v2: Investigate which limits/configs are useful for intake v2 #1299 Investigate which limits/configs are useful for intake v2
• Ensure common problem section is valid for both versions or specifies the version.
• v2: rename the remaining 'metric' to 'metricset' #1409 follow up on [v2] Rename "metric" entity to "metricset" #1356, rename code symbols from "metrics" to "metricset"
• v2: Require either span.start or span.timestamp #1340 (comment) v2: Require either span.start or span.timestamp

Non-blockers:

• Remove null values from Json Spec #1410 Deny null values in JSON spec - Postponed to 7.0
• [v2] Update Instrumentation #1385 Update instrumentation
• Intake v2: investigate parallelizing decode and validate #1285 Investigate parallelizing decode and validate
• Add Intake v2 related documentation #1329 Add proper documentation around v2 to APM Server docs and clarify which errors could happen in which Intake API version.
• [v2] accept unix epoch timestamps in microseconds #1413 (review) make sure model/transaction/event_test.go has a TestEventTransform that covers v2 events

[felixbarny]

To see all changes in the intake API, open https://github.com/elastic/apm-server/compare/v2 and search for docs/spec/.

[roncohen]

all blockers are resolved. Closing