[WIP] Update breaking changes doc for 7.0 (#10632)

* Update breaking changes doc for 7.0

Update structure to allow for previous and current changes

* Update docs/static/breaking-changes.asciidoc

Adds info about field reference parser

Co-Authored-By: karenzone <35154725+karenzone@users.noreply.github.com>

* Populate content for plugin changes

Co-Authored-By: karenzone <35154725+karenzone@users.noreply.github.com>

* Fix asciidoc formatting

* Incorporate review comments and new content

* Minor change for clarity

* add anchors for linking

* Add anchor for ecs-beats

* Incorporate review comments

* [DOCS] Adds tagged region for notable breaking changes

* Incorporate review comments

Remove link

docs/static/breaking-changes.asciidoc

@@ -1,32 +1,232 @@
 [[breaking-changes]]
 == Breaking Changes
 
-We strive to maintain backward compatibility between minor versions (6.x to 6.y,
+We strive to maintain backward compatibility between minor versions (7.x to 7.y,
 for example) so that you can upgrade without changing any configuration files.
-Breaking changes are usually introduced only between major versions (such as 5.x
-to 6.y). On occasion, we are forced to break compatibility within a given major release
+Breaking changes are usually introduced only between major versions (such as 6.x
+to 7.y). On occasion, we are forced to break compatibility within a given major release
 to ensure correctness of operation.
 
 This section covers the changes that you need to be aware of when migrating to
-Logstash 6.0.0 and later.
+Logstash 7.0.0 and later.
 
-NOTE: Migrating directly between non-consecutive major versions (1.x to
-6.x) is not recommended.
+NOTE: Migrating directly between non-consecutive major versions (5.x to
+7.x) is not recommended.
 
 See these topics for a description of breaking changes:
 
+* <<breaking-7.0>>
 * <<breaking-pq>>
 * <<breaking-6.0>>
 
 See also <<releasenotes>>. 
 
 //NOTE: The notable-breaking-changes tagged regions are re-used in the
 //Installation and Upgrade Guide
+
+[[breaking-7.0]]
+=== Breaking changes in 7.0
+
+coming[7.0.0]
+
+Here are the breaking changes for {ls} 7.0. 
+
 // tag::notable-breaking-changes[]
+[float]
+==== Changes in Logstash Core
 
-// end::notable-breaking-changes[]
+These changes can affect any instance of Logstash that uses impacted features.
+Changes to Logstash Core are plugin agnostic.
+
+[float]
+[[java-exec-default]]
+===== Java execution engine enabled by default
+
+The new Java execution engine is enabled by default. It features faster
+performance, reduced memory usage, and lower config startup and reload times.
+
+For more information, see the blog post about the
+https://www.elastic.co/blog/meet-the-new-logstash-java-execution-engine[initial
+release of the Java execution engine].  
+
+We went to considerable lengths to make this change seamless. Still, it's a big
+change. If you notice different behaviors that might be related, please 
+https://github.com/elastic/logstash/issues[open a GitHub issue] to let us
+know.
+
+[float]
+[[beats-ecs]]
+===== Beats conform to the Elastic Common Schema (ECS)
+
+As of 7.0, Beats fields conform to the {ecs-ref}/index.html[Elastic Common
+Schema (ECS)].
+
+If you upgrade Logstash before you upgrade Beats, the payloads continue to use
+the pre-ECS schema. If you upgrade your Beats before you upgrade Logstash, then
+you'll get payloads with the ECS schema in advance of any Logstash upgrade.
+
+If you see mapping conflicts after upgrade, that is an indication that the
+Beats/ECS change is influencing the data reaching existing indices. 
+
+See the *{beats} Platform Reference* for more information on
+{beats-ref}/upgrading-6-to-7.html#enable-ecs-compatibility[Beats and ECS].
+
+[float]
+[[field-ref-strict]]
+===== Field Reference parser is more strict
+
+The Field Reference parser, which is used to interpret references to fields in
+your pipelines and plugins, was made to be more strict and will now reject
+inputs that are either ambiguous or illegal. Since 6.4, Logstash has emitted
+warnings when encountering input that is ambiguous, and allowed an early opt-in
+of strict-mode parsing either by providing the command-line flag
+`--field-reference-parser STRICT` or by adding `config.field_reference.parser:
+STRICT` to `logstash.yml`.
+
+Here's an example. 
+
+*Before*
+
+[source,txt]
+-----
+logstash-6.7.0 % echo "hello"| bin/logstash -e 'filter { mutate { replace => { "message" => "%{[[]]message]} you" } } }'
+[2019-04-05T16:52:18,691][WARN ][org.logstash.FieldReference] Detected ambiguous Field Reference `[[]]message]`, which we expanded to the path `[message]`; in a future release of Logstash, ambiguous Field References will not be expanded.
+{
+       "message" => "hello you",
+      "@version" => "1",
+    "@timestamp" => 2019-04-05T15:52:18.546Z,
+          "type" => "stdin",
+          "host" => "overcraft.lan"
+}
+-----
+
+*After* 
+
+[source,txt]
+-----
+logstash-7.0.0 % echo "hello"| bin/logstash -e 'filter { mutate { replace => { "message" => "%{[[]]message]} you" } } }'
+[2019-04-05T16:48:09,135][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `[[]]message]`
+[2019-04-05T16:48:09,167][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
+-----
+
+
+[float]
+==== Changes in Logstash Plugins
+
+With 7.0.0, we have taken the opportunity to upgrade a number of bundled plugins
+to their newest major version, absorbing their breaking changes into the
+Logstash distribution.
+
+While these upgrades included new features and important fixes, only the
+breaking changes are called out below.
+
+NOTE: The majority of the changes to plugins are the removal of previously-deprecated
+and now-obsolete options. Please ensure that your pipeline
+configurations do not use these removed options before upgrading.
+
+[float]
+===== Codec Plugins
+
+Here are the breaking changes for codec plugins.
+
+*CEF Codec*
+
+* Removed obsolete `sev` option
+* Removed obsolete `deprecated_v1_fields` option
+
+*Netflow Codec*
+
+* Changed decoding of application_id to implement RFC6759; the format changes from a pair of colon-separated ids (e.g. `0:40567`) to a variable number of double-dot-separated ids (e.g. `0..12356..40567`).
 
 [float]
+===== Filter Plugins
+
+Here are the breaking changes for filter plugins.
+
+*Clone Filter*
+
+* Make `clones` a required option
+
+*Geoip Filter*
+
+* Removed obsolete `lru_cache_size` option
+
+*HTTP Filter*
+
+* Removed obsolete `ssl_certificate_verify` option
+
+[float]
+===== Input Plugins
+
+Here are the breaking changes for  input plugins.
+
+*Beats Input*
+
+* Removed obsolete `congestion_threshold` option
+* Removed obsolete `target_field_for_codec` option
+* Changed default value of `add_hostname` to false
+
+NOTE: In Beats 7.0.0, the fields exported by Beats _to_ the Logstash Beats Input
+conform to the {ecs-ref}/index.html[Elastic Common Schema (ECS)]. Many of the
+exported fields have been renamed, so you may need to modify your pipeline
+configurations to access them at their new locations prior to upgrading your
+Beats. See {beats-ref}/breaking-changes-7.0.html[Beats Breaking changes in 7.0]
+for the full list of changed names.
+
+*HTTP Input*
+
+* Removed obsolete `ssl_certificate_verify` option
+
+*HTTP Poller Input*
+
+* Removed obsolete `interval` option
+* Removed obsolete `ssl_certificate_verify` option
+
+*Tcp Input*
+
+* Removed obsolete `data_timeout` option
+* Removed obsolete `ssl_cacert` option
+
+[float]
+===== Output Plugins
+
+Here are the breaking changes for output plugins.
+
+*Elasticsearch Output*
+
+* {es} {ref}/index-lifecycle-management.html[Index lifecycle management (ILM)] is
+auto-detected and enabled by default if your {es} cluster supports it.
+* Remove support for parent/child (still support join data type) since we don't
+support multiple document types any more
+* Removed obsolete `flush_size` option
+* Removed obsolete `idle_flush_time` option
+
+*HTTP Output*
+
+* Removed obsolete `ssl_certificate_verify` option
+
+*Kafka Output*
+
+* Removed obsolete `block_on_buffer_full` option
+* Removed obsolete `ssl` option
+* Removed obsolete `timeout_ms` option
+
+*Redis Output*
+
+* Removed obsolete `queue` option
+* Removed obsolete `name` option
+
+*Sqs Output*
+
+* Removed obsolete `batch` option
+* Removed obsolete `batch_timeout` option
+
+*Tcp Output*
+
+* Removed obsolete `message_format` option
+
+// end::notable-breaking-changes[]
+
 [[breaking-pq]]
 === Breaking change across PQ versions prior to Logstash 6.3.0
 
@@ -35,7 +235,6 @@ and have the persistent queue enabled, we strongly recommend that you drain or
 delete the persistent queue before you upgrade. See <<upgrading-logstash-pqs>>
 for information and instructions.
 
-[float]
 [[breaking-6.0]]
 === Breaking changes in 6.0
 
@@ -44,7 +243,8 @@ Here are the breaking changes for 6.0.
 [float]
 ==== Changes in Logstash Core
 
-These changes can impact any instance of Logstash and are plugin agnostic, but only if you are using the features that are impacted.
+These changes can affect any instance of Logstash that uses impacted features.
+Changes to Logstash Core are plugin agnostic.
 
 [float]
 ===== Application Settings
@@ -68,7 +268,7 @@ These changes can impact any instance of Logstash and are plugin agnostic, but o
 [float]
 ===== List of plugins bundled with Logstash
 
-The following plugins were removed from the 5.0 default bundle based on usage data. You can still install these plugins manually:
+The following plugins were removed from the default bundle based on usage data. You can still install these plugins manually:
 
 * logstash-codec-oldlogstashjson
 * logstash-input-couchdb_changes