Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Update breaking changes doc for 7.0 #10632

Merged
merged 12 commits into from
Apr 8, 2019
Merged

[WIP] Update breaking changes doc for 7.0 #10632

Changes from 5 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
6cd2fb5
Update breaking changes doc for 7.0
karenzone Apr 3, 2019
ec3f75e
Update docs/static/breaking-changes.asciidoc
yaauie Apr 4, 2019
78d37bf
Populate content for plugin changes
yaauie Apr 4, 2019
1bd73b3
Fix asciidoc formatting
karenzone Apr 4, 2019
124be0f
Incorporate review comments and new content
karenzone Apr 4, 2019
b2f0d32
Minor change for clarity
karenzone Apr 5, 2019
e0ecc8e
add anchors for linking
karenzone Apr 5, 2019
bf3c60f
Add anchor for ecs-beats
karenzone Apr 5, 2019
5e2de91
Incorporate review comments
karenzone Apr 5, 2019
5cd6131
[DOCS] Adds tagged region for notable breaking changes
lcawl Apr 8, 2019
ff14f95
Merge branch 'master' into breaking-7.0
lcawl Apr 8, 2019
fe41afe
Incorporate review comments
karenzone Apr 8, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
193 changes: 184 additions & 9 deletions docs/static/breaking-changes.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,200 @@
[[breaking-changes]]
== Breaking Changes

We strive to maintain backward compatibility between minor versions (6.x to 6.y,
We strive to maintain backward compatibility between minor versions (7.x to 7.y,
for example) so that you can upgrade without changing any configuration files.
Breaking changes are usually introduced only between major versions (such as 5.x
to 6.y). On occasion, we are forced to break compatibility within a given major release
Breaking changes are usually introduced only between major versions (such as 6.x
to 7.y). On occasion, we are forced to break compatibility within a given major release
to ensure correctness of operation.

This section covers the changes that you need to be aware of when migrating to
Logstash 6.0.0 and later.
Logstash 7.0.0 and later.

NOTE: Migrating directly between non-consecutive major versions (1.x to
6.x) is not recommended.
NOTE: Migrating directly between non-consecutive major versions (5.x to
7.x) is not recommended.

See these topics for a description of breaking changes:

* <<breaking-7.0>>
yaauie marked this conversation as resolved.
Show resolved Hide resolved
* <<breaking-pq>>
* <<breaking-6.0>>

See also <<releasenotes>>.

[[breaking-7.0]]
=== Breaking changes in 7.0

coming[7.0.0]

Here are the breaking changes for {ls} 7.0.

[float]
==== Changes in Logstash Core

These changes can affect any instance of Logstash that uses impacted features.
Changes to Logstash Core are plugin agnostic.

[float]
===== Field Reference parser is more strict

The Field Reference parser, which is used to interpret references to fields in
your pipelines and plugins, was made to be more strict and will now reject
inputs that are either ambiguous or illegal. Since 6.4, Logstash has emitted
warnings when encountering input that is ambiguous, and allowed an early opt-in
of strict-mode parsing either by providing the command-line flag
`--field-reference-parser STRICT` or by adding `config.field_reference.parser:
STRICT` to `logstash.yml`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we maybe have an example here? not sure if it will be too verbose:

Before:

logstash-6.7.0 % echo "hello"| bin/logstash -e 'filter { mutate { replace => { "message" => "%{[[]]message]} you" } } }'
[2019-04-05T16:52:18,691][WARN ][org.logstash.FieldReference] Detected ambiguous Field Reference `[[]]message]`, which we expanded to the path `[message]`; in a future release of Logstash, ambiguous Field References will not be expanded.
{
       "message" => "hello you",
      "@version" => "1",
    "@timestamp" => 2019-04-05T15:52:18.546Z,
          "type" => "stdin",
          "host" => "overcraft.lan"
}

After:

logstash-7.0.0 % echo "hello"| bin/logstash -e 'filter { mutate { replace => { "message" => "%{[[]]message]} you" } } }'
[2019-04-05T16:48:09,135][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `[[]]message]`
[2019-04-05T16:48:09,167][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit


[float]
===== Java execution engine enabled by default

The new Java execution engine is enabled by default. It features faster
performance, reduced memory usage, and lower config startup and reload times.

For more information, see the blog post about the
https://www.elastic.co/blog/meet-the-new-logstash-java-execution-engine[initial
release of the Java execution engine].

We went to considerable lengths to make this change seamless. Still, it's a big
change. If you notice different behaviors that might be related, please let us
know.
karenzone marked this conversation as resolved.
Show resolved Hide resolved

[float]
===== Beats conform to the Elastic Common Schema (ECS)

As of 7.0, Beats fields conform to the {ecs-ref}/index.html[Elastic Common
Schema (ECS)].

If you upgrade Logstash before you upgrade Beats, the payloads continue to use
the pre-ECS schema. If you upgrade your Beats before you upgrade Logstash, then
you'll get payloads with the ECS schema in advance of any Logstash upgrade.

If you see mapping conflicts after upgrade, that is an indication that the
Beats/ECS change is influencing the data reaching existing indices.

See the *{beats} Platform Reference* for more information on
{beats-ref}/upgrading-6-to-7.html#enable-ecs-compatibility[Beats and ECS].

[float]
==== Changes in Logstash Plugins

With 7.0.0, we have taken the opportunity to upgrade a number of bundled plugins
to their newest major version, absorbing their breaking changes into the
Logstash distribution.

While these upgrades included new features and important fixes, only the
breaking changes are called out below.

NOTE: The majority of the changes to plugins are the removal of previously-deprecated
and now-obsolete options. Please ensure that your pipeline
configurations do not use these removed options before upgrading.

Please review the
https://www.elastic.co/support/matrix#matrix_logstash_plugins[Support Matrix for Logstash plugins]
for the latest on supported plugins.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can take out this link if y'all don't think it is needed or appropriate.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes let's remove this reference to supported plugins please

[float]
===== Codec Plugins

Here are the breaking changes for codec plugins.

*CEF Codec*

* Removed obsolete `sev` option
* Removed obsolete `deprecated_v1_fields` option

*Netflow Codec*

* Changed decoding of application_id to implement RFC6759; the format changes from a pair of colon-separated ids (e.g. `0:40567`) to a variable number of double-dot-separated ids (e.g. `0..12356..40567`).

[float]
===== Filter Plugins

Here are the breaking changes for filter plugins.

*Clone Filter*

* Make `clones` a required option

*Geoip Filter*

* Removed obsolete `lru_cache_size` option

*HTTP Filter*

* Removed obsolete `ssl_certificate_verify` option

[float]
===== Input Plugins

Here are the breaking changes for input plugins.

*Beats Input*

* Removed obsolete `congestion_threshold` option
* Removed obsolete `target_field_for_codec` option
* Changed default value of `add_hostname` to false

NOTE: In Beats 7.0.0, the fields exported by Beats _to_ the Logstash Beats Input
conform to the {ecs-ref}/index.html[Elastic Common Schema (ECS)]. Many of the
exported fields have been renamed, so you may need to modify your pipeline
configurations to access them at their new locations prior to upgrading your
Beats. See {beats-ref}/breaking-changes-7.0.html[Beats Breaking changes in 7.0]
for the full list of changed names.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a note here that seeing a mapping conflict after upgrade is an indication that is change is influencing the data reaching existing indices.


*HTTP Input*

karenzone marked this conversation as resolved.
Show resolved Hide resolved
* Removed obsolete `ssl_certificate_verify` option

*HTTP Poller Input*

* Removed obsolete `interval` option
* Removed obsolete `ssl_certificate_verify` option

*Tcp Input*

* Removed obsolete `data_timeout` option
* Removed obsolete `ssl_cacert` option

[float]
===== Output Plugins

Here are the breaking changes for output plugins.

*Elasticsearch Output*

* {es} {ref}/index-lifecycle-management.html[Index lifecycle management (ILM)] is
auto-detected and enabled by default if your {es} cluster supports it.
* Remove support for parent/child (still support join data type) since we don't
support multiple document types any more
* Removed obsolete `flush_size` option
* Removed obsolete `idle_flush_time` option

*HTTP Output*

* Removed obsolete `ssl_certificate_verify` option

*Kafka Output*

* Removed obsolete `block_on_buffer_full` option
* Removed obsolete `ssl` option
* Removed obsolete `timeout_ms` option

*Redis Output*

* Removed obsolete `queue` option
* Removed obsolete `name` option

*Sqs Output*

* Removed obsolete `batch` option
* Removed obsolete `batch_timeout` option

*Tcp Output*

* Removed obsolete `message_format` option


[[breaking-pq]]
=== Breaking change across PQ versions prior to Logstash 6.3.0

Expand All @@ -29,7 +203,7 @@ and have the persistent queue enabled, we strongly recommend that you drain or
delete the persistent queue before you upgrade. See <<upgrading-logstash-pqs>>
for information and instructions.

[float]

[[breaking-6.0]]
=== Breaking changes in 6.0

Expand All @@ -38,7 +212,8 @@ Here are the breaking changes for 6.0.
[float]
==== Changes in Logstash Core

These changes can impact any instance of Logstash and are plugin agnostic, but only if you are using the features that are impacted.
These changes can affect any instance of Logstash that uses impacted features.
Changes to Logstash Core are plugin agnostic.

[float]
===== Application Settings
Expand All @@ -62,7 +237,7 @@ These changes can impact any instance of Logstash and are plugin agnostic, but o
[float]
===== List of plugins bundled with Logstash

The following plugins were removed from the 5.0 default bundle based on usage data. You can still install these plugins manually:
The following plugins were removed from the 6.0 default bundle based on usage data. You can still install these plugins manually:

* logstash-codec-oldlogstashjson
* logstash-input-couchdb_changes
Expand Down