Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Update breaking changes doc for 7.0 #10632

Merged
merged 12 commits into from
Apr 8, 2019
163 changes: 156 additions & 7 deletions docs/static/breaking-changes.asciidoc
Original file line number Diff line number Diff line change
@@ -1,21 +1,22 @@
[[breaking-changes]]
== Breaking Changes

We strive to maintain backward compatibility between minor versions (6.x to 6.y,
We strive to maintain backward compatibility between minor versions (7.x to 7.y,
for example) so that you can upgrade without changing any configuration files.
Breaking changes are usually introduced only between major versions (such as 5.x
to 6.y). On occasion, we are forced to break compatibility within a given major release
Breaking changes are usually introduced only between major versions (such as 6.x
to 7.y). On occasion, we are forced to break compatibility within a given major release
to ensure correctness of operation.

This section covers the changes that you need to be aware of when migrating to
Logstash 6.0.0 and later.
Logstash 7.0.0 and later.

NOTE: Migrating directly between non-consecutive major versions (1.x to
6.x) is not recommended.
NOTE: Migrating directly between non-consecutive major versions (5.x to
7.x) is not recommended.

See these topics for a description of breaking changes:

* <<breaking-pq>>
* <<breaking-7.0>>
yaauie marked this conversation as resolved.
Show resolved Hide resolved
* <<breaking-6.0>>

See also <<releasenotes>>.
Expand All @@ -29,7 +30,155 @@ and have the persistent queue enabled, we strongly recommend that you drain or
delete the persistent queue before you upgrade. See <<upgrading-logstash-pqs>>
for information and instructions.


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

section should be above breaking-pq section to maintain newest-first ordering

[[breaking-7.0]]
=== Breaking changes in 7.0

coming[7.0.0]

Here are the breaking changes for 7.0.

[float]
==== Changes in Logstash Core

These changes can impact any instance of Logstash that uses impacted features.
Changes to Logstash Core are plugin agnostic.


[float]
==== Changes in Logstash Plugins

With 7.0.0, we have taken the opportunity to upgrade a number of bundled plugins to their newest major version, absorbing their breaking changes into the Logstash distribution.

While these upgrades included new features and important fixes, only the breaking changes are called out below.

NOTE: The majority of the changes to plugins are the removal of previously-deprecated and now-obsolete options; please be careful to ensure that your pipeline configurations do not use these removed options before upgrading.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can take out this link if y'all don't think it is needed or appropriate.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes let's remove this reference to supported plugins please

[float]
===== Codec Plugins

*CEF Codec*

* Removed obsolete `sev` option
* Removed obsolete `deprecated_v1_fields` option

*Netflow Codec*

* Changed decoding of application_id to implement RFC6759; the format changes from a pair of colon-separated ids (e.g. `0:40567`) to a variable number of double-dot-separated ids (e.g. `0..12356..40567`).

[float]
===== Filter Plugins

*Clone Filter*

* Make `clones` a required option

*Geoip Filter*

* Removed obsolete `lru_cache_size` option

*HTTP Filter*

* Removed obsolete `ssl_certificate_verify` option

[float]
===== Input Plugins

*Beats Input*

* Removed obsolete `congestion_threshold` option
* Removed obsolete `target_field_for_codec` option
* Changed default value of `add_hostname` to false

NOTE: In Beats 7.0.0, the fields exported by Beats _to_ the Logstash Beats Input
conform to the {ecs-ref}/index.html[Elastic Common Schema (ECS)]. Many of the
exported fields have been renamed, so you may need to modify your pipeline
configurations to access them at their new locations prior to upgrading your
Beats. See {beats-ref}/breaking-changes-7.0.html[Beats Breaking changes in 7.0]
for the full list of changed names.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a note here that seeing a mapping conflict after upgrade is an indication that is change is influencing the data reaching existing indices.


*HTTP Input*

* Removed obsolete `ssl_certificate_verify` option

*HTTP Poller Input*

* Removed obsolete `interval` option
* Removed obsolete `ssl_certificate_verify` option

*Tcp Input*

* Removed obsolete `data_timeout` option
* Removed obsolete `ssl_cacert` option

[float]
===== Output Plugins

*Elasticsearch Output*

* Remove support for parent/child (still support join data type) since we don't support multiple document types any more
* Removed obsolete `flush_size` option
* Removed obsolete `idle_flush_time` option

*HTTP Output*

* Removed obsolete `ssl_certificate_verify` option

*Kafka Output*

* Removed obsolete `block_on_buffer_full` option
* Removed obsolete `ssl` option
* Removed obsolete `timeout_ms` option

*Redis Output*

* Removed obsolete `queue` option
* Removed obsolete `name` option

*Sqs Output*

* Removed obsolete `batch` option
* Removed obsolete `batch_timeout` option

*Tcp Output*

* Removed obsolete `message_format` option

[float]
===== Beats input

Beats fields have been renamed to map to Elastic Common Schema (ECS).

Starting with 7.0, the fields exported by {beats} conform to the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔

These breaking changes aren't in the Beats Input itself, but in the payloads that Beats sends. If a user upgrades Logstash before upgrading Beats, the payloads will continue to have the pre-ECS schema, and if they upgrade their Beats before upgrading Logstash they'll get payloads with the ECS schema in advance of any Logstash upgrade.

I would rather add this as a call-out than in bullet-points. See my proposed diff above for a WARNING call-out.

{ecs-ref}/index.html[Elastic Common Schema (ECS)]. Many of the exported fields
have been renamed. See {beats-ref}/breaking-changes-7.0.html[Beats breaking
changes in 7.0] for the full list of changed names.

See the *{beats} Platform Reference* for more information on
{beats-ref}/upgrading-6-to-7.html#enable-ecs-compatibility[Beats and ECS].

TODO: What action does user need to take?

[float]
===== Elasticsearch output

ILM on by default
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yaauie can we add here that in 7.0 ILM will be auto detected and enabled by default if the ES cluster supports it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's a peek at what's coming in my next commit:

* {es} {ref}/index-lifecycle-management.html[Index lifecycle management (ILM)] is
auto-detected and enabled by default if your {es} cluster supports it.

Comments are welcomed, pre- or post-commit.


TODO: What action does user need to take?

karenzone marked this conversation as resolved.
Show resolved Hide resolved
- The Field Reference parser, which is used to interpret references to fields in your pipelines and plugins, was made to be more strict and will now reject inputs that are either ambiguous or illegal; since 6.4, Logstash has emitted warnings when encountering input that is ambiguous, and allowed an early opt-in of strict-mode parsing either by providing the command-line flag `--field-reference-parser STRICT` or by adding `config.field_reference.parser: STRICT` to `logstash.yml`.


[float]
===== List of plugins bundled with Logstash

These plugins were removed from the 7.0 default bundle based on usage data.
You can still install these plugins manually.

* removed-plugin-1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only the graphite output stopped being supported but it's still in the package

Copy link
Contributor Author

@karenzone karenzone Apr 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any notation required around that?
I'll remove this section entirely.

Would it be helpful to suggest that the user revisit the list of supported plugins as of 7.0? Or am I overthinking this. :-)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can remove this section, commercial support is something only for our customers, which can be checked in our support matrix page

* removed-plugin-2


[[breaking-6.0]]
=== Breaking changes in 6.0

Expand Down Expand Up @@ -62,7 +211,7 @@ These changes can impact any instance of Logstash and are plugin agnostic, but o
[float]
===== List of plugins bundled with Logstash

The following plugins were removed from the 5.0 default bundle based on usage data. You can still install these plugins manually:
The following plugins were removed from the 6.0 default bundle based on usage data. You can still install these plugins manually:

* logstash-codec-oldlogstashjson
* logstash-input-couchdb_changes
Expand Down