[WIP] Update breaking changes doc for 7.0 #10632

karenzone · 2019-04-03T15:39:55Z

No description provided.

karenzone · 2019-04-03T15:40:36Z

Update structure to allow for previous and current changes

yaauie

I've added a proposal for the plugin breaking changes, and a note about how we should call out the upstream breaking changes to the payloads Beats sends that I hope adds more clarity.

docs/static/breaking-changes.asciidoc

yaauie · 2019-04-03T18:49:40Z

docs/static/breaking-changes.asciidoc

@@ -29,7 +30,30 @@ and have the persistent queue enabled, we strongly recommend that you drain or
 delete the persistent queue before you upgrade. See <<upgrading-logstash-pqs>>
 for information and instructions.

+


section should be above breaking-pq section to maintain newest-first ordering

docs/static/breaking-changes.asciidoc

yaauie · 2019-04-04T16:46:26Z

docs/static/breaking-changes.asciidoc

+
+Beats fields have been renamed to map to Elastic Common Schema (ECS).
+
+Starting with 7.0, the fields exported by {beats} conform to the


🤔

These breaking changes aren't in the Beats Input itself, but in the payloads that Beats sends. If a user upgrades Logstash before upgrading Beats, the payloads will continue to have the pre-ECS schema, and if they upgrade their Beats before upgrading Logstash they'll get payloads with the ECS schema in advance of any Logstash upgrade.

I would rather add this as a call-out than in bullet-points. See my proposed diff above for a WARNING call-out.

Adds info about field reference parser Co-Authored-By: karenzone <35154725+karenzone@users.noreply.github.com>

Co-Authored-By: karenzone <35154725+karenzone@users.noreply.github.com>

jsvd · 2019-04-04T20:56:34Z

docs/static/breaking-changes.asciidoc

+exported fields have been renamed, so you may need to modify your pipeline
+configurations to access them at their new locations prior to upgrading your
+Beats. See {beats-ref}/breaking-changes-7.0.html[Beats Breaking changes in 7.0]
+for the full list of changed names.


Let's add a note here that seeing a mapping conflict after upgrade is an indication that is change is influencing the data reaching existing indices.

jsvd · 2019-04-04T20:59:58Z

docs/static/breaking-changes.asciidoc

+These plugins were removed from the 7.0 default bundle based on usage data. 
+You can still install these plugins manually.
+
+* removed-plugin-1


only the graphite output stopped being supported but it's still in the package

Any notation required around that?
I'll remove this section entirely.

Would it be helpful to suggest that the user revisit the list of supported plugins as of 7.0? Or am I overthinking this. :-)

I think we can remove this section, commercial support is something only for our customers, which can be checked in our support matrix page

karenzone · 2019-04-04T21:10:08Z

@robbavey @yaauie @jsvd What about ILM on by default? Does that belong in the elasticsearch-output plugin? Or is that more inline with the Beats/ECS info (in that it should be more prominent)?

jsvd · 2019-04-04T21:45:36Z

docs/static/breaking-changes.asciidoc

+[float]
+===== Elasticsearch output
+
+ILM on by default


@yaauie can we add here that in 7.0 ILM will be auto detected and enabled by default if the ES cluster supports it?

Here's a peek at what's coming in my next commit:

* {es} {ref}/index-lifecycle-management.html[Index lifecycle management (ILM)] is auto-detected and enabled by default if your {es} cluster supports it.

Comments are welcomed, pre- or post-commit.

karenzone · 2019-04-04T23:02:21Z

docs/static/breaking-changes.asciidoc

+Please review the 
+https://www.elastic.co/support/matrix#matrix_logstash_plugins[Support Matrix for Logstash plugins]
+for the latest on supported plugins.
+


I can take out this link if y'all don't think it is needed or appropriate.

yes let's remove this reference to supported plugins please

karenzone · 2019-04-05T01:53:54Z

I was making changes while comments came in. If you don't see your comment implement, the omission was not intentional. Please let me know and I'll handle it.

docs/static/breaking-changes.asciidoc

jsvd · 2019-04-05T15:56:15Z

docs/static/breaking-changes.asciidoc

+warnings when encountering input that is ambiguous, and allowed an early opt-in
+of strict-mode parsing either by providing the command-line flag
+`--field-reference-parser STRICT` or by adding `config.field_reference.parser:
+STRICT` to `logstash.yml`.


can we maybe have an example here? not sure if it will be too verbose:

Before:

logstash-6.7.0 % echo "hello"| bin/logstash -e 'filter { mutate { replace => { "message" => "%{[[]]message]} you" } } }' [2019-04-05T16:52:18,691][WARN ][org.logstash.FieldReference] Detected ambiguous Field Reference `[[]]message]`, which we expanded to the path `[message]`; in a future release of Logstash, ambiguous Field References will not be expanded. { "message" => "hello you", "@version" => "1", "@timestamp" => 2019-04-05T15:52:18.546Z, "type" => "stdin", "host" => "overcraft.lan" }

After:

logstash-7.0.0 % echo "hello"| bin/logstash -e 'filter { mutate { replace => { "message" => "%{[[]]message]} you" } } }' [2019-04-05T16:48:09,135][FATAL][logstash.runner ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `[[]]message]` [2019-04-05T16:48:09,167][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

karenzone · 2019-04-05T22:28:11Z

Rendered here for your viewing/reviewing pleasure: https://preview-docs.firebaseapp.com/logstash-breaking/breaking-changes.html

lcawl · 2019-04-08T15:42:49Z

I fixed the merge conflict with #10654 and tagged all of the 7.0 items as "notable" for inclusion in the Installation and Upgrade Guide. If you only consider a subset of those items noteworthy, we can tag individual sections.

jsvd

added a minor comment about removing the commercial support link otherwise LGTM

Remove link

* Update breaking changes doc for 7.0 Update structure to allow for previous and current changes * Update docs/static/breaking-changes.asciidoc Adds info about field reference parser Co-Authored-By: karenzone <35154725+karenzone@users.noreply.github.com> * Populate content for plugin changes Co-Authored-By: karenzone <35154725+karenzone@users.noreply.github.com> * Fix asciidoc formatting * Incorporate review comments and new content * Minor change for clarity * add anchors for linking * Add anchor for ecs-beats * Incorporate review comments * [DOCS] Adds tagged region for notable breaking changes * Incorporate review comments Remove link

* Update breaking changes doc for 7.0 Update structure to allow for previous and current changes * Update docs/static/breaking-changes.asciidoc Adds info about field reference parser Co-Authored-By: karenzone <35154725+karenzone@users.noreply.github.com> * Populate content for plugin changes Co-Authored-By: karenzone <35154725+karenzone@users.noreply.github.com> * Fix asciidoc formatting * Incorporate review comments and new content * Minor change for clarity * add anchors for linking * Add anchor for ecs-beats * Incorporate review comments * [DOCS] Adds tagged region for notable breaking changes * Incorporate review comments Remove link Fixes #10666

karenzone added the docs label Apr 3, 2019

lcawl mentioned this pull request Apr 3, 2019

[DOCS] Supporting Upgrades from 6.x to 7.x elastic/stack-docs#250

Open

16 tasks

Update breaking changes doc for 7.0

6cd2fb5

Update structure to allow for previous and current changes

yaauie reviewed Apr 4, 2019

View reviewed changes

karenzone force-pushed the breaking-7.0 branch from 75afce0 to 6cd2fb5 Compare April 4, 2019 16:51

yaauie and others added 3 commits April 4, 2019 13:42

Update docs/static/breaking-changes.asciidoc

ec3f75e

Adds info about field reference parser Co-Authored-By: karenzone <35154725+karenzone@users.noreply.github.com>

Populate content for plugin changes

78d37bf

Co-Authored-By: karenzone <35154725+karenzone@users.noreply.github.com>

Fix asciidoc formatting

1bd73b3

jsvd reviewed Apr 4, 2019

View reviewed changes

Incorporate review comments and new content

124be0f

karenzone commented Apr 4, 2019

View reviewed changes

karenzone marked this pull request as ready for review April 5, 2019 12:23

karenzone added 2 commits April 5, 2019 11:02

Minor change for clarity

b2f0d32

add anchors for linking

e0ecc8e

karenzone requested review from jsvd and yaauie April 5, 2019 15:32

jsvd reviewed Apr 5, 2019

View reviewed changes

karenzone added 2 commits April 5, 2019 14:50

Add anchor for ecs-beats

bf3c60f

Incorporate review comments

5e2de91

lcawl added 2 commits April 8, 2019 08:38

[DOCS] Adds tagged region for notable breaking changes

5cd6131

Merge branch 'master' into breaking-7.0

ff14f95

jsvd approved these changes Apr 8, 2019

View reviewed changes

Incorporate review comments

fe41afe

Remove link

karenzone merged commit d96854a into elastic:master Apr 8, 2019

karenzone deleted the breaking-7.0 branch April 8, 2019 20:02

karenzone mentioned this pull request Apr 9, 2019

Update breaking changes doc for 7.0 (backport of #10632) #10666

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Update breaking changes doc for 7.0 #10632

[WIP] Update breaking changes doc for 7.0 #10632

karenzone commented Apr 3, 2019

karenzone commented Apr 3, 2019

yaauie left a comment

yaauie Apr 3, 2019

yaauie Apr 4, 2019

jsvd Apr 4, 2019

jsvd Apr 4, 2019

karenzone Apr 4, 2019 •

edited

jsvd Apr 4, 2019

karenzone commented Apr 4, 2019

jsvd Apr 4, 2019

karenzone Apr 4, 2019

karenzone Apr 4, 2019

jsvd Apr 8, 2019

karenzone commented Apr 5, 2019 •

edited

jsvd Apr 5, 2019

karenzone commented Apr 5, 2019

lcawl commented Apr 8, 2019

jsvd left a comment

		@@ -29,7 +30,30 @@ and have the persistent queue enabled, we strongly recommend that you drain or
		delete the persistent queue before you upgrade. See <<upgrading-logstash-pqs>>
		for information and instructions.


		Beats fields have been renamed to map to Elastic Common Schema (ECS).

		Starting with 7.0, the fields exported by {beats} conform to the

[WIP] Update breaking changes doc for 7.0 #10632

[WIP] Update breaking changes doc for 7.0 #10632

Conversation

karenzone commented Apr 3, 2019

karenzone commented Apr 3, 2019

yaauie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karenzone Apr 4, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karenzone commented Apr 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karenzone commented Apr 5, 2019 • edited

Choose a reason for hiding this comment

karenzone commented Apr 5, 2019

lcawl commented Apr 8, 2019

jsvd left a comment

Choose a reason for hiding this comment

karenzone Apr 4, 2019 •

edited

karenzone commented Apr 5, 2019 •

edited