Add upgrade docs #11378
Add upgrade docs #11378
Conversation
There was a problem hiding this comment.
Leaving a few notes on other potential migration pitfalls we should document:
- Users using Logstash and having just 1 index or template. Going to 7.0 things will stop ingesting because of field conflict. Users need to use versioned templates and indices.
- Dashboard upgrade: If users want to run 6.7 and 7.0 metricbeat in parallel, migration flag has to be used and old dashboards have to stay around.
Overwriting index pattern: Users upgrading all Beats to 7.0 should / must overwrite the Kibana index patter.
@ruflin Edited this list so I can mark off the items I feel we've covered or that we've decided not to cover. Please confirm by reviewing the changes I've pushed for this PR.
|
@karenzone FYI |
|
There are some upgrade-related comments in this issue that need to be addressed: #11276 |
|
@dedemorton For the field changes we have a generated list in https://www.elastic.co/guide/en/beats/libbeat/master/breaking-changes-7.0.html Instead of just having 1 list, we potentially need it per Beat or a different place. We could also do a different format. Let me know what is the best way to document these and make it visible also in the upgrade docs. |
|
Noting here that there is a related discussion about upgrading dashboards in #11269. |
Let's start out with the list where it's stored currently, and I can point to it from the release notes. I think we'll get a lot of feedback next week from the group that's testing the upgrade docs. Then we can focus on improving the structure and flow based on specific problems that testers encounter. |
I feel a little dense for asking, but when you say that users must overwrite the index pattern, do you mean that they must set |
|
Leaving a few more notes here after helping some upgrades:
Minor edit to make these items a checklist for easier tracking ^^ [DM] |
|
@karenzone Not Nicolas' comment above:
|
|
Some follow-up questions:
Can you be more specific so I can add this to the doc?
By wiping the old data, do you mean deleting the index, or is there a less dramatic way to do this? |
|
Some more notes from an other issue:
|
|
@dedemorton I was thinking about the index pattern in Kibana. But it turned out I don't think we need it. For the migration steps with other outputs:
For the deletion: Yes, the index has to be deleted with the incorrect mapping. I don't think there is an other way :-( |
If users are ingesting Beats data via Logstash, they also need to ensure that the resulting index name (defined by the Logstash configuration) will match the index pattern in the Beats index template. So for example, if the Beats index template expects indices to match |
|
@ruflin A few more follow up questions for your solution-docs list:
Do users on Metricbeat versions < 6.6 need to set up the aliases manually? If so, that's not a very good experience for the type of user we are targeting. We should provide a migration script. Is that something we can get in place for 7.0?
Can you identify the fields and breaking changes that only affect solutions? For the list of fields, we can probably take the existing list and add some conditional logic to create something specific for solutions. Not sure how to best do the breaking changes. IMO, we can't expect folks using our curated UIs to spend time digging through the breaking changes in the release notes. We need to make the upgrade as simple and clear as possible (for all users, but especially folks using our curated UIs). |
libbeat/docs/upgrading.asciidoc
Outdated
| configuration options is renamed to `<beatname>.reference.yml` starting with | ||
| Beats 6.0. We recommend using this file as a reference only. It's not intended | ||
| to be used in production. | ||
| Before upgrading to 7.0, make sure {ls} uses versioned templates and indices. |
There was a problem hiding this comment.
I know it's implied, but maybe it's worth explicitly stating, "If you're using Logstash to ingest your Beats data..." so that people won't think LS is required for upgrading.
There was a problem hiding this comment.
Also, what does "versioned templates" mean in this case? The link below goes to plugins-inputs-beats.html which currently only mentions versioned indices.
There was a problem hiding this comment.
I'm modifying this section a bit to show the settings (including manage_template => false, which let's Beats manage the template).
libbeat/docs/upgrading.asciidoc
Outdated
|
|
||
| We recommend that you fully upgrade {es} and {kib} to version 7.0 | ||
| before upgrading {beats}. If you're on Beats 6.0 through 6.6, | ||
| <<upgrading-to-6.7,upgrade to Beats 6.7>> *before* doing the {es} upgrade. |
There was a problem hiding this comment.
Really minor, but when I was reading the generated doc, I saw this link and immediately thought to open it in a new tab because it looks like a link to a related resource. However, it's a link to the anchor tag a few lines down. Does it help the user that much or do we think that they'll find the content naturally and intuitively without the link? It might be worth removing it if so. Just a thought.
There was a problem hiding this comment.
yah that's a valid point. I'll remove the link.
libbeat/docs/upgrading.asciidoc
Outdated
| configuration file. | ||
| // REVIEWERS: I'm not sure the following statement is correct. I was not able | ||
| // to use the dashboards I saved from 6.x with 7.0. I did a very quick and dirty | ||
| // upgrade of the stack tho so might have missed something critical. |
There was a problem hiding this comment.
@ruflin Any gotchas/limitations we want to mention here?
dedemorton
added
the
needs_backport
libbeat/docs/upgrading.asciidoc
Outdated
|
|
||
| [source,shell] | ||
| ------------------------------------------------------------------------------ | ||
| ---- | ||
| curl -XPUT -H'Content-Type: application/json' http://localhost:9200/_template/metricbeat -d @metricbeat.template.json |
There was a problem hiding this comment.
We should not recommend this as users must put in the correct version for the template. Already this example is wihtout the version so we rather remove it completely and force users to use the beat directly to load it which will to the right thing.
There was a problem hiding this comment.
If we're going to drop this, it might be worth adding a note that it's no longer recommended:
Note: In previous versions, we advised users to create the Metricbeat index template from the command line. This is no longer recommended. Use the
metricbeatcommand outlined above.
There was a problem hiding this comment.
@ruflin Do we want to remove this section from the getting started, too? https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-template.html#load-template-manually-alternate
There was a problem hiding this comment.
At least there we have version numbers but I would rather recommend loading it with running the beat TBH.
libbeat/docs/upgrading.asciidoc
Outdated
|
|
||
| [source,shell] | ||
| ------------------------------------------------------------------------------ | ||
| ---- | ||
| curl -XPUT -H'Content-Type: application/json' http://localhost:9200/_template/metricbeat -d @metricbeat.template.json |
There was a problem hiding this comment.
If we're going to drop this, it might be worth adding a note that it's no longer recommended:
Note: In previous versions, we advised users to create the Metricbeat index template from the command line. This is no longer recommended. Use the
metricbeatcommand outlined above.
There was a problem hiding this comment.
I've noted a few small things and one bigger worry about Kibana index templates.
As discussed with @dedemorton, we can proceed to merge & backport, regardless of my review comments.
We need to publish a preliminary version of this page today, for people to be able to start testing.
We'll keep iterating in another PR.
| [source,json] | ||
| ---- | ||
| manage_template => false | ||
| index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" |
There was a problem hiding this comment.
Will @metadata still be populated in 7.0? This was a Logstash-only concept, correct?
There was a problem hiding this comment.
No, I believe the metadata is just part of the event...not just a LS concept.
* Add upgrade docs * Apply suggestions from code review * Add more detail based on review feedback * A few clarifications * Remove old file * Rename file to original name * More changes from review * Add changes from review before moving stuff around * Move some content and clean it up * Add more fixes from the review * Add more changes based on review comments
* Add upgrade docs * Apply suggestions from code review * Add more detail based on review feedback * A few clarifications * Remove old file * Rename file to original name * More changes from review * Add changes from review before moving stuff around * Move some content and clean it up * Add more fixes from the review * Add more changes based on review comments
dedemorton
removed
the
needs_backport
Initial draft of upgrade docs. This version simply modifies the existing content from 6.x. It's something we can build on for 7.x. We can even change for format if we feel it didn't work well for 6.x.
@urso I know there's a lot missing/incorrect here. I'm sure ECS has implications on migration. I hope that we will provide some kind of tooling to help folks migrate their configs. Can you get the team to help me flesh out this content? They can suggest changes, add comments, or whatever works best. I can create a feature branch, or give people permission to push to my branch.
I haven't hooked this into the platform reference yet because I don't want it to be in the published docs until we're further along.
To do: (feel free to edit this list or add a comment below):
migration.6_to_7.enabledsetting for Beats 7.x documentation pages should mentionmigration.enabled:true#10772