7.0 stack upgrade docs feedback

[cjcenizal]

Context

As @bleskes coordinates testers for the 7.0 stack upgrade process, we'll gather feedback on the docs and record it here.

• The page we're using to guide us: https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html
• Notes for testers: https://docs.google.com/spreadsheets/d/1LtmjfCgtohS_H6fMk3fSKh3tmEqyjioZbFk2giFDeQA/edit#gid=590440521

[cjcenizal]

"Upgrade Elasticsearch" docs

I think we should begin these docs with a single section titled "Upgrading the Elastic Stack?" which explains that if you're upgrading the Elastic Stack to visit the "Upgrade Elastic Stack" docs first. By giving this section its own title we can separate it from the rest of the other content, and ES-only upgraders can ignore it more easily. Might it also be worth removing the link to the Interactive Upgrade Guide, to avoid giving the user too many options?

Backups

The first step of preparing to upgrade is to back-up your data, which links to Snapshot and Restore docs. These docs attempt to explain how to define where ES will create snapshots on the filesystem:

Assuming that the shared filesystem is mounted to /mount/backups/my_fs_backup_location...

I find this wording a bit confusing because it assumes the reader knows a) what makes a filesystem "shared" and b) what it means to have a shared filesystem "mounted" somewhere. I used this third party tutorial to learn what this meant. I'd suggest rephrasing this line to be:

Assuming you want to store your snapshots inside of the directory at the absolute path /backups/my_fs_backup_location...

And if we have an official tutorial on snapshot and restore, I recommend we link directly to it from the docs.

Deprecations

The link to the deprecation log doesn't seem to take you to a useful location... it's pointed at https://www.elastic.co/guide/en/elasticsearch/reference/6.6/settings.html, but I'm not sure how that relates?

Breaking changes

The link for Elasticsearch should point to the version-specific page.

Upgrading internal indices

If Kibana and X-Pack are part of your stack, upgrade the internal Kibana and X-Pack indices

What does it mean to upgrade the internal indices? How do you do that?

[bleskes]

Thx for opening this @cjcenizal.

[droberts195]

what makes a filesystem "shared"

When that question was asked on the Discuss forums the answer that made it clear was https://discuss.elastic.co/t/backups-need-a-shared-mount/85605/4. So I think the thing the docs need to say to make it clear for people who don't work as sysadmins is that the backup directory must be accessible at the same path on every node in the cluster.

[bleskes]

@jasontedor brought to my attention that we are not clear enough about version dependencies between stack elements. The following is supported:

Versions of the same major will speak to each other, cross major version is latest (6.7) to latest (7.0 first, 7.1 when released etc). Only the data indexed by the previous major will be read by the next version.

An example: say that someone starts with all stack elements on 6.5. In order to get to 7.0, they need to do the following:

1. First upgrade ES, followed by Kibana, followed by Beats (or other shippers) to 6.7.
2. Now that all the components are on 6.7, you can start upgrading ES to 7.0 (and then Kibana, followed by Beats).

Upgrading ES directly to 7.0 will cause beats and Kibana to fail.

Note that this may have implications for other things as well. For example, the monitoring docs say:

A monitoring cluster cannot monitor production clusters running newer versions of the stack. If necessary, the monitoring cluster can monitor production clusters running older versions, but the versions cannot differ by more than one major version.

The last part is wrong. It should say - the same major version, or major-1.latest to major.latest (in some more human-readable form).

[bleskes]

@debadair I'm posting this here as it may be relevant for other products - the ES 7.0 and master upgrade pages are full of references to incorrect versions. For example:

Rolling upgrades can be performed between minor versions. Elasticsearch 6.x supports rolling upgrades from Elasticsearch 5.6. Upgrading from earlier 5.x versions requires a full cluster restart. You must reindex to upgrade from versions prior to 5.x.

TBC these docs are for the 8.0 version. I don't know how to structurally fix this as there is always something that's version specific but I wish we could come up with some process to make sure we never forget to update these pages.

[ppf2]

Few additional suggestions:

• The current doc only talks about checking using the 5.6 upgrade assistant if coming from 5.6. I think they will want to run the upgrade assistant in 5.6 (for catching breaking changes between 5.6 and 6.0), and then again in 6.7 (for catching breaking changes between 6.7 and 7.0)?
  • Related: It's unclear from the doc if they have to use the upgrade assistant's reindex helper to reindex/upgrade the internal Kibana and X-Pack indices on 5.6, and then again on 6.7 (or if the reindex on 6.7 will take care of all incremental changes between the major versions).
• "Make sure real passwords are configured for the built-in elasticsearch, kibana, and logstash_system users. They cannot use the 5.n default password (changeme). For more information, see Built-in users." It can be helpful to provide some specific instructions (e.g. run setup passwords once all the nodes are on 7.0?)
• The way the doc is written today suggests that index formats are compatible between 5.6 and 7.0 - which they are not.
• "Rolling upgrades are supported when upgrading from Elasticsearch 5.6 and Elasticsearch 6.0-6.2 to 7.0.0-beta1". It provides no explanation on what it means for 6.3+. I think this one will be addressed by Boaz's notes above.
• It may be helpful to mention the different approaches for 5.6 to 7.0 upgrade
  • 5.6 to 6.7 as an interim step for the upgrade to 7.0 (so they can review the breaking changes incrementally, and reindex on 6.7, etc..).
  • 5.6 to 7.0 which requires a full cluster restart and possibly separate clusters to reduce downtime
• This document has different sections/headings that do not have a particular flow, so when reading from top to bottom, it can get confusing since all of this information resides on the same page with anchor links. Have we considered breaking this up into different pages linking from the main page (these are arbitrary heading names), e.g.,
  • Upgrading from 6.x to 7.0
  • Upgrading from 5.x to 7.0
  • Upgrading from pre-5.x to 7.0
  • Steps after upgrading to 7.0
    • For any additional steps required after installing 7.0 on all nodes
    • This can then be cross-referenced from each of the "Upgrading from ..." pages
  • Considerations for OSS users when upgrading from pre-6.3
    • This can then be cross-referenced from each of the "Upgrading from ..." pages
  • Upgrading on Elastic Cloud
  • Upgrading on ECE (currently missing from our docs)

[cjcenizal]

I created elastic/beats#11276 to track some issues I had with the Metricbeat docs.

[cjcenizal]

Created elastic/kibana#33543 to track some issues with the Kibana docs.

[lcawl]

Created elastic/elasticsearch#40663 to follow up on need for backup data steps per #226 (comment)

[lcawl]

Created #263 for the inaccurate monitoring detail described in #226 (comment)