Need more doc about upgrading from 2.2 to 2.3

[dedemorton]

The current upgrade documentation needs to discuss how the change to the default output workers that we made in 2.3 (see issue #4877) might affect customers who are upgrading from 2.2 to 2.3.

We need to:

• Update the Breaking Changes doc to indicate that the change to output workers is potentially a breaking change that may affect throughput.
• Also mention in the Release Notes that this change might affect throughput.
• Add a topic about upgrading to 2.3 and mention that in order to get the same configuration in 2.3 as you had in 2.2, you will need to specify the number of workers in the actual output configuration (note the value specified for output workers depends on whether they were using -w.
• In the plugin docs, it would also be good to remove or reword the following sentence that appears under the description for workers because it's so vague that it's virtually meaningless: "Note that this setting may not be useful for all outputs."

[GlenRSmith]

👍

[suyograo]

Add a topic about upgrading to 2.3 and mention that in order to get the same configuration in 2.3 as you had in 2.2, you will need to specify the number of workers in the actual output configuration (note the value specified for output workers depends on whether they were using -w.

I think this statement needs to be qualified to say output workers will increase resource usage. See explanation below:

Its a bug that in 2.2 we made output workers = pipeline workers = # of cores. Its perfectly fine to set pipeline workers = # of cores or even greater than # of cores. Outputs in Logstash are not thread safe in terms of underlying resources it uses. Increasing output workers (for Kafka, Elasticsearch) may increase the throughput but it will also result in resource issues like too many file handles being used, too many sockets being opened etc. For reference see #4754. This has also been pointed out in the blog: https://www.elastic.co/blog/upgrade-guide-for-logstash-2-2. See section "New Pipeline and Outputs".

[dedemorton]

Wanted to note that changing the value of output workers (the fix described above) is not generally recommended. The dev team is working on a better explanation to help users understand and resolve this problem if they encounter it when moving to 2.3.

[dedemorton]

Closing because we decided to address the concerns that prompted this issue by adding a troubleshooting guide instead.