Refactors for 7.0 (Breaking changes I want to make)

[andrewkroh]

This is a running list of things to consider changing for 7.0. Feel free to add your own with attribution.

• Refactor the logging configuration (remove all the to_files, to_syslog settings etc.).
• Make Windows event log the default log output for Windows.
• Set escape_html to false: Html escaping #7445 Do not escape HTML by default anymore #9914 @urso
• Remove notion of prospector in the filebeat config. (@ph) PR Remove prospector support in Filebeat #8909
• Cleanup the methods used by go-plugins
• Default Template Mappings for IP fields are keyword type Default Template Mappings for IP fields are keyword type #7847 [Packetbeat] Restructure client/server and process fields #9303 @ruflin
• Remove ports from LS and Redis Deprecated LS/redis ports followup #7881 Remove port settings from LS/redis output #9934 @urso
• Dissect * and named skip field change Dissect: * and & reference modifier, breaking change for 7.0 #8054, PR At Allow to use a star (*) to extract a value that can be referenced. #8750 (@ph)
• Make beats log to journald by default on systems with systemd (add -e by default to systemd unit) (@jsoriano) Log by default to journald on systems with systemd #8942
• Dedot docker and k8s labels by default, or use new queryable object type Use new json type for labels and annotations #9286 (@kaiyan-sheng)
• Remove calls to cfgwarn.CheckRemoved5xSetting(s) (@ph) Remove 5.x obsolete options checks #9002
• Remove 5.x dashboard directories, add 7 directories? Remove kibana 5 dashboards and index patterns #8927
• Assert any 'deprecated' comment in beats. (@ph)
• Change types to double in metricbeat modules for big gauges (@jsoriano, see Mask uint64 with 63bit mask by default #8991, Check Beats overflow handling #9271)
• Rename custom.* to journald.custom.* fields (@kvch, already on master, needs changelog entry)
• New location of registry of Journalbeat (@kvch, already on master, needs BC and changelog)
• Deprecate fields/tags and introduce add_fields/add_tags processors Provide add_fields processor in filebeat #4518 Introduce add_labels and add_tags processors #9973 (@urso)
• Remove ES 1.x and 5.x support in template/dashboard loading code
• Unify module names for the same services module names for the same service different between beats (apache and apache2) #9395 (@jsoriano)
• Fixed any issues with the _xpack deprecation Elasticsearch _xpack endpoint is getting deprecated #9424

Would be nice to have

[ph]

Added #7881

[jsoriano]

I have added a point to make beats log to journald by default when running on systemd, open to discussion, I'll open a PR.

@exekias I have added also a point about dedoting labels so we don't forget about this.

[ruflin]

I added a point about removing the 5.x dashboard directories to the above list. We should probably add a directory for 7 as the data structure for Kibana will change. @elastic/apm-server This might also effect you.

I put it on the service meeting agenda today.

[ph]

@jsoriano

I have added a point to make beats log to journald by default when running on systemd, open to discussion, I'll open a PR.

Can you create an issue for that, I don't know how big of a task it is.

[ph]

@ruflin I believe your ECS migration will take care of: Default Template Mappings for IP fields are keyword type #7847

[ph]

@urso @ruflin @exekias I have added your name to the list of items you have added to this list, feel free to either remove the items or delegate the work.

[ruflin]

The IP stuff is not directly related to ECS but parts of it might be resolved through ECS.

[jsoriano]

@ph for the systemd unit logging to journald I mean something like #8942

[ph]

FYI I've added the following methods to fails on removed options, if you want to provide better feedback use them and it make it easy for us to remove them in 8.0.

beats/libbeat/common/cfgwarn/removed.go

Lines 77 to 85 in 74779e4

	// CheckRemoved6xSettings prints a warning if the obsolete setting is used.
	func CheckRemoved6xSettings(cfg *common.Config, settings ...string) error {
	return checkRemovedSettings(cfg, settings...)
	}
	// CheckRemoved6xSetting prints a warning if the obsolete setting is used.
	func CheckRemoved6xSetting(cfg *common.Config, setting string) error {
	return checkRemovedSetting(cfg, setting)
	}

[urso]

recent changes:

• Deprecate fields/tags and introduce add_fields/add_tags processors Provide add_fields processor in filebeat #4518 (@urso)
• Remove ES 1.x and 5.x support in template/dashboard loading code

[elasticmachine]

Pinging @elastic/infrastructure

[ph]

Added a note about _xpack's rest endpoint deprecation #9424

[urso]

I think we can close this one.

@andrewkroh @ruflin @ph @kaiyan-sheng @kvch @jsoriano Any of the items mentioned still open?

[jsoriano]

I didn't check if we should change some potentially big gauges from long to float, but I think this wouldn't be so breaking.

[kaiyan-sheng]

Nothing open on my side 👍For dedot we do have #10773 to track some cleanup work.

[urso]

Not sure if all check boxes can be ticked off. But 7.0 is out for quite a while. Closing this.