Write deprecation logs to a data stream #58924

pugnascotia · 2020-07-02T14:13:10Z

Closes #46106. Introduce a mechanism for writing deprecation logs to a data stream as well as to disk.

Test by running ./gradlew run and then:

curl -u elastic-admin:elastic-password http://localhost:9200/_flush/synced?pretty
curl -u elastic-admin:elastic-password http://localhost:9200/logs-deprecation-elasticsearch/_search?pretty

This implementation reworks deprecation logging to rely on log4j by introducing filters and appenders, and a custom log level. The existing deprecation X-Pack plugin simply configures an extra appender to capture messages to an index. A "component" class manages most of the interface between the cluster and the appender.

This approach can be extended for other types of logging, e.g. security by introducing further custom log levels.

elasticmachine · 2020-07-02T14:13:12Z

Pinging @elastic/es-core-infra (:Core/Infra/Logging)

server/src/main/java/org/elasticsearch/common/logging/DeprecationLogger.java

pugnascotia · 2020-07-02T14:20:32Z

I should add, the set of indexed fields in this PR may need to evolve, as it becomes clearer what we need to support the Upgrade Assistant. It should be a reasonable starting point however.

…-logs-v2

server/src/main/java/org/elasticsearch/common/logging/DeprecationIndexingService.java

jakelandis · 2020-07-02T20:03:33Z

server/src/main/java/org/elasticsearch/common/logging/DeprecationIndexingService.java

+
+    private static final String DEPRECATION_ORIGIN = "deprecation";
+
+    public static final Setting<Boolean> WRITE_DEPRECATION_LOGS_TO_INDEX = Setting.boolSetting(


Should the setting allow to flex between writing to a log, an index, or both ?

pugnascotia · 2020-07-02T20:12:31Z

Moving to draft status while we reconsider some things.

server/src/main/java/org/elasticsearch/common/logging/DeprecationIndexingService.java

…-logs-v2

.../rest/src/main/java/org/elasticsearch/xpack/deprecation/TestDeprecationHeaderRestAction.java

...deprecation/qa/rest/src/test/java/org/elasticsearch/xpack/deprecation/DeprecationHttpIT.java

pugnascotia · 2020-08-14T10:28:55Z

@elasticmachine update branch

pgomulka

it looks really good now. Thank you for working on this.
I left two minor comments.

test/framework/src/main/java/org/elasticsearch/test/ESSingleNodeTestCase.java

pgomulka · 2020-08-17T07:57:12Z

server/src/main/java/org/elasticsearch/common/logging/HeaderWarningAppender.java

+import org.apache.logging.log4j.core.config.plugins.PluginFactory;
+import org.apache.logging.log4j.message.Message;
+
+@Plugin(name = "HeaderWarningAppender", category = Core.CATEGORY_NAME, elementType = Appender.ELEMENT_TYPE)


@pugnascotia any views on creating a logger?

…-logs-v2

pugnascotia · 2020-08-19T09:05:14Z

@elasticmachine update branch

pugnascotia · 2020-08-19T15:21:09Z

@rjernst @jakelandis CI is now green, can you please take another look?

rjernst

I left a couple comments, but overall, would it be possible to split this up? The change pretty large, and doing several things, which I think are semi unrelated concerns:

Using log4j as backend of deprecation logs
Using log4j as backend of header warnings
Adding deprecation log messages from log4j to a data stream

rjernst · 2020-08-13T23:55:43Z

x-pack/plugin/deprecation/build.gradle

@@ -9,6 +8,15 @@ esplugin {
 }
 archivesBaseName = 'x-pack-deprecation'

+// add all sub-projects of the qa sub-project
+gradle.projectsEvaluated {


Why is this necessary?

rjernst · 2020-08-14T00:02:08Z

libs/x-content/src/main/resources/log4j2.properties

@@ -0,0 +1,8 @@
+appender.header_warning.type = HeaderWarningAppender


Can you explain why this file is needed?

rjernst · 2020-08-21T02:47:12Z

server/src/main/java/org/elasticsearch/common/logging/HeaderWarningAppender.java

+import org.apache.logging.log4j.core.config.plugins.PluginFactory;
+import org.apache.logging.log4j.message.Message;
+
+@Plugin(name = "HeaderWarningAppender", category = Core.CATEGORY_NAME, elementType = Appender.ELEMENT_TYPE)


I'm failing to understand what the advantage is of using log4j for header warnings? In either case, the warnings are stored in the thread context still right? Is it for formatting?

If we are going to use it, then I think it should be hidden, like deprecation logging is behind DeprecationLogger. But with header warnings, we already have the HeaderWarning class, and the logger retrieved will be the same for all locations, so we could just do this statically? Using a logger for header warnings directly is odd because the log level doesn't actual matter?

pugnascotia · 2020-08-24T14:18:20Z

I'm breaking up the PR as follows:

Deprecation logging through log4j Implement deprecation logging using log4j #61474
Index deprecation logs Write deprecation logs to a data stream #61484.

Regarding header warnings: although I thought carefully before reimplementing header warnings with log4j...now I'm not so sure. That part of the work was relatively straightforward, so now I'm thinking that we only go down that road if/when we think we really need to.

pugnascotia · 2020-08-24T16:00:06Z

Closing in favour of #61474 and #61484.

Backport of #58924. Closes #46106. Introduce a mechanism for writing deprecation logs to a data stream as well as to disk.

The implementation for indexing deprecation logs to a data stream (#58924) relied on the Stack template for `logs-*-*`. This meant that if the user disabled the stack templates, the templates would also be unavailable for the deprecation logs. Change the implementation so that: * There is a separate template for deprecation logging * The data stream is marked as hidden * The data stream name is prefixed with a period (`.`)

The implementation for indexing deprecation logs to a data stream (elastic#58924) relied on the Stack template for `logs-*-*`. This meant that if the user disabled the stack templates, the templates would also be unavailable for the deprecation logs. Change the implementation so that: * There is a separate template for deprecation logging * The data stream is marked as hidden * The data stream name is prefixed with a period (`.`)

Backport of #64417. The implementation for indexing deprecation logs to a data stream (#58924) relied on the Stack template for `logs-*-*`. This meant that if the user disabled the stack templates, the templates would also be unavailable for the deprecation logs. Change the implementation so that: * There is a separate template for deprecation logging * The data stream is marked as hidden * The data stream name is prefixed with a period (`.`)

pugnascotia added 6 commits June 30, 2020 15:54

Introduce DeprecationIndexingService

36ac092

First go at ensuring the data stream template is loaded

cf5a677

Tweaks index name

86c3d7d

WIP - Adding tests for DeprecationIndexingService

d74dc9a

More tests

6457f97

Fix template index pattern

f0e0192

pugnascotia added >enhancement :Core/Infra/Logging Log management and logging utilities v8.0.0 v7.9.0 labels Jul 2, 2020

pugnascotia requested review from rjernst, jakelandis and pgomulka July 2, 2020 14:13

elasticmachine added the Team:Core/Infra Meta label for core/infra team label Jul 2, 2020

pugnascotia commented Jul 2, 2020

View reviewed changes

server/src/main/java/org/elasticsearch/common/logging/DeprecationLogger.java Outdated Show resolved Hide resolved

Merge remote-tracking branch 'upstream/master' into index-deprecation…

581be1f

…-logs-v2

jakelandis reviewed Jul 2, 2020

View reviewed changes

server/src/main/java/org/elasticsearch/common/logging/DeprecationIndexingService.java Outdated Show resolved Hide resolved

jakelandis reviewed Jul 2, 2020

View reviewed changes

server/src/main/java/org/elasticsearch/common/logging/DeprecationIndexingService.java Outdated Show resolved Hide resolved

jakelandis reviewed Jul 2, 2020

View reviewed changes

pugnascotia marked this pull request as draft July 2, 2020 20:12

pgomulka reviewed Jul 3, 2020

View reviewed changes

server/src/main/java/org/elasticsearch/common/logging/DeprecationIndexingService.java Outdated Show resolved Hide resolved

server/src/main/java/org/elasticsearch/common/logging/DeprecationIndexingService.java Outdated Show resolved Hide resolved

pugnascotia added 6 commits July 3, 2020 13:04

Rework deprecation code so that log handlers are more generic

14c929a

Move DeprecationIndexingService to x-pack depreacation plugin

e902fb9

Don't need a custom template any more 🎉

40bea2c

Remove unused import

013bf55

Merge remote-tracking branch 'upstream/master' into index-deprecation…

4c189da

…-logs-v2

License header fix

d61b145

pugnascotia commented Aug 13, 2020

View reviewed changes

.../rest/src/main/java/org/elasticsearch/xpack/deprecation/TestDeprecationHeaderRestAction.java Show resolved Hide resolved

pugnascotia commented Aug 13, 2020

View reviewed changes

...deprecation/qa/rest/src/test/java/org/elasticsearch/xpack/deprecation/DeprecationHttpIT.java Show resolved Hide resolved

pugnascotia added 2 commits August 13, 2020 14:41

Formatting

2dac9a3

Remove redundant line

f0f3edb

Merge branch 'master' into index-deprecation-logs-v2

9877d57

pgomulka approved these changes Aug 17, 2020

View reviewed changes

pugnascotia added 3 commits August 17, 2020 12:53

Remove unused import

1309355

Merge remote-tracking branch 'upstream/master' into index-deprecation…

bd8e3c9

…-logs-v2

Add warning headers solely through the log4j appender

76993c9

elasticmachine and others added 2 commits August 19, 2020 03:05

Merge branch 'master' into index-deprecation-logs-v2

e695753

Fixes to JsonLoggerTest

3ff1015

gwbrown mentioned this pull request Aug 20, 2020

Check Security Roles in Deprecation Info API #49212

Closed

rjernst reviewed Aug 21, 2020

View reviewed changes

pugnascotia closed this Aug 24, 2020

pugnascotia mentioned this pull request Sep 4, 2020

Write deprecation logs to a data stream #61966

Merged

pgomulka mentioned this pull request Sep 8, 2020

Write Slow Logs to a data stream #62077

Closed

pugnascotia added a commit that referenced this pull request Sep 9, 2020

Write deprecation logs to a data stream (#61966)

b7fd7cf

Backport of #58924. Closes #46106. Introduce a mechanism for writing deprecation logs to a data stream as well as to disk.

jsvd mentioned this pull request Sep 24, 2020

improved logging when pipeline terminates by error elastic/logstash#12274

Draft

5 tasks

pugnascotia mentioned this pull request Oct 30, 2020

Change deprecation indexing to use a custom template #64417

Merged

pugnascotia mentioned this pull request Nov 5, 2020

Change deprecation indexing to use a custom template #64627

Merged

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Write deprecation logs to a data stream #58924

Write deprecation logs to a data stream #58924

pugnascotia commented Jul 2, 2020 •

edited

Loading

elasticmachine commented Jul 2, 2020

pugnascotia commented Jul 2, 2020

jakelandis Jul 2, 2020

pugnascotia commented Jul 2, 2020

pugnascotia commented Aug 14, 2020

pgomulka left a comment

pgomulka Aug 17, 2020

pugnascotia commented Aug 19, 2020

pugnascotia commented Aug 19, 2020

rjernst left a comment

rjernst Aug 13, 2020

rjernst Aug 14, 2020

rjernst Aug 21, 2020

pugnascotia commented Aug 24, 2020 •

edited

Loading

pugnascotia commented Aug 24, 2020


		private static final String DEPRECATION_ORIGIN = "deprecation";

		public static final Setting<Boolean> WRITE_DEPRECATION_LOGS_TO_INDEX = Setting.boolSetting(

		@@ -0,0 +1,8 @@
		appender.header_warning.type = HeaderWarningAppender

Write deprecation logs to a data stream #58924

Write deprecation logs to a data stream #58924

Conversation

pugnascotia commented Jul 2, 2020 • edited Loading

elasticmachine commented Jul 2, 2020

pugnascotia commented Jul 2, 2020

jakelandis Jul 2, 2020

Choose a reason for hiding this comment

pugnascotia commented Jul 2, 2020

pugnascotia commented Aug 14, 2020

pgomulka left a comment

Choose a reason for hiding this comment

pgomulka Aug 17, 2020

Choose a reason for hiding this comment

pugnascotia commented Aug 19, 2020

pugnascotia commented Aug 19, 2020

rjernst left a comment

Choose a reason for hiding this comment

rjernst Aug 13, 2020

Choose a reason for hiding this comment

rjernst Aug 14, 2020

Choose a reason for hiding this comment

rjernst Aug 21, 2020

Choose a reason for hiding this comment

pugnascotia commented Aug 24, 2020 • edited Loading

pugnascotia commented Aug 24, 2020

pugnascotia commented Jul 2, 2020 •

edited

Loading

pugnascotia commented Aug 24, 2020 •

edited

Loading