Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log write failures for watcher history document. #44129

Merged
merged 3 commits into from
Jul 15, 2019

Conversation

jakelandis
Copy link
Contributor

@jakelandis jakelandis commented Jul 9, 2019

The failure is correctly getting propagated, however, it is ignored.
This commit adds support to explicitly look for .watch-history failures
using the same logging strategy as triggered watch failures.


This is pretty difficult to add a test (and the change is just an additional log message).

To test manually...

add "index.mapping.total_fields.limit" : "150" to watch-history-no-ilm.json and/or watch-history.json in x-pack/plugin/core/src/main/resources

create a watch that will add a bunch of mappings

{
	"metadata" : {
		"a" : 0,
		"b" : 0,
		"c" : 0,
		"d" : 0,
		"e" : 0,
		"f" : 0,
		"g" : 0,
		"h" : 0,
		"i" : 0,
		"j" : 0,
		"k" : 0,
		"l" : 0,
		"m" : 0,
		"n" : 0,
		"o" : 0,
		"p" : 0,
		"q" : 0,
		"r" : 0,
		"s" : 0,
		"t" : 0,
		"u" : 0,
		"v" : 0,
		"w" : 0,
		"x" : 0,
		"y" : 0,
		"z" : 0,
		"aa" : 0,
		"ab" : 0,
		"ac" : 0,
		"ad" : 0,
		"ae" : 0,
		"af" : 0,
		"ag" : 0,
		"ah" : 0,
		"ai" : 0,
		"aj" : 0,
		"ak" : 0,
		"al" : 0,
		"am" : 0,
		"an" : 0,
		"ao" : 0,
		"ap" : 0,
		"aq" : 0,
		"ar" : 0,
		"as" : 0,
		"at" : 0,
		"au" : 0,
		"av" : 0,
		"aw" : 0,
		"ax" : 0,
		"ay" : 0,
		"az" : 0
	},
	"trigger": {
		"schedule": {
			"interval": "30s"
		}
	},
	"actions": {
		"my-logging-action": {
			"logging": {
				"level": "info",
				"category": "foo",
				"text": "Payload is {{ctx.payload}}"
			}
		}
	},
  "throttle_period_in_millis": 0
}

The error will now be displayed in log and will look like:

[2019-07-09T12:21:55,323][ERROR][o.e.x.w.Watcher          ] [node1] watch history could not be written [watch3_46a3f5a6-c925-454b-aaa7-d99d7b7f7644-2019-07-09T17:21:55.154236Z], failure [java.lang.IllegalArgumentException: Limit of total fields [150] in index [.watcher-history-10-2019.07.09] has been exceeded]

To see the mapping error from the origin, you can set

{
  "transient": {
    "logger.org.elasticsearch.action.admin.indices.mapping.put.TransportPutMappingAction": "debug"
  }
}

The failure is correctly getting propagated, this commit adds support to
explicitly look for .watch-history failures using the same logging strategy
as triggered watch failures.
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

@jakelandis jakelandis requested a review from hub-cap July 9, 2019 17:31
@jakelandis jakelandis changed the title Expose failures to write to watcher history document. Log write failures for watcher history document. Jul 9, 2019
Copy link
Contributor

@hub-cap hub-cap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM just one question about the if / else

@jakelandis
Copy link
Contributor Author

@elasticmachine update branch

@jakelandis jakelandis merged commit b4c60e5 into elastic:master Jul 15, 2019
jakelandis added a commit to jakelandis/elasticsearch that referenced this pull request Jul 15, 2019
The failure is correctly getting propagated, this commit adds support to
explicitly look for .watch-history failures using the same logging strategy
as triggered watch failures.
jakelandis added a commit that referenced this pull request Jul 16, 2019
The failure is correctly getting propagated, this commit adds support to
explicitly look for .watch-history failures using the same logging strategy
as triggered watch failures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants