New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using default:null
for _timestamp field creates a index loss on restart
#9233
Conversation
… on restart Step to reproduce: * Create new index and type. ``` DELETE new_index PUT new_index { "mappings": { "power": { "_timestamp" : { "enabled" : true, "default": null } } } } ``` * Add a document ``` PUT new_index/power/1 { "foo": "bar" } ``` * Restart cluster ... and **index is missing**... ``` GET new_index ``` Gives IndexMissingException Closes elastic#9223.
@@ -354,7 +354,7 @@ private void initMappers(Map<String, Object> withoutType) { | |||
path = fieldNode.toString(); | |||
} else if (fieldName.equals("format")) { | |||
format = fieldNode.toString(); | |||
} else if (fieldName.equals("default")) { | |||
} else if (fieldName.equals("default") && fieldNode != null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't the same NPE possible above in all the other conditions? Should we just continue the loop before these checks?
if (fieldNode == null) continue;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Valid point. Though I think null values might be rejected at mapping creation time for other values.
But as it's a safer option, I'll modify the PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
original reporter here who discovered this issue after using _timestamp with default=null.
For "default" field in particular, null actually means something (which is why i used it) - to ignore the normal _timestamp default of now if _timestamp is not provided, thereby making _timestamp (or the field providing it via path property) a required field. seems like "default" property needs separate handling from other fields that might inadvertently have been passed a null - as in this case passing a null has a meaning (that being: clear any default behavior that the _timestamp should have).
From _timestamp doc, section on Default.
By default, the default value is now which means the date the document was processed by the indexing chain.
You can disable that default value by setting default to null. It means that timestamp is mandatory:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
btw -in the case where you inject a record without provided path property being there, and have set default=null, you'll properly get this error, as expected per the documentation quoted above.
{
"error": "TimestampParsingException[failed to parse timestamp [timestamp is required by mapping]]",
"status": 500
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
logic subscribing to what is described in the docs (ignoring null assigned to any other properties) would be more like this... making defaultTimestamp null if so, else the string provided.
else if (fieldName.equals("default")) {
defaultTimestamp = (fieldNode == null ? null : fieldNode.toString());
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In 1.5 we will change this logic: see #9104
@jpountz please review this before it gets merged |
I think we should also modify the log level if we can not open an index. For now this error could only be seen when in DEBUG level. |
LGTM |
regardless of 1.5 changing the nature of the feature, for 1.4.x, I still say this fix is going to break the intended default=null behavior that the docs state, as the "default" property will continue to default to "now" even when user sets default=null to disable that behavior (per doc snippit I quoted above). |
default:null
for _timestamp field creates a index loss on restart
Step to reproduce:
Gives IndexMissingException
Closes #9223.