Default position_offset_gap to 100 #12544

nik9000 · 2015-07-29T21:11:59Z

This is much more fiddly than it looks because of the way position_offset_gap
is applied in StringFieldMapper. Instead of setting the default to 10 its
simpler to make sure that all the analyzers default to 10 and that
StringFieldMapper doesn't override the default unless the user specifies
something different.

Closes #7268

nik9000 · 2015-07-29T21:12:25Z

WIP because it needs tests and more testing. This is way way harder then it ought to be.

nik9000 · 2015-07-29T21:12:45Z

Oh and it needs documentation changes and breaking changes documentation.

rmuir · 2015-07-30T00:01:45Z

Which one is it? the position gap or the offset gap? Lucene has both, and they both have different meanings. Changing the former to this value makes sense, the latter will break many things.

rmuir · 2015-07-30T00:02:11Z

And please, please, we have to rename this, because its totally meaningless. positions and offsets are separate things...

nik9000 · 2015-07-30T00:24:50Z

I understand the rename. It's the positions.
On Jul 29, 2015 8:02 PM, "Robert Muir" notifications@github.com wrote:

And please, please, we have to rename this, because its totally
meaningless. positions and offsets are separate things...

—
Reply to this email directly or view it on GitHub
#12544 (comment)
.

nik9000 · 2015-08-03T13:44:37Z

Does anyone have an opinion on what would be ok behavior from a backwards compatibility standpoint?

nik9000 · 2015-08-03T18:01:04Z

Does anyone have an opinion on what would be ok behavior from a backwards compatibility standpoint?

Talked with @dakrone and we decided that the most correct thing as to make position_offset_gap be immutable once the index is created even if it isn't set. So indexes created before 2.0.0-beta1 will always have a default position_offset_gap of 0 regardless of which version of elasticsearch operates on them. Indexes created by versions of Elasticsearch on or after 2.0.0-beta1 will default to 10 like the issue says they should.

nik9000 · 2015-08-03T18:14:34Z

I believe this is ready for review. @jpountz - its more fiddly than I expected at first so it might be worth a review from you or someone who's pretty experienced with mapping.

@mute - I think this is the right way to do #12538. Its much more complicated than it looked and certainly wasn't really low hanging fruit.

nik9000 · 2015-08-11T14:41:35Z

I'd love this in 2.0.0 at some point. It'll need a review soon if it is going to.

mikemccand · 2015-08-11T16:13:49Z

docs/reference/mapping/types/core-types.asciidoc

@@ -150,7 +150,7 @@ limit to `32766 / 3 = 10922` since UTF-8 characters may occupy at most 3
 bytes.

 |`position_offset_gap` |Position increment gap between field instances
-with the same field name. Defaults to 0.
+with the same field name. Defaults to the default of the analyzer which is 10.


10 seems so arbitrary, but I'm OK with it. I assume you didn't pick 1 because then a sloppy phrase query with slop=1 would incorrectly match...

But, can we explain why it's non-zero here? E.g. just add "... so that proximity queries like PhraseQuery do not incorrectly match across 2 values of a multi-valued field"?

Yeah! I'll do that.

10 is totally arbitrary. Its much more than 1 which is too low. I'd be happy to change it to 1024 or some other magic number. I just want it much much more than 1.

clintongormley · 2015-08-11T16:20:40Z

I agree that 10 is small, and could quite easily overlap with a typical slop value. Personally i'd go for eg 100

mikemccand · 2015-08-11T16:21:43Z

core/src/main/java/org/elasticsearch/index/mapper/core/StringFieldMapper.java

@@ -155,6 +173,9 @@ public StringFieldMapper build(BuilderContext context) {
                    iterator.remove();
                } else if (propName.equals("position_offset_gap")) {
                    builder.positionOffsetGap(XContentMapValues.nodeIntegerValue(propNode, -1));
+                    if (builder.positionOffsetGap < 0) {
+                        throw new MapperParsingException("positions_offset_gap less than 0 aren't allowed.");


I know the exc we throw here effectively discards builder but I think it's less smelly to first assign to a local "untrusted" var first, e.g. int newPositionOffsetGap, and check whether that's < 0, before calling builder.positionOffsetGap(newPositionOffsetGap).

Yeah - I thought about that. I'll make that change.

nik9000 · 2015-08-11T16:30:19Z

Ok - I'll make these changes. I don't think I'll have time today because I'm in class and can't really concentrate. But tonight or tomorrow "morning".

mikemccand · 2015-08-11T16:36:04Z

core/src/main/java/org/elasticsearch/index/analysis/AnalysisService.java

                analyzer = (NamedAnalyzer) analyzerF;
+                if (overridePositionOffsetGap >= 0 && analyzer.getPositionIncrementGap(analyzer.name()) != overridePositionOffsetGap) {


Maybe add some comments here explaining? I'm confused...

Yeah - this was confusing when I wrote it. Its funky and deserves more comments....

Ok - added some comments above.

mikemccand · 2015-08-11T16:39:22Z

Thanks @nik9000 this is a great change (so prox aware queries never match across 2 values of a multi-valued field), I just left some minor comments.

Can you open a follow-on issue to rename positionOffsetGap? Need not block this good change, but this is extremely confusing :) As @rmuir said, position gap and offset gap are wildly different things!

nik9000 · 2015-08-11T16:41:43Z

Can you open a follow-on issue to rename positionOffsetGap? Need not block this good change, but this is extremely confusing :) As @rmuir said, position gap and offset gap are wildly different things!

#12562

nik9000 · 2015-08-24T16:25:32Z

Swapped 2.0 for 2.1 because we're too late in the 2.0 release cycle to get this merged there.

I'm going to pick this one back up in a few minutes and have another read through. I'll see if I can address that last open comment of @mikemccand and rebase. And I'll change all the 2.0s into 2.1s in the code.

This is much more fiddly than you'd expect it to be because of the way position_offset_gap is applied in StringFieldMapper. Instead of setting the default to 100 its simpler to make sure that all the analyzers default to 100 and that StringFieldMapper doesn't override the default unless the user specifies something different. Unless the index was created before 2.1, in which case the old default of 0 has to take. Also postition_offset_gaps less than 0 aren't allowed at all. New tests test that: 1. the new default doesn't match phrases across values with reasonably low slop (5) 2. the new default doest match phrases across values with reasonably high slop (50) 3. you can override the value and phrases work as you'd expect 4. if you leave the value undefined in the mapping and define it on a custom analyzer the the value from the custom analyzer shines through Closes elastic#7268

nik9000 · 2015-08-24T19:42:40Z

docs/reference/mapping/types/core-types.asciidoc

@@ -0,0 +1,651 @@
+[[mapping-core-types]]


grrr - how did I add a whole new file....

Ah. I figured it out - rebase woes. Fixing.

My rebased added core-types back which was removed in the mean time. This properly re-removes the file and moves the updated documentation to the string field mapper docs.

nik9000 · 2015-08-24T19:53:18Z

@mikemccand - would you mind having another look at this? Its now ready for review again based on 2.1.

mikemccand · 2015-08-24T22:37:01Z

LGTM, thanks @nik9000!

s1monw · 2015-08-25T08:25:14Z

@nik9000 this looks good I think this should go into 2.0 though

nik9000 · 2015-08-25T14:59:37Z

@nik9000 this looks good I think this should go into 2.0 though

Ok - I'll merge to 2.1 now and backport it. I'm not super comfortable sticking it in 2.0 but I'll try because you want it in there.

nik9000 · 2015-08-25T18:29:48Z

I merged this into master but github doesn't recognize it - probably because I merged it locally and pushed. That failed so I rebased onto elastic's master and squashed. At this point github's must have lost any connection to the patch.

I'll leave this open to work the backport to 2.0. I'll close it once I've merged there.

xuzha · 2015-08-25T18:37:11Z

core/src/main/java/org/elasticsearch/index/mapper/core/StringFieldMapper.java

+         * values.
+         */
+        public static final int POSITION_OFFSET_GAP = 100;
+        public static final int POSITION_OFFSET_GAP_PRE_2_1 = 0;


This change is going to 2.0 branch right? Do we need to change this to 2_0?

Sadly, yup. I had originally targeted 2.0, then gave up on making it in, then @s1monw said we should jam it in there anyway. So I got what I had reviewed merged as quickly as I could. It's looking like it'll be reasonably quick to merge to 2.0 so I'll do that and then go fix master so it makes sense.

xuzha · 2015-08-25T19:03:58Z

docs/reference/migration/migrate_2_1.asciidoc

+=== Mapping changes
+
+==== position_offset_gap
+The default `position_offset_grap` is now 100. Indexes created in Elasticsearch


position_offset_grap -> position_offset_gap

Ok - Will fix this one right quick.

nik9000 · 2015-08-25T19:57:12Z

And merged to 2.0. For my last trick on this pull request I'll fix master so that it has the same version range checks as 2.0. And then finally, finally, this is done.

nik9000 · 2015-08-25T20:05:25Z

And merged to 2.0. For my last trick on this pull request I'll fix master so that it has the same version range checks as 2.0. And then finally, finally, this is done.

Scratch that, I'll send that as another pull request.

nik9000 added v2.0.0-beta1 WIP labels Jul 29, 2015

nik9000 mentioned this pull request Jul 29, 2015

Updating default position_offset_gap to 10. #12538

Closed

nik9000 mentioned this pull request Jul 30, 2015

Rename position_offset_gap to position_increment_gap #12562

Closed

nik9000 added >feature >breaking review :Search/Mapping Index mappings, including merging and defining field types and removed WIP labels Aug 3, 2015

nik9000 mentioned this pull request Aug 3, 2015

Fix plugin script to allow spaces in ES_HOME #12610

Merged

clintongormley removed the >feature label Aug 5, 2015

nik9000 added v2.0.0 and removed v2.0.0-beta1 labels Aug 11, 2015

mikemccand reviewed Aug 11, 2015
View reviewed changes

nik9000 added v2.1.0 and removed v2.0.0 labels Aug 24, 2015

nik9000 changed the title ~~Mapping: Default position_offset_gap to 10~~ Mapping: Default position_offset_gap to 100 Aug 24, 2015

nik9000 force-pushed the position_offset branch from d3378af to 41d0199 Compare August 24, 2015 18:55

nik9000 force-pushed the position_offset branch from 41d0199 to e49b98c Compare August 24, 2015 19:36

nik9000 reviewed Aug 24, 2015
View reviewed changes

Fix rebase mistake

ab75e6d

My rebased added core-types back which was removed in the mean time. This properly re-removes the file and moves the updated documentation to the string field mapper docs.

nik9000 changed the title ~~Mapping: Default position_offset_gap to 100~~ Default position_offset_gap to 100 Aug 25, 2015

nik9000 mentioned this pull request Aug 25, 2015

Rename position_offset_gap to position_increment_gap #13056

Merged

xuzha reviewed Aug 25, 2015
View reviewed changes

nik9000 added the v2.0.0 label Aug 25, 2015

xuzha reviewed Aug 25, 2015
View reviewed changes

nik9000 closed this Aug 25, 2015

nik9000 mentioned this pull request Aug 25, 2015

Should the default position_offset_gap be higher then 0? #7268

Closed

clintongormley added v2.0.0-beta2 and removed v2.0.0 labels Sep 14, 2015

clintongormley removed the v2.1.0 label Nov 22, 2015

		analyzer = (NamedAnalyzer) analyzerF;
		if (overridePositionOffsetGap >= 0 && analyzer.getPositionIncrementGap(analyzer.name()) != overridePositionOffsetGap) {

Default position_offset_gap to 100 #12544

Default position_offset_gap to 100 #12544

Conversation

nik9000 commented Jul 29, 2015

nik9000 commented Jul 29, 2015

nik9000 commented Jul 29, 2015

rmuir commented Jul 30, 2015

rmuir commented Jul 30, 2015

nik9000 commented Jul 30, 2015

nik9000 commented Aug 3, 2015

nik9000 commented Aug 3, 2015

nik9000 commented Aug 3, 2015

nik9000 commented Aug 11, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clintongormley commented Aug 11, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nik9000 commented Aug 11, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikemccand commented Aug 11, 2015

nik9000 commented Aug 11, 2015

nik9000 commented Aug 24, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nik9000 commented Aug 24, 2015

mikemccand commented Aug 24, 2015

s1monw commented Aug 25, 2015

nik9000 commented Aug 25, 2015

nik9000 commented Aug 25, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nik9000 commented Aug 25, 2015

nik9000 commented Aug 25, 2015