Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update default precision step, modulo tests #5908

Closed
wants to merge 4 commits into from
Closed

Update default precision step, modulo tests #5908

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0bee444
update default precision step, modulo tests
Apr 22, 2014
67fc81c
cleanups from pr comments
Apr 22, 2014
80efba6
fix documentation to refer to the new defaults, also fix description …
Apr 22, 2014
31a9c09
add tests for precision steps, fix mapper to actually set this in the…
Apr 23, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
12 changes: 11 additions & 1 deletion core-signatures.txt
View file
Open in desktop
Expand Up @@ -34,6 +34,16 @@ org.apache.lucene.index.IndexWriter#forceMergeDeletes(boolean) @ use Merges#forc
@defaultMessage QueryWrapperFilter is cachable by default - use Queries#wrap instead
org.apache.lucene.search.QueryWrapperFilter#<init>(org.apache.lucene.search.Query)

@defaultMessage Pass the precision step from the mappings explicitly instead
org.apache.lucene.search.NumericRangeQuery#newDoubleRange(java.lang.String,java.lang.Double,java.lang.Double,boolean,boolean)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

man those seem to be trappy - maybe we should deprecate them in Lucene as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, see https://issues.apache.org/jira/browse/LUCENE-5605

org.apache.lucene.search.NumericRangeQuery#newFloatRange(java.lang.String,java.lang.Float,java.lang.Float,boolean,boolean)
org.apache.lucene.search.NumericRangeQuery#newIntRange(java.lang.String,java.lang.Integer,java.lang.Integer,boolean,boolean)
org.apache.lucene.search.NumericRangeQuery#newLongRange(java.lang.String,java.lang.Long,java.lang.Long,boolean,boolean)
org.apache.lucene.search.NumericRangeFilter#newDoubleRange(java.lang.String,java.lang.Double,java.lang.Double,boolean,boolean)
org.apache.lucene.search.NumericRangeFilter#newFloatRange(java.lang.String,java.lang.Float,java.lang.Float,boolean,boolean)
org.apache.lucene.search.NumericRangeFilter#newIntRange(java.lang.String,java.lang.Integer,java.lang.Integer,boolean,boolean)
org.apache.lucene.search.NumericRangeFilter#newLongRange(java.lang.String,java.lang.Long,java.lang.Long,boolean,boolean)

@defaultMessage Only use wait / notify when really needed try to use concurrency primitives, latches or callbacks instead.
java.lang.Object#wait()
java.lang.Object#wait(long)
Expand All @@ -46,4 +56,4 @@ java.lang.Math#abs(int)
java.lang.Math#abs(long)

@defaultMessage Use Long.compare instead we are on Java7
com.google.common.primitives.Longs#compare(long,long)
com.google.common.primitives.Longs#compare(long,long)
9 changes: 5 additions & 4 deletions docs/reference/mapping/types/core-types.asciidoc
View file
Open in desktop
Expand Up @@ -238,8 +238,9 @@ in `_source`, have `include_in_all` enabled, or `store` be set to
|`doc_values` |Set to `true` to store field values in a column-stride fashion.
Automatically set to `true` when the fielddata format is `doc_values`.

|`precision_step` |The precision step (number of terms generated for
each number value). Defaults to `4`.
|`precision_step` |The precision step (influences the number of terms
generated for each number value). Defaults to `16` for `long`, `double`,
`8` for `short`, `integer`, `float`, and `2147483647` for `byte`.

|`boost` |The boost value. Defaults to `1.0`.

Expand Down Expand Up @@ -347,8 +348,8 @@ in `_source`, have `include_in_all` enabled, or `store` be set to
|`doc_values` |Set to `true` to store field values in a column-stride fashion.
Automatically set to `true` when the fielddata format is `doc_values`.

|`precision_step` |The precision step (number of terms generated for
each number value). Defaults to `4`.
|`precision_step` |The precision step (influences the number of terms
generated for each number value). Defaults to `16`.

|`boost` |The boost value. Defaults to `1.0`.

Expand Down
7 changes: 4 additions & 3 deletions docs/reference/mapping/types/geo-point-type.asciidoc
View file
Open in desktop
Expand Up @@ -155,9 +155,10 @@ is `true`).

|`normalize_lon` |Set to `true` to normalize longitude.

|`precision_step` |The precision step (number of terms generated for
each number value) for `.lat` and `.lon` fields if `lat_lon` is set to `true`.
Defaults to `4`.
|`precision_step` |The precision step (influences the number of terms
generated for each number value) for `.lat` and `.lon` fields
if `lat_lon` is set to `true`.
Defaults to `16`.
|=======================================================================

[float]
Expand Down
4 changes: 2 additions & 2 deletions docs/reference/mapping/types/ip-type.asciidoc
View file
Open in desktop
Expand Up @@ -21,8 +21,8 @@ and it can be retrieved from it).
`store` should be set to `true`, since if it's not indexed and not
stored, there is nothing to do with it.

|`precision_step` |The precision step (number of terms generated for
each number value). Defaults to `4`.
|`precision_step` |The precision step (influences the number of terms
generated for each number value). Defaults to `16`.

|`boost` |The boost value. Defaults to `1.0`.

Expand Down
5 changes: 0 additions & 5 deletions src/main/java/org/elasticsearch/index/analysis/NumericDateAnalyzer.java
View file
Open in desktop
Expand Up @@ -19,7 +19,6 @@

package org.elasticsearch.index.analysis;

import org.apache.lucene.util.NumericUtils;
import org.joda.time.format.DateTimeFormatter;

import java.io.IOException;
Expand All @@ -34,10 +33,6 @@ public class NumericDateAnalyzer extends NumericAnalyzer<NumericDateTokenizer> {

private final DateTimeFormatter dateTimeFormatter;

public NumericDateAnalyzer(DateTimeFormatter dateTimeFormatter) {
this(NumericUtils.PRECISION_STEP_DEFAULT, dateTimeFormatter);
}

public NumericDateAnalyzer(int precisionStep, DateTimeFormatter dateTimeFormatter) {
this.precisionStep = precisionStep;
this.dateTimeFormatter = dateTimeFormatter;
Expand Down
5 changes: 0 additions & 5 deletions src/main/java/org/elasticsearch/index/analysis/NumericDoubleAnalyzer.java
View file
Open in desktop
Expand Up @@ -20,7 +20,6 @@
package org.elasticsearch.index.analysis;

import com.carrotsearch.hppc.IntObjectOpenHashMap;
import org.apache.lucene.util.NumericUtils;

import java.io.IOException;
import java.io.Reader;
Expand Down Expand Up @@ -50,10 +49,6 @@ public static NamedAnalyzer buildNamedAnalyzer(int precisionStep) {

private final int precisionStep;

public NumericDoubleAnalyzer() {
this(NumericUtils.PRECISION_STEP_DEFAULT);
}

public NumericDoubleAnalyzer(int precisionStep) {
this.precisionStep = precisionStep;
}
Expand Down
5 changes: 0 additions & 5 deletions src/main/java/org/elasticsearch/index/analysis/NumericFloatAnalyzer.java
View file
Open in desktop
Expand Up @@ -20,7 +20,6 @@
package org.elasticsearch.index.analysis;

import com.carrotsearch.hppc.IntObjectOpenHashMap;
import org.apache.lucene.util.NumericUtils;

import java.io.IOException;
import java.io.Reader;
Expand Down Expand Up @@ -50,10 +49,6 @@ public static NamedAnalyzer buildNamedAnalyzer(int precisionStep) {

private final int precisionStep;

public NumericFloatAnalyzer() {
this(NumericUtils.PRECISION_STEP_DEFAULT);
}

public NumericFloatAnalyzer(int precisionStep) {
this.precisionStep = precisionStep;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems unused anyways I think we should just drop these default ctors?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good. I didnt yet look to see how they are used.

Expand Down
5 changes: 0 additions & 5 deletions src/main/java/org/elasticsearch/index/analysis/NumericIntegerAnalyzer.java
View file
Open in desktop
Expand Up @@ -20,7 +20,6 @@
package org.elasticsearch.index.analysis;

import com.carrotsearch.hppc.IntObjectOpenHashMap;
import org.apache.lucene.util.NumericUtils;

import java.io.IOException;
import java.io.Reader;
Expand Down Expand Up @@ -50,10 +49,6 @@ public static NamedAnalyzer buildNamedAnalyzer(int precisionStep) {

private final int precisionStep;

public NumericIntegerAnalyzer() {
this(NumericUtils.PRECISION_STEP_DEFAULT);
}

public NumericIntegerAnalyzer(int precisionStep) {
this.precisionStep = precisionStep;
}
Expand Down
5 changes: 0 additions & 5 deletions src/main/java/org/elasticsearch/index/analysis/NumericLongAnalyzer.java
View file
Open in desktop
Expand Up @@ -20,7 +20,6 @@
package org.elasticsearch.index.analysis;

import com.carrotsearch.hppc.IntObjectOpenHashMap;
import org.apache.lucene.util.NumericUtils;

import java.io.IOException;
import java.io.Reader;
Expand Down Expand Up @@ -50,10 +49,6 @@ public static NamedAnalyzer buildNamedAnalyzer(int precisionStep) {

private final int precisionStep;

public NumericLongAnalyzer() {
this(NumericUtils.PRECISION_STEP_DEFAULT);
}

public NumericLongAnalyzer(int precisionStep) {
this.precisionStep = precisionStep;
}
Expand Down
6 changes: 3 additions & 3 deletions src/main/java/org/elasticsearch/index/mapper/core/ByteFieldMapper.java
View file
Open in desktop
Expand Up @@ -78,7 +78,7 @@ public static class Builder extends NumberFieldMapper.Builder<Builder, ByteField
protected Byte nullValue = Defaults.NULL_VALUE;

public Builder(String name) {
super(name, new FieldType(Defaults.FIELD_TYPE));
super(name, new FieldType(Defaults.FIELD_TYPE), Defaults.PRECISION_STEP_8_BIT);
builder = this;
}

Expand All @@ -91,7 +91,7 @@ public Builder nullValue(byte nullValue) {
public ByteFieldMapper build(BuilderContext context) {
fieldType.setOmitNorms(fieldType.omitNorms() && boost == 1.0f);
ByteFieldMapper fieldMapper = new ByteFieldMapper(buildNames(context),
precisionStep, boost, fieldType, docValues, nullValue, ignoreMalformed(context),
fieldType.numericPrecisionStep(), boost, fieldType, docValues, nullValue, ignoreMalformed(context),
coerce(context), postingsProvider, docValuesProvider, similarity, normsLoading,
fieldDataSettings, context.indexSettings(), multiFieldsBuilder.build(this, context), copyTo);
fieldMapper.includeInAll(includeInAll);
Expand Down Expand Up @@ -346,7 +346,7 @@ public void merge(Mapper mergeWith, MergeContext mergeContext) throws MergeMappi
protected void doXContentBody(XContentBuilder builder, boolean includeDefaults, Params params) throws IOException {
super.doXContentBody(builder, includeDefaults, params);

if (includeDefaults || precisionStep != Defaults.PRECISION_STEP) {
if (includeDefaults || precisionStep != Defaults.PRECISION_STEP_8_BIT) {
builder.field("precision_step", precisionStep);
}
if (includeDefaults || nullValue != null) {
Expand Down
6 changes: 3 additions & 3 deletions src/main/java/org/elasticsearch/index/mapper/core/DateFieldMapper.java
View file
Open in desktop
Expand Up @@ -97,7 +97,7 @@ public static class Builder extends NumberFieldMapper.Builder<Builder, DateField
private Locale locale;

public Builder(String name) {
super(name, new FieldType(Defaults.FIELD_TYPE));
super(name, new FieldType(Defaults.FIELD_TYPE), Defaults.PRECISION_STEP_64_BIT);
builder = this;
// do *NOT* rely on the default locale
locale = Locale.ROOT;
Expand Down Expand Up @@ -130,7 +130,7 @@ public DateFieldMapper build(BuilderContext context) {
dateTimeFormatter = new FormatDateTimeFormatter(dateTimeFormatter.format(), dateTimeFormatter.parser(), dateTimeFormatter.printer(), locale);
}
DateFieldMapper fieldMapper = new DateFieldMapper(buildNames(context), dateTimeFormatter,
precisionStep, boost, fieldType, docValues, nullValue, timeUnit, roundCeil, ignoreMalformed(context), coerce(context),
fieldType.numericPrecisionStep(), boost, fieldType, docValues, nullValue, timeUnit, roundCeil, ignoreMalformed(context), coerce(context),
postingsProvider, docValuesProvider, similarity, normsLoading, fieldDataSettings, context.indexSettings(),
multiFieldsBuilder.build(this, context), copyTo);
fieldMapper.includeInAll(includeInAll);
Expand Down Expand Up @@ -523,7 +523,7 @@ public void merge(Mapper mergeWith, MergeContext mergeContext) throws MergeMappi
protected void doXContentBody(XContentBuilder builder, boolean includeDefaults, Params params) throws IOException {
super.doXContentBody(builder, includeDefaults, params);

if (includeDefaults || precisionStep != Defaults.PRECISION_STEP) {
if (includeDefaults || precisionStep != Defaults.PRECISION_STEP_64_BIT) {
builder.field("precision_step", precisionStep);
}
builder.field("format", dateTimeFormatter.format());
Expand Down
6 changes: 3 additions & 3 deletions src/main/java/org/elasticsearch/index/mapper/core/DoubleFieldMapper.java
View file
Open in desktop
Expand Up @@ -82,7 +82,7 @@ public static class Builder extends NumberFieldMapper.Builder<Builder, DoubleFie
protected Double nullValue = Defaults.NULL_VALUE;

public Builder(String name) {
super(name, new FieldType(Defaults.FIELD_TYPE));
super(name, new FieldType(Defaults.FIELD_TYPE), Defaults.PRECISION_STEP_64_BIT);
builder = this;
}

Expand All @@ -95,7 +95,7 @@ public Builder nullValue(double nullValue) {
public DoubleFieldMapper build(BuilderContext context) {
fieldType.setOmitNorms(fieldType.omitNorms() && boost == 1.0f);
DoubleFieldMapper fieldMapper = new DoubleFieldMapper(buildNames(context),
precisionStep, boost, fieldType, docValues, nullValue, ignoreMalformed(context), coerce(context), postingsProvider,
fieldType.numericPrecisionStep(), boost, fieldType, docValues, nullValue, ignoreMalformed(context), coerce(context), postingsProvider,
docValuesProvider, similarity, normsLoading, fieldDataSettings, context.indexSettings(), multiFieldsBuilder.build(this, context), copyTo);
fieldMapper.includeInAll(includeInAll);
return fieldMapper;
Expand Down Expand Up @@ -348,7 +348,7 @@ public void merge(Mapper mergeWith, MergeContext mergeContext) throws MergeMappi
protected void doXContentBody(XContentBuilder builder, boolean includeDefaults, Params params) throws IOException {
super.doXContentBody(builder, includeDefaults, params);

if (includeDefaults || precisionStep != Defaults.PRECISION_STEP) {
if (includeDefaults || precisionStep != Defaults.PRECISION_STEP_64_BIT) {
builder.field("precision_step", precisionStep);
}
if (includeDefaults || nullValue != null) {
Expand Down
6 changes: 3 additions & 3 deletions src/main/java/org/elasticsearch/index/mapper/core/FloatFieldMapper.java
View file
Open in desktop
Expand Up @@ -83,7 +83,7 @@ public static class Builder extends NumberFieldMapper.Builder<Builder, FloatFiel
protected Float nullValue = Defaults.NULL_VALUE;

public Builder(String name) {
super(name, new FieldType(Defaults.FIELD_TYPE));
super(name, new FieldType(Defaults.FIELD_TYPE), Defaults.PRECISION_STEP_32_BIT);
builder = this;
}

Expand All @@ -96,7 +96,7 @@ public Builder nullValue(float nullValue) {
public FloatFieldMapper build(BuilderContext context) {
fieldType.setOmitNorms(fieldType.omitNorms() && boost == 1.0f);
FloatFieldMapper fieldMapper = new FloatFieldMapper(buildNames(context),
precisionStep, boost, fieldType, docValues, nullValue, ignoreMalformed(context), coerce(context), postingsProvider,
fieldType.numericPrecisionStep(), boost, fieldType, docValues, nullValue, ignoreMalformed(context), coerce(context), postingsProvider,
docValuesProvider, similarity, normsLoading, fieldDataSettings, context.indexSettings(), multiFieldsBuilder.build(this, context), copyTo);
fieldMapper.includeInAll(includeInAll);
return fieldMapper;
Expand Down Expand Up @@ -354,7 +354,7 @@ public void merge(Mapper mergeWith, MergeContext mergeContext) throws MergeMappi
protected void doXContentBody(XContentBuilder builder, boolean includeDefaults, Params params) throws IOException {
super.doXContentBody(builder, includeDefaults, params);

if (includeDefaults || precisionStep != Defaults.PRECISION_STEP) {
if (includeDefaults || precisionStep != Defaults.PRECISION_STEP_32_BIT) {
builder.field("precision_step", precisionStep);
}
if (includeDefaults || nullValue != null) {
Expand Down
6 changes: 3 additions & 3 deletions src/main/java/org/elasticsearch/index/mapper/core/IntegerFieldMapper.java
View file
Open in desktop
Expand Up @@ -79,7 +79,7 @@ public static class Builder extends NumberFieldMapper.Builder<Builder, IntegerFi
protected Integer nullValue = Defaults.NULL_VALUE;

public Builder(String name) {
super(name, new FieldType(Defaults.FIELD_TYPE));
super(name, new FieldType(Defaults.FIELD_TYPE), Defaults.PRECISION_STEP_32_BIT);
builder = this;
}

Expand All @@ -91,7 +91,7 @@ public Builder nullValue(int nullValue) {
@Override
public IntegerFieldMapper build(BuilderContext context) {
fieldType.setOmitNorms(fieldType.omitNorms() && boost == 1.0f);
IntegerFieldMapper fieldMapper = new IntegerFieldMapper(buildNames(context), precisionStep, boost, fieldType, docValues,
IntegerFieldMapper fieldMapper = new IntegerFieldMapper(buildNames(context), fieldType.numericPrecisionStep(), boost, fieldType, docValues,
nullValue, ignoreMalformed(context), coerce(context), postingsProvider, docValuesProvider, similarity, normsLoading, fieldDataSettings,
context.indexSettings(), multiFieldsBuilder.build(this, context), copyTo);
fieldMapper.includeInAll(includeInAll);
Expand Down Expand Up @@ -349,7 +349,7 @@ public void merge(Mapper mergeWith, MergeContext mergeContext) throws MergeMappi
protected void doXContentBody(XContentBuilder builder, boolean includeDefaults, Params params) throws IOException {
super.doXContentBody(builder, includeDefaults, params);

if (includeDefaults || precisionStep != Defaults.PRECISION_STEP) {
if (includeDefaults || precisionStep != Defaults.PRECISION_STEP_32_BIT) {
builder.field("precision_step", precisionStep);
}
if (includeDefaults || nullValue != null) {
Expand Down
6 changes: 3 additions & 3 deletions src/main/java/org/elasticsearch/index/mapper/core/LongFieldMapper.java
View file
Open in desktop
Expand Up @@ -79,7 +79,7 @@ public static class Builder extends NumberFieldMapper.Builder<Builder, LongField
protected Long nullValue = Defaults.NULL_VALUE;

public Builder(String name) {
super(name, new FieldType(Defaults.FIELD_TYPE));
super(name, new FieldType(Defaults.FIELD_TYPE), Defaults.PRECISION_STEP_64_BIT);
builder = this;
}

Expand All @@ -91,7 +91,7 @@ public Builder nullValue(long nullValue) {
@Override
public LongFieldMapper build(BuilderContext context) {
fieldType.setOmitNorms(fieldType.omitNorms() && boost == 1.0f);
LongFieldMapper fieldMapper = new LongFieldMapper(buildNames(context), precisionStep, boost, fieldType, docValues, nullValue,
LongFieldMapper fieldMapper = new LongFieldMapper(buildNames(context), fieldType.numericPrecisionStep(), boost, fieldType, docValues, nullValue,
ignoreMalformed(context), coerce(context), postingsProvider, docValuesProvider, similarity, normsLoading,
fieldDataSettings, context.indexSettings(), multiFieldsBuilder.build(this, context), copyTo);
fieldMapper.includeInAll(includeInAll);
Expand Down Expand Up @@ -331,7 +331,7 @@ public void merge(Mapper mergeWith, MergeContext mergeContext) throws MergeMappi
protected void doXContentBody(XContentBuilder builder, boolean includeDefaults, Params params) throws IOException {
super.doXContentBody(builder, includeDefaults, params);

if (includeDefaults || precisionStep != Defaults.PRECISION_STEP) {
if (includeDefaults || precisionStep != Defaults.PRECISION_STEP_64_BIT) {
builder.field("precision_step", precisionStep);
}
if (includeDefaults || nullValue != null) {
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/org/elasticsearch/index/mapper/core/Murmur3FieldMapper.java
View file
Open in desktop
Expand Up @@ -50,15 +50,15 @@ public static class Defaults extends LongFieldMapper.Defaults {
public static class Builder extends NumberFieldMapper.Builder<Builder, Murmur3FieldMapper> {

public Builder(String name) {
super(name, new FieldType(Defaults.FIELD_TYPE));
super(name, new FieldType(Defaults.FIELD_TYPE), Integer.MAX_VALUE);
builder = this;
builder.precisionStep(Integer.MAX_VALUE);
}

@Override
public Murmur3FieldMapper build(BuilderContext context) {
fieldType.setOmitNorms(fieldType.omitNorms() && boost == 1.0f);
Murmur3FieldMapper fieldMapper = new Murmur3FieldMapper(buildNames(context), precisionStep, boost, fieldType, docValues, ~0L,
Murmur3FieldMapper fieldMapper = new Murmur3FieldMapper(buildNames(context), fieldType.numericPrecisionStep(), boost, fieldType, docValues, ~0L,
ignoreMalformed(context), coerce(context), postingsProvider, docValuesProvider, similarity, normsLoading,
fieldDataSettings, context.indexSettings(), multiFieldsBuilder.build(this, context), copyTo);
fieldMapper.includeInAll(includeInAll);
Expand Down