Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolves #249: Collating index #307

Merged
merged 5 commits into from Feb 11, 2019
Merged
Changes from all commits
Commits
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -48,6 +48,9 @@
<module name="fdb-extensions_integrationTest" target="1.8" />
<module name="fdb-extensions_main" target="1.8" />
<module name="fdb-extensions_test" target="1.8" />
<module name="fdb-record-layer-icu_integrationTest" target="1.8" />
<module name="fdb-record-layer-icu_main" target="1.8" />
<module name="fdb-record-layer-icu_test" target="1.8" />
<module name="fdb-record-layer-core-shaded_integrationTest" target="1.8" />
<module name="fdb-record-layer-core-shaded_main" target="1.8" />
<module name="fdb-record-layer-core-shaded_test" target="1.8" />
@@ -58,4 +61,4 @@
<module name="fdb-record-layer_test" target="1.8" />
</bytecodeTargetLevel>
</component>
</project>
</project>
@@ -12,10 +12,11 @@
<option value="$PROJECT_DIR$/fdb-extensions" />
<option value="$PROJECT_DIR$/fdb-record-layer-core" />
<option value="$PROJECT_DIR$/fdb-record-layer-core-shaded" />
<option value="$PROJECT_DIR$/fdb-record-layer-icu" />
</set>
</option>
<option name="useAutoImport" value="true" />
</GradleProjectSettings>
</option>
</component>
</project>
</project>
@@ -162,3 +162,36 @@ QOS.ch (slf4j)
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Unicode, Inc (ICU4J)

Copyright © 1991-2018 Unicode, Inc. All rights reserved.
Distributed under the Terms of Use in http://www.unicode.org/copyright.html.

Permission is hereby granted, free of charge, to any person obtaining
a copy of the Unicode data files and any associated documentation
(the "Data Files") or Unicode software and any associated documentation
(the "Software") to deal in the Data Files or Software
without restriction, including without limitation the rights to use,
copy, modify, merge, publish, distribute, and/or sell copies of
the Data Files or Software, and to permit persons to whom the Data Files
or Software are furnished to do so, provided that either
(a) this copyright and permission notice appear with all copies
of the Data Files or Software, or
(b) this copyright and permission notice appear in associated
Documentation.

THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF
ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT OF THIRD PARTY RIGHTS.
IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS
NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL
DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE,
DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THE DATA FILES OR SOFTWARE.

Except as contained in this notice, the name of a copyright holder
shall not be used in advertising or otherwise to promote the sale,
use or other dealings in these Data Files or Software without prior
written authorization of the copyright holder.
@@ -19,7 +19,7 @@ As the [versioning guide](Versioning.md) details, it cannot always be determined
* **Performance** Improvement 4 [(Issue #NNN)](https://github.com/FoundationDB/fdb-record-layer/issues/NNN)
* **Performance** Improvement 5 [(Issue #NNN)](https://github.com/FoundationDB/fdb-record-layer/issues/NNN)
* **Feature** Feature 1 [(Issue #NNN)](https://github.com/FoundationDB/fdb-record-layer/issues/NNN)
* **Feature** Feature 2 [(Issue #NNN)](https://github.com/FoundationDB/fdb-record-layer/issues/NNN)
* **Feature** Collating indexes [(Issue #249)](https://github.com/FoundationDB/fdb-record-layer/issues/249)
* **Feature** Feature 3 [(Issue #NNN)](https://github.com/FoundationDB/fdb-record-layer/issues/NNN)
* **Feature** Feature 4 [(Issue #NNN)](https://github.com/FoundationDB/fdb-record-layer/issues/NNN)
* **Feature** Feature 5 [(Issue #NNN)](https://github.com/FoundationDB/fdb-record-layer/issues/NNN)
@@ -0,0 +1,183 @@
/*
* CollateFunctionKeyExpression.java
*
* This source file is part of the FoundationDB open source project
*
* Copyright 2015-2018 Apple Inc. and the FoundationDB project authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.apple.foundationdb.record.metadata.expressions;

import com.apple.foundationdb.API;
import com.apple.foundationdb.record.metadata.Key;
import com.apple.foundationdb.record.metadata.MetaDataException;
import com.apple.foundationdb.record.provider.common.text.TextCollator;
import com.apple.foundationdb.record.provider.common.text.TextCollatorRegistry;
import com.apple.foundationdb.record.provider.foundationdb.FDBRecord;
import com.google.protobuf.Message;

import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.Collections;
import java.util.List;
import java.util.function.Function;

/**
* {@code COLLATE} function.
*
* Turns a string into a locale-specific sort key.
*
* <p>
* The name of the function is determined by the underlying collation rules chosen.
* For the Java Platform's own collations, this is {@code collate_jre}.
*
* <p>
* The function takes the following arguments:<ol>
* <li>text field to be collated (mandatory)</li>
* <li>collation to use (optional)</li>
* <li>strength of character matching (optional)</li>
* </ol>
*
* <p>
* For example,
* <pre>
* Key.Expressions.function("collate_jre")
* </pre>
* is the default locale (from the JVM) and the default (weakest -- ignoring almost all character modifiers) strength.
*
* <pre>
* Key.Expressions.function("collate_jre", Key.Expressions.concat(Key.Expressions.field("text_field"),
* Key.Expressions.value("fr_CA"), Key.Expressions.value(TextCollator.Strength.SECONDARY)))
* </pre>
* is case-insensitive, accent-sensitive, Canadian French.
*
* @see TextCollator
*/
@API(API.Status.EXPERIMENTAL)
public class CollateFunctionKeyExpression extends FunctionKeyExpression implements QueryableKeyExpression {
@Nonnull
private final TextCollatorRegistry collatorRegistry;
@Nullable
private TextCollator invariableCollator;

protected CollateFunctionKeyExpression(@Nonnull TextCollatorRegistry collatorRegistry,
@Nonnull String name, @Nonnull KeyExpression arguments) {
super(name, arguments);
this.collatorRegistry = collatorRegistry;
this.invariableCollator = getInvariableCollator(collatorRegistry, arguments);
}

@Nullable
protected static TextCollator getInvariableCollator(@Nonnull TextCollatorRegistry collatorRegistry,
@Nonnull KeyExpression arguments) {
// If the locale and strength are missing or literals, we can lookup immediately.
if (arguments.getColumnSize() < 2) {
return collatorRegistry.getTextCollator();
}
if (arguments instanceof ThenKeyExpression) {
final List<KeyExpression> children = ((ThenKeyExpression)arguments).getChildren();
if (children.size() > 1 && !(children.get(1) instanceof LiteralKeyExpression<?>)) {
return null;
}
if (children.size() > 2 && !(children.get(2) instanceof LiteralKeyExpression<?>)) {
return null;
}
String locale = null;
Integer strength = null;
if (children.size() > 1) {
Object literal1 = ((LiteralKeyExpression<?>)children.get(1)).getValue();
if (literal1 instanceof String) {
locale = (String)literal1;
} else if (literal1 != null) {
return null;
}
}
if (children.size() > 2) {
Object literal2 = ((LiteralKeyExpression<?>)children.get(2)).getValue();
if (literal2 instanceof Number) {
strength = ((Number)literal2).intValue();
} else if (literal2 != null) {
return null;
}
}
if (locale == null) {
return strength == null ? collatorRegistry.getTextCollator() : collatorRegistry.getTextCollator(strength);
} else {
return strength == null ? collatorRegistry.getTextCollator(locale) : collatorRegistry.getTextCollator(locale, strength);

}
}
return null;
}

protected TextCollator getTextCollator(@Nonnull Key.Evaluated arguments) {
if (invariableCollator != null) {
return invariableCollator;
}
if (arguments.size() < 2) {
return collatorRegistry.getTextCollator();
}
final String locale = arguments.getString(1);
if (arguments.size() < 3) {
return locale == null ? collatorRegistry.getTextCollator() : collatorRegistry.getTextCollator(locale);
}
final int strength = (int)arguments.getLong(2);
return locale == null ? collatorRegistry.getTextCollator(strength) : collatorRegistry.getTextCollator(locale, strength);
}

@Override
public int getMinArguments() {
return 1;
}

@Override
public int getMaxArguments() {
return 3;
}

@Nonnull
@Override
public <M extends Message> List<Key.Evaluated> evaluateFunction(@Nullable FDBRecord<M> record,
@Nullable Message message,
@Nonnull Key.Evaluated arguments) {
final String value = arguments.getString(0);
if (value == null) {
return Collections.singletonList(Key.Evaluated.NULL);
}

final TextCollator textCollator = getTextCollator(arguments);
return Collections.singletonList(Key.Evaluated.scalar(textCollator.getKey(value)));
}

@Override
public boolean createsDuplicates() {
return getArguments().createsDuplicates();
}

@Override
public int getColumnSize() {
return 1;
}

@Nullable
@Override
public Function<Object, Object> getComparandConversionFunction() {
final TextCollator textCollator = invariableCollator;
if (textCollator == null) {
throw new MetaDataException("can only be used in queries when collation name is constant");
}
return o -> textCollator.getKey((String)o);
}
}
@@ -0,0 +1,53 @@
/*
* CollateFunctionKeyExpressionFactoryJRE.java
*
* This source file is part of the FoundationDB open source project
*
* Copyright 2015-2018 Apple Inc. and the FoundationDB project authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.apple.foundationdb.record.metadata.expressions;

import com.apple.foundationdb.API;
import com.google.auto.service.AutoService;
import com.apple.foundationdb.record.provider.common.text.TextCollatorRegistryJRE;

import javax.annotation.Nonnull;
import java.util.Collections;
import java.util.List;

/**
* Implemention of {@link CollateFunctionKeyExpression} using {@link TextCollatorRegistryJRE}.
*
*/
@AutoService(FunctionKeyExpression.Factory.class)
@API(API.Status.EXPERIMENTAL)
@SuppressWarnings("checkstyle:abbreviationaswordinname") // Allow JRE here.
public class CollateFunctionKeyExpressionFactoryJRE implements FunctionKeyExpression.Factory {
public static final String FUNCTION_NAME = "collate_jre";

@Nonnull
@Override
public List<FunctionKeyExpression.Builder> getBuilders() {
return Collections.singletonList(
new FunctionKeyExpression.BiFunctionBuilder(FUNCTION_NAME, CollateFunctionKeyExpressionJRE::new));
}

protected static class CollateFunctionKeyExpressionJRE extends CollateFunctionKeyExpression {
protected CollateFunctionKeyExpressionJRE(@Nonnull String name, @Nonnull KeyExpression arguments) {
super(TextCollatorRegistryJRE.instance(), name, arguments);
}
}
}
@@ -0,0 +1,69 @@
/*
* QueryableKeyExpression.java
*
* This source file is part of the FoundationDB open source project
*
* Copyright 2015-2018 Apple Inc. and the FoundationDB project authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.apple.foundationdb.record.metadata.expressions;

import com.apple.foundationdb.API;
import com.apple.foundationdb.record.EvaluationContext;
import com.apple.foundationdb.record.RecordCoreException;
import com.apple.foundationdb.record.metadata.Key;
import com.apple.foundationdb.record.provider.foundationdb.FDBRecord;
import com.apple.foundationdb.record.provider.foundationdb.FDBRecordStoreBase;
import com.google.protobuf.Message;

import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
import java.util.function.Function;

/**
* A {@link KeyExpression} that can be used with a {@link com.apple.foundationdb.record.query.expressions.QueryKeyExpressionWithComparison}.
*
* The index entries generated by the key expression are matched to the application of the key expression to the query record,
* with an optional conversion of any comparison operand.
*/
@API(API.Status.EXPERIMENTAL)
public interface QueryableKeyExpression extends KeyExpression {
@Nonnull
String getName();

@Nullable
default <M extends Message> Object evalForQuery(@Nonnull FDBRecordStoreBase<M> store, @Nonnull EvaluationContext context, @Nullable FDBRecord<M> record, @Nullable Message message) {
List<Key.Evaluated> keys = evaluateMessage(record, message);
if (keys.size() != 1) {
throw new RecordCoreException("Should evaluate to single key only");
}
Key.Evaluated key = keys.get(0);
if (keys.size() != 1) {
throw new RecordCoreException("Should evaluate to single key only");
}
return key.getObject(0);
}

/**
* Get a function to be applied to the comparison operand before compairing it with the application of the key expression
* to the record.
* @return a conversion function or {@code null} for no conversion
*/
@Nullable
default Function<Object, Object> getComparandConversionFunction() {
return null;
}
}
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.