New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make boolean conversion strict #22200

Merged
merged 26 commits into from Jan 19, 2017

Conversation

Projects
None yet
6 participants
@danielmitterdorfer
Member

danielmitterdorfer commented Dec 15, 2016

This PR removes all leniency in the conversion of Strings to booleans: "true" is converted to the boolean value true, "false" is converted to the boolean value false. Everything else raises an error.

This applies to:

  • All settings (index settings, cluster settings)
  • Boolean values in the REST API
  • Documents

Exceptions are:

  • For indices created before Elasticsearch 6, we still apply the old coercion logic (but we add deprecation logging).
  • We allow to omit the value "true" for request parameters in the REST API, i.e. ?pretty is interpreted as ?pretty=true.

Noteworthy / to discuss:

  • Bucket term aggregations on boolean fields continue to use 1 and 0 for the key, and the strings "true" and "false" for the key_as_string (I think that's fine).
  • Existing index templates can be read after an upgrade but creating a new index based on an index template with invalid boolean values will fail (see below for a discussion of the tradeoffs).

Example scenario (the lowercase field of my_custom_pattern_analyzer is a boolean field):

  1. Create the following index template on Elasticsearch 5.x:
DELETE /_template/default_template

PUT /_template/default_template
{
  "template": "*",
    "settings": {
        "analysis": {
            "analyzer": {
                "my_custom_pattern_analyzer": {
                    "type": "pattern",
                    "lowercase": 0
                }
            }
        }    
    }  
}
  1. Upgrade to Elasticsearch 6

  2. Create the following index:

DELETE /test

PUT /test

Result: Creating the index will fail, because the lowercase setting is invalid for a 6.x index.

Analysis: When a new index is created, MetaDataCreateIndexService will load all matching index templates, merge them settings and apply request settings overrides. By the time the index is created we have no clue where the affected setting came from and before that we don't have enough information to know a value's type.

Discussion:

I can think of the following options to address this:

  1. Keep the code as it is but highlight this specific behavior in the migration docs (and the migration plugin?) to ensure that users are aware and can take appropriate action before the upgrade.
  2. Be lenient when we find a matching index template (I'd rather avoid this).
  3. Add additional metadata to index templates to store the Elasticsearch version when an index template has been created. Also reimplement the merging logic so we know a setting's origin.

To me, option 1 seems the only feasible one (and the doc update is contained in the PR) but I want to put that up for discussion.

Make boolean conversion strict
This PR removes all leniency in the conversion of Strings to booleans: "true"
is converted to the boolean value `true`, "false" is converted to the boolean
value `false`. Everything else raises an error.
@jasontedor

This comment has been minimized.

Show comment
Hide comment
@jasontedor

jasontedor Dec 15, 2016

Member

+1

This:

on -> true
no -> false

has always offended me; it's too easy to make a typo.

Member

jasontedor commented Dec 15, 2016

+1

This:

on -> true
no -> false

has always offended me; it's too easy to make a typo.

@danielmitterdorfer

This comment has been minimized.

Show comment
Hide comment
@danielmitterdorfer

danielmitterdorfer Dec 16, 2016

Member

@nik9000 Thanks for your comments. They all make sense and I'll address them.

Member

danielmitterdorfer commented Dec 16, 2016

@nik9000 Thanks for your comments. They all make sense and I'll address them.

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley Dec 16, 2016

Member

I don't care about strict booleans in settings, mappings, query string params, etc. The one place I do care about it is in boolean fields in documents.

My initial thought was: we allow coercion on other field types by default, why are we suddenly strict on boolean fields? Why not also accept 0/1 as false/true? Of course in Perl, the numbers 0 and 1 might be rendered as strings in JSON, so you'd need to accept "0" and "1" as well. But "0" is true in many languages, and in fact 0 is true in Ruby as well.

In other words, whatever coercion you allow, somebody is going to be surprised. Much better to shout loud, shout early. So what I'd like to see:

  • Deprecation logging for any use of boolean values which are not true, false,"true", or "false"
  • This change is made for 6.0, but only for new indices.
  • The boolean semantics don't change for indices created in 5.x, but they continue to emit deprecation logging
Member

clintongormley commented Dec 16, 2016

I don't care about strict booleans in settings, mappings, query string params, etc. The one place I do care about it is in boolean fields in documents.

My initial thought was: we allow coercion on other field types by default, why are we suddenly strict on boolean fields? Why not also accept 0/1 as false/true? Of course in Perl, the numbers 0 and 1 might be rendered as strings in JSON, so you'd need to accept "0" and "1" as well. But "0" is true in many languages, and in fact 0 is true in Ruby as well.

In other words, whatever coercion you allow, somebody is going to be surprised. Much better to shout loud, shout early. So what I'd like to see:

  • Deprecation logging for any use of boolean values which are not true, false,"true", or "false"
  • This change is made for 6.0, but only for new indices.
  • The boolean semantics don't change for indices created in 5.x, but they continue to emit deprecation logging
@nik9000

This comment has been minimized.

Show comment
Hide comment
@nik9000

nik9000 Dec 16, 2016

Contributor

Deprecation logging for any use of boolean values which are not true, false,"true", or "false"

Everywhere. In the API and in documents.

This change is made for 6.0, but only for new indices.

That makes this much more complicated but I think it is worth it. @danielmitterdorfer, for testing see this trick that lets you create an index on the current version that "looks like" it was made on a previous version. It makes testing this kind of thing much easier.

Contributor

nik9000 commented Dec 16, 2016

Deprecation logging for any use of boolean values which are not true, false,"true", or "false"

Everywhere. In the API and in documents.

This change is made for 6.0, but only for new indices.

That makes this much more complicated but I think it is worth it. @danielmitterdorfer, for testing see this trick that lets you create an index on the current version that "looks like" it was made on a previous version. It makes testing this kind of thing much easier.

@danielmitterdorfer

This comment has been minimized.

Show comment
Hide comment
@danielmitterdorfer

danielmitterdorfer Dec 19, 2016

Member

I've addressed all initial review comments now (although I did not comment on each one). Thanks to @nik9000 for the pointer to the BWC trick. I'll start now to add BWC.

Member

danielmitterdorfer commented Dec 19, 2016

I've addressed all initial review comments now (although I did not comment on each one). Thanks to @nik9000 for the pointer to the BWC trick. I'll start now to add BWC.

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley Dec 19, 2016

Member

This change is made for 6.0, but only for new indices.

To clarify, I meant this comment for boolean fields only, not for other booleans associated with an index (including mappings, settings etc).

Member

clintongormley commented Dec 19, 2016

This change is made for 6.0, but only for new indices.

To clarify, I meant this comment for boolean fields only, not for other booleans associated with an index (including mappings, settings etc).

@danielmitterdorfer danielmitterdorfer added review and removed WIP labels Jan 3, 2017

@danielmitterdorfer

This comment has been minimized.

Show comment
Hide comment
@danielmitterdorfer

danielmitterdorfer Jan 3, 2017

Member

After having worked on BWC, I've updated the ticket description. Cleanup of the backwards compatibility code will be performed in #22298. The backport to 5.x will require a separate PR.

Member

danielmitterdorfer commented Jan 3, 2017

After having worked on BWC, I've updated the ticket description. Cleanup of the backwards compatibility code will be performed in #22298. The backport to 5.x will require a separate PR.

@danielmitterdorfer

This comment has been minimized.

Show comment
Hide comment
@danielmitterdorfer

danielmitterdorfer Jan 10, 2017

Member

@nik9000 Can you spare some cycles on the review?

Member

danielmitterdorfer commented Jan 10, 2017

@nik9000 Can you spare some cycles on the review?

@nik9000

This comment has been minimized.

Show comment
Hide comment
@nik9000

nik9000 Jan 10, 2017

Contributor

@nik9000 Can you spare some cycles on the review?

Sure! I was hoping someone else would give it a deep dive but I will. Not today, though. I'll save it for tomorrow I think!

Contributor

nik9000 commented Jan 10, 2017

@nik9000 Can you spare some cycles on the review?

Sure! I was hoping someone else would give it a deep dive but I will. Not today, though. I'll save it for tomorrow I think!

@nik9000

I left a bunch of comments, mixing the mundane ("please add a space here") with the important ("we should remove the default here" and "I think we should support this behavior you've documented as unsupported").

Show outdated Hide outdated ...earch/action/admin/cluster/snapshots/restore/RestoreSnapshotRequest.java
* @param offset offset to start
* @param length length to check
*
* @deprecated Only kept to provide automatic upgrades for pre 6.0 indices. Use {@link #isBoolean(char[], int, int)} instead.

This comment has been minimized.

@nik9000

nik9000 Jan 11, 2017

Contributor

I don't believe this is only kept for BWC. You use this to parse _source above.

@nik9000

nik9000 Jan 11, 2017

Contributor

I don't believe this is only kept for BWC. You use this to parse _source above.

This comment has been minimized.

@danielmitterdorfer

danielmitterdorfer Jan 12, 2017

Member

I checked usages of all Booleans.*lenient() but I didn't see the code you're referring to. However, if it's just about the wording of the Javadoc I'm fine with changing it. The gist of the Javadocs should be: "Don't use it, we just need it during a transition phase and it will be removed."

@danielmitterdorfer

danielmitterdorfer Jan 12, 2017

Member

I checked usages of all Booleans.*lenient() but I didn't see the code you're referring to. However, if it's just about the wording of the Javadoc I'm fine with changing it. The gist of the Javadocs should be: "Don't use it, we just need it during a transition phase and it will be removed."

This comment has been minimized.

@nik9000

nik9000 Jan 12, 2017

Contributor

I think you use it indirectly through parser. isBooleanValueLenient.

@nik9000

nik9000 Jan 12, 2017

Contributor

I think you use it indirectly through parser. isBooleanValueLenient.

This comment has been minimized.

@danielmitterdorfer

danielmitterdorfer Jan 13, 2017

Member

Oh, that one. XContentParser#isBooleanValueLenient() is here for BWC as well so I think the comment is fine.

@danielmitterdorfer

danielmitterdorfer Jan 13, 2017

Member

Oh, that one. XContentParser#isBooleanValueLenient() is here for BWC as well so I think the comment is fine.

*/
@Deprecated
public Boolean getAsBooleanLenientForPreEs6Indices(Version indexVersion, String setting, Boolean defaultValue) {
if (indexVersion.before(Version.V_6_0_0_alpha1_UNRELEASED)) {

This comment has been minimized.

@nik9000

nik9000 Jan 11, 2017

Contributor

In the test for this you can assert that the minimum supported version is before 6.0.0_alpha1 with a message to remove the whole method after the assertion fails.

@nik9000

nik9000 Jan 11, 2017

Contributor

In the test for this you can assert that the minimum supported version is before 6.0.0_alpha1 with a message to remove the whole method after the assertion fails.

This comment has been minimized.

@danielmitterdorfer

danielmitterdorfer Jan 12, 2017

Member

Clever! I'll do that.

@danielmitterdorfer

danielmitterdorfer Jan 12, 2017

Member

Clever! I'll do that.

*/
boolean isBooleanValue() throws IOException;
@Deprecated
boolean isBooleanValueLenient() throws IOException;

This comment has been minimized.

@nik9000

nik9000 Jan 11, 2017

Contributor

Same thing as last comment.

@nik9000

nik9000 Jan 11, 2017

Contributor

Same thing as last comment.

//Only emit a warning if the setting's value is not a proper boolean
final String value = get(setting, "false");
if (Booleans.isBoolean(value) == false) {
@SuppressWarnings("deprecation")

This comment has been minimized.

@nik9000

nik9000 Jan 11, 2017

Contributor

I don't believe it does anything to suppress deprecated warnings when you are in a deprecated method.

@nik9000

nik9000 Jan 11, 2017

Contributor

I don't believe it does anything to suppress deprecated warnings when you are in a deprecated method.

This comment has been minimized.

@danielmitterdorfer

danielmitterdorfer Jan 12, 2017

Member

My rationale around this whole suppression was: deprecated methods get the @Deprecate annotation but I don't want to spit out unnecessary additional deprecation warnings when I (intentionally) call deprecated API in these methods.

@danielmitterdorfer

danielmitterdorfer Jan 12, 2017

Member

My rationale around this whole suppression was: deprecated methods get the @Deprecate annotation but I don't want to spit out unnecessary additional deprecation warnings when I (intentionally) call deprecated API in these methods.

This comment has been minimized.

@nik9000

nik9000 Jan 12, 2017

Contributor

Fine by me.

@nik9000

nik9000 Jan 12, 2017

Contributor

Fine by me.

public void testAddWithValidSourceValueIsAccepted() throws Exception {
XContentParser parser = createParser(XContentFactory.jsonBuilder()
.startObject()

This comment has been minimized.

@nik9000

nik9000 Jan 11, 2017

Contributor

Thanks for indenting!

@nik9000

nik9000 Jan 11, 2017

Contributor

Thanks for indenting!

.addMapping("type1", XContentFactory.jsonBuilder().startObject().startObject("type1").startObject("properties")
.startObject("field2").field("type", "text").field("store", "no").endObject()
.endObject().endObject().endObject())
.addMapping("type1", XContentFactory.jsonBuilder()

This comment has been minimized.

@nik9000

nik9000 Jan 11, 2017

Contributor

👍

@nik9000

nik9000 Jan 11, 2017

Contributor

👍

Show outdated Hide outdated core/src/test/java/org/elasticsearch/rest/BaseRestHandlerTests.java
Show outdated Hide outdated docs/reference/migration/migrate_6_0/mappings.asciidoc
Show outdated Hide outdated docs/reference/migration/migrate_6_0/mappings.asciidoc
@danielmitterdorfer

This comment has been minimized.

Show comment
Hide comment
@danielmitterdorfer

danielmitterdorfer Jan 13, 2017

Member

@nik9000 Thanks for your comments, very helpful! I've pushed a few more commits that address all your comments.

Member

danielmitterdorfer commented Jan 13, 2017

@nik9000 Thanks for your comments, very helpful! I've pushed a few more commits that address all your comments.

@nik9000

LGTM but I'm glad it'll get another set of eyes before merging.

Show outdated Hide outdated core/src/main/java/org/elasticsearch/cluster/metadata/IndexMetaData.java
Show outdated Hide outdated core/src/main/java/org/elasticsearch/common/TriFunction.java
@@ -256,7 +257,7 @@ public void addIndexStore(String type, Function<IndexSettings, IndexStore> provi
* @param name Name of the SimilarityProvider
* @param similarity SimilarityProvider to register

This comment has been minimized.

@nik9000

nik9000 Jan 17, 2017

Contributor

I think I'd prefer to have an interface for building the similarity. That way you can document what the two Settings are.

@nik9000

nik9000 Jan 17, 2017

Contributor

I think I'd prefer to have an interface for building the similarity. That way you can document what the two Settings are.

This comment has been minimized.

@danielmitterdorfer

danielmitterdorfer Jan 17, 2017

Member

I thought about this too but I figured that given it is temporary (i.e. this will be again a BiFunction after the cleanup task #22298 is done) it might be ok. But I might need to ponder once more.

@danielmitterdorfer

danielmitterdorfer Jan 17, 2017

Member

I thought about this too but I figured that given it is temporary (i.e. this will be again a BiFunction after the cleanup task #22298 is done) it might be ok. But I might need to ponder once more.

@dakrone

LGTM, I left pretty minor comments which are pretty much optional, happy to have this go in!

Show outdated Hide outdated core/src/main/java/org/elasticsearch/cluster/metadata/IndexMetaData.java
Show outdated Hide outdated core/src/main/java/org/elasticsearch/common/Booleans.java
Show outdated Hide outdated core/src/main/java/org/elasticsearch/common/Booleans.java
* @deprecated Only kept to provide automatic upgrades for pre 6.0 indices. Use {@link #parseBoolean(char[], int, int, boolean)} instead
*/
@Deprecated
public static boolean parseBooleanLenient(char[] text, int offset, int length, boolean defaultValue) {

This comment has been minimized.

@dakrone

dakrone Jan 17, 2017

Member

Does this need to be public, or can we make it private to this class and force everyone to go through the String version of the parseBooleanLenient? (I did a cursory glance and didn't see usages, but it's possible I missed some)

@dakrone

dakrone Jan 17, 2017

Member

Does this need to be public, or can we make it private to this class and force everyone to go through the String version of the parseBooleanLenient? (I did a cursory glance and didn't see usages, but it's possible I missed some)

This comment has been minimized.

@dakrone

dakrone Jan 17, 2017

Member

Ah nevermind, I see where we use it :-/

@dakrone

dakrone Jan 17, 2017

Member

Ah nevermind, I see where we use it :-/

Show outdated Hide outdated core/src/main/java/org/elasticsearch/common/TriFunction.java
Show outdated Hide outdated ...in/java/org/elasticsearch/common/xcontent/support/XContentMapValues.java
Show outdated Hide outdated ...in/java/org/elasticsearch/common/xcontent/support/XContentMapValues.java
// TODO: remove this leniency in 6.0
if (BOOLEAN_STRINGS.contains(node.toString()) == false) {
DEPRECATION_LOGGER.deprecated("Expected a boolean for property [{}] but got [{}]", name, node);
//TODO 22298: Remove this method and have all call-sites use <code>XContentMapValues.nodeBooleanValue(node)</code> directly.

This comment has been minimized.

@dakrone

dakrone Jan 17, 2017

Member

You could add an assert that it's not ES 7.x here so we know to remove it

@dakrone

dakrone Jan 17, 2017

Member

You could add an assert that it's not ES 7.x here so we know to remove it

Show outdated Hide outdated core/src/main/java/org/elasticsearch/index/mapper/TypeParsers.java
Show outdated Hide outdated core/src/main/java/org/elasticsearch/index/mapper/TypeParsers.java

@danielmitterdorfer danielmitterdorfer merged commit aece89d into elastic:master Jan 19, 2017

2 checks passed

CLA Commit author has signed the CLA
Details
elasticsearch-ci Build finished.
Details

tlrx added a commit to tlrx/elasticsearch that referenced this pull request Jan 19, 2017

Since #22200 the Settings class now uses a DeprecationLogger. It basi…
…cally means that no settings should be used before Log4j's status logger has been configured without configuration by LogConfigurator.configureWithoutConfig() method.

danielmitterdorfer added a commit to elastic/rally that referenced this pull request Jan 20, 2017

@Grinnz Grinnz referenced this pull request Dec 8, 2017

Closed

Allow 1 and 0 as booleans? #27733

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment