Add support for specifying if a property is indexed

[silvolu]

There is no support for specified if a property is indexed. Any user who uses gcloud-ruby with existing data will have all their indexing specifications reset when doing a put.

In gcloud-node this is exposed as an 'excludeFromIndexes' boolean property that get's translated to the opposite value for 'Indexed' (the current name for the property in the API). This is because 'Indexed' can be misleading regarding the effect of setting the property (discussion here)

[blowmage]

Ugh, this comment should have been posted yesterday...

Yeah, I'm seeing the issue here. In the protobuf definition the default value for the Value's indexed is true. So every new Value is being flagged as indexed.

We can identify if the Value was flagged as indexed when updating an entity that was retrieved from the datastore:

entity = dataset.find "User", "mike"
entity["description"] = "New description"
entity.indexed? "description" #=> false

But, if you create a new Entity, how do we know whether property should be indexed or not? I mean, other than making the user configure it each time. Is there some sort of manifest we can get?

entity = Gcloud::Datastore::Entity.new
entity.key.kind = "User"
entity.key.name = "mike"
entity["name"] = "Mike Moore"
entity["description"] = "New description"
# How do we determine if description should be indexed when saved or not?
dataset.save entity

Here is what I'm thinking: we have an exclude_from_indexes method that is an array. Property names can be added to the exclude_from_indexes array so the property's protobuf Value object will be created with indexed = false when saving. When creating an Entity from a proto response the exclude_from_indexes array will be populated with the properties that had a Value object with indexed = false.

entity = Gcloud::Datastore::Entity.new
entity.key.kind = "User"
entity.key.name = "mike"
entity["name"] = "Mike Moore"
entity["description"] = "New description"
entity.exclude_from_indexes << "description"
dataset.save entity # description won't be indexed.

entity = dataset.find "User", "mike"
entity["description"] = "New description"
entity.exclude_from_indexes.include? "description" #=> true

[blowmage]

In discussion with @silvolu the exclude_from_indexes as an array approach fails on properties that are lists. How does the user specify which list values are flagged as indexed when being saved to Datastore? So in the following example the user may want just the first and last tag to be added to the index:

entity["tags"] = ["ruby", "c0dez", "h4x0rz", "software"]
entity.exclude_from_indexes << "tags" # Wait, not *all* of the tags! Fail...

This is a problem and so we need a new approach.

Proposal.new

Here is my updated proposal: we have two new methods on Entity, exclude_from_indexes?(name) that checks if the property gets the flag, and exclude_from_indexes!(name, value) which sets the flag.

entity["title"] = "Hello World!"
entity.exclude_from_indexes? "title" #=> nil/false
entity.exclude_from_indexes! "title", true
entity.exclude_from_indexes? "title" #=> true

We should also support setting this flag when the property value is set:

entity["title"] = "Hello World!", exclude_from_indexes: true

What about lists?

So far so good. But what about lists? The way we can support lists is to accept value that are an array. So, let's assume we want to add some tags to the index, but not all of them. We will set the tags in an array to use a Datastore List, and we can call exclude_from_indexes!(name, value) with an array of values to use for each item in the array.

entity["tags"] = ["ruby", "c0dez", "h4x0rz", "software"]
entity.exclude_from_indexes! "tags", [false, true, true, false]

If the flag set for an array property is not an array, then the value will be used for all elements in an array:

entity["tags"] = ["ruby", "c0dez", "h4x0rz", "software"]
# Add all tags to the index
# This would be the same as setting [false, false, false, false]
entity.exclude_from_indexes! "tags", false
# Exclude all tags from the index
# This would be the same as setting [true, true, true, true]
entity.exclude_from_indexes! "tags", true

We will also support setting the index flag in the same call that sets the property's value:

entity = Gcloud::Datastore::Entity.new
entity["tags"] = []
entity.exclude_from_indexes? "tags" #=> nil/false
entity["tags"] = ["ruby", "c0dez", "h4x0rz", "software"], exclude_from_indexes: [false, true, true, false]
entity.exclude_from_indexes? "tags" #=> [false, true, true, false]

What about index flags from Datastore?

And we populate the flags off of the protobuf data when returning objects from Datastore. So if a user wants to change a value and its indexed flag, they can:

entity = dataset.find "Post", "helloworld"
entity["tags"] #=> ["ruby", "c0dez", "h4x0rz", "software"]
entity.exclude_from_indexes? "tags" #=> [false, true, true, false]
entity["tags"] = ["ruby", "c0dez", "h4x0rz", "software", "n00b"]
entity.exclude_from_indexes! "tags", [false, true, true, false, true]
dataset.save entity

What if there are not enough flags for a list?

What happens when a array property changes and the flags are out of date? Then the default index value will be used. Most values will have the indexed flag to true, so an exclude flag of false.

entity["tags"] #=> ["ruby", "c0dez", "h4x0rz", "software"]
entity.exclude_from_indexes? "tags" #=> [false, true, true, false]
entity["tags"] = ["ruby", "c0dez", "h4x0rz", "software", "n00b"]
# The tag "n00b" will not be excluded
# This would be the same as setting [false, true, true, false, false]
dataset.save entity

[silvolu]

entity["tags"] = ["ruby", "c0dez", "h4x0rz", "software"], exclude_from_indexes: [false, true, true, false]
entity.exclude_from_indexes? "tags" #=> [false, true, true, false]

This is my only doubt: to manipulate indexes in a list, the developer will need to iterate two different lists. Something like entity["tags"]["h4x0rz"].exclude_from_indexed! true would be ideal.

[blowmage]

Something like entity["tags"]["h4x0rz"].exclude_from_indexed! true would be ideal.

I don't like the idea of wrapping every property value in an Entity in some Value object that knows if it can be excluded from the index. Or worse, monkey-patching Object. :) How about allowing entity.exclude_from_indexes! to also accept a block that will take each value in the list (or just the value in non-lists) and return true/false?

entity["tags"] = ["ruby", "c0dez", "h4x0rz", "software", "n00b"]
entity.exclude_from_indexes! "tags" do |tag|
  tag =~ /\A[a-z]*\z/i
end

I think in the case where there are multiple values in a list and only some of those values should be in the index, it is highly likely that there is a way to express the rule in code. This approach would allow the users to add the rule that would be evaluated when the entity is persisted, while still allowing a second explicit list to be maintained.

[silvolu]

Ok, let's try this approach!