Add schema update + index reloading [JIRA: RIAK-1591]

[rzezeski]

After a conversation with someone on IRC it came to my attention that Yokozuna should support schema updates + index reloading sooner rather than later. The user was performing these basic steps that would better be handled by Yokozuna itself.

1. Modify the schema.
2. Copy the schema to /data/yz//conf/.xml on each node.
3. Reboot each Riak node to reload the schema.
4. New objects can use new fields. New fields can be queried on.

This could be done in Yokozuan like so:

1. Modify schema.
2. Update in Yokozuna: curl -X PUT -H 'content-type: application/json' 'http://host:port/yz/schema/<schema name>' --data-binary @<schema name>.xml
3. Tell Yokozuna to reload the index: curl -X POST -H 'content-type: text/plain' 'http://host:port/yz/index/<index name>' RELOAD

This would use Solr's core RELOAD command and wouldn't require a restart.

[arobson]

@rzezeski - so I've been working on getting yokozuna support for an upcoming feature and having to bounce the cluster every time we add a schema or make a schema change is a real concern. Any chance this might land in the 2.0 release proper?

[rzezeski]

Yes, this should land in 2.0 proper.

[rzezeski]

Some thought needs to be giving to Solr's new managed schema API.

http://wiki.apache.org/solr/SchemaRESTAPI

Should Yokozuna try to mimic this? I'm not sure this plays well with Yokozuna's distributed bits.

Consideration for #219 must also be given. If schemaless mode is used the schema is updating automatically in Solr itself. How do Yokozuna and Solr keep a consistent view of what the schema is for an index?

[rzezeski]

The server now has the ability to reload indexes on the fly thanks to #260. But there is still no HTTP, PB, or riak-admin interface yet.

[krestenkrab]

This is a real issue for the usability of Yokozuna, as developing anything usually requires modifying the schema as you go along. We really need at least HTTP access to this, so it can be scripted.

Also, it's a bit unclear how to push an updated schema. If you just re-PUT an updated schema to the HTTP /search/schema/schemaName url then the schema does not get rewritten into the corresponding indexName/conf directory for indexes that use the schema. That would seem like an obvious semantics, I was a bit baffled about this at least.

A related issue is that it would be nice to have a way to PUT a document, and get an error back if it doesn't conform to a schema in the back end. Maybe not as a default behaviour -- as it would be really slow -- but as a way to avoid having to scavenge the log/solr.log files for errors when queries subsequently do not return the expected results.

In short, the edit-deploy-test cycle for working with a schema needs to be improved. IMHO, this needs done before you can reasonably include Yokozuna in a release. I would mark this issue as [must] for 1.0.0.

[rzezeski]

This is a real issue for the usability of Yokozuna, as developing anything usually requires modifying the schema as you go along. We really need at least HTTP access to this, so it can be scripted.

Also, it's a bit unclear how to push an updated schema. If you just re-PUT an updated schema to the HTTP /search/schema/schemaName url then the schema does not get rewritten into the corresponding indexName/conf directory for indexes that use the schema. That would seem like an obvious semantics, I was a bit baffled about this at least.

The current admin API for Yokozuna is awkward, but it is
workable. Exposing the reload command would avoid the need create a
new index just to edit the schema but modifying the schema of an index
with existing data also has many issues. There is related discussion
in the following issues:

• Catch-all "*" in default schema XML is problematic #316
• Schemaless Mode [JIRA: RIAK-1707] #219
• Schema Admin via HTTP [JIRA: RIAK-1715] #10

In short: mutating or removing fields will break your index. Adding
fields will not retroactively re-index all your previously written
objects. Modifying analyzer chains will break queries. This is not
solely a Yokozuna issue; the same problems exist in
ElasticSearch and SolrCloud.

A related issue is that it would be nice to have a way to PUT a document, and get an error back if it doesn't conform to a schema in the back end. Maybe not as a default behaviour -- as it would be really slow -- but as a way to avoid having to scavenge the log/solr.log files for errors when queries subsequently do not return the expected results.

I agree, this would be nice and would like to add something like
this. There is no time for 2.0.

In short, the edit-deploy-test cycle for working with a schema needs to be improved. IMHO, this needs done before you can reasonably include Yokozuna in a release. I would mark this issue as [must] for 1.0.0.

I agree with everything but the last part. I have many ideas for
improvement but Riak 2.0 is in code freeze and there is not enough
time to rework the entire API. It's awkward to use right now but it is
possible. There are people who have been using Yokozuna with success
for months now. This is an area that needs much improvement but I
disagree that it should prevent it from being shipped in Riak 2.0. It
just may not be ready according to your standards; and I can respect
that. You are making reasonable requests but the time is simply not
there.

I hope to put out an RFC soon about my ideas for a better
administration API. It attempts to solve a lot of the issues around
mutating schemas, allow synchronous index creation, get rid of
separate notion of schema (an index has a schema, not a pointer to a
schema), allow testing of index/schema before committing, allow staged
planning of schema changes, and move everything into a riak-admin
sub-command to provide a unified interface. It's one of N things I'd
like to see improved in Yokozuna.

Thank you for taking the time to comment on this issue and bring up
your concerns.

[krestenkrab]

I can understand that. I'm sorry this was mostly negative feedback :-) Once things are all aligned right, things worked just fine. You just have to walk a narrow path as things are now, so we should make sure the "good path" is well documented.