Add IndexManager #2879
Add IndexManager #2879
Conversation
This makes them accessible to both appscale-admin and appscale-datastore.
This allows each datastore server to assume a management role when needed without relying on a separate thread.
The IndexManager maintains a cache of datastore index definitions that's kept up to date by ZooKeeper watches. It also keeps track of composite index status and backfills indexes when needed.
|
I think the life cycle of the index metadata no longer matches that of the index data. If you undeploy and then deploy an application the index metadata will be recreated and the index data will then be duplicated. |
|
@sjones4 That's a good point. Given that we don't delete a project's datastore data on undeploy, it seems that an undeploy on AppScale is more similar to disabling the project than deleting it. Do you think it would be sufficient if we just left the metadata intact on an undeploy? We could remove the services node instead of the entire project tree. An alternative would be do implement a servingStatus field on the project node. We may also want to rename the undeploy operation to make it clearer what it does. The easiest change in the short term would probably be allowing deletions only on the service level. |
|
@cdonati leaving the project level metadata intact on undeploy sounds fine to me. The metadata for services would then reflect what was currently deployed. |
| existing_indexes = [DatastoreIndex.from_dict(project_id, index) | ||
| for index in json.loads(existing_indexes)] | ||
| except (TypeError, ValueError): | ||
| existing_indexes = [] |
There was a problem hiding this comment.
Is this part of a normal workflow? It's not clear why listed errors should be ignored silently. Should they?
This makes it clearer that a non-existent indexes node is not an exceptional circumstance.
After ensure_path, the node data is an empty string.
The IndexManager maintains a cache of datastore index definitions that's kept up to date by ZooKeeper watches.
It also keeps track of composite index status and backfills indexes when needed.
This also introduces a slight delay for newly defined indexes while they are backfilled.
Resolves #2874
Resolves #2878
Future work in this area includes