Introduce vshard.storage.service_call() #227
Comments
Gerold103
added
storage
feature
Gerold103
added a commit
that referenced
this issue
Mar 21, 2020
Since vshard creation it never needed any schema changes. Version
of vshard schema was always '1', stored in _schema as
'oncevshard:storage:1', because it was created by
box.once('vshard:storage:1').
Now it is time to make some changes to the schema. First is
necessity to simplify introduction and removal of internal
functions. There are 3 of them at the moment: bucket_recv()/
rebalancer_apply_routes()/rebalancer_request_state(). Users see
them in vshard.storage.* namespace, and in _func space. They are
going to be replaced by one vshard.storage._service_call(). In
that way it will be easy to update them without schema changes, to
add new functions, remove them. In _service_call() it will be
possible to validate versions, return proper errors in case a
requested function does not exist.
Secondly, _service_call() is going to be used by new discovery on
routers. It is not possible to modify the existing
vshard.storage.buckets_discovery() because it is a pubic function.
But a new discovery can be added to _service_call().
Thirdly, there is a bug in rebalancing related to possible big TCP
delays, which requires a change of _bucket space format in future.
Fourthly, _service_call() would allow to introduce new functions
on read-only replica, using code reload only. This may be needed
for the bug about bucket_ref() not preventing bucket move.
The patch introduces versioning using 4 numbers: x.x.x.x. The
first 3 numbers are major, minor, patch, the same as in the tags.
The last value is a number increased when first 3 numbers can't be
changed, but the schema is modified. That happens when users take
master branch between 2 tags, and then the schema is changed again
before the new tag is published.
Upgrade between two adjacent tags is supposed to be always safe
and automatic when reload or reconfiguration happen. However
currently there are too few versions so as any upgrade could be
not safe.
Needed for #227
Needed for #210
Gerold103
added a commit
that referenced
this issue
Mar 21, 2020
It is a new internal function which is supposed to absorb all the other internal functions to make it easier to change them, to simplify adding new and drop old functions. This also hides them from vshard.storage.* namespace so users can't try to use them not knowing they are private. Additionally, _call() allows to change internal function set even on read-only instances, because it has nothing to do with the schema. Internal function set can be updated by reload. Closes #227 Part of #210
Gerold103
added a commit
that referenced
this issue
Mar 22, 2020
Since vshard creation it never needed any schema changes. Version
of vshard schema was always '1', stored in _schema as
'oncevshard:storage:1', because it was created by
box.once('vshard:storage:1').
Now it is time to make some changes to the schema. First is
necessity to simplify introduction and removal of internal
functions. There are 3 of them at the moment: bucket_recv()/
rebalancer_apply_routes()/rebalancer_request_state(). Users see
them in vshard.storage.* namespace, and in _func space. They are
going to be replaced by one vshard.storage._service_call(). In
that way it will be easy to update them without schema changes, to
add new functions, remove them. In _service_call() it will be
possible to validate versions, return proper errors in case a
requested function does not exist.
Secondly, _service_call() is going to be used by new discovery on
routers. It is not possible to modify the existing
vshard.storage.buckets_discovery() because it is a pubic function.
But a new discovery can be added to _service_call().
Thirdly, there is a bug in rebalancing related to possible big TCP
delays, which requires a change of _bucket space format in future.
Fourthly, _service_call() would allow to introduce new functions
on read-only replica, using code reload only. This may be needed
for the bug about bucket_ref() not preventing bucket move.
The patch introduces versioning using 4 numbers: x.x.x.x. The
first 3 numbers are major, minor, patch, the same as in the tags.
The last value is a number increased when first 3 numbers can't be
changed, but the schema is modified. That happens when users take
master branch between 2 tags, and then the schema is changed again
before the new tag is published.
Upgrade between two adjacent tags is supposed to be always safe
and automatic when reload or reconfiguration happen. However
currently there are too few versions so as any upgrade could be
not safe.
Needed for #227
Needed for #210
Gerold103
added a commit
that referenced
this issue
Mar 22, 2020
It is a new internal function which is supposed to absorb all the other internal functions to make it easier to change them, to simplify adding new and drop old functions. This also hides them from vshard.storage.* namespace so users can't try to use them not knowing they are private. Additionally, _call() allows to change internal function set even on read-only instances, because it has nothing to do with the schema. Internal function set can be updated by reload. Closes #227 Part of #210
Gerold103
added a commit
that referenced
this issue
Mar 23, 2020
Since vshard creation it never needed any schema changes. Version
of vshard schema was always '1', stored in _schema as
'oncevshard:storage:1', because it was created by
box.once('vshard:storage:1').
Now it is time to make some changes to the schema. First is
necessity to simplify introduction and removal of internal
functions. There are 3 of them at the moment: bucket_recv()/
rebalancer_apply_routes()/rebalancer_request_state(). Users see
them in vshard.storage.* namespace, and in _func space. They are
going to be replaced by one vshard.storage._service_call(). In
that way it will be easy to update them without schema changes, to
add new functions, remove them. In _service_call() it will be
possible to validate versions, return proper errors in case a
requested function does not exist.
Secondly, _service_call() is going to be used by new discovery on
routers. It is not possible to modify the existing
vshard.storage.buckets_discovery() because it is a pubic function.
But a new discovery can be added to _service_call().
Thirdly, there is a bug in rebalancing related to possible big TCP
delays, which requires a change of _bucket space format in future.
Fourthly, _service_call() would allow to introduce new functions
on read-only replica, using code reload only. This may be needed
for the bug about bucket_ref() not preventing bucket move.
The patch introduces versioning using 4 numbers: x.x.x.x. The
first 3 numbers are major, minor, patch, the same as in the tags.
The last value is a number increased when first 3 numbers can't be
changed, but the schema is modified. That happens when users take
master branch between 2 tags, and then the schema is changed again
before the new tag is published.
Upgrade between two adjacent tags is supposed to be always safe
and automatic when reload or reconfiguration happen. However
currently there are too few versions so as any upgrade could be
not safe.
Needed for #227
Needed for #210
Gerold103
added a commit
that referenced
this issue
Mar 23, 2020
It is a new internal function which is supposed to absorb all the other internal functions to make it easier to change them, to simplify adding new and drop old functions. This also hides them from vshard.storage.* namespace so users can't try to use them not knowing they are private. Additionally, _call() allows to change internal function set even on read-only instances, because it has nothing to do with the schema. Internal function set can be updated by reload. Closes #227 Part of #210
Gerold103
added a commit
that referenced
this issue
Mar 24, 2020
Since vshard creation it never needed any schema changes. Version
of vshard schema was always '1', stored in _schema as
'oncevshard:storage:1', because it was created by
box.once('vshard:storage:1').
Now it is time to make some changes to the schema. First is
necessity to simplify introduction and removal of internal
functions. There are 3 of them at the moment: bucket_recv()/
rebalancer_apply_routes()/rebalancer_request_state(). Users see
them in vshard.storage.* namespace, and in _func space. They are
going to be replaced by one vshard.storage._call(). In that way
it will be easy to update them without schema changes, to add new
functions, remove them. In _call() it will be possible to
validate versions, return proper errors in case a requested
function does not exist.
Secondly, _call() is going to be used by new discovery on
routers. It is not possible to modify the existing
vshard.storage.buckets_discovery() because it is a pubic function.
But a new discovery can be added to _call().
Thirdly, there is a bug in rebalancing related to possible big TCP
delays, which requires a change of _bucket space format in future.
Fourthly, _call() would allow to introduce new functions on
read-only replica, using code reload only. This may be needed
for the bug about bucket_ref() not preventing bucket move.
The patch introduces versioning using 4 numbers: x.x.x.x. The
first 3 numbers are major, minor, patch, the same as in the tags.
The last value is a number increased when first 3 numbers can't be
changed, but the schema is modified. That happens when users take
master branch between 2 tags, and then the schema is changed again
before the new tag is published.
Upgrade between two adjacent tags is supposed to be always safe
and automatic when reload or reconfiguration happen. However
currently there are too few versions so as any upgrade could be
not safe.
Needed for #227
Needed for #210
Gerold103
added a commit
that referenced
this issue
Mar 24, 2020
It is a new internal function which is supposed to absorb all the other internal functions to make it easier to change them, to simplify adding new and drop old functions. This also hides them from vshard.storage.* namespace so users can't try to use them not knowing they are private. Additionally, _call() allows to change internal function set even on read-only instances, because it has nothing to do with the schema. Internal function set can be updated by reload. Closes #227 Part of #210
Gerold103
added a commit
that referenced
this issue
Mar 28, 2020
Since vshard creation it never needed any schema changes. Version
of vshard schema was always '1', stored in _schema as
'oncevshard:storage:1', because it was created by
box.once('vshard:storage:1').
Now it is time to make some changes to the schema. First is
necessity to simplify introduction and removal of internal
functions. There are 3 of them at the moment: bucket_recv()/
rebalancer_apply_routes()/rebalancer_request_state(). Users see
them in vshard.storage.* namespace, and in _func space. They are
going to be replaced by one vshard.storage._call(). In that way
it will be easy to update them without schema changes, to add new
functions, remove them. In _call() it will be possible to
validate versions, return proper errors in case a requested
function does not exist.
Secondly, _call() is going to be used by new discovery on
routers. It is not possible to modify the existing
vshard.storage.buckets_discovery() because it is a pubic function.
But a new discovery can be added to _call().
Thirdly, there is a bug in rebalancing related to possible big TCP
delays, which requires a change of _bucket space format in future.
Fourthly, _call() would allow to introduce new functions on
read-only replica, using code reload only. This may be needed
for the bug about bucket_ref() not preventing bucket move.
The patch introduces versioning using 4 numbers: x.x.x.x. The
first 3 numbers are major, minor, patch, the same as in the tags.
The last value is a number increased when first 3 numbers can't be
changed, but the schema is modified. That happens when users take
master branch between 2 tags, and then the schema is changed again
before the new tag is published.
Upgrade between two adjacent tags is supposed to be always safe
and automatic when reload or reconfiguration happen. However
currently there are too few versions so as any upgrade could be
not safe.
Needed for #227
Needed for #210
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently all functions need to be registered in
_func, even private functions used only by routers and storages internally, not by user code. This is dangerous - users can discover these functions, start using them for something, and have problems when the functions are changed or removed. This is hard to extend - every time when a new function is needed, the schema should be changed (because of change in_func). Every time when new parameters are needed, the existing function needs to be changed, and sometimes functions are public. Change of their arguments may break existing user code.vshard.storage.service_call()can replace all private functions. It could take operation name as one of its arguments, and be changed anytime, without schema changes, with hot code reload, without breaking anything.The text was updated successfully, but these errors were encountered: