New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Divert reads away from data nodes #2104
Labels
Comments
erimatnor
changed the title
Divert reads away from cordoned data nodes
Divert reads away from data nodes
Mar 10, 2021
Related issue #3502 |
A proposed API could be
|
In a kubernetes setup, ideally there should be an operator that detects the loss of the node and automatically applies the cordon |
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 15, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a node as not available, which means other nodes will be used for, e.g., reads instead (a different "default" data node is set for each chunk). For this to work, all the chunks on the unavailable data node must have a replica on other data nodes. If some chunks are not replicated, a warning will be raised. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 15, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a node as not available, which means other nodes will be used for, e.g., reads instead (a different "default" data node is set for each chunk). For this to work, all the chunks on the unavailable data node must have a replica on other data nodes. If some chunks are not replicated, a warning will be raised. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 15, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a node as not available, which means other nodes will be used for, e.g., reads instead (a different "default" data node is set for each chunk). For this to work, all the chunks on the unavailable data node must have a replica on other data nodes. If some chunks are not replicated, a warning will be raised. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 15, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a node as "unavailable", which means other nodes will be used for, e.g., reads instead (a different "default" data node is set for each chunk). For this to work, all the chunks on the unavailable data node must have a replica on other data nodes. If some chunks are not replicated, a warning will be raised. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 15, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a node as "unavailable", which means other data nodes will be used for queries. The function finds all the chunks for which the unavailable data node is the "default" query target and sets another data node as the default. For this to work, all the chunks on the unavailable data node must have a replica on another data node. If some chunks are not replicated, a warning will be raised. When a data node is available again, the function should once again use the data node for queries. However, this part is not yet implemented and is left for a future change. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 15, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a node as "unavailable", which means other data nodes will be used for queries. The function finds all the chunks for which the unavailable data node is the "default" query target and sets another data node as the default. For this to work, all the chunks on the unavailable data node must have a replica on another data node. If some chunks are not replicated, a warning will be raised. When a data node is available again, the function should allow switching back to once again use the data node for queries. However, this part is not yet implemented and is left for a future change. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 17, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a node as "unavailable", which means other data nodes will be used for queries. The function finds all the chunks for which the unavailable data node is the "default" query target and sets another data node as the default. For this to work, all the chunks on the unavailable data node must have a replica on another data node. If some chunks are not replicated, a warning will be raised. When a data node is available again, the function should allow switching back to once again use the data node for queries. However, this part is not yet implemented and is left for a future change. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 17, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a node as "unavailable", which means other data nodes will be used for queries. The function finds all the chunks for which the unavailable data node is the "default" query target and sets another data node as the default. For this to work, all the chunks on the unavailable data node must have a replica on another data node. If some chunks are not replicated, a warning will be raised. When a data node is available again, the function should allow switching back to once again use the data node for queries. However, this part is not yet implemented and is left for a future change. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a node as "unavailable", which means other data nodes will be used for queries. The function finds all the chunks for which the unavailable data node is the "default" query target and sets another data node as the default. For this to work, all the chunks on the unavailable data node must have a replica on another data node. If some chunks are not replicated, a warning will be raised. When a data node is available again, the function switches back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a node as "unavailable", which means other data nodes will be used for queries. The function finds all the chunks for which the unavailable data node is the "default" query target and sets another data node as the default. For this to work, all the chunks on the unavailable data node must have a replica on another data node. If some chunks are not replicated, a warning will be raised. When a data node is available again, the function switches back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a node as "unavailable", which means other data nodes will be used for queries. The function finds all the chunks for which the unavailable data node is the "default" query target and sets another data node as the default. For this to work, all the chunks on the unavailable data node must have a replica on another data node. If some chunks are not replicated, a warning will be raised. When a data node is available again, the function switches back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a node as "unavailable", which means other data nodes will be used for queries. The function finds all the chunks for which the unavailable data node is the "default" query target and sets another data node as the default. For this to work, all the chunks on the unavailable data node must have a replica on another data node. If some chunks are not replicated, a warning will be raised. When a data node is available again, the function switches back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a data node as "unavailable", which means the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and sets a chunk replica on another data node to use instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a data node as "unavailable", which means the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and sets a chunk replica on another data node to use instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a data node as "unavailable", which means the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and sets a chunk replica on another data node to use instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a data node as "unavailable", which means the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and sets a chunk replica on another data node to use instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a data node as "unavailable", which means the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and sets a chunk replica on another data node to use instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a data node as "unavailable", which means the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and sets a chunk replica on another data node to use instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a data node as "unavailable", which means the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and sets a chunk replica on another data node to use instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a data node as "unavailable", which means the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and sets a chunk replica on another data node to use instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a data node as "unavailable", which means the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and sets a chunk replica on another data node to use instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 18, 2022
Add a new function, `alter_data_node()`, which can be used to change a data node's configuration on the access node, including connection information like host, port, and database. In addition, the function allows marking a data node as "unavailable", which means the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and sets a chunk replica on another data node to use instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to once again use the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 20, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The common configuration options include connection information like host, port, and database. In addition, the function allows setting _advanced_ options via a generic "options" array parameter. These options might be esoteric (and/or experimental) and therefore do not warrant an explicit function parameter. A new advanced option "available" allows configuring the availability of the data node. Setting "available=false" means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 21, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The common configuration options include connection information like host, port, and database. In addition, the function allows setting _advanced_ options via a generic "options" array parameter. These options might be esoteric (and/or experimental) and therefore do not warrant an explicit function parameter. A new advanced option "available" allows configuring the availability of the data node. Setting "available=false" means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Oct 21, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The common configuration options include connection information like host, port, and database. In addition, the function allows setting _advanced_ options via a generic "options" array parameter. These options might be esoteric (and/or experimental) and therefore do not warrant an explicit function parameter. A new advanced option "available" allows configuring the availability of the data node. Setting "available=false" means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Nov 7, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The new functions introduces a new option "available" that allows configuring the availability of the data node. Setting "available=false" means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data node instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Nov 7, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The new functions introduces a new option "available" that allows configuring the availability of the data node. Setting `available=>false` means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data node instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Nov 7, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The new functions introduces a new option "available" that allows configuring the availability of the data node. Setting `available=>false` means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data node instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Nov 7, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The new functions introduces a new option "available" that allows configuring the availability of the data node. Setting `available=>false` means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data node instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Nov 7, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The new functions introduces a new option "available" that allows configuring the availability of the data node. Setting `available=>false` means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data node instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Nov 9, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The new functions introduces a new option "available" that allows configuring the availability of the data node. Setting `available=>false` means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data node instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes timescale#2104
erimatnor
added a commit
that referenced
this issue
Nov 10, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The new functions introduces a new option "available" that allows configuring the availability of the data node. Setting `available=>false` means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data node instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes #2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Nov 10, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The new functions introduces a new option "available" that allows configuring the availability of the data node. Setting `available=>false` means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data node instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes timescale#2104
erimatnor
added a commit
to erimatnor/timescaledb
that referenced
this issue
Nov 11, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The new functions introduces a new option "available" that allows configuring the availability of the data node. Setting `available=>false` means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data node instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes timescale#2104
erimatnor
added a commit
that referenced
this issue
Nov 11, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The new functions introduces a new option "available" that allows configuring the availability of the data node. Setting `available=>false` means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data node instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes #2104
SachinSetiya
pushed a commit
that referenced
this issue
Nov 28, 2022
Add a new function, `alter_data_node()`, which can be used to change the data node's configuration originally set up via `add_data_node()` on the access node. The new functions introduces a new option "available" that allows configuring the availability of the data node. Setting `available=>false` means that the node should no longer be used for reads and writes. Only read "failover" is implemented as part of this change, however. To fail over reads, the alter data node function finds all the chunks for which the unavailable data node is the "primary" query target and "fails over" to a chunk replica on another data node instead. If some chunks do not have a replica to fail over to, a warning will be raised. When a data node is available again, the function can be used to switch back to using the data node for queries. Closes #2104
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When a node is unavailable, we should not only block new chunks from being created on it; we should also divert away reads from it. In other words, we should make sure the we set another "primary" data node for the chunks no the data node that is unavailable. The primary data node for a chunk is the foreign server referenced in the chunks foreign table.
The text was updated successfully, but these errors were encountered: