Make copy fetcher more async #5362
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## main #5362 +/- ##
==========================================
- Coverage 90.67% 90.64% -0.03%
==========================================
Files 226 226
Lines 52524 52533 +9
==========================================
- Hits 47626 47619 -7
- Misses 4898 4914 +16
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
erimatnor
force-pushed
the
async-copy-fetcher
branch
from
ccc0946
to
7f051d5
Compare
akuzm
approved these changes
Feb 28, 2023
erimatnor
force-pushed
the
async-copy-fetcher
branch
3 times, most recently
from
af918a3
to
853ed85
Compare
Make the copy fetcher more asynchronous by separating the sending of the request for data from the receiving of the response. By doing that, the async append node can send the request to each data node before it starts reading the first response. This can massively improve the performance because the response isn't returned until the remote node has finished executing the query and is ready to return the first tuple.
erimatnor
force-pushed
the
async-copy-fetcher
branch
from
853ed85
to
e31bd33
Compare
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Mar 6, 2023
This release contains bug fixes since the 2.10.0 release. We recommend that you upgrade at the next available opportunity. **Bugfixes** * timescale#5364 Fix num_chunks inconsistency in hypertables view * timescale#5362 Make copy fetcher more async * timescale#5336 Use NameData and namestrcpy for names * timescale#5317 Fix some incorrect memory handling * timescale#5367 Rename columns in old-style continuous aggregates * timescale#5336 Use NameData and namestrcpy for names * timescale#5343 Set PortalContext when starting job * timescale#5360 Fix uninitialized bucket_info variable * timescale#5362 Make copy fetcher more async * timescale#5364 Fix num_chunks inconsistency in hypertables view * timescale#5367 Fix column name handling in old-style continuous aggregates * timescale#5378 Fix multinode DML HA performance regression * timescale#5384 Fix Hierarchical Continuous Aggregates chunk_interval_size * timescale#5153 Fix concurrent locking with chunk_data_node table **Thanks** * @justinozavala for reporting an issue with PL/Python procedures in the background worker * @Medvecrab for discovering an issue with copying NameData when forming heap tuples. * @pushpeepkmonroe for discovering an issue in upgrading old-style continuous aggregates with renamed columns * @pushpeepkmonroe for discovering an issue in upgrading old-style continuous aggregates with renamed columns
Merged
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Mar 6, 2023
This release contains bug fixes since the 2.10.0 release. We recommend that you upgrade at the next available opportunity. **Bugfixes** * timescale#5364 Fix num_chunks inconsistency in hypertables view * timescale#5362 Make copy fetcher more async * timescale#5336 Use NameData and namestrcpy for names * timescale#5317 Fix some incorrect memory handling * timescale#5367 Rename columns in old-style continuous aggregates * timescale#5336 Use NameData and namestrcpy for names * timescale#5343 Set PortalContext when starting job * timescale#5360 Fix uninitialized bucket_info variable * timescale#5362 Make copy fetcher more async * timescale#5364 Fix num_chunks inconsistency in hypertables view * timescale#5367 Fix column name handling in old-style continuous aggregates * timescale#5378 Fix multinode DML HA performance regression * timescale#5384 Fix Hierarchical Continuous Aggregates chunk_interval_size * timescale#5153 Fix concurrent locking with chunk_data_node table **Thanks** * @justinozavala for reporting an issue with PL/Python procedures in the background worker * @Medvecrab for discovering an issue with copying NameData when forming heap tuples. * @pushpeepkmonroe for discovering an issue in upgrading old-style continuous aggregates with renamed columns * @pushpeepkmonroe for discovering an issue in upgrading old-style continuous aggregates with renamed columns
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Mar 6, 2023
This release contains bug fixes since the 2.10.0 release. We recommend that you upgrade at the next available opportunity. **Bugfixes** * timescale#5159 Support Continuous Aggregates names in hypertable_(detailed_)size * timescale#5226 Fix concurrent locking with chunk_data_node table * timescale#5317 Fix some incorrect memory handling * timescale#5336 Use NameData and namestrcpy for names * timescale#5343 Set PortalContext when starting job * timescale#5360 Fix uninitialized bucket_info variable * timescale#5362 Make copy fetcher more async * timescale#5364 Fix num_chunks inconsistency in hypertables view * timescale#5367 Fix column name handling in old-style continuous aggregates * timescale#5378 Fix multinode DML HA performance regression * timescale#5384 Fix Hierarchical Continuous Aggregates chunk_interval_size **Thanks** * @justinozavala for reporting an issue with PL/Python procedures in the background worker * @Medvecrab for discovering an issue with copying NameData when forming heap tuples. * @pushpeepkmonroe for discovering an issue in upgrading old-style continuous aggregates with renamed columns * @pushpeepkmonroe for discovering an issue in upgrading old-style continuous aggregates with renamed columns
svenklemm
added a commit
that referenced
this pull request
Mar 7, 2023
This release contains bug fixes since the 2.10.0 release. We recommend that you upgrade at the next available opportunity. **Bugfixes** * #5159 Support Continuous Aggregates names in hypertable_(detailed_)size * #5226 Fix concurrent locking with chunk_data_node table * #5317 Fix some incorrect memory handling * #5336 Use NameData and namestrcpy for names * #5343 Set PortalContext when starting job * #5360 Fix uninitialized bucket_info variable * #5362 Make copy fetcher more async * #5364 Fix num_chunks inconsistency in hypertables view * #5367 Fix column name handling in old-style continuous aggregates * #5378 Fix multinode DML HA performance regression * #5384 Fix Hierarchical Continuous Aggregates chunk_interval_size **Thanks** * @justinozavala for reporting an issue with PL/Python procedures in the background worker * @Medvecrab for discovering an issue with copying NameData when forming heap tuples. * @pushpeepkmonroe for discovering an issue in upgrading old-style continuous aggregates with renamed columns * @pushpeepkmonroe for discovering an issue in upgrading old-style continuous aggregates with renamed columns
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Mar 7, 2023
This release contains bug fixes since the 2.10.0 release. We recommend that you upgrade at the next available opportunity. **Bugfixes** * timescale#5159 Support Continuous Aggregates names in hypertable_(detailed_)size * timescale#5226 Fix concurrent locking with chunk_data_node table * timescale#5317 Fix some incorrect memory handling * timescale#5336 Use NameData and namestrcpy for names * timescale#5343 Set PortalContext when starting job * timescale#5360 Fix uninitialized bucket_info variable * timescale#5362 Make copy fetcher more async * timescale#5364 Fix num_chunks inconsistency in hypertables view * timescale#5367 Fix column name handling in old-style continuous aggregates * timescale#5378 Fix multinode DML HA performance regression * timescale#5384 Fix Hierarchical Continuous Aggregates chunk_interval_size **Thanks** * @justinozavala for reporting an issue with PL/Python procedures in the background worker * @Medvecrab for discovering an issue with copying NameData when forming heap tuples. * @pushpeepkmonroe for discovering an issue in upgrading old-style continuous aggregates with renamed columns * @pushpeepkmonroe for discovering an issue in upgrading old-style continuous aggregates with renamed columns
svenklemm
added a commit
that referenced
this pull request
Mar 7, 2023
This release contains bug fixes since the 2.10.0 release. We recommend that you upgrade at the next available opportunity. **Bugfixes** * #5159 Support Continuous Aggregates names in hypertable_(detailed_)size * #5226 Fix concurrent locking with chunk_data_node table * #5317 Fix some incorrect memory handling * #5336 Use NameData and namestrcpy for names * #5343 Set PortalContext when starting job * #5360 Fix uninitialized bucket_info variable * #5362 Make copy fetcher more async * #5364 Fix num_chunks inconsistency in hypertables view * #5367 Fix column name handling in old-style continuous aggregates * #5378 Fix multinode DML HA performance regression * #5384 Fix Hierarchical Continuous Aggregates chunk_interval_size **Thanks** * @justinozavala for reporting an issue with PL/Python procedures in the background worker * @Medvecrab for discovering an issue with copying NameData when forming heap tuples. * @pushpeepkmonroe for discovering an issue in upgrading old-style continuous aggregates with renamed columns * @pushpeepkmonroe for discovering an issue in upgrading old-style continuous aggregates with renamed columns
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Make the copy fetcher more asynchronous by separating the sending of the request for data from the receiving of the response. By doing that, the async append node can send the request to each data node before it starts reading the first response. This can massively improve the performance because the response isn't returned until the remote node has finished executing the query and is ready to return the first tuple.