Set status for backend in background jobs #2850

mkindahl · 2021-01-21T14:55:44Z

The status of background workers is recorded using
pgstat_record_activity which allow the status of the worker to be
inspected using the status field in pg_stat_activity. This is used
by the timescaledb_information.job_stats view to show if a job is
running, scheduled, or paused.

However, the activity was never recorded in the code and hence the
field was always NULL. Since this was the default for the job_status
field of timescaledb_information.job_stats it would always show as
Scheduled.

This commit fixes this by adding calls to pgstat_record_activity and
suitable locations and also add a default of Undefined for the
job_status field to capture missing calls of
pgstat_record_activity. This is more of a precaution and is unlikely
to ever show.

Fixes #2831

codecov · 2021-01-22T21:32:57Z

Codecov Report

Merging #2850 (3d59938) into master (99e450d) will increase coverage by 0.14%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2850      +/-   ##
==========================================
+ Coverage   90.09%   90.23%   +0.14%     
==========================================
  Files         212      212              
  Lines       34775    34738      -37     
==========================================
+ Hits        31330    31346      +16     
+ Misses       3445     3392      -53

Impacted Files	Coverage Δ
src/bgw/job_stat.c	`88.94% <100.00%> (+0.11%)`	⬆️
src/bgw/scheduler.c	`82.74% <100.00%> (+0.15%)`	⬆️
src/chunk.c	`95.73% <100.00%> (+0.01%)`	⬆️
tsl/src/bgw_policy/job.c	`96.82% <100.00%> (+0.08%)`	⬆️
tsl/src/fdw/shippable.c	`82.85% <0.00%> (-11.43%)`	⬇️
src/loader/bgw_message_queue.c	`84.51% <0.00%> (-2.59%)`	⬇️
src/process_utility.c	`93.86% <0.00%> (+0.12%)`	⬆️
tsl/src/chunk_api.c	`94.71% <0.00%> (+0.17%)`	⬆️
src/chunk_insert_state.c	`97.67% <0.00%> (+0.46%)`	⬆️
src/chunk_append/explain.c	`83.07% <0.00%> (+1.53%)`	⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e3c6725...5dd0a95. Read the comment docs.

pmwkaa · 2021-01-25T16:39:36Z

tsl/src/bgw_policy/job.c

+	query = makeStringInfo();
+	appendStringInfo(query,
+					 "CALL %s.%s()",
+					 NameStr(job->fd.proc_schema),


Does it need quotes? in case if name contains spaces, etc.

Well, it is just user viewed output, not something that is used for execution, but changed it anyway since it looks nicer.

svenklemm · 2021-01-25T19:21:07Z

tsl/src/bgw_policy/job.c

+					 "CALL %s.%s()",
+					 NameStr(job->fd.proc_schema),
+					 NameStr(job->fd.proc_name));
+	pgstat_report_activity(STATE_RUNNING, query->data);


this needs a test

Added one. I actually wanted to add a more elaborate test that shows what statement is being executed inside a custom procedure, but that proved to be difficult.

The status of background workers is recorded in PostgreSQL using the function `pgstat_record_activity` which allow the status of the background worker to be inspected using the `status` field in `pg_stat_activity`. This is used by the TimescaleDB `job_stats` view to show if a job is running, scheduled, or paused. However, the activity was never recorded in the code and hence the field was always NULL. Since this was the default for the `job_status` field of `timescaledb_information.job_stats` it would always show as `Scheduled`. This commit fixes this by adding calls to `pgstat_record_activity` at suitable locations. Fixes timescale#2831

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.2 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Bugfixes** * timescale#2850 Set status for backend in background jobs * timescale#2883 Fix join qual propagation for nested joins * timescale#2884 Add GUC to control join qual propagation * timescale#2885 Fix compressed chunk check when disabling compression * timescale#2908 Fix changing column type of clustered hypertables **Thanks** * @zeeshanshabbir93 for reporting an issue with joins * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status * @zeeshanshabbir93 for reporting the issue with full outer joins

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.2 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Bugfixes** * timescale#2850 Set status for backend in background jobs * timescale#2883 Fix join qual propagation for nested joins * timescale#2884 Add GUC to control join qual propagation * timescale#2885 Fix compressed chunk check when disabling compression * timescale#2908 Fix changing column type of clustered hypertables * timescale#2942 Validate continuous aggregate policy **Thanks** * @zeeshanshabbir93 for reporting the issue with full outer joins * @Antiarchitect for reporting the issue with slow refreshes of * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.2 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Bugfixes** * #2850 Set status for backend in background jobs * #2883 Fix join qual propagation for nested joins * #2884 Add GUC to control join qual propagation * #2885 Fix compressed chunk check when disabling compression * #2908 Fix changing column type of clustered hypertables * #2942 Validate continuous aggregate policy **Thanks** * @zeeshanshabbir93 for reporting the issue with full outer joins * @Antiarchitect for reporting the issue with slow refreshes of * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.2 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Minor features** * timescale#2926 Optimize cagg refresh for small invalidations **Bugfixes** * timescale#2850 Set status for backend in background jobs * timescale#2883 Fix join qual propagation for nested joins * timescale#2884 Add GUC to control join qual propagation * timescale#2885 Fix compressed chunk check when disabling compression * timescale#2908 Fix changing column type of clustered hypertables * timescale#2942 Validate continuous aggregate policy **Thanks** * @zeeshanshabbir93 for reporting the issue with full outer joins * @Antiarchitect for reporting the issue with slow refreshes of * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.1 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Minor features** * timescale#2926 Optimize cagg refresh for small invalidations **Bugfixes** * timescale#2850 Set status for backend in background jobs * timescale#2883 Fix join qual propagation for nested joins * timescale#2884 Add GUC to control join qual propagation * timescale#2885 Fix compressed chunk check when disabling compression * timescale#2908 Fix changing column type of clustered hypertables * timescale#2942 Validate continuous aggregate policy **Thanks** * @zeeshanshabbir93 for reporting the issue with full outer joins * @Antiarchitect for reporting the issue with slow refreshes of * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.1 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Minor features** * timescale#2926 Optimize cagg refresh for small invalidations **Bugfixes** * timescale#2850 Set status for backend in background jobs * timescale#2883 Fix join qual propagation for nested joins * timescale#2884 Add GUC to control join qual propagation * timescale#2885 Fix compressed chunk check when disabling compression * timescale#2908 Fix changing column type of clustered hypertables * timescale#2942 Validate continuous aggregate policy **Thanks** * @zeeshanshabbir93 for reporting the issue with full outer joins * @Antiarchitect for reporting the issue with slow refreshes of * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.1 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Minor features** * #2926 Optimize cagg refresh for small invalidations **Bugfixes** * #2850 Set status for backend in background jobs * #2883 Fix join qual propagation for nested joins * #2884 Add GUC to control join qual propagation * #2885 Fix compressed chunk check when disabling compression * #2908 Fix changing column type of clustered hypertables * #2942 Validate continuous aggregate policy **Thanks** * @zeeshanshabbir93 for reporting the issue with full outer joins * @Antiarchitect for reporting the issue with slow refreshes of * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.1 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Minor features** * #2926 Optimize cagg refresh for small invalidations **Bugfixes** * #2850 Set status for backend in background jobs * #2883 Fix join qual propagation for nested joins * #2884 Add GUC to control join qual propagation * #2885 Fix compressed chunk check when disabling compression * #2908 Fix changing column type of clustered hypertables * #2942 Validate continuous aggregate policy **Thanks** * @zeeshanshabbir93 for reporting the issue with full outer joins * @Antiarchitect for reporting the issue with slow refreshes of * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.1 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Minor features** * #2926 Optimize cagg refresh for small invalidations **Bugfixes** * #2850 Set status for backend in background jobs * #2883 Fix join qual propagation for nested joins * #2884 Add GUC to control join qual propagation * #2885 Fix compressed chunk check when disabling compression * #2908 Fix changing column type of clustered hypertables * #2942 Validate continuous aggregate policy **Thanks** * @zeeshanshabbir93 for reporting the issue with full outer joins * @Antiarchitect for reporting the issue with slow refreshes of * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.1 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Minor features** * timescale#2926 Optimize cagg refresh for small invalidations **Bugfixes** * timescale#2850 Set status for backend in background jobs * timescale#2883 Fix join qual propagation for nested joins * timescale#2884 Add GUC to control join qual propagation * timescale#2885 Fix compressed chunk check when disabling compression * timescale#2908 Fix changing column type of clustered hypertables * timescale#2942 Validate continuous aggregate policy **Thanks** * @zeeshanshabbir93 for reporting the issue with full outer joins * @Antiarchitect for reporting the issue with slow refreshes of * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.1 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Minor features** * timescale#2926 Optimize cagg refresh for small invalidations **Bugfixes** * timescale#2850 Set status for backend in background jobs * timescale#2883 Fix join qual propagation for nested joins * timescale#2884 Add GUC to control join qual propagation * timescale#2885 Fix compressed chunk check when disabling compression * timescale#2908 Fix changing column type of clustered hypertables * timescale#2942 Validate continuous aggregate policy **Thanks** * @zeeshanshabbir93 for reporting the issue with full outer joins * @Antiarchitect for reporting the issue with slow refreshes of * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.1 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Minor features** * #2926 Optimize cagg refresh for small invalidations **Bugfixes** * #2850 Set status for backend in background jobs * #2883 Fix join qual propagation for nested joins * #2884 Add GUC to control join qual propagation * #2885 Fix compressed chunk check when disabling compression * #2908 Fix changing column type of clustered hypertables * #2942 Validate continuous aggregate policy **Thanks** * @zeeshanshabbir93 for reporting the issue with full outer joins * @Antiarchitect for reporting the issue with slow refreshes of * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.1 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Minor features** * timescale#2926 Optimize cagg refresh for small invalidations **Bugfixes** * timescale#2850 Set status for backend in background jobs * timescale#2883 Fix join qual propagation for nested joins * timescale#2884 Add GUC to control join qual propagation * timescale#2885 Fix compressed chunk check when disabling compression * timescale#2908 Fix changing column type of clustered hypertables * timescale#2942 Validate continuous aggregate policy **Thanks** * @zeeshanshabbir93 for reporting the issue with full outer joins * @Antiarchitect for reporting the issue with slow refreshes of * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status

@zeeshanshabbir93

This maintenance release contains bugfixes since the 2.0.1 release. We deem it high priority for upgrading. The bug fixes in this release address issues with joins, the status of background jobs, and disabling compression. It also includes enhancements to continuous aggregates, including improved validation of policies and optimizations for faster refreshes when there are a lot of invalidations. **Minor features** * #2926 Optimize cagg refresh for small invalidations **Bugfixes** * #2850 Set status for backend in background jobs * #2883 Fix join qual propagation for nested joins * #2884 Add GUC to control join qual propagation * #2885 Fix compressed chunk check when disabling compression * #2908 Fix changing column type of clustered hypertables * #2942 Validate continuous aggregate policy **Thanks** * @zeeshanshabbir93 for reporting the issue with full outer joins * @Antiarchitect for reporting the issue with slow refreshes of * @diego-hermida for reporting the issue about being unable to disable compression * @mtin for reporting the issue about wrong job status

mkindahl force-pushed the job_status_view branch 3 times, most recently from cdd8cbc to 7382ac5 Compare January 22, 2021 10:19

mkindahl closed this Jan 22, 2021

mkindahl force-pushed the job_status_view branch from 7382ac5 to 41cfabd Compare January 22, 2021 10:50

mkindahl reopened this Jan 22, 2021

mkindahl force-pushed the job_status_view branch 2 times, most recently from 7a7000d to ca9b055 Compare January 22, 2021 21:20

mkindahl marked this pull request as ready for review January 25, 2021 10:28

mkindahl requested a review from a team as a code owner January 25, 2021 10:28

mkindahl requested review from erimatnor, k-rus, svenklemm and gayyappan and removed request for a team, erimatnor and k-rus January 25, 2021 10:28

pmwkaa approved these changes Jan 25, 2021

View reviewed changes

svenklemm reviewed Jan 25, 2021

View reviewed changes

mkindahl force-pushed the job_status_view branch 3 times, most recently from 6436aba to f7581fa Compare February 2, 2021 08:54

mkindahl requested a review from svenklemm February 2, 2021 08:55

svenklemm approved these changes Feb 2, 2021

View reviewed changes

mkindahl force-pushed the job_status_view branch from f7581fa to 5dd0a95 Compare February 2, 2021 15:22

mkindahl merged commit 954c775 into timescale:master Feb 2, 2021

mkindahl deleted the job_status_view branch February 2, 2021 16:03

erimatnor mentioned this pull request Feb 18, 2021

Release 2.0.2 #2961

Merged

erimatnor mentioned this pull request Feb 19, 2021

Release 2.0.2 #2965

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set status for backend in background jobs #2850

Set status for backend in background jobs #2850

mkindahl commented Jan 21, 2021 •

edited

codecov bot commented Jan 22, 2021 •

edited

pmwkaa Jan 25, 2021

mkindahl Feb 2, 2021

svenklemm Jan 25, 2021

mkindahl Feb 2, 2021

Set status for backend in background jobs #2850

Set status for backend in background jobs #2850

Conversation

mkindahl commented Jan 21, 2021 • edited

codecov bot commented Jan 22, 2021 • edited

Codecov Report

pmwkaa Jan 25, 2021

Choose a reason for hiding this comment

mkindahl Feb 2, 2021

Choose a reason for hiding this comment

svenklemm Jan 25, 2021

Choose a reason for hiding this comment

mkindahl Feb 2, 2021

Choose a reason for hiding this comment

mkindahl commented Jan 21, 2021 •

edited

codecov bot commented Jan 22, 2021 •

edited