adding support for using background workers instead of libpq + adding support for an audit table #111

bdrouvotAWS · 2020-08-26T07:05:01Z

This PR is mainly made of 4 commits:

background worker support
audit table support
job and task wording
renamed bgworker scheduler to launcher

Let's explain each commit

background worker support

This commit is adding support for using background workers instead of libpq sessions to execute the jobs.
This behavior is guarded behind a GUC and disabled by default so that it does not impact backward compatibility.
The background workers use a similar technique to the pg_background extension to run commands, and things like cron.host, nodename, and nodeport have no effect in this mode.

audit table support

This commit is adding support for an audit table.
It works for either libpq or background-worker mode.

content looks like:

postgres=# select * from cron.job_run_details;
 jobid | runid | job_pid | database | username |                  command                   |  status   |         return_message         |          start_time           |           end_time
-------+-------+---------+----------+----------+--------------------------------------------+-----------+--------------------------------+-------------------------------+-------------------------------
     6 |     2 |   15073 | postgres | bdt      | insert into bdtcron values(1)              | succeeded | INSERT 0 1                     | 2020-08-18 12:43:01.004976+00 | 2020-08-18 12:43:01.009695+00
     2 |     3 |   15074 | postgres | bdt      | update bdtcron set a = a -1                | succeeded | UPDATE 0                       | 2020-08-18 12:43:01.005747+00 | 2020-08-18 12:43:01.010155+00
     4 |     4 |   15075 | postgres | bdt      | create table tutu as select * from bdtcron | succeeded | SELECT 1                       | 2020-08-18 12:43:01.006427+00 | 2020-08-18 12:43:01.011015+00
     5 |     5 |   15076 | postgres | bdt      | select * from bdtcron                      | succeeded | SELECT 1                       | 2020-08-18 12:43:01.00724+00  | 2020-08-18 12:43:01.011435+00
     1 |     6 |   15077 | postgres | bdt      | delete from bdtcron                        | succeeded | DELETE 1                       | 2020-08-18 12:43:01.008254+00 | 2020-08-18 12:43:01.011857+00
     3 |     7 |   15078 | postgres | bdt      | vacuum bdtcron                             | succeeded | VACUUM                         | 2020-08-18 12:43:01.00887+00  | 2020-08-18 12:43:01.022352+00
     6 |     8 |   15080 | postgres | bdt      | insert into bdtcron values(1)              | succeeded | INSERT 0 1                     | 2020-08-18 12:44:01.004167+00 | 2020-08-18 12:44:01.009513+00
     2 |     9 |   15081 | postgres | bdt      | update bdtcron set a = a -1                | succeeded | UPDATE 0                       | 2020-08-18 12:44:01.004883+00 | 2020-08-18 12:44:01.010053+00
     4 |    10 |   15082 | postgres | bdt      | create table tutu as select * from bdtcron | failed    | relation "tutu" already exists | 2020-08-18 12:44:01.005524+00 | 2020-08-18 12:44:01.010601+00
     5 |    11 |   15083 | postgres | bdt      | select * from bdtcron                      | succeeded | SELECT 1                       | 2020-08-18 12:44:01.006221+00 | 2020-08-18 12:44:01.011112+00
     1 |    12 |   15084 | postgres | bdt      | delete from bdtcron                        | succeeded | DELETE 1                       | 2020-08-18 12:44:01.00723+00  | 2020-08-18 12:44:01.011554+00

job and task wording

This commit is adding some wording about job and task.

renamed bgworker scheduler to launcher

This commit is renaming the current background worker from "scheduler" to "launcher" (to go with autovac nomenclature) to avoid confusion with the new workers.

ghost · 2020-08-26T07:05:15Z

All CLA requirements met.

src/pg_cron.c

src/job_metadata.c

src/pg_cron.c

marcocitus · 2020-09-01T12:59:48Z

src/pg_cron.c

@@ -3,6 +3,9 @@
 * src/pg_cron.c
 *
 * Implementation of the pg_cron task scheduler.
+ * Wording:
+ *	   - A job is a scheduling definition of a task
+ *	   - A task is what is actually executed within the database engine


I admit that this wasn't the greatest use of words.

src/pg_cron.c

pg_cron--1.2--1.3.sql

src/pg_cron.c

marcocitus

This looks very polished already.

The only issue I found so far was that a WARNING results in "failed" status. We could also do a little more to restrict concurrency within the bounds of max_worker_processes.

src/pg_cron.c

marcocitus

This one is a bit mean. It hits an assert failure.

SELECT cron.schedule('* * * * *', 'drop extension pg_cron');

2020-09-01 16:15:00.007 CEST [3775] LOG:  cron job 2 starting: drop extension pg_cron
TRAP: FailedAssertion("!(update_result)", File: "src/pg_cron.c", Line: 1647)

…back, adapt max running tasks

marcocitus · 2020-09-10T11:37:19Z

src/job_metadata.c

+
+	appendStringInfo(&querybuf,
+		"update %s.%s set status = '%s', return_message = 'server restarted' where status in ('%s','%s')"
+		, CRON_SCHEMA_NAME, JOB_RUN_DETAILS_TABLE_NAME, GetCronStatus(CRON_STATUS_FAILED), GetCronStatus(CRON_STATUS_STARTING), GetCronStatus(CRON_STATUS_RUNNING));


it's usually nicer to use quote_qualified_identifier to generate qualified table names.

marcocitus · 2020-09-10T11:44:34Z

src/job_metadata.c

@@ -275,6 +281,43 @@ NextJobId(void)
 	return jobId;
 }

+int64
+NextRunId(void)


There are some style things like having comments on all functions, but I think it's easiest to merge and adjust them later.

marcocitus · 2020-09-10T11:45:52Z

src/pg_cron.c

+						if (edata.elevel >= ERROR)
+							UpdateJobRunDetail(task->runId, NULL, GetCronStatus(CRON_STATUS_FAILED), display_msg.data, NULL, &end_time);
+						else
+							UpdateJobRunDetail(task->runId, NULL, GetCronStatus(CRON_STATUS_SUCCEEDED), display_msg.data, NULL, &end_time);


It's kind of neat to see the notices appear in the table. They do get overwritten once the task is done. Is that intentional? (I don't mind this behaviour)

marcocitus · 2020-09-10T11:49:51Z

src/pg_cron.c

+				ereport(WARNING,
+					(errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+					errmsg("out of background worker slots"),
+					errhint("You might need to increase max_worker_processes.")));


any reason to not put this relevant info in the errorMessage?

marcocitus

This all seems fine and I'm happy to merge as is and address remaining comments myself.

I do want to confirm: Is the current warning/notice behaviour as intended?

bdrouvotAWS · 2020-09-10T18:02:00Z

Hi Marco,

Thanks for the review!
The intention was to lay the foundation to display intermediate feedback (specially for long running tasks).
But as it is right now, I agree that we can update job_run_details only when elevel >= ERROR.

Bertrand

marcocitus · 2020-09-15T08:49:08Z

Found some compilation issues on PG13:

gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -ggdb -Og -g3 -fno-omit-frame-pointer -fPIC -std=c99 -Wall -Wextra -Werror -Wno-unused-parameter -Wno-implicit-fallthrough -Iinclude -I/home/marco/pgsql-13/include -I. -I./ -I/home/marco/pgsql-13/include/server -I/home/marco/pgsql-13/include/internal  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o src/pg_cron.o src/pg_cron.c -MMD -MP -MF .deps/pg_cron.Po
src/pg_cron.c: In function ‘ExecuteSqlString’:
src/pg_cron.c:2000:4: error: too many arguments to function ‘set_ps_display’
 2000 |    set_ps_display(GetCommandTagName(commandTag), false);
      |    ^~~~~~~~~~~~~~
In file included from src/pg_cron.c:72:
/home/marco/pgsql-13/include/server/utils/ps_status.h:21:13: note: declared here
   21 | extern void set_ps_display(const char *activity);
      |             ^~~~~~~~~~~~~~
src/pg_cron.c:2026:54: error: passing argument 3 of ‘pg_plan_queries’ makes integer from pointer without a cast [-Werror=int-conversion]
 2026 |   plantree_list = pg_plan_queries(querytree_list, 0, NULL);
      |                                                      ^~~~
      |                                                      |
      |                                                      void *
In file included from /home/marco/pgsql-13/include/server/tcop/utility.h:18,
                 from src/pg_cron.c:82:
/home/marco/pgsql-13/include/server/tcop/tcopprot.h:58:14: note: expected ‘int’ but argument is of type ‘void *’
   58 | extern List *pg_plan_queries(List *querytrees, const char *query_string,
      |              ^~~~~~~~~~~~~~~
src/pg_cron.c:2026:19: error: too few arguments to function ‘pg_plan_queries’
 2026 |   plantree_list = pg_plan_queries(querytree_list, 0, NULL);
      |                   ^~~~~~~~~~~~~~~
In file included from /home/marco/pgsql-13/include/server/tcop/utility.h:18,
                 from src/pg_cron.c:82:
/home/marco/pgsql-13/include/server/tcop/tcopprot.h:58:14: note: declared here
   58 | extern List *pg_plan_queries(List *querytrees, const char *query_string,
      |              ^~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make: *** [/home/marco/pgsql-13/lib/pgxs/src/makefiles/../../src/Makefile.global:920: src/pg_cron.o] Error 1

Will merge into a separate branch to fix.

marcocitus · 2020-09-15T13:43:57Z

Other issue I'll try to fix is to degrade more gracefully when the schema is outdated. pg_cron starts reading from the metadata as soon as the server starts, so even if the user knows to do ALTER EXTENSION pg_cron UPDATE immediately after installing new binaries, it might crash and burn before then.

rauanmayemir · 2020-09-26T15:05:05Z

These are exciting changes, when can a new release be expected?

bdt added 4 commits August 26, 2020 06:48

background worker support

febacd7

audit table support

62f1744

job and task wording

8f9fe9b

renamed bgworker scheduler to launcher

8077138