Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Single procrastinate task #271

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Single procrastinate task #271

wants to merge 2 commits into from

Conversation

vadim-zabolotniy
Copy link
Collaborator

We have two setting for Procrastination tasks:
queueing_lock - this blocks spam of tasks in queue. It allows only one task with status todo. This means that in DB at any time can be one(doing) or two tasks(doing + todo).
lock - forbidden only one doing task in any time.

If we use:
Only queueing_lock will stop todo task spam. But we will have 2 executing same tasks.
Only lock will have only one executing of some kind task, but will have todo task spam.
queueing_lock + lock gives as optimal result we have no spam in task queue and always only one executing task.

Now about problem: Logic of working lock pretty simple if task with status doing exists is skips start of executing new one. And this leas to "dead" task after server restart. Worker killed, status doing and new execution will never start.

This is significant issue that need to be discus. For that reason I have created this MR as draft.

@vadim-zabolotniy vadim-zabolotniy self-assigned this Feb 6, 2023
@vadim-zabolotniy vadim-zabolotniy linked an issue Feb 6, 2023 that may be closed by this pull request
Single Procrastination tasks #267
Open
@vadim-zabolotniy
Copy link
Collaborator Author

Hi @rtpro!

I have pushed commit to remove stalled jobs before Procrastination worker start.
I want to warn you about PostgreSQL message in logs:

postgres_1       | 2023-02-09 14:36:00.528 UTC [35] ERROR:  duplicate key value violates unique constraint "procrastinate_jobs_queueing_lock_idx"
postgres_1       | 2023-02-09 14:36:00.528 UTC [35] DETAIL:  Key (queueing_lock)=(application__check_health) already exists.
postgres_1       | 2023-02-09 14:36:00.528 UTC [35] CONTEXT:  SQL statement "INSERT INTO procrastinate_jobs (queue_name, task_name, lock, queueing_lock, args, scheduled_at)
postgres_1       |          VALUES (queue_name, task_name, lock, queueing_lock, args, scheduled_at)
postgres_1       |          RETURNING id"
postgres_1       |      PL/pgSQL function procrastinate_defer_job(character varying,character varying,text,text,jsonb,timestamp with time zone) line 5 at SQL statement
postgres_1       |      SQL statement "UPDATE procrastinate_periodic_defers
postgres_1       |              SET job_id = procrastinate_defer_job(
postgres_1       |                      _queue_name,
postgres_1       |                      _task_name,
postgres_1       |                      _lock,
postgres_1       |                      _queueing_lock,
postgres_1       |                      _args,
postgres_1       |                      NULL
postgres_1       |                  )
postgres_1       |              WHERE id = _defer_id
postgres_1       |              RETURNING job_id"
postgres_1       |      PL/pgSQL function procrastinate_defer_periodic_job(character varying,character varying,character varying,character varying,character varying,bigint,jsonb) line 17 at SQL statement
postgres_1       | 2023-02-09 14:36:00.528 UTC [35] STATEMENT:  SELECT procrastinate_defer_periodic_job('default', 'application__check_health', 'application__check_health', 'application__check_health', '', 1675953360, '{"timestamp": 1675953360}') AS id;

This error raised by constraint that was created by Procrastination. Procrastination knows about it and correctly handling it in Python code. You can consider it as INFO level message.

@vadim-zabolotniy vadim-zabolotniy marked this pull request as ready for review February 9, 2023 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rtpro rtpro Awaiting requested review from rtpro

At least 1 approving review is required to merge this pull request.

Assignees

@vadim-zabolotniy vadim-zabolotniy

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Single Procrastination tasks
1 participant
@vadim-zabolotniy