Add Django Q2 for task management and scheduling

[tompollard]

Why switch to Django Q2

We have a need to be able to:

1. Move tasks to the background, e.g. when uploading files to cloud and sending emails.
2. Run scheduled tasks, e.g. scrubbing spam accounts, logging metrics from services like CrossRef, DataCite, Google Scholar, etc.

Right now, we have:

• Django Cron tasks that don't appear to run because Django Cron is not installed.
• Django Background Tasks which appears to have been last updated 5 years ago!

Django Q2 is well documented and under active development. It is compatible with Django 5 (which I hope we can move to soon-ish).

What's in this pull request

This pull request:

1. Adds the Django Q2 dependency
2. Adds configuration to Django settings.
3. Sets the backend to the Django ORM (rather than external tools like Redis/Celery), for simplicity.
4. Provides instructions for running the Django Q2 server.

After the dependency is added, I would like to:

• migrate all background-tasks and cron tasks to Django Q2.
• remove the dependency on the other django tasks management packages.

[bemoody]

This all looks good and django-q2 sounds like a good way to go; thanks for find it.

Django Cron tasks that don't appear to run because Django Cron is not installed.

Just to be clear: you make it sound like we're not running any periodic background tasks at the moment. We do run periodic tasks using cron. We do not use django-cron to do so.

I'm happy to merge this so you can experiment, but please don't change the existing cron jobs until I have a chance to look at them in detail.

[tompollard]

Thanks!

Django Cron tasks that don't appear to run because Django Cron is not installed.

Just to be clear: you make it sound like we're not running any periodic background tasks at the moment. We do run periodic tasks using cron. We do not use django-cron to do so.

Yep, I'm not saying that we don't run cron tasks. e.g. we're purging old accounts here:

physionet-build/deploy/production/etc/cron.d/physionet

Lines 11 to 15 in 83c80c5

	## Daily tasks ##
	# Remove expired Django sessions
	31 23 * * * www-data env DJANGO_SETTINGS_MODULE=physionet.settings.production /physionet/python-env/physionet/bin/python3 /physionet/physionet-build/physionet-django/manage.py clearsessions
	35 23 * * * www-data env DJANGO_SETTINGS_MODULE=physionet.settings.production /physionet/python-env/physionet/bin/python3 /physionet/physionet-build/physionet-django/manage.py purgeaccounts

My point was just that we have django-cron tasks defined in https://github.com/MIT-LCP/physionet-build/blob/dev/physionet-django/physionet/cron.py that (as far as I can see) are not being run:

physionet-build/physionet-django/physionet/cron.py

Lines 14 to 73 in 83c80c5

	class RemoveUnverifiedEmails(CronJobBase):
	RUN_EVERY_MINS = 1
	RETRY_AFTER_FAILURE_MINS = 5
	schedule = Schedule(run_every_mins=RUN_EVERY_MINS,
	retry_after_failure_mins=RETRY_AFTER_FAILURE_MINS)
	code = 'physionet.RemoveUnverifiedEmails'
	def do(self):
	associated_emails = AssociatedEmail.objects.filter(
	is_primary_email=False, is_verified=False,
	added_date__lt=timezone.now() - timezone.timedelta(
	days=UNVERIFIED_DAY_LIMIT))
	for ae in associated_emails:
	ae.delete()
	print('{}: Deleted email {} from user {}'.format(
	timezone.now().strftime('%Y-%m-%d %H:%M:%S'), ae.email,
	ae.user.email))
	subject = "PhysioNet Unverified Email Removal"
	context = {
	'name': ae.user.get_full_name(),
	'email': ae.email,
	'SITE_NAME': settings.SITE_NAME,
	}
	body = loader.render_to_string(
	'user/email/unverified_email_removal.html', context)
	send_mail(subject, body, settings.DEFAULT_FROM_EMAIL,
	[ae.user.email], fail_silently=False)
	class RemoveOutstandingInvites(CronJobBase):
	RUN_EVERY_MINS = 1
	RETRY_AFTER_FAILURE_MINS = 5
	schedule = Schedule(run_every_mins=RUN_EVERY_MINS,
	retry_after_failure_mins=RETRY_AFTER_FAILURE_MINS)
	code = 'physionet.RemoveOutstandingInvites'
	def do(self):
	invitations = AuthorInvitation.objects.filter(is_active=True,
	request_datetime__lt=timezone.now() - timezone.timedelta(
	days=UNVERIFIED_DAY_LIMIT))
	for invitation in invitations:
	invitation.response_datetime = timezone.now()
	invitation.is_active = False
	invitation.save()
	subject = 'PhysioNet Expired Author Invitation'
	context = {'name':invitation.inviter.get_full_name(),
	'email':invitation.email, 'title':invitation.project.title}
	body = loader.render_to_string(
	'project/email/outstanding_invitation_removal.html', context)
	send_mail(subject, body, settings.DEFAULT_FROM_EMAIL,
	[invitation.inviter.email], fail_silently=False)
	print('{}: Removed author invitation for project {} from {} to {}'.format(
	timezone.now().strftime('%Y-%m-%d %H:%M:%S'),
	invitation.project.title, invitation.inviter.email,
	invitation.email))