Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make user not nullable in DailyXFormSubmissionCounter #900

Merged
merged 7 commits into from
Oct 13, 2023
Merged

Make user not nullable in DailyXFormSubmissionCounter #900

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b2bb881
delete daily submission counters without a user
LMNTL Oct 12, 2023
f0da3c3
make DailyXFormSubmissionCounter.user non-nullable
LMNTL Oct 12, 2023
675a8e9
don't delete counters if we can associate them with a user
LMNTL Oct 12, 2023
ba90d45
refactor logger.0030 and logger.0031
LMNTL Oct 13, 2023
fb4ff4a
rename old logger.0031 to logger.0032 to keep schema alteration atomic
LMNTL Oct 13, 2023
7abba8d
move delete_null_user_daily_counters to loggers/utils.py
LMNTL Oct 13, 2023
a82302e
use batch_size in iterator
LMNTL Oct 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
14 changes: 14 additions & 0 deletions onadata/apps/logger/migrations/0030_backfill_lost_monthly_counters.py
View file
Open in desktop
Expand Up @@ -15,6 +15,20 @@ def populate_missing_monthly_counters(apps, schema_editor):
if not DailyXFormSubmissionCounter.objects.all().exists():
return

# Associate each daily counter with user=None with a user based on its xform
for counter in DailyXFormSubmissionCounter.objects.filter(user=None).iterator:
if counter.xform and counter.xform.user:
has_duplicate = DailyXFormSubmissionCounter.objects.filter(
date=counter.date, xform=counter.xform
).exclude(user=None).exists()
# don't add a user to duplicate counters, so they get deleted in the next step
if not has_duplicate:
counter.user = counter.xform.user
counter.save()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to use bulk_update() to be more efficient.

Something like that:

    batch = []
    batch_size = 5000 
    for counter in (
        DailyXFormSubmissionCounter.objects.filter(user=None)
        .exclude(xform=None)
        .iterator(chunk_size=batch_size)
    ):
        counter.user = counter.xform.user
        if DailyXFormSubmissionCounter.objects.filter(
            date=counter.date, xform=counter.xform
        ).exclude(user=None).exists():
            continue
        batch.append(counter)
        if len(batch) >= batch_size:
            DailyXFormSubmissionCounter.objects.bulk_update(batch, ['user_id'])
            batch = []
    if batch:
        DailyXFormSubmissionCounter.objects.bulk_update(batch, ['user_id'])

What do you think?

# Delete daily counters without a user to avoid creating invalid monthly counters
DailyXFormSubmissionCounter.objects.filter(user=None).delete()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if those lines should be present in the 0031 too for users who have already run 0030 on an earlier date.

Example:

  • Upgrade to 2.023.37g (which contains 0030) on Oct 5.
  • Upgrade to 2.023.37h (which contains 0031) on Oct 13.

0030 won't be reapplied and null could have been inserted between both patch releases.

What about adding adding a RunPython in 0031 which would block the migration if null is detected in user_id field? We could provided instruction to rollback to 0029 and upgrade to 0031. Otherwise, it would apply without any message.

Copy link
Contributor Author

@LMNTL LMNTL Oct 13, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a very good point. I refactored 0030 to delete the daily counters and then repopulate the monthly counters, as well as creating a new 0031 migration that just deletes daily counters. The previous 0031 migration that alters the schema is now 0032.

This covers the edge case where a user upgraded to 2.023.37g on the same date as they upgrade to 2.023.37h. (Also, it means one less manual step needed to deploy.)

previous_migration = MigrationRecorder.Migration.objects.filter(
app='logger', name='0029_populate_daily_xform_counters_for_year'
).first()
Expand Down
18 changes: 18 additions & 0 deletions onadata/apps/logger/migrations/0031_alter_daily_submission_counter_user.py
View file
Open in desktop
@@ -0,0 +1,18 @@
from django.conf import settings
from django.db import migrations, models


class Migration(migrations.Migration):

dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
('logger', '0030_backfill_lost_monthly_counters'),
]

operations = [
migrations.AlterField(
model_name='dailyxformsubmissioncounter',
name='user',
field=models.ForeignKey('auth.User', related_name='daily_users', null=False, on_delete=models.CASCADE),
),
]
2 changes: 1 addition & 1 deletion onadata/apps/logger/models/daily_xform_submission_counter.py
View file
Open in desktop
Expand Up @@ -7,7 +7,7 @@

class DailyXFormSubmissionCounter(models.Model):
date = models.DateField()
user = models.ForeignKey(User, related_name='daily_counts', null=True, on_delete=models.CASCADE)
user = models.ForeignKey(User, related_name='daily_counts', on_delete=models.CASCADE)
xform = models.ForeignKey(
'logger.XForm', related_name='daily_counters', null=True, on_delete=models.CASCADE
)
Expand Down