Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make user not nullable in DailyXFormSubmissionCounter #900

Merged
merged 7 commits into from
Oct 13, 2023
Merged

Make user not nullable in DailyXFormSubmissionCounter #900

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b2bb881
delete daily submission counters without a user
LMNTL Oct 12, 2023
f0da3c3
make DailyXFormSubmissionCounter.user non-nullable
LMNTL Oct 12, 2023
675a8e9
don't delete counters if we can associate them with a user
LMNTL Oct 12, 2023
ba90d45
refactor logger.0030 and logger.0031
LMNTL Oct 13, 2023
fb4ff4a
rename old logger.0031 to logger.0032 to keep schema alteration atomic
LMNTL Oct 13, 2023
7abba8d
move delete_null_user_daily_counters to loggers/utils.py
LMNTL Oct 13, 2023
a82302e
use batch_size in iterator
LMNTL Oct 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
11 changes: 11 additions & 0 deletions onadata/apps/logger/migrations/0030_backfill_lost_monthly_counters.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,17 @@ def populate_missing_monthly_counters(apps, schema_editor):
if not DailyXFormSubmissionCounter.objects.all().exists():
return

# Associate each daily counter with user=None with a user based on its xform
for counter in DailyXFormSubmissionCounter.objects.filter(user=None).iterator:
if counter.xform and counter.xform.user:
has_duplicate = DailyXFormSubmissionCounter.objects.filter(
date=counter.date, xform=counter.xform
).exclude(user=None).exists()
# don't add a user to duplicate counters, so they get deleted in the next step
if not has_duplicate:
counter.user = counter.xform.user
counter.save()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to use bulk_update() to be more efficient.

Something like that:

    batch = []
    batch_size = 5000 
    for counter in (
        DailyXFormSubmissionCounter.objects.filter(user=None)
        .exclude(xform=None)
        .iterator(chunk_size=batch_size)
    ):
        counter.user = counter.xform.user
        if DailyXFormSubmissionCounter.objects.filter(
            date=counter.date, xform=counter.xform
        ).exclude(user=None).exists():
            continue
        batch.append(counter)
        if len(batch) >= batch_size:
            DailyXFormSubmissionCounter.objects.bulk_update(batch, ['user_id'])
            batch = []
    if batch:
        DailyXFormSubmissionCounter.objects.bulk_update(batch, ['user_id'])

What do you think?

# Delete daily counters without a user to avoid creating invalid monthly counters
DailyXFormSubmissionCounter.objects.filter(user=None).delete()

Expand Down