Skip to content

Add bulk add functionality for disposable email domains#23726

Merged
KevinMind merged 7 commits intomasterfrom
kevinmind/AMOENG-1872
Jul 28, 2025
Merged

Add bulk add functionality for disposable email domains#23726
KevinMind merged 7 commits intomasterfrom
kevinmind/AMOENG-1872

Conversation

@KevinMind
Copy link
Contributor

@KevinMind KevinMind commented Jul 24, 2025

  • Introduced a new CSV file containing disposable email domains.
  • Implemented the bulk_add_disposable_email_domains task to handle the addition of these domains.
  • Added a management command to facilitate bulk addition from a CSV file.
  • Created unit tests to ensure the functionality works as expected, including handling of duplicate entries.

Description

Add a task + command to execute the task, taking a csv file (via the command) path, extracting the (domain, provider) values and passing them to the task.

The task bulk adds the records in batches to handle gracefully large payloads. Currently, it logs errors that fail django model validation, removes the m from the list of domains to add and continues.

Context

We want an easy way to process large numbers of domain block requests.

Testing

TBD: see the automated tests, we should be covering the following edge cases:

Task level:

  • duplicate same domain and provider
  • duplicate same domain different provider
  • existing model validation for DisposableEmailRestriction

To run the task, from a django shell run something like

from olympia.users.tasks import bulk_add_disposable_domains 

entries = [
('foo.com', 'fooprovider')
]

result = bulk_add_disposable_domains.apply(args=[entries])

You can inspect result and check the result.status to see if it was ran successfully, you can verify the records exist in the database.

Command level:

  • Invalid or missing file provided.

To execute the command run from a shell

./manage.py bulk_add_disposable_domains disposable_domains_2025-07-22.csv
  • invalid file (remove the file path before checking this) should be a required argument
  • missing file (delete the file but pass the path) should raise command error that the file must exist.

Checklist

  • Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
  • Successfully verified the change locally.
  • The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
  • Add before and after screenshots (Only for changes that impact the UI).
  • Add or update relevant docs reflecting the changes made.

@KevinMind KevinMind marked this pull request as draft July 24, 2025 00:39
@KevinMind KevinMind force-pushed the kevinmind/AMOENG-1872 branch 2 times, most recently from 24ccff1 to f476c04 Compare July 24, 2025 12:39
- Implemented the `bulk_add_disposable_email_domains` task to handle the addition of these domains.
- Added a management command to facilitate bulk addition from a CSV file.
- Created unit tests to ensure the functionality works as expected, including handling of duplicate entries.
@KevinMind KevinMind force-pushed the kevinmind/AMOENG-1872 branch from f476c04 to 54fdf4c Compare July 24, 2025 13:23
@KevinMind KevinMind requested a review from eviljeff July 24, 2025 16:30
@KevinMind KevinMind marked this pull request as ready for review July 24, 2025 16:30
@KevinMind KevinMind requested a review from eviljeff July 28, 2025 08:38
Copy link
Member

@eviljeff eviljeff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be an issue and a link to the issue in the description.

From testing (I didn't notice it before): we should probably .trim() at least the domain - we can't guarantee all the .csv files will be clean

In the interests of getting this priority command merged I'm approving, though the outstanding code quality comments remain unaddressed.

@KevinMind KevinMind force-pushed the kevinmind/AMOENG-1872 branch from 742bb9d to 80cb236 Compare July 28, 2025 12:33
@KevinMind KevinMind merged commit 4fdc216 into master Jul 28, 2025
46 checks passed
@KevinMind KevinMind deleted the kevinmind/AMOENG-1872 branch July 28, 2025 12:52
eviljeff pushed a commit that referenced this pull request Jul 28, 2025
* Add bulk add functionality for disposable email domains

- Implemented the `bulk_add_disposable_email_domains` task to handle the addition of these domains.
- Added a management command to facilitate bulk addition from a CSV file.
- Created unit tests to ensure the functionality works as expected, including handling of duplicate entries.

* Fix comments

* TMP: Respond to comments.

* Robustify the testimajigger.

* A great offering to the oracle of beautiful python.

* Strip whitespace from domain

* Fix broken test
eviljeff pushed a commit that referenced this pull request Jul 28, 2025
* Add bulk add functionality for disposable email domains

- Implemented the `bulk_add_disposable_email_domains` task to handle the addition of these domains.
- Added a management command to facilitate bulk addition from a CSV file.
- Created unit tests to ensure the functionality works as expected, including handling of duplicate entries.

* Fix comments

* TMP: Respond to comments.

* Robustify the testimajigger.

* A great offering to the oracle of beautiful python.

* Strip whitespace from domain

* Fix broken test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants