Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement special POST :batchDelete callbacks #1810

Merged
merged 2 commits into from
Dec 15, 2023
Merged

Conversation

DL6ER
Copy link
Member

@DL6ER DL6ER commented Dec 7, 2023

What does this implement/fix?

This PR implements new special POST :batchDelete callbacks for /api/groups, /api/domains, /api/clients, and /api/lists. While not being immediately obvious, this is actually acceptable in the REST-world, see e.g., Custom methods | Cloud APIs | Google Cloud for a similar implementation.

While batch-deletion could indeed by very simple in the sense of

DELETE FROM adlist WHERE id IN (:some_list);

this is unfortunately not possible with SQLite3 as the IN () operator lacks support for binding variables. So I ended up implementing the following workaround that uses parameter binding to eliminate any possibility of SQL injections:

  1. Start a transaction
  2. Create a TEMPORARY table (lives only in memory) called deltable
  3. Iterate through the user input and INSERT each row into this table (values are bound to the table)
  4. Then run SQL like
    DELETE FROM gravity WHERE adlist_id = (SELECT id FROM adlist WHERE address IN (SELECT item FROM deltable));
    DELETE FROM antigravity WHERE adlist_id = (SELECT id FROM adlist WHERE address IN (SELECT item FROM deltable));
    DELETE FROM adlist WHERE address IN (SELECT item FROM deltable);
  5. Delete the temporary table
  6. Commit the transaction

If anything fails, we ROLLBACK to undo anything. Surely this SQL will not win a beauty contest due to the involved subqueries but they are so fast and their results are so small that I couldn't be bothered to use JOIN instead.

Addendum:

It is actually more tricky for domains as they are identified in addition by a type, so their temporary database is (item, type) and the SQL looks like:

DELETE FROM domainlist WHERE domain IN (SELECT item FROM deltable WHERE type = 0) AND type = 0;
DELETE FROM domainlist WHERE domain IN (SELECT item FROM deltable WHERE type = 1) AND type = 1;
DELETE FROM domainlist WHERE domain IN (SELECT item FROM deltable WHERE type = 2) AND type = 2;
DELETE FROM domainlist WHERE domain IN (SELECT item FROM deltable WHERE type = 3) AND type = 3;

This PR should still work fine with web on branch development-v6 which does not use :batchDelete. However, to really get the speed benefits, you'll really need to switch web to the accompanying branch (it has the same name). This ensures that your deletions - regardless how many things you delete - will all be done at once.


Related issue or feature (if applicable): N/A

Pull request in docs with documentation (if applicable): N/A


By submitting this pull request, I confirm the following:

  1. I have read and understood the contributors guide, as well as this entire template. I understand which branch to base my commits and Pull Requests against.
  2. I have commented my proposed changes within the code.
  3. I am willing to help maintain this change if there are issues with it later.
  4. It is compatible with the EUPL 1.2 license
  5. I have squashed any insignificant commits. (git rebase)

Checklist:

  • The code change is tested and works locally.
  • I based my code and PRs against the repositories developmental branch.
  • I signed off all commits. Pi-hole enforces the DCO for all contributions
  • I signed all my commits. Pi-hole requires signatures to verify authorship
  • I have read the above and my PR is ready for review.

…omains/, /api/clients, and /api/lists

Signed-off-by: DL6ER <dl6er@dl6er.de>
@pralor-bot
Copy link

This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/lange-wartezeit-beim-hinzufugen-und-entfernen-von-adblock-listen/66713/27

@DL6ER DL6ER merged commit 1a0921c into development-v6 Dec 15, 2023
17 checks passed
@DL6ER DL6ER deleted the new/batchDelete branch December 15, 2023 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants