Fix broken link checker

[timobrembeck]

Short description

Fix some problems in the broken link checker

Proposed changes

• Fix change of pagination size in broken link checker
• Fix pagination in broken link checker
• Show same URLs only once in broken link checker
• Delete outdated link objects
• Translate link checker status
• Fix uncaught TypeError in bulk action script
• Add chunk size option of 500

Note: I just noticed I forgot to delete old links on the XLIFF import - this I will have to change as well, otherwise outdated links might be shown in the list.

Resolved issues

Fixes: #1405
Fixes: #1413
Fixes: #1422

[codeclimate]

Code Climate has analyzed commit 7a33db0 and detected 3 issues on this pull request.

Here's the issue category breakdown:

Category	Count
Complexity	3

The test coverage on the diff in this pull request is 61.8% (50% is the threshold).

This pull request will bring the total coverage in the repository to 73.4% (0.0% change).

View more on Code Climate.

[svenseeberg]

Thanks, I cannot see any immediate problems with your code changes. However, I have some more general comments and I stumbled upon one stack trace while updating a broken link with 3 occurrences but I'm not sure how to reproduce:

May 14 12:19:39 ERROR django.request - Internal Server Error: /augsburg/analytics/linkcheck/invalid/3/
Traceback (most recent call last):
  File "/home/sven/Code/integreat-cms-django/.venv/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
    return self.connection.commit()
psycopg2.errors.ForeignKeyViolation: update or delete on table "linkcheck_url" violates foreign key constraint "linkcheck_link_url_id_c5f791e9_fk" on table "linkcheck_link"
DETAIL:  Key (id)=(1) is still referenced from table "linkcheck_link".


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/sven/Code/integreat-cms-django/.venv/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/sven/Code/integreat-cms-django/.venv/lib64/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/sven/Code/integreat-cms-django/.venv/lib64/python3.9/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/sven/Code/integreat-cms-django/.venv/lib64/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/sven/Code/integreat-cms-django/integreat_cms/cms/views/linkcheck/linkcheck.py", line 161, in post
    translation.links.all().delete()
  File "/home/sven/Code/integreat-cms-django/.venv/lib64/python3.9/site-packages/django/db/models/query.py", line 746, in delete
    deleted, _rows_count = collector.delete()
  File "/home/sven/Code/integreat-cms-django/.venv/lib64/python3.9/site-packages/django/db/models/deletion.py", line 435, in delete
    signals.post_delete.send(
  File "/home/sven/Code/integreat-cms-django/.venv/lib64/python3.9/site-packages/django/db/transaction.py", line 246, in __exit__
    connection.commit()
  File "/home/sven/Code/integreat-cms-django/.venv/lib64/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/home/sven/Code/integreat-cms-django/.venv/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 266, in commit
    self._commit()
  File "/home/sven/Code/integreat-cms-django/.venv/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
    return self.connection.commit()
  File "/home/sven/Code/integreat-cms-django/.venv/lib64/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/sven/Code/integreat-cms-django/.venv/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
    return self.connection.commit()
django.db.utils.IntegrityError: update or delete on table "linkcheck_url" violates foreign key constraint "linkcheck_link_url_id_c5f791e9_fk" on table "linkcheck_link"
DETAIL:  Key (id)=(1) is still referenced from table "linkcheck_link".

[timobrembeck]

However, I have some more general comments and I stumbled upon one stack trace while playing updating a broken link with 3 occurrences but I'm not sure how to reproduce

I think this might have been caused by a race condition - the post_save signal is called asynchronously in a separate thread - when old links are deleted while the post_save signal is running, the database integrity seems to be violated.
I think I solved this by acquiring linkcheck's internal lock for this - could you play around with it again and see whether you observe any errors?

[svenseeberg]

I tested a little more and could not find any issues.