Storing deleted messages fails because of "not null" constraint

[SebastiaanZ]

Currently, some of our moderation tools fail because our database rejects messages without a value for the attachment field. The issue is caused by the fact that null=True is not set for this field, while we do not always provide a value:

site/pydis_site/apps/api/models/bot/message.py

Lines 54 to 60 in 824566b

	attachments = pgfields.ArrayField(
	models.URLField(
	max_length=512
	),
	blank=True,
	help_text="Attachments attached to this message."
	)

(Note that blank=True means that the field won't be required in form validation, but it does not mean the field can be blank in the database.)

This bug is critical as it causes some of our moderation tools to fail halfway through their execution; most notably, when our anti-spam feature issues a mute, it will mute the user but not alert moderators. Since there are quite a few false positives, we rely heavily on those alerts to inspect the trigger. The bug also means the deleted messages are not retained for inspection.

---

Full traceback:

Feb. 3, 2020, 10:06 a.m.	Website	Error	Internal Server Error: /bot/deleted-messages 
Traceback (most recent call last): 
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.NotNullViolation: null value in column "attachments" violates not-null constraint DETAIL: Failing row contains (673831740799844352, 607247579608121354, .savethedate, {}, 271315596405112833, 164, null).

The above exception was the direct cause of the following exception:

Traceback (most recent call last): 
  File "/usr/local/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/viewsets.py", line 116, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 495, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 455, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 492, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/mixins.py", line 21, in create
    self.perform_create(serializer)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/mixins.py", line 26, in perform_create
    serializer.save()
  File "/usr/local/lib/python3.7/site-packages/rest_framework/serializers.py", line 214, in save
    self.instance = self.create(validated_data)
  File "/app/pydis_site/apps/api/serializers.py", line 84, in create
    **message
  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 422, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 741, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 779, in save_base
    force_update, using, update_fields,
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 870, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 908, in _do_insert
    using=using, raw=raw)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1368, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execut
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__ 
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: null value in column "attachments" violates not-null constraint DETAIL: Failing row contains (673831740799844352, 607247579608121354, .savethedate, {}, 271315596405112833, 164, null). 

[Akarys42]

python-discord/bot#630 will solve this, the two PRs should have been merged at the same time.

[SebastiaanZ]

Okay, what's blocking the other PR?

[Akarys42]

Some small issues, but @MarkKoz worked on it in the meantime, and from my testing, the bot doesn’t seem to send an empty list when there is no attachments.

[SebastiaanZ]

This has been solved now, right?

[Akarys42]

Should be fine yep

[SebastiaanZ]

Thanks! I'll close this, then.