Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Data too long for column 'code' at row 1" #929

Closed
techsolo12 opened this issue Dec 16, 2023 · 10 comments
Closed

"Data too long for column 'code' at row 1" #929

techsolo12 opened this issue Dec 16, 2023 · 10 comments
Labels

Comments

@techsolo12
Copy link

Hello everyone,

unfortunately, Heathchecks no longer works in conjunction with a MariaDB. The problem occurred after the update to version 3.1. MariaDB version 10.8.8 is used. In my understanding the string is too long for the column? But my knowledge from coding is very limited.

docker-compose.yml

services:
  healthchecks:
    image: healthchecks/healthchecks:latest
    container_name: healthchecks
    user: "1001:1001"
    restart: always
    environment:
      ADMINS: admin@domain.tld
      # Used for running as non prod
      DEBUG: "False"
      # Disable http request logging
      UWSGI_DISABLE_LOGGING: "1"
      # Database settings
      DB: mysql
      DB_HOST: mariadb
      DB_NAME: healthchecks
      DB_USER: healthchecks
      DB_PASSWORD: "---------"
      # Email settings
      EMAIL_HOST: smtp.domain.tld
      EMAIL_PORT: 587
      EMAIL_HOST_USER: admin@domain.tld
      EMAIL_HOST_PASSWORD: "-----------"
      EMAIL_USE_TLS: "True"
      DEFAULT_FROM_EMAIL: healthchecks@domain.tld
      # Force email verifications
      EMAIL_USE_VERIFICATION: "True"
      # Set to FQDN for container
      ALLOWED_HOSTS: hc.domain.tld
      # Ping body size (email attached max. log files size)
      PING_BODY_LIMIT: 5000000
      # Domain for ping email generation
      PING_EMAIL_DOMAIN: hc.domain.tld
      PUSHOVER_API_TOKEN: -------------
      PUSHOVER_SUBSCRIPTION_URL: https://pushover.net/subscribe/Healthchecks---------
      # Open registration for everyone
      REGISTRATION_OPEN: "False"
      SITE_NAME: Healthchecks
      SITE_ROOT: https://hc.domain.tld
      # Random creted secret key
      SECRET_KEY: "------"
      SMTPD_PORT: 2525
    volumes:
      # maps the custom logo.png and custom apple-touch-180.png to healthchecks static-collected dir
      - ./data/apple-touch-180.png:/opt/healthchecks/static-collected/img/apple-touch-180.png
      - ./data/logo.png:/opt/healthchecks/static-collected/img/logo.png
    ports:
    #  - "8000:8000"
      - "2525:2525"
    networks:
      - npm
      - sql

networks:
  npm:
    name: npm
    external: true
  sql:
    name: sql
    external: true

Logs:

[uwsgi-daemons] respawning "./manage.py sendalerts" (uid: 1001 gid: 1001)
sendalerts is now running
f8290de5-b561-456e-97a7-9304bc1871f4 goes down, notifying via po
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/mysql/base.py", line 75, in execute
    return self.cursor.execute(query, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/MySQLdb/cursors.py", line 331, in _query
    self._do_get_result(db)
  File "/usr/local/lib/python3.11/site-packages/MySQLdb/cursors.py", line 136, in _do_get_result
    self._result = result = self._get_result()
                            ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/MySQLdb/cursors.py", line 363, in _get_result
    return self._get_db().store_result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
MySQLdb.DataError: (1406, "Data too long for column 'code' at row 1")
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/healthchecks/./manage.py", line 17, in <module>
    raise e
  File "/opt/healthchecks/./manage.py", line 13, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/healthchecks/hc/api/management/commands/sendalerts.py", line 164, in handle
    if self.process_one_flip():
       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/healthchecks/hc/api/management/commands/sendalerts.py", line 89, in process_one_flip
    notify(flip.id, self.stdout)
  File "/opt/healthchecks/hc/api/management/commands/sendalerts.py", line 42, in notify
    error = ch.notify(check)
            ^^^^^^^^^^^^^^^^
  File "/opt/healthchecks/hc/api/models.py", line 919, in notify
    n.save()
  File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 901, in save_base
    updated = self._save_table(
              ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1059, in _save_table
    results = self._do_insert(
              ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1100, in _do_insert
    return manager._insert(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1845, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/mysql/base.py", line 75, in execute
    return self.cursor.execute(query, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/MySQLdb/cursors.py", line 331, in _query
    self._do_get_result(db)
  File "/usr/local/lib/python3.11/site-packages/MySQLdb/cursors.py", line 136, in _do_get_result
    self._result = result = self._get_result()
                            ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/MySQLdb/cursors.py", line 363, in _get_result
    return self._get_db().store_result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/MySQLdb/cursors.py", line 363, in _get_result
    return self._get_db().store_result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.DataError: (1406, "Data too long for column 'code' at row 1")
@samuelkadolph
Copy link

Also running into this issue. Can't load any projects (getting a 404) and have Data too long for column 'code' at row 1 trying to create a new project.

@samuelkadolph
Copy link

After comparing my current install's schema versus a fresh install, I had the run the following MySQL commands to fix my install. Was there missing migrations?

ALTER TABLE accounts_credential MODIFY code uuid NOT NULL;
ALTER TABLE accounts_project MODIFY code uuid NOT NULL;
ALTER TABLE api_channel MODIFY code uuid NOT NULL;
ALTER TABLE api_check MODIFY code uuid NOT NULL;
ALTER TABLE api_check MODIFY last_start_rid uuid DEFAULT NULL;
ALTER TABLE api_notification MODIFY code uuid DEFAULT NULL;
ALTER TABLE api_ping MODIFY rid uuid DEFAULT NULL;
ALTER TABLE payments_subscription ADD next_billing_date date DEFAULT NULL;
ALTER TABLE payments_subscription ADD renew_notice_date date DEFAULT NULL;
ALTER TABLE payments_subscription ADD setup_date date DEFAULT NULL;

@cuu508
Copy link
Member

cuu508 commented Dec 17, 2023

Not sure yet but this may be the culprit: https://docs.djangoproject.com/en/5.0/releases/5.0/#migrating-uuidfield

@cuu508
Copy link
Member

cuu508 commented Dec 18, 2023

The Data too long for column 'code' at row 1 error is indeed related to a change in how Django stores UUID values in the database.

Up until Django 4.2.8, UUID fields were stored in varchar(32) columns. MariaDB 10.7 introduced a native UUID data type, and Django 5.0 switched to using that. The problem is that existing databases still have varchar(32) columns. The Django documentation proposes a migration path: create a subclass of UUIDField class backed by char(32). I think this would be problematic in Healthchecks case:

  • There are existing Healthchecks installations that use PostgreSQL, MariaDB, and SQLite databases. If we subclass the UUIDField, we would need to make sure the change does not negatively affect the other database types.
  • Healthchecks 3.1 is already released, and by now there may be MariaDB installations that were initialized from scratch and use the native UUID data type. Switching back to varchar(32) would break those.
  • We would be left with a hack in the codebase. The varchar(32) -> UUID migration would still need to happen at some point anyway

Here's the Django ticket where the varchar(32) -> UUID change was proposed and discussed: https://code.djangoproject.com/ticket/33507

The developers do discuss an alternative migration path: "alter table ... modify column ... uuid". They conclude this approach would be tricky when UUIDs are used as foreign keys. Constraints would need to be dropped, then recreated after migration – ugh! But luckily in Healthchecks case, we do not use UUIDs as foreign keys anywhere, so perhaps this could still work for us?

I reproduced the Data too long for column 'code' at row 1 error with the following steps:

  1. I started Healthchecks 3.0 and MariaDB 10.10 containers using this docker-compose.yml:
version: "3"
services:
  db:
    image: mariadb:10.10
    environment:
      - MARIADB_DATABASE=$DB_NAME
      - MARIADB_ROOT_PASSWORD=$DB_PASSWORD
  web:
    image: healthchecks/healthchecks:v3.0
    env_file:
        - .env
    ports:
        - "8000:8000"
    depends_on:
        - db
    command: bash -c 'sleep 5; uwsgi /opt/healthchecks/docker/uwsgi.ini'
  1. I opened http://localhost:8000, created an account, created a check.
  2. I bumped Healthchecks version to 3.1 in docker-compose.yml and restarted containers:
[...]
  web:
    image: healthchecks/healthchecks:v3.1
[...]
  1. In the web UI I indeed started seeing 404 and 500 errors.

I then dropped into the database shell, and ran the following commands to convert varchar(32) columns to UUID:

ALTER TABLE accounts_credential MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE accounts_project MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE api_channel MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE api_check MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE api_check MODIFY COLUMN last_start_rid UUID;
ALTER TABLE api_notification MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE api_ping MODIFY COLUMN rid UUID;

After this, the Healthchecks instance appears to be nominally working, that is, I don't see any obvious issues yet.

So I'm thinking about the following path forward:

  • If you need a stable, production Healthchecks instance, stay on 3.0 for now
  • If you are willing to test experimental changes, make a database backup, upgrade to 3.1 and try the above "alter table" commands
  • If the above commands fix the problem and do not cause other issues, I'll look into packaging them into a management command

PS. Why not run the "alter table" commands automatically in a migration? Because they should only be run on MariaDB databases 10.7+. Suppose you are currently using MariaDB 10.6 (this is what ships with Ubuntu 22.04 for example). A migration that converts varchar(32) to UUID would fail, because MariaDB 10.6 does not have UUID datatypes yet. So you would be stuck unable to complete migrations.

@cuu508 cuu508 added the bug label Dec 18, 2023
@cuu508
Copy link
Member

cuu508 commented Jan 19, 2024

No feedback and no other problem reports, so I will leave this as-is for now.

@cuu508 cuu508 closed this as completed Jan 19, 2024
@techsolo12
Copy link
Author

I have tested the database changes now about 4 weeks with 3 instances of heathchecks.
I don't get any error or warning, so i think we can add the database changes in the migration function.

What do you mean?

@cuu508
Copy link
Member

cuu508 commented Feb 3, 2024

I think this cannot be a migration–

PS. Why not run the "alter table" commands automatically in a migration? Because they should only be run on MariaDB databases 10.7+. Suppose you are currently using MariaDB 10.6 (this is what ships with Ubuntu 22.04 for example). A migration that converts varchar(32) to UUID would fail, because MariaDB 10.6 does not have UUID datatypes yet. So you would be stuck unable to complete migrations.

This could be be packaged into a management command that the site operator runs manually when/if needed though.

@techsolo12
Copy link
Author

techsolo12 commented Feb 3, 2024

I think it should be possible to check the mariadb version with a "SELECT VERSION()" statemant after the connection to the db is initialized. If the result under version 10.7.x we can do a automated migration.

Otherwise we should note this behavior in the docs or set a recommend version like 10.7.x
By the way you do a fantastic job! <3

@abcqwertz
Copy link

The Data too long for column 'code' at row 1 error is indeed related to a change in how Django stores UUID values in the database.

Up until Django 4.2.8, UUID fields were stored in varchar(32) columns. MariaDB 10.7 introduced a native UUID data type, and Django 5.0 switched to using that. The problem is that existing databases still have varchar(32) columns. The Django documentation proposes a migration path: create a subclass of UUIDField class backed by char(32). I think this would be problematic in Healthchecks case:

* There are existing Healthchecks installations that use PostgreSQL, MariaDB, and SQLite databases. If we subclass the UUIDField, we would need to make sure the change does not negatively affect the other database types.

* Healthchecks 3.1 is already released, and by now there may be MariaDB installations that were initialized from scratch and use the native UUID data type. Switching back to varchar(32) would break those.

* We would be left with a hack in the codebase. The varchar(32) -> UUID migration would still need to happen _at some point anyway_

Here's the Django ticket where the varchar(32) -> UUID change was proposed and discussed: https://code.djangoproject.com/ticket/33507

The developers do discuss an alternative migration path: "alter table ... modify column ... uuid". They conclude this approach would be tricky when UUIDs are used as foreign keys. Constraints would need to be dropped, then recreated after migration – ugh! But luckily in Healthchecks case, we do not use UUIDs as foreign keys anywhere, so perhaps this could still work for us?

I reproduced the Data too long for column 'code' at row 1 error with the following steps:

1. I started Healthchecks 3.0 and MariaDB 10.10 containers using this docker-compose.yml:
version: "3"
services:
  db:
    image: mariadb:10.10
    environment:
      - MARIADB_DATABASE=$DB_NAME
      - MARIADB_ROOT_PASSWORD=$DB_PASSWORD
  web:
    image: healthchecks/healthchecks:v3.0
    env_file:
        - .env
    ports:
        - "8000:8000"
    depends_on:
        - db
    command: bash -c 'sleep 5; uwsgi /opt/healthchecks/docker/uwsgi.ini'
2. I opened http://localhost:8000, created an account, created a check.

3. I bumped Healthchecks version to 3.1 in docker-compose.yml and restarted containers:
[...]
  web:
    image: healthchecks/healthchecks:v3.1
[...]
4. In the web UI I indeed started seeing 404 and 500 errors.

I then dropped into the database shell, and ran the following commands to convert varchar(32) columns to UUID:

ALTER TABLE accounts_credential MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE accounts_project MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE api_channel MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE api_check MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE api_check MODIFY COLUMN last_start_rid UUID;
ALTER TABLE api_notification MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE api_ping MODIFY COLUMN rid UUID;

After this, the Healthchecks instance appears to be nominally working, that is, I don't see any obvious issues yet.

So I'm thinking about the following path forward:

* If you need a stable, production Healthchecks instance, stay on 3.0 for now

* If you are willing to test experimental changes, make a database backup, upgrade to 3.1 and try the above "alter table" commands

* If the above commands fix the problem and do not cause other issues, I'll look into packaging them into a management command

PS. Why not run the "alter table" commands automatically in a migration? Because they should only be run on MariaDB databases 10.7+. Suppose you are currently using MariaDB 10.6 (this is what ships with Ubuntu 22.04 for example). A migration that converts varchar(32) to UUID would fail, because MariaDB 10.6 does not have UUID datatypes yet. So you would be stuck unable to complete migrations.

Had the same problem. The SQL statements fixed that for me as well. Thank you!

@cuu508
Copy link
Member

cuu508 commented Feb 9, 2024

I added a system check which tests if we're running on MariaDB 10.7+ with varchar instead of uuid datatypes. If positive, it outputs:

SystemCheckError: System check identified some issues:

ERRORS:
?: (hc.api.E001) Detected MariaDB >= 10.7, a manual migration to UUID datatypes required
	HINT: See https://github.com/healthchecks/healthchecks/issues/929 for details

To anyone finding this comment by following the link in the error message, the recommended steps are:

  • Make a backup of your database
  • From the database shell, run the following commands:
ALTER TABLE accounts_credential MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE accounts_project MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE api_channel MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE api_check MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE api_check MODIFY COLUMN last_start_rid UUID;
ALTER TABLE api_notification MODIFY COLUMN code UUID NOT NULL;
ALTER TABLE api_ping MODIFY COLUMN rid UUID;

@cuu508 cuu508 closed this as completed Feb 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants