"User has no profile" error after migration

[stanislav-brabec]

Describe the bug
I am trying to upgrade Weblate 3.6.1 to the lastest Weblate. Following the documentation, I picked Weblate 4.1.1 as an intermediate step.

After upgrade from 3.6.1 to 4.1.1, anonymous access works as expected. But any attempt to login (for both regular user and admin) ends by error:

Oct 23 04:03:58 i18n-staging weblate[2985]: ERROR Internal Server Error: /
                                            Traceback (most recent call last):
                                              File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
                                                response = get_response(request)
                                              File "/usr/share/weblate/weblate/accounts/middleware.py", line 63, in __call__
                                                if user.is_authenticated and user.profile.language:
                                              File "/usr/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 424, in __get__
                                                self.related.get_accessor_name()
                                            weblate.auth.models.User.profile.RelatedObjectDoesNotExist: User has no profile.

The web itself displays internal server error and login and it does not show logged user (there is the Log in button).

A clear and concise description of what the bug is.

There were no errors reported by manage.py migrate

I also tried manage.py createadmin, but it sees the user:

CommandError: User exists, specify --update to update existing

To Reproduce the bug

Description should look similar to this:

Steps to reproduce the behavior:

1. Try to upgrade from 3.6.1 to 4.1.1
2. Do all migration steps.
3. Try to login.
4. See error.

Expected behavior

No error.

Screenshots

If applicable, add screenshots to better explain your problem.

Server configuration and status

./manage.py list_versions

 * Weblate: 4.1.1
 * Django: 3.1.1
 * siphashc: 1.3
 * Whoosh: 2.7.4
 * translate-toolkit: 3.0.0
 * lxml: 4.4.2
 * Pillow: 7.2.0
 * bleach: 3.1.1
 * python-dateutil: 2.8.1
 * social-auth-core: 3.3.3
 * social-auth-app-django: 4.0.0
 * django-crispy-forms: 1.9.0
 * oauthlib: 2.0.6
 * django-compressor: 2.4
 * djangorestframework: 3.11.0
 * django-filter: 2.3.0
 * django-appconf: 1.0.3
 * user-agents: 2.1
 * filelock: 3.0.12
 * setuptools: 40.5.0
 * jellyfish: 0.7.2
 * openpyxl: 3.0.3
 * celery: 4.4.6
 * kombu: 4.6.11
 * translation-finder: 2.1
 * html2text: 2019.8.11
 * pycairo: 1.19.1
 * pygobject: 3.34.0
 * diff-match-patch: 20181111
 * requests: 2.23.0
 * django-redis: 4.11.0
 * hiredis: 1.0.1
 * sentry_sdk: 0.14.4
 * Cython: 0.29.14
 * misaka: 2.1.1
 * GitPython: 3.0.5
 * borgbackup: 1.1.13
 * pyparsing: 2.4.7
 * Python: 3.6.10
 * Git: 2.26.2
 * psycopg2: 2.8.5
 * chardet: 3.0.4
 * ruamel.yaml: 0.16.10
 * Redis server: 6.0.4
 * PostgreSQL server: 12.4
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: localhost
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: redis://localhost:6379, redis://localhost:6379, regular
 * Platform: Linux 5.3.18-lp152.47-default (x86_64)

./manage.py check --deploy

SystemCheckError: System check identified some issues:

CRITICALS:
?: (weblate.E003) Cannot send e-mail ([Errno 111] Connection refused), please check EMAIL_* settings.
	HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/install.html#out-mail

WARNINGS:
?: (weblate.W025.ass) Failure in loading handler for ass file format: aeidon or gaupol package required for Subtitle support
	HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/install.html#optional-deps
?: (weblate.W025.ini) Failure in loading handler for ini file format: Missing iniparse library.
	HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/install.html#optional-deps
?: (weblate.W025.islu) Failure in loading handler for islu file format: Missing iniparse library.
	HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/install.html#optional-deps
?: (weblate.W025.laravel) Failure in loading handler for laravel file format: No module named 'phply'
	HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/install.html#optional-deps
?: (weblate.W025.php) Failure in loading handler for php file format: No module named 'phply'
	HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/install.html#optional-deps
?: (weblate.W025.srt) Failure in loading handler for srt file format: aeidon or gaupol package required for Subtitle support
	HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/install.html#optional-deps
?: (weblate.W025.ssa) Failure in loading handler for ssa file format: aeidon or gaupol package required for Subtitle support
	HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/install.html#optional-deps
?: (weblate.W025.sub) Failure in loading handler for sub file format: aeidon or gaupol package required for Subtitle support
	HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/install.html#optional-deps
?: (weblate.W033.Gerrit) Failure in loading VCS module for Gerrit: git: 'review' is not a git command. See 'git --help'.
 (1)
	HINT: https://docs.weblate.org/en/weblate-4.1.1/vcs.html
?: (weblate.W033.GitHub) Failure in loading VCS module for GitHub: [Errno 2] No such file or directory: 'hub': 'hub'
	HINT: https://docs.weblate.org/en/weblate-4.1.1/vcs.html
?: (weblate.W033.GitLab) Failure in loading VCS module for GitLab: [Errno 2] No such file or directory: 'lab': 'lab'
	HINT: https://docs.weblate.org/en/weblate-4.1.1/vcs.html
?: (weblate.W033.Mercurial) Failure in loading VCS module for Mercurial: [Errno 2] No such file or directory: 'hg': 'hg'
	HINT: https://docs.weblate.org/en/weblate-4.1.1/vcs.html
?: (weblate.W033.Subversion) Failure in loading VCS module for Subversion: git: 'svn' is not a git command. See 'git --help'.

The most similar commands are
	fsck
	mv
	show
 (1)
	HINT: https://docs.weblate.org/en/weblate-4.1.1/vcs.html

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
	HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
	HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/backup.html
?: (weblate.I031) New Weblate version is available, please upgrade to 4.3.1.
	HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/upgrade.html

System check identified 17 issues (1 silenced).

Additional context

Add any other context about the problem here.

[stanislav-brabec]

Not sure whether it makes any difference, but the database source was a dump of the database of a production server, where the database name is weblate, and the contents of the database was cloned to a database named weblate-staging. The target server uses database named weblate-staging. (I already had to solve #4664 with this setup.)

[nijel]

That's strange. All newly created users since 2.6 should have profile (see 7919a61). The previously imported ones were handled on the fly before and since the 3.4 there was migration to add profile for all existing users (see 04b9fa8). So there should be no user without profile unless it was manually created in the database.

This SQL should give you list of affected users to investigate further:

SELECT * FROM weblate_auth_user WHERE id NOT IN (SELECT user_id FROM accounts_profile);

[github-actions]

This issue looks more like a support question than an issue. We strive to answer these reasonably fast, but purchasing the support subscription is not only more responsible and faster for your business but also makes Weblate stronger. In case your question is already answered, making a donation is the right way to say thank you!

[github-actions]

This issue has been automatically marked as stale because it has not had any recent activity.

It will be closed soon if no further action occurs.

Thank you for your contributions!

[MeggyCal]

Hi @nijel , thanks for your answer. We checked and all users in the database are affected.

[nijel]

So, there is no recordds in the accounts_profile table? That sounds weird because no user would have the settings in that case. Are you confident that your staging environment matches production in this?

[stanislav-brabec]

I just checked "SELECT * FROM accounts_profile;" in the production weblate. There are 707 rows. In the staging, there are none.

The staging database was created by pg_dump of the whole weblate database. The staging was restored by psql on the new database.

To be sure that the source problem is not outside of Weblate, we can restore the database to the original state again, then run the upper mentioned query, and then run migration again and check accounts_profile again.

Please note that there is a one non-standard bit in our setup. The database on the production server is called weblate. The database on the staging server is called weblate-staging. (Note that it triggered issue #4664.)

[stanislav-brabec]

I just repeated the problem and found the real source of the problem. It is the database duplication.

For an unknown reason, some values don't fit the constraints. As a result, psql completely ignores the whole command in the dump.

Actually, there are following problems:

ERROR:  value too long for type character varying(30)
KONTEXT:  COPY accounts_profile, line 685, column special_chars: "▸⁄⌘⌥⇧⇪↹↵−×ᵉʳᵘᵒˢ"
84      es      0       3667    86      f       t       f       \N      1               ▸⁄⌘⌥⇧⇪↹↵−×ᵉʳᵘᵒˢ 0       0
ERROR:  value too long for type character varying(30)
KONTEXT:  COPY auth_user, line 68, column first_name: "Дмитрий Горбачев"

Looking at the Weblate admin interface, all names contain full name in both entries USER and FULL NAME (and both are the same for all names). But only this name overflows, and it overflows only in one table.

ERROR:  value too long for type character varying(200)
KONTEXT:  COPY django_admin_log, line 335, column object_repr: "The translation contains large help strings (Source string location is src/.include/.iplb/.hel..."
341	2018-02-23 19:59:15.157771+00	107	The translation contains large help strings (Source string location is src/​include/​iplb/​helps.rb). If you want to save your work, you can decide to skip all of them. The application will be still f	1	[{"added": {}}]	50	4

Now I am trying cat weblate-backup/weblate.psql | sed 's/▸⁄⌘⌥⇧⇪↹↵−×ᵉʳᵘᵒˢ/0/;s/Дмитрий Горбачев/Дм. Горбачев/' ) | psql

[stanislav-brabec]

I just checked all these constraint violations one by one and it seems that there is no programming bug. All size limits there were since beginning.

It seems to be a side effect of a new Django feature: database constraints https://docs.djangoproject.com/en/3.1/ref/contrib/postgres/constraints/
Before that, it was possible to enter oversized strings into the database.
In all cases, these constraint existed even in old versions and they did not change over time:

KONTEXT: COPY accounts_profile, line 685, column special_chars: "▸⁄⌘⌥⇧⇪↹↵−×ᵉʳᵘᵒˢ"
Introduced by 9b1ebf7 with size 30. There is no missing migration.

It seems that 30 bytes is not enough for some users.

KONTEXT: COPY auth_user, line 68, column first_name: "Дмитрий Горбачев"
This table probably comes from django. Even Django-1.11.8 released in 2017 already had a limit 30. There is no missing migration.

KONTEXT: COPY django_admin_log, line 335, column object_repr: "The translation contains large help strings
This table comes from django. Even Django-1.11.8 released in 2017 already had a limit 200. But it allowed to write oversized string to the database. There is no missing migration.

[nijel]

Weblate doesn't use PostgreSQL specific database constraints at all (as we still support MySQL as well).

Anyway the only important table out of these is accounts_profile - you won't notice others being empty. The auth_user is no longer used, but we're still using some django.contrib.auth features, so it cannot be deleted. django_admin_log contains log of actions done in the Django admin interface and is not that useful with Weblate (as most of the management happens outside the Django admin anyway).

How is the accounts_profile table defined in your production environment? The special_chars column should be character varying(30) and I see no reason why PostgreSQL should allow more than 30 chars there.

It might be encoding issue as well, are both your servers using UTF-8?

[github-actions]

This issue has been automatically marked as stale because it has not had any recent activity.

It will be closed soon if no further action occurs.

Thank you for your contributions!

[MeggyCal]

@stanislav-brabec any new discoveries? I don't want it closed again :) .

[stanislav-brabec]

It is apparently an UTF-8 destination server issue in the destination database.

Production (source):

weblate=> SHOW SERVER_ENCODING;
 server_encoding 
-----------------
 UTF8
(1 row)

Staging (destination):

weblate-staging=> SHOW SERVER_ENCODING;
 server_encoding 
-----------------
 SQL_ASCII
(1 řádka)

It has nothing to do with Weblate. Weblate can do nothing. Maybe Django should refuse to use SQL_ASCII database.

This is an excerpt from the dump:

-- Dumped from database version 11.10
-- Dumped by pg_dump version 11.1
...
SET client_encoding = 'UTF8';
...
CREATE TABLE public.accounts_profile (
    id integer NOT NULL,
    language character varying(10) NOT NULL,
    suggested integer NOT NULL,
    translated integer NOT NULL,
    user_id integer NOT NULL,
    hide_completed boolean NOT NULL,
    secondary_in_zen boolean NOT NULL,
    hide_source_secondary boolean NOT NULL,
    dashboard_component_list_id integer,
    dashboard_view integer NOT NULL,
    editor_link character varying(200) NOT NULL,
    special_chars character varying(30) NOT NULL,
    uploaded integer NOT NULL,
    translate_mode integer NOT NULL
);
...
COPY public.accounts_profile (id, language, suggested, translated, user_id, hide_completed, secondary_in_zen, hide_source_secondary, dashboard_component_list_id, dashboard_view, editor_link,
 special_chars, uploaded, translate_mode) FROM stdin;
...
84      es      0       3667    86      f       t       f       \N      1               ▸⁄⌘⌥⇧⇪↹↵−×ᵉʳᵘᵒˢ 0       0

[github-actions]

The issue you have reported is resolved now. If you don’t feel it’s right, please follow it’s labels to get a clue and take further steps.

• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.

[stanislav-brabec]

Reading PostgreSQL documentation again, it seems to be safer to explicitly enforce UTF-8. Otherwise the encoding depends on the database template encoding.
0001-Docs-Update-postgres-createdb.patch.txt

[nijel]

Thanks, committed as c09c477.