Django "flush" management command fails because of "wagtailsearch_editorspick" table #1824
Comments
|
It sounds like you've already pieced together the background to this, but just to fill in any gaps... the EditorsPick model was taken out of the However, it relied on the assumption that having an unmanaged table sitting around in the database wouldn't cause any problems - and evidently that isn't the case, mainly due to it having a foreign key to Page. (You're the second person to report this problem, incidentally - on the first occasion, it was someone who had upgraded from a pre-1.1 version. If this was something that only affected upgrades, then we could handle it in a reasonably satisfactory way by adding some detailed troubleshooting documentation in our upgrade notes. If it's a problem on fresh installs too, then that's rather more annoying...) This is clearly something that's going to take a lot of careful planning to solve. Perhaps the way to tackle it is to divide it into two steps:
I don't currently have the answers to the above, and I'm not even sure that this is a fully solved problem within Django. (To give a sort of scaled-down version of the kind of problem we're dealing with: what's the official Django-recommended procedure for removing an app from INSTALLED_APPS and ensuring that all of its tables get garbage-collected?) |
|
I am facing the same bug, the relevant versions from pip-freeze are: |
|
We faced the same problem, but in our case it was just that our test suite started to fail when integrating wagtail into our existing django project. Here is a workaround for someone that want to use wagtail, but not include wagtailsearchpromotions. Please note that this might/will cause problems if you later on want to use wagtailsearchpromotions, as the wagtailsearch_editorspick no longer exists, and the wagtailsearchpromotions initial migration will/might fail. Note that I haven't tried that path, but I suspect it might fail. This migration needs to be added to your project. Please note the danger of this migration as well, in case you already have data in wagtailsearch_editorspick as it will remove related rows. |
|
I'm hitting the same issue when running tests with pytest and pytest-django. |
|
To answer the questions above:
|
|
To minimize the damage at this point, a solution might be to remove the FK constraint on the |
|
FWIW, this issue isn't a blocker for me anymore. We ended up not needing to run But I endorse @aaugustin's proposed solution - it seems like a good way out of this pickle. |
|
Any updates on the implementation, I am having trouble running tests with fixtures? |
|
I'm having this same issue whenever I try and run tests here is my output. This is making it so I can't run any tests that work with Page models. I really need a fix for this or we might have to stop using wagtail |
|
I am noticing the errors go away if i use TestCase instead of TransactionTestCase but that still a bad sign. |
|
I ran into the same issue where the table wagtailsearch_editorspick did not live in my database but was defined as a model. So when I ran
Looking at the output of
When I execute that command in the dbshell using
When I run the same command with "CASCADE;" at the end, it works.
The django flush.py management command on line 34 (of django 2.1 anyway) has built in support to allow the cascade option.
But it is a "stealth option" and set to False by default. You cannot currently specify something like --allow_cascade=True on the So, using the django instructions for over-riding a management module I copied flush.py to myapp/management/commands/flush.py and inserted the following in the add_arguments() function (at line 26) And now, I can simply run Hope this helps someone! |
|
I faced the same problem. When will it be fixed? Requirements: |
|
this is clunky, what can we do to fix this? |
|
As most of you, I had this problem while integrating wagtail in a existing project. After reading this thread I found out that only my Selenium tests were failing and that was due to cascade not being specified. I simply overrode the # Allow TRUNCATE ... CASCADE and don't emit the post_migrate signal
# when flushing only a subset of the apps
for db_name in self._databases_names(include_mirrors=False):
# Flush the database
inhibit_post_migrate = (
self.available_apps is not None or
( # Inhibit the post_migrate signal when using serialized
# rollback to avoid trying to recreate the serialized data.
self.serialized_rollback and
hasattr(connections[db_name], '_test_serialized_contents')
)
)
call_command('flush', verbosity=0, interactive=False,
database=db_name, reset_sequences=False,
allow_cascade=True,
inhibit_post_migrate=inhibit_post_migrate) |
|
Apologies for the delay in dealing with this - I'll aim to get this looked at for version 2.5 (but in the meantime anyone is welcome to pick this up). I think the solution will be something along the lines that @aaugustin describes - either dropping and re-adding the foreign key constraint, or having the wagtailsearch app 'reclaim' the current Whatever we go for, we'll need to check the solution carefully to make sure that all before/after states of the database are accounted for - retrospectively changing the migration setup is a tricky business, and there's a risk of making things worse if we're not careful. |
That worked for me. Thanks! |
|
I don't know if this is useful information or not, but I only hit this when using LiveServerTestcase. TestCase and the WagtailTestCase don't trigger it for me. Assuming it is related to TransactionTestCase rollback optimizations. |
|
Can confirm this still exists in Wagtail 2.16.2, and Wagtail 3.0rc1. Note as mentioned above (#1824 (comment)) this seems to prevent use of Django LiveServerTestCase when running with a PostgreSQL database, due to the way that it calls sqlflush. Adding Minimal test case to reproduce with 3.0rc1:
Click to expand full stack trace% ./manage.py test
Found 1 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.E
======================================================================
ERROR: test_something (home.tests.MyTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/db/backends/utils.py", line 87, in _execute
return self.cursor.execute(sql)
psycopg2.errors.FeatureNotSupported: cannot truncate a table referenced in a foreign key constraint
DETAIL: Table "wagtailsearch_editorspick" references "wagtailsearch_query".
HINT: Truncate table "wagtailsearch_editorspick" at the same time, or use TRUNCATE ... CASCADE.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/core/management/commands/flush.py", line 73, in handle
connection.ops.execute_sql_flush(sql_list)
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/db/backends/base/operations.py", line 442, in execute_sql_flush
cursor.execute(sql)
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/db/backends/utils.py", line 87, in _execute
return self.cursor.execute(sql)
django.db.utils.NotSupportedError: cannot truncate a table referenced in a foreign key constraint
DETAIL: Table "wagtailsearch_editorspick" references "wagtailsearch_query".
HINT: Truncate table "wagtailsearch_editorspick" at the same time, or use TRUNCATE ... CASCADE.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/test/testcases.py", line 299, in _setup_and_call
self._post_teardown()
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/test/testcases.py", line 1199, in _post_teardown
self._fixture_teardown()
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/test/testcases.py", line 1233, in _fixture_teardown
call_command(
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/core/management/__init__.py", line 198, in call_command
return command.execute(*args, **defaults)
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/core/management/base.py", line 460, in execute
output = self.handle(*args, **options)
File "/Users/chosaka/.virtualenvs/testwagtail/lib/python3.8/site-packages/django/core/management/commands/flush.py", line 75, in handle
raise CommandError(
django.core.management.base.CommandError: Database test_testwagtail couldn't be flushed. Possible reasons:
* The database isn't running or isn't configured correctly.
* At least one of the expected database tables doesn't exist.
* The SQL was invalid.
Hint: Look at the output of 'django-admin sqlflush'. That's the SQL this command wasn't able to run.
----------------------------------------------------------------------
Ran 1 test in 0.514s
FAILED (errors=1)
Destroying test database for alias 'default'... |
|
Also stumbled across this in Wagtail 3.0.1. Why is it so hard to fix? What happened when looking into this for version 2.5 in 2019? |
|
Also stuck here. I need to use Has anybody figured out a work around this bug to make it compatible with |
|
@GabrielBogo As @chosak mentioned, adding Working towards a proper solution... The migrations that brought about the current situation are:
All of these migrations have been around since Wagtail 1.1, and at this point we can reasonably assume that nobody is going to upgrade directly from Wagtail 1.1 to 4.x without deploying any intermediate version (or, at least, if they're foolhardy enough to do that, the onus is on them to devise their own migration plan), so we can dismiss the possibility of a project being in some historical state where migrations are only partially applied, or where a This means that any prospective bugfix will need to handle all of these initial states:
From these, we want to arrive at a state where
My first pass at a solution would be:
However, this fails on scenario 3, because then the wagtailsearch.0007 migration will end up trying to delete a table that doesn't exist. I expect there's a way to specify "drop table if it exists" to work around that, but it feels like I'm missing a more elegant solution - if we got into this mess without any conditional logic, surely we can get out of it again? |
On further consideration, probably not. Essentially I'm trying to engineer a sort of symmetrical process to undo our mistakes in the reverse of the order they happened in - and since wagtailsearchpromotions.0001 is an optional final step that runs after the wagtailsearch migrations to remove So, I'm pretty sure the "drop table if exists" route is the way to go... looks like https://stackoverflow.com/questions/40546762/django-migrations-how-to-check-if-table-exists-in-migrations gives us a recipe for doing that. |
…ound Fixes wagtail#1824, as per the outline in wagtail#1824 (comment). Revise the existing migrations so that rather than keeping the wagtailsearch_editorspick table in place and unmanaged for the wagtail.contrib.search_promotions app to potentially rename and adopt, search_promotions creates its own table. This then leaves wagtailsearch free to delete wagtailsearch_editorspick properly in a new migration, to be run on both new and upgraded projects (although this needs to be done inside an existence check, in case the old version of the search_promotions app migrations have already run and renamed the table).
Thanks, @gasman - that worked |
|
Thanks @gasman |
…ound Fixes wagtail#1824, as per the outline in wagtail#1824 (comment). Revise the existing migrations so that rather than keeping the wagtailsearch_editorspick table in place and unmanaged for the wagtail.contrib.search_promotions app to potentially rename and adopt, search_promotions creates its own table. This then leaves wagtailsearch free to delete wagtailsearch_editorspick properly in a new migration, to be run on both new and upgraded projects (although this needs to be done inside an existence check, in case the old version of the search_promotions app migrations have already run and renamed the table).
and others
I'm gearing up to use Wagtail for a new project and ran into a small issue flushing a Postgres database.
With a stock Wagtail installation (no contrib apps), running
python manage.py flushfails because of thewagtailsearch_editorspicktable:After doing some digging, it looks like the key is that this migration leaves the
wagtailsearch_editorspicktable in the database for the contrib app to pick up:https://github.com/torchbox/wagtail/blob/master/wagtail/wagtailsearch/migrations/0003_remove_editors_pick.py
Relevant comment:
I'm new to Wagtail (and really liking it so far!), so I'm not sure what the fix is here. If someone can point me in the right direction, though, I'm happy to put together a PR.
Thanks!
Additional background:
Installed apps:
Requirements:
The text was updated successfully, but these errors were encountered: